MongoDB 自动分片 auto sharding
MongoDB部署实验系列文章
MongoDB做为NoSQL数据库，最近几年持续升温，越来越多的企业都开始尝试用MongoDB代替原有Database做一些事情。MongoDB也在集群，分片，复制上也有相当不错的的表现。我通过将做各种MongoDB的部署实验进行介绍。
原网址：
http://blog.fens.me/mongodb-shard/
第三篇 MongoDB 自动分片 auto sharding，分为6个部分
- 初始化文件目录
- 启动shard节点
- 配置shard节点
- 插入数据分片实验
- 删除主分片
- 重置主分片
系统环境介绍：
Ubuntu 12.04. LTS 64bit Server
1. 初始化文件目录
创建目录
- config1,config2,config3是配置节点
- shard1,shard2,shard3是分片节点
~ pwd /home/conan/dbs ~ mkdir config1 config2 config3 shard1 shard2 shard3 ~/dbs$ ls -l drwxrwxr-x 3 conan conan 4096 May 31 11:27 config1 drwxrwxr-x 3 conan conan 4096 May 31 11:27 config2 drwxrwxr-x 3 conan conan 4096 May 31 11:27 config3 drwxrwxr-x 3 conan conan 4096 May 31 11:28 shard1 drwxrwxr-x 3 conan conan 4096 May 31 11:29 shard2 drwxrwxr-x 3 conan conan 4096 May 31 11:29 shard3
2. 启动shard节点
启动config节点
mongod --dbpath config1 --port 20001 --fork --logpath config1.log mongod --dbpath config2 --port 20002 --fork --logpath config2.log mongod --dbpath config3 --port 20003 --fork --logpath config3.log
启动mongos节点
mongos --configdb localhost:20001,localhost:20002,localhost:20003 --port 30001 --fork --logpath mongos1.log mongos --configdb localhost:20001,localhost:20002,localhost:20003 --port 30002 --fork --logpath mongos2.log
启动shard节点
mongod --dbpath shard1 --port 10001 --fork --logpath shard1.log mongod --dbpath shard2 --port 10002 --fork --logpath shard2.log mongod --dbpath shard3 --port 10003 --fork --logpath shard3.log
查看端口
~ netstat -nlt
Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:10001 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:30001 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:10002 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:30002 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:10003 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:20001 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:20002 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:20003 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN tcp 0 0 :::22 :::* LISTEN tcp 0 0 ::1:631 :::* LISTEN tcp 0 0 ::1:25 :::* LISTEN
3. 配置shard节点
连接mongos1，在mongos中添加分片：
# mongo localhost:30001/admin MongoDB shell version: 3.0.5 connecting to: localhost:30001/admin Server has startup warnings: 2015-08-12T23:37:56.997-0700 I CONTROL ** WARNING: You are running this process as the root user, which is not recommended. 2015-08-12T23:37:56.997-0700 I CONTROL mongos> db.runCommand({addshard : "localhost:10001", allowLocal : true}) { "shardAdded" : "shard0000", "ok" : 1 } mongos> db.runCommand({addshard : "localhost:10002", allowLocal : true}) { "shardAdded" : "shard0001", "ok" : 1 } mongos> db.runCommand({addshard : "localhost:10003", allowLocal : true}) { "shardAdded" : "shard0002", "ok" : 1 }
查看分片信息
mongos> db.runCommand({listshards:1})
{
"shards" : [
{
"_id" : "shard0000",
"host" : "localhost:10001"
},
{
"_id" : "shard0001",
"host" : "localhost:10002"
},
{
"_id" : "shard0002",
"host" : "localhost:10003"
}
],
"ok" : 1
}
启用数据库分片：fensme
mongos> db.runCommand({"enablesharding" : "fensme"})
{ "ok" : 1 }
注：一旦enable了个数据库，mongos将会把数据库里的不同数据集放在不同的分片上。只有数据集也被分片，否则一个数据集的所有数据将放在一个分片上。
启用数据集分片：fensme.users
db.runCommand({"shardcollection" : "fensme.users", "key" : {"_id" : 1,"uid":1}})
{ "collectionsharded" : "fensme.users", "ok" : 1 }
查看分片状态
mongos> db.printShardingStatus() --- Sharding Status --- sharding version: { "_id" : 1, "minCompatibleVersion" : 5, "currentVersion" : 6, "clusterId" : ObjectId("55cc3b46396d4ce739e9eacc") } shards: { "_id" : "shard0000", "host" : "localhost:10001" } { "_id" : "shard0001", "host" : "localhost:10002" } { "_id" : "shard0002", "host" : "localhost:10003" } balancer: Currently enabled: yes Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: No recent migrations databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "fensme", "partitioned" : true, "primary" : "shard0000" } fensme.users shard key: { "_id" : 1, "uid" : 1 } chunks: shard0000 1 { "_id" : { "$minKey" : 1 }, "uid" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 }, "uid" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0)
fensme数据库是支持shard的，主shard是shard0000，对应host是localhost:10001
再查看config数据库
mongos> use config switched to db config mongos> show collections actionlog changelog chunks collections databases lockpings locks mongos settings shards system.indexes tags version mongos> db.shards.find() { "_id" : "shard0000", "host" : "localhost:10001" } { "_id" : "shard0001", "host" : "localhost:10002" } { "_id" : "shard0002", "host" : "localhost:10003" } mongos> db.chunks.find() { "_id" : "fensme.users-_id_MinKeyuid_MinKey", "lastmod" : Timestamp(1, 0), "lastmodEpoch" : ObjectId("55cc4b46396d4ce739e9edcf"), "ns" : "fensme.users", "min" : { "_id" : { "$minKey" : 1 }, "uid" : { "$minKey" : 1 } }, "max" : { "_id" : { "$maxKey" : 1 }, "uid" : { "$maxKey" : 1 } }, "shard" : "shard0000" }
shards配置成功，分片信息都正确
4. 插入数据分片实验
向fensme.users插入数据, 批量插入10w条记录
mongos> use fensme switched to db fensme mongos> for(var i=0; i<10000; i++){db.users.insert({_id:i*1597,uid:i});} WriteResult({ "nInserted" : 1 }) mongos> db.users.find() { "_id" : 0, "uid" : 0 } { "_id" : 1597, "uid" : 1 } { "_id" : 33537, "uid" : 21 } { "_id" : 3194, "uid" : 2 } { "_id" : 35134, "uid" : 22 } { "_id" : 4791, "uid" : 3 } { "_id" : 36731, "uid" : 23 } { "_id" : 6388, "uid" : 4 } { "_id" : 38328, "uid" : 24 } { "_id" : 7985, "uid" : 5 } { "_id" : 39925, "uid" : 25 } { "_id" : 9582, "uid" : 6 } { "_id" : 41522, "uid" : 26 } { "_id" : 11179, "uid" : 7 } { "_id" : 43119, "uid" : 27 } { "_id" : 12776, "uid" : 8 } { "_id" : 44716, "uid" : 28 } { "_id" : 14373, "uid" : 9 } { "_id" : 46313, "uid" : 29 } { "_id" : 15970, "uid" : 10 } Type "it" for more
查看数据分片存储信息
mongos> db.users.stats() { "sharded" : true, "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.", "userFlags" : 1, "capped" : false, "ns" : "fensme.users", "count" : 10000, "numExtents" : 6, "size" : 480048, "storageSize" : 712704, "totalIndexSize" : 711312, "indexSizes" : { "_id_" : 302512, "_id_1_uid_1" : 408800 }, "avgObjSize" : 48.0048, "nindexes" : 2, "nchunks" : 3, "shards" : { "shard0000" : { "ns" : "fensme.users", "count" : 9979, "size" : 479040, "avgObjSize" : 48, "numExtents" : 4, "storageSize" : 696320, "lastExtentSize" : 524288, "paddingFactor" : 1, "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.", "userFlags" : 1, "capped" : false, "nindexes" : 2, "totalIndexSize" : 678608, "indexSizes" : { "_id_" : 286160, "_id_1_uid_1" : 392448 }, "ok" : 1 }, "shard0001" : { "ns" : "fensme.users", "count" : 1, "size" : 48, "avgObjSize" : 48, "numExtents" : 1, "storageSize" : 8192, "lastExtentSize" : 8192, "paddingFactor" : 1, "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.", "userFlags" : 1, "capped" : false, "nindexes" : 2, "totalIndexSize" : 16352, "indexSizes" : { "_id_" : 8176, "_id_1_uid_1" : 8176 }, "ok" : 1 }, "shard0002" : { "ns" : "fensme.users", "count" : 20, "size" : 960, "avgObjSize" : 48, "numExtents" : 1, "storageSize" : 8192, "lastExtentSize" : 8192, "paddingFactor" : 1, "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.", "userFlags" : 1, "capped" : false, "nindexes" : 2, "totalIndexSize" : 16352, "indexSizes" : { "_id_" : 8176, "_id_1_uid_1" : 8176 }, "ok" : 1 } }, "ok" : 1 }
查看users数据集信息，shard0000上面9979条，shard0001上面1条，shard0002上面20条。分片很不均匀。
分别连接到shard1,shard2,shard3查看数据分片存储的情况
连接shard1
# mongo localhost:10001 MongoDB shell version: 3.0.5 connecting to: localhost:10001/test> show dbsshow dbs fensme 0.078GB local 0.078GB > use fensme switched to db fensme > show collections system.indexes users > db.users.count() 9979
连接shard2, 1条记录
# mongo localhost:10002 MongoDB shell version: 3.0.5 connecting to: localhost:10002/test> use fensme switched to db fensme > db.users.count() 1
连接shard3，20
# mongo localhost:10003 MongoDB shell version: 3.0.5 connecting to: localhost:10003/test> use fensme switched to db fensme > db.users.count() 20
注：分片数据分配不均匀，应该重新规化分片的key。
5. 删除主分片
移除Primary分片shard1，localhost:10001
mongos> db.runCommand({"removeshard":"localhost:10001"})
{
"msg" : "draining started successfully",
"state" : "started",
"shard" : "shard0000",
"note" : "you need to drop or movePrimary these databases",
"dbsToMove" : [
"fensme"
],
"ok" : 1
}
提示shard0000是主分片，要手动迁移主分片。
查看分片状态
mongos> db.runCommand({listshards:1})
{
"shards" : [
{
"_id" : "shard0000",
"host" : "localhost:10001",
"draining" : true
},
{
"_id" : "shard0001",
"host" : "localhost:10002"
},
{
"_id" : "shard0002",
"host" : "localhost:10003"
}
],
"ok" : 1
}
draining为正在迁移过程中。。。。
mongos> db.printShardingStatus() --- Sharding Status --- sharding version: { "_id" : 1, "minCompatibleVersion" : 5, "currentVersion" : 6, "clusterId" : ObjectId("55cc3b46396d4ce739e9eacc") } shards: { "_id" : "shard0000", "host" : "localhost:10001", "draining" : true } { "_id" : "shard0001", "host" : "localhost:10002" } { "_id" : "shard0002", "host" : "localhost:10003" } balancer: Currently enabled: yes Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: 3 : Success 1 : Failed with error 'could not acquire collection lock for fensme.users to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in fensme.users is taken.', from shard0000 to shard0002 databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "fensme", "partitioned" : true, "primary" : "shard0000" } fensme.users shard key: { "_id" : 1, "uid" : 1 } chunks: shard0001 2 shard0002 1 { "_id" : { "$minKey" : 1 }, "uid" : { "$minKey" : 1 } } -->> { "_id" : 1597, "uid" : 1 } on : shard0001 Timestamp(2, 0) { "_id" : 1597, "uid" : 1 } -->> { "_id" : 33537, "uid" : 21 } on : shard0002 Timestamp(3, 0) { "_id" : 33537, "uid" : 21 } -->> { "_id" : { "$maxKey" : 1 }, "uid" : { "$maxKey" : 1 } } on : shard0001 Timestamp(4, 0)
draining为正在迁移过程中。。。。
查看db.users的分片存储分布
mongos> use fensme switched to db fensme mongos> db.users.stats() { "sharded" : true, "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.", "userFlags" : 1, "capped" : false, "ns" : "fensme.users", "count" : 10000, "numExtents" : 9, "size" : 480048, "storageSize" : 1400832, "totalIndexSize" : 711312, "indexSizes" : { "_id_" : 302512, "_id_1_uid_1" : 408800 }, "avgObjSize" : 48.0048, "nindexes" : 2, "nchunks" : 3, "shards" : { "shard0000" : { "ns" : "fensme.users", "count" : 0, "size" : 0, "numExtents" : 4, "storageSize" : 696320, "lastExtentSize" : 524288, "paddingFactor" : 1, "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.", "userFlags" : 1, "capped" : false, "nindexes" : 2, "totalIndexSize" : 16352, "indexSizes" : { "_id_" : 8176, "_id_1_uid_1" : 8176 }, "ok" : 1 }, "shard0001" : { "ns" : "fensme.users", "count" : 9980, "size" : 479088, "avgObjSize" : 48, "numExtents" : 4, "storageSize" : 696320, "lastExtentSize" : 524288, "paddingFactor" : 1, "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.", "userFlags" : 1, "capped" : false, "nindexes" : 2, "totalIndexSize" : 678608, "indexSizes" : { "_id_" : 286160, "_id_1_uid_1" : 392448 }, "ok" : 1 }, "shard0002" : { "ns" : "fensme.users", "count" : 20, "size" : 960, "avgObjSize" : 48, "numExtents" : 1, "storageSize" : 8192, "lastExtentSize" : 8192, "paddingFactor" : 1, "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.", "userFlags" : 1, "capped" : false, "nindexes" : 2, "totalIndexSize" : 16352, "indexSizes" : { "_id_" : 8176, "_id_1_uid_1" : 8176 }, "ok" : 1 } }, "ok" : 1 }
发现shard0000的数据，都被自动迁移到了shard0001
连接node2，发现9980条数据全在这里。
# mongo localhost:10002 MongoDB shell version: 3.0.5 connecting to: localhost:10002/test> use fensme switched to db fensme > db.users.count() 9980
6. 重置主分片
重新设置主分片为shard0002
mongos> use adminuse admin switched to db admin mongos> db.runCommand({"moveprimary":"fensme","to":"localhost:10003"}) { "primary " : "shard0002:localhost:10003", "ok" : 1 }
再次删除node1, shard0000
mongos> use adminuse admin switched to db admin mongos> db.runCommand({"removeshard":"localhost:10001"}) { "msg" : "removeshard completed successfully", "state" : "completed", "shard" : "shard0000", "ok" : 1 }
删除分片成功。
查看分片信息
mongos> use adminuse admin switched to db admin mongos> db.printShardingStatus() --- Sharding Status --- sharding version: { "_id" : 1, "minCompatibleVersion" : 5, "currentVersion" : 6, "clusterId" : ObjectId("55cc3b46396d4ce739e9eacc") } shards: { "_id" : "shard0001", "host" : "localhost:10002" } { "_id" : "shard0002", "host" : "localhost:10003" } balancer: Currently enabled: yes Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: 3 : Success 1 : Failed with error 'could not acquire collection lock for fensme.users to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in fensme.users is taken.', from shard0000 to shard0002 databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "fensme", "partitioned" : true, "primary" : "shard0002" } fensme.users shard key: { "_id" : 1, "uid" : 1 } chunks: shard0001 2 shard0002 1 { "_id" : { "$minKey" : 1 }, "uid" : { "$minKey" : 1 } } -->> { "_id" : 1597, "uid" : 1 } on : shard0001 Timestamp(2, 0) { "_id" : 1597, "uid" : 1 } -->> { "_id" : 33537, "uid" : 21 } on : shard0002 Timestamp(3, 0) { "_id" : 33537, "uid" : 21 } -->> { "_id" : { "$maxKey" : 1 }, "uid" : { "$maxKey" : 1 } } on : shard0001 Timestamp(4, 0)
fensme的数据库的primary分片，成功的改为了shard0002.
再看数据的分布情况：
mongos> use fensme switched to db fensme mongos> db.users.stats() { "sharded" : true, "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.", "userFlags" : 1, "capped" : false, "ns" : "fensme.users", "count" : 10000, "numExtents" : 5, "size" : 480048, "storageSize" : 704512, "totalIndexSize" : 694960, "indexSizes" : { "_id_" : 294336, "_id_1_uid_1" : 400624 }, "avgObjSize" : 48.0048, "nindexes" : 2, "nchunks" : 3, "shards" : { "shard0001" : { "ns" : "fensme.users", "count" : 9980, "size" : 479088, "avgObjSize" : 48, "numExtents" : 4, "storageSize" : 696320, "lastExtentSize" : 524288, "paddingFactor" : 1, "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.", "userFlags" : 1, "capped" : false, "nindexes" : 2, "totalIndexSize" : 678608, "indexSizes" : { "_id_" : 286160, "_id_1_uid_1" : 392448 }, "ok" : 1 }, "shard0002" : { "ns" : "fensme.users", "count" : 20, "size" : 960, "avgObjSize" : 48, "numExtents" : 1, "storageSize" : 8192, "lastExtentSize" : 8192, "paddingFactor" : 1, "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.", "userFlags" : 1, "capped" : false, "nindexes" : 2, "totalIndexSize" : 16352, "indexSizes" : { "_id_" : 8176, "_id_1_uid_1" : 8176 }, "ok" : 1 } }, "ok" : 1 }
shard0001包括9980条数据，shard0002包括20数据，shard0000已经不存储
