【八】MongoDB管理之分片集群实践
MongoDB中集群有三种:主从复制、副本集、分片集群。目前副本集已经替代主从复制架构,成为官方建议采用的架构,而分片集群相较于前两种,更加复杂。
下面是生产环境中常用的分片集群架构:
我们知道,分片集群由三个组件构成:
【分片】:官方建议采用副本集,提供数据冗余和高可用,主要存储业务数据。
【配置服务器】:官方建议采用副本集,存储集群的元数据。很重要,能够影响集群的使用。
【mongos instances】:是应用程序的接口,通过它,应用程序与整个集群是透明的,故一般每个应用服务器对应一个实例,可以跟应用部署到一台服务器上。它主要读取或缓存配置服务器中元数据,提供查询路由到每个分片的功能。
一、分片集群部署
下面我们就根据上面的架构搭建个用于开发测试的环境。当然作为测试环境,为了简单方便,这里就不用副本集了。具体测试环境描述如下:
搭建本环境前提是:所有服务器上mongodb已经安装完成。
1、启动配置服务器(192.168.245.132:10000)
mongod --configsvr --port 10000 --dbpath /data/config/db --logpath /data/config/log/mongodb.log --fork
2、启动mongos实例(192.168.245.132:20000)
mongos --configdb 192.168.245.132:10000 --port 20000 --logpath /data/config/log/mongodb.log --fork
3、启动各个分片
mongod --dbpath=/data/27017/db --fork --logpath=/data/27017/log/mongodb.log --port 27017 mongod --dbpath=/data/27018/db --fork --logpath=/data/27018/log/mongodb.log --port 27018
4、添加分片到集群中
[root@node3 log]# mongo --host 192.168.245.132 --port 20000 #连接到mongos实例 mongos> sh.addShard("192.168.245.129:27017") #添加单个主机,如果要添加副本集:sh.addShard( "<repl_name>/<ip>:<port>" ) { "shardAdded" : "shard0000", "ok" : 1 } mongos> sh.addShard("192.168.245.129:27018") { "shardAdded" : "shard0001", "ok" : 1 } mongos> sh.addShard("192.168.245.131:27018") { "shardAdded" : "shard0002", "ok" : 1 } mongos> sh.addShard("192.168.245.131:27017") { "shardAdded" : "shard0003", "ok" : 1 } mongos>
5、开启数据库分片功能
mongos> sh.enableSharding("test") #让test库可以分片 { "ok" : 1 }
在为collection分片前,必须让该集合所属的数据库具有分片的功能,一旦你开启了某个数据库的分片,MongoDB会分配一个主片。
6、为集合分片
mongos> sh.shardCollection("test.user",{"_id":1}) #以_id字段为shard key进行分片 { "collectionsharded" : "test.user", "ok" : 1 }
#查看分片后的情况
mongos> use config
switched to db config
mongos> db.databases.find()
{ "_id" : "test", "primary" : "shard0000", "partitioned" : true }
mongos> db.chunks.find()
{ "_id" : "test.user-_id_MinKey", "ns" : "test.user", "min" : { "_id" : { "$minKey" : 1 } }, "max" : { "_id" : { "$maxKey" : 1 } }, "shard" : "shard0000", "lastmod" : Timestamp(1, 0), "lastmodEpoch" : ObjectId("5677cc4015fdf4f1ffbb15bd") }
mongos>
到这里,整个测试的分片集群就搭建完成了,下面进行测试。
二、分片集群测试
1、对新插入的数据是否正常分片测试
mongos> for(i=0;i<100000;i++){ db.user.insert({"_id":i,"Name":"darren","Age":20,"Date":new Date()}); } WriteResult({ "nInserted" : 1 }) mongos> use config switched to db config mongos> db.chunks.find() # { "_id" : "test.user-_id_MinKey", "lastmod" : Timestamp(2, 0), "lastmodEpoch" : ObjectId("5677cc4015fdf4f1ffbb15bd"), "ns" : "test.user", "min" : { "_id" : { "$minKey" : 1 } }, "max" : { "_id" : 1 }, "shard" : "shard0001" } { "_id" : "test.user-_id_1.0", "lastmod" : Timestamp(3, 0), "lastmodEpoch" : ObjectId("5677cc4015fdf4f1ffbb15bd"), "ns" : "test.user", "min" : { "_id" : 1 }, "max" : { "_id" : 17 }, "shard" : "shard0002" } { "_id" : "test.user-_id_17.0", "lastmod" : Timestamp(3, 1), "lastmodEpoch" : ObjectId("5677cc4015fdf4f1ffbb15bd"), "ns" : "test.user", "min" : { "_id" : 17 }, "max" : { "_id" : { "$maxKey" : 1 } }, "shard" : "shard0000" }
到此为止,貌似有点不正常啊,我明明有4个分片的,为什么这里仅仅有三个呢,而且数据的范围也不对啊,数据明显分配得不均匀,这是为什么呢?
通过查阅文档发现,默认chunk的大小是64M,config.settings.find()可以看到这个值,而我们刚才插入的数据量不大,估计也不会产生几个chunks,而且chunk迁移需要满足一定的条件:
![](https://images2015.cnblogs.com/blog/818283/201512/818283-20151222144342718-1383644055.png)
所以,为了能够测试看的很清楚,我们调整下chunk的大小为1M:
mongos> db.settings.save( { _id:"chunksize", value: 1 } ) WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 }) mongos> db.settings.find() { "_id" : "chunksize", "value" : 1 }
这时再次进行测试:
mongos> use admin switched to db admin mongos> db.runCommand({"shardcollection":"test.students3","key":{"Uid":1}}) { "collectionsharded" : "test.students3", "ok" : 1 } mongos> use test switched to db test mongos> for(i=0;i<100000;i++){ db.students3.insert({"Uid":i,"Name":"darren","Age":21,"Date":new Date()}); } WriteResult({ "nInserted" : 1 }) mongos> use config switched to db config mongos> db.chunks.find() { "_id" : "test.students3-Uid_MinKey", "lastmod" : Timestamp(6, 1), "lastmodEpoch" : ObjectId("5678e209e02c4f2c17a4bfeb"), "ns" : "test.students3", "min" : { "Uid" : { "$minKey" : 1 } }, "max" : { "Uid" : 1 }, "shard" : "shard0001" }
{ "_id" : "test.students3-Uid_1.0", "lastmod" : Timestamp(7, 1), "lastmodEpoch" : ObjectId("5678e209e02c4f2c17a4bfeb"), "ns" : "test.students3", "min" : { "Uid" : 1 }, "max" : { "Uid" : 13 }, "shard" : "shard0002" }
{ "_id" : "test.students3-Uid_13.0", "lastmod" : Timestamp(8, 1), "lastmodEpoch" : ObjectId("5678e209e02c4f2c17a4bfeb"), "ns" : "test.students3", "min" : { "Uid" : 13 }, "max" : { "Uid" : 6649 }, "shard" : "shard0000" }
{ "_id" : "test.students3-Uid_6649.0", "lastmod" : Timestamp(3, 3), "lastmodEpoch" : ObjectId("5678e209e02c4f2c17a4bfeb"), "ns" : "test.students3", "min" : { "Uid" : 6649 }, "max" : { "Uid" : 14804 }, "shard" : "shard0000" }
{ "_id" : "test.students3-Uid_14804.0", "lastmod" : Timestamp(9, 1), "lastmodEpoch" : ObjectId("5678e209e02c4f2c17a4bfeb"), "ns" : "test.students3", "min" : { "Uid" : 14804 }, "max" : { "Uid" : 21440 }, "shard" : "shard0003" }
{ "_id" : "test.students3-Uid_21440.0", "lastmod" : Timestamp(4, 3), "lastmodEpoch" : ObjectId("5678e209e02c4f2c17a4bfeb"), "ns" : "test.students3", "min" : { "Uid" : 21440 }, "max" : { "Uid" : 29158 }, "shard" : "shard0003" }
以上测试正常了,几个分片上都有数据了,比如shard0001分片有1个文档,shard0002有1到13个文档,shard0000有:13-14804个文档等等。mongodb分片并不能做到数据非常的均匀。也可以通过sh.status()直观的查看分片情况:
--- Sharding Status --- sharding version: { "_id" : 1, "minCompatibleVersion" : 5, "currentVersion" : 6, "clusterId" : ObjectId("5677bf1e37ac37662f7982ed") } shards: { "_id" : "shard0000", "host" : "192.168.245.129:27017" } { "_id" : "shard0001", "host" : "192.168.245.129:27018" } { "_id" : "shard0002", "host" : "192.168.245.131:27018" } { "_id" : "shard0003", "host" : "192.168.245.131:27017" } active mongoses: "3.2.0" : 1 balancer: Currently enabled: yes Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: No recent migrations databases: { "_id" : "test", "primary" : "shard0000", "partitioned" : true } test.students3 shard key: { "Uid" : 1 } unique: false balancing: true chunks: shard0000 4 shard0001 4 shard0002 3 shard0003 4 { "Uid" : { "$minKey" : 1 } } -->> { "Uid" : 1 } on : shard0001 Timestamp(6, 1) { "Uid" : 1 } -->> { "Uid" : 13 } on : shard0002 Timestamp(7, 1) { "Uid" : 13 } -->> { "Uid" : 6649 } on : shard0000 Timestamp(8, 1) { "Uid" : 6649 } -->> { "Uid" : 14804 } on : shard0000 Timestamp(3, 3) { "Uid" : 14804 } -->> { "Uid" : 21440 } on : shard0003 Timestamp(9, 1) { "Uid" : 21440 } -->> { "Uid" : 29158 } on : shard0003 Timestamp(4, 3) { "Uid" : 29158 } -->> { "Uid" : 35794 } on : shard0001 Timestamp(5, 2) { "Uid" : 35794 } -->> { "Uid" : 42899 } on : shard0001 Timestamp(5, 3) { "Uid" : 42899 } -->> { "Uid" : 49535 } on : shard0002 Timestamp(6, 2) { "Uid" : 49535 } -->> { "Uid" : 56475 } on : shard0002 Timestamp(6, 3) { "Uid" : 56475 } -->> { "Uid" : 63111 } on : shard0000 Timestamp(7, 2) { "Uid" : 63111 } -->> { "Uid" : 71485 } on : shard0000 Timestamp(7, 3) { "Uid" : 71485 } -->> { "Uid" : 78121 } on : shard0003 Timestamp(8, 2) { "Uid" : 78121 } -->> { "Uid" : 85826 } on : shard0003 Timestamp(8, 3) { "Uid" : 85826 } -->> { "Uid" : { "$maxKey" : 1 } } on : shard0001 Timestamp(9, 0)
2、对已存在数据分片测试(最常用)
一般而言,数据是由少到多,架构发展也是一样,由简单到复杂。之前是直接搭建个分片集群,考虑到维护成本等原因,现实中这样的做法一般不常用。当一个服务器上的数据库增长到瓶颈时,需要对其进行分片,这个是我们常遇到的情况。
假设现在存在一个数据很大的children数据库,在192.168.245.129:27019上面,需要将这些数据进行分片,下面经过以下几个步骤:
1)连接到mongos实例,将192.168.245.129:27019添加到分片集群中。
mongos> sh.addShard("192.168.245.129:27019") { "shardAdded" : "shard0004", "ok" : 1 }
注意集群分片中不能与新添加的分片中有相同的数据库,否则报错。假如新加入的分片中有test库且有文档数据,集群分片中也存在一个test库,那么这时添加分片时就会报错了,这时可以删除test库。
mongos> sh.addShard("192.168.245.129:27019") { "ok" : 0, "errmsg" : "can't add shard '192.168.245.129:27019' because a local database 'test' exists in another shard0000", "code" : 96 }
2)在需要的数据库上开启分片功能
mongos> sh.enableSharding("children") { "ok" : 1 }
3)对children数据库下的集合进行分片。注意:对已存在的数据进行分片,一定要保证shard key字段是索引,否则报下面的错误。
mongos> sh.shardCollection("children.children",{"Uid":1}) { "proposedKey" : { "Uid" : 1 }, "curIndexes" : [ { "v" : 1, "key" : { "_id" : 1 }, "name" : "_id_", "ns" : "children.children" } ], "ok" : 0, "errmsg" : "please create an index that starts with the shard key before sharding." }
在Uid上创建个索引,然后再进行分片:
#要到那个192.168.245.129:27019上创建 > db.children.createIndex({"Uid":1}) { "createdCollectionAutomatically" : false, "numIndexesBefore" : 1, "numIndexesAfter" : 2, "ok" : 1 } #重新分片 mongos> sh.shardCollection("children.children",{"Uid":1}) { "collectionsharded" : "children.children", "ok" : 1 }
这个时候查看具体分片情况:
mongos> db.shards.find()
{ "_id" : "shard0000", "host" : "192.168.245.129:27017" }
{ "_id" : "shard0001", "host" : "192.168.245.129:27018" }
{ "_id" : "shard0002", "host" : "192.168.245.131:27018" }
{ "_id" : "shard0003", "host" : "192.168.245.131:27017" }
{ "_id" : "shard0004", "host" : "192.168.245.129:27019" } #新加入的分片
mongos> db.databases.find()
{ "_id" : "test", "primary" : "shard0000", "partitioned" : true }
{ "_id" : "OSSP10", "primary" : "shard0001", "partitioned" : true }
{ "_id" : "children", "primary" : "shard0004", "partitioned" : true } #这个分片的大本营是0004
mongos> use config switched to db config mongos> db.chunks.find() { "_id" : "children.children-Uid_MinKey", "lastmod" : Timestamp(2, 0), "lastmodEpoch" : ObjectId("56790924e02c4f2c17a4c7c1"), "ns" : "children.children", "min" : { "Uid" : { "$minKey" : 1 } }, "max" : { "Uid" : 6316 }, "shard" : "shard0000" } { "_id" : "children.children-Uid_6316.0", "lastmod" : Timestamp(3, 0), "lastmodEpoch" : ObjectId("56790924e02c4f2c17a4c7c1"), "ns" : "children.children", "min" : { "Uid" : 6316 }, "max" : { "Uid" : 12633 }, "shard" : "shard0001" } { "_id" : "children.children-Uid_12633.0", "lastmod" : Timestamp(4, 0), "lastmodEpoch" : ObjectId("56790924e02c4f2c17a4c7c1"), "ns" : "children.children", "min" : { "Uid" : 12633 }, "max" : { "Uid" : 18950 }, "shard" : "shard0002" } { "_id" : "children.children-Uid_18950.0", "lastmod" : Timestamp(5, 0), "lastmodEpoch" : ObjectId("56790924e02c4f2c17a4c7c1"), "ns" : "children.children", "min" : { "Uid" : 18950 }, "max" : { "Uid" : 25267 }, "shard" : "shard0003" } { "_id" : "children.children-Uid_25267.0", "lastmod" : Timestamp(6, 0), "lastmodEpoch" : ObjectId("56790924e02c4f2c17a4c7c1"), "ns" : "children.children", "min" : { "Uid" : 25267 }, "max" : { "Uid" : 31584 }, "shard" : "shard0000" } { "_id" : "children.children-Uid_31584.0", "lastmod" : Timestamp(7, 0), "lastmodEpoch" : ObjectId("56790924e02c4f2c17a4c7c1"), "ns" : "children.children", "min" : { "Uid" : 31584 }, "max" : { "Uid" : 37901 }, "shard" : "shard0001" } { "_id" : "children.children-Uid_37901.0", "lastmod" : Timestamp(7, 1), "lastmodEpoch" : ObjectId("56790924e02c4f2c17a4c7c1"), "ns" : "children.children", "min" : { "Uid" : 37901 }, "max" : { "Uid" : 44218 }, "shard" : "shard0004" } { "_id" : "children.children-Uid_44218.0", "ns" : "children.children", "min" : { "Uid" : 44218 }, "max" : { "Uid" : { "$maxKey" : 1 } }, "shard" : "shard0004", "lastmod" : Timestamp(1, 7), "lastmodEpoch" : ObjectId("56790924e02c4f2c17a4c7c1") }
3、哈希分片测试
以上都是基于range的分片,这种方式优点是:对于一些基于范围的查询速度很快;缺点是在各分片上数据分配不均匀。而哈希分片恰恰相反,它牺牲了范围查询的性能,能够让数据相对均匀的分配到各个分片上。下面我们进行测试:
mongos> sh.enableSharding("HashTest") { "ok" : 1 } mongos> sh.shardCollection("HashTest.HashTest",{"Uid":"hashed"}) #与上面的范围分片就这个红色的区别 { "collectionsharded" : "HashTest.HashTest", "ok" : 1 } mongos> use HashTest switched to db HashTest mongos> for(i=0;i<100000;i++){ db.HashTest.insert({"Uid":i,"Name":"darren","Age":21,"Date":new Date()}); } #插入10万条文档数据 WriteResult({ "nInserted" : 1 }) mongos> use config switched to db config mongos> db.chunks.find() #不像范围分片那样,有具体的行数,hash貌似还看不出,都是hash值,那么就具体到每个分片上数数数据记录条数吧。
{ "_id" : "HashTest.HashTest-Uid_MinKey", "lastmod" : Timestamp(5, 2), "lastmodEpoch" : ObjectId("56790d30e02c4f2c17a4c89e"), "ns" : "HashTest.HashTest", "min" : { "Uid" : { "$minKey" : 1 } }, "max" : { "Uid" : NumberLong("-7378697629483820640") }, "shard" : "shard0000" }
{ "_id" : "HashTest.HashTest-Uid_-5534023222112865480", "lastmod" : Timestamp(5, 4), "lastmodEpoch" : ObjectId("56790d30e02c4f2c17a4c89e"), "ns" : "HashTest.HashTest", "min" : { "Uid" : NumberLong("-5534023222112865480") }, "max" : { "Uid" : NumberLong("-3689348814741910320") }, "shard" : "shard0001" }
{ "_id" : "HashTest.HashTest-Uid_-1844674407370955160", "lastmod" : Timestamp(5, 6), "lastmodEpoch" : ObjectId("56790d30e02c4f2c17a4c89e"), "ns" : "HashTest.HashTest", "min" : { "Uid" : NumberLong("-1844674407370955160") }, "max" : { "Uid" : NumberLong(0) }, "shard" : "shard0002" }
{ "_id" : "HashTest.HashTest-Uid_1844674407370955160", "lastmod" : Timestamp(5, 8), "lastmodEpoch" : ObjectId("56790d30e02c4f2c17a4c89e"), "ns" : "HashTest.HashTest", "min" : { "Uid" : NumberLong("1844674407370955160") }, "max" : { "Uid" : NumberLong("3689348814741910320") }, "shard" : "shard0003" }
{ "_id" : "HashTest.HashTest-Uid_5534023222112865480", "lastmod" : Timestamp(5, 10), "lastmodEpoch" : ObjectId("56790d30e02c4f2c17a4c89e"), "ns" : "HashTest.HashTest", "min" : { "Uid" : NumberLong("5534023222112865480") }, "max" : { "Uid" : NumberLong("7378697629483820640") }, "shard" : "shard0004" }
{ "_id" : "HashTest.HashTest-Uid_-7378697629483820640", "lastmod" : Timestamp(5, 3), "lastmodEpoch" : ObjectId("56790d30e02c4f2c17a4c89e"), "ns" : "HashTest.HashTest", "min" : { "Uid" : NumberLong("-7378697629483820640") }, "max" : { "Uid" : NumberLong("-5534023222112865480") }, "shard" : "shard0000" }
{ "_id" : "HashTest.HashTest-Uid_-3689348814741910320", "lastmod" : Timestamp(5, 5), "lastmodEpoch" : ObjectId("56790d30e02c4f2c17a4c89e"), "ns" : "HashTest.HashTest", "min" : { "Uid" : NumberLong("-3689348814741910320") }, "max" : { "Uid" : NumberLong("-1844674407370955160") }, "shard" : "shard0001" }
{ "_id" : "HashTest.HashTest-Uid_0", "lastmod" : Timestamp(5, 7), "lastmodEpoch" : ObjectId("56790d30e02c4f2c17a4c89e"), "ns" : "HashTest.HashTest", "min" : { "Uid" : NumberLong(0) }, "max" : { "Uid" : NumberLong("1844674407370955160") }, "shard" : "shard0002" }
{ "_id" : "HashTest.HashTest-Uid_3689348814741910320", "lastmod" : Timestamp(5, 9), "lastmodEpoch" : ObjectId("56790d30e02c4f2c17a4c89e"), "ns" : "HashTest.HashTest", "min" : { "Uid" : NumberLong("3689348814741910320") }, "max" : { "Uid" : NumberLong("5534023222112865480") }, "shard" : "shard0003" }
{ "_id" : "HashTest.HashTest-Uid_7378697629483820640", "lastmod" : Timestamp(5, 11), "lastmodEpoch" : ObjectId("56790d30e02c4f2c17a4c89e"), "ns" : "HashTest.HashTest", "min" : { "Uid" : NumberLong("7378697629483820640") }, "max" : { "Uid" : { "$maxKey" : 1 } }, "shard" : "shard0004" }
最后查了下:shard0000:19833 shard0001:20132 shard0002:20310 shard0003:19916 shard0004:19809 应该算是均匀分布了!!!
三、分片集群常用的管理命令
1、添加分片
sh.addShard( "<ip>:<27017>" )
2、删除分片
#需要运行两次,如果删除的是大本营,必须先要把数据库移到别的分片上或者删除该数据库
db.runCommand({"removeshard":"192.168.245.131:27017"})
3、修改chunk的大小
db.settings.save( { _id:"chunksize", value: 1 } ) db.settings.find()
4、刷新config服务器路由
use admin db.runCommand("flushRouterConfig");
5、对数据库/集合进行分片
sh.enableSharding("HashTest") 或者 db.runCommand({"enablesharding":"test"})
sh.shardCollection("HashTest.HashTest",{"Uid":1}) 或者 db.runCommand({"shardcollection":"test.students2","key":{"Uid":1}})
sh.shardCollection("HashTest.HashTest",{"Uid":"hashed"}) #hash分片
6、查看分片集群的状态
sh.status()
7、查看config库信息
mongos> use config switched to db config mongos> show collections actionlog changelog chunks collections databases lockpings locks mongos settings shards tags testss version #查看分片信息 mongos> db.shards.find() { "_id" : "shard0000", "host" : "192.168.245.129:27017" } { "_id" : "shard0001", "host" : "192.168.245.129:27018" } { "_id" : "shard0002", "host" : "192.168.245.131:27018" } { "_id" : "shard0003", "host" : "192.168.245.131:27017" } { "_id" : "shard0004", "host" : "192.168.245.129:27019" } #查看分片数据库信息
mongos> db.databases.find() { "_id" : "test", "primary" : "shard0000", "partitioned" : true } { "_id" : "OSSP10", "primary" : "shard0001", "partitioned" : true } { "_id" : "children", "primary" : "shard0004", "partitioned" : true } { "_id" : "HashTest", "primary" : "shard0002", "partitioned" : true } #查看块信息
mongos> db.chunks.find() { "_id" : "test.user-_id_MinKey", "lastmod" : Timestamp(2, 0), "lastmodEpoch" : ObjectId("5677cc4015fdf4f1ffbb15bd"), "ns" : "test.user", "min" : { "_id" : { "$minKey" : 1 } }, "max" : { "_id" : 1 }, "shard" : "shard0001" } { "_id" : "test.user-_id_1.0", "lastmod" : Timestamp(3, 0), "lastmodEpoch" : ObjectId("5677cc4015fdf4f1ffbb15bd"), "ns" : "test.user", "min" : { "_id" : 1 }, "max" : { "_id" : 17 }, "shard" : "shard0002" } { "_id" : "test.user-_id_17.0", "lastmod" : Timestamp(3, 1), "lastmodEpoch" : ObjectId("5677cc4015fdf4f1ffbb15bd"), "ns" : "test.user", "min" : { "_id" : 17 }, "max" : { "_id" : { "$maxKey" : 1 } }, "shard" : "shard0000" }