Docker mongodb 3.4 分片 一主 一副 一仲 鉴权集群部署.
非docker部署
为了避免过分冗余,并且在主节点挂了,还能顺利自动提升,所以加入仲裁节点
为什么要用docker部署,因为之前直接在虚拟机启动10个mongod 进程.多线程并发测试的时候,mongo的进程经常会挂掉.
所以这次使用docker部署测试
之前用的是7200转的固态硬盘
这次用nvme 的固态硬盘,性能是飞跃提升,使用批量插入,每次插入100个文档 ,每个文档里面有内嵌又内嵌文档1=10个文档, 算上内嵌总数相当于关系型数据库5000W数据量,总耗时130秒,而且非常稳定,未见异常情况!
镜像版本: 3.4.10-jessie, 3.4-jessie, 3-jessie, jessie (3.4/Dockerfile)
环境:一台虚拟机
三个configsvr 副本: 端口为 27020,27021,27022
两个分片:
shard1:-> 三个副本,端口为 27017,27018,27019
shard2:-> 三个副本,端口为 27024,27025,27026
一个路由:mongos -> 端口为 27023
前置条件:
注意:configsvr 的 data 目录是 /data/configdb 而数据节点的目录是 /data/db
创建数据存储文件的目录
mkdir /usr/local/mongodb/data cd /usr/local/mongodb/data mkdir c0 && mkdir c1 && mkdir c2 && mkdir s100 && mkdir s101 && mkdir s102 && mkdir s200 && mkdir s201 && mkdir s202
生成鉴权需要的keyfile,keyfile 内容不能太长,否则启动不了,权限不能太大,否则也是启动不了
openssl rand -base64 512 > /usr/local/mongodb/keyfile chmod 600 /usr/local/mongodb/keyfile chown 999 /usr/local/mongodb/keyfile
副本模式启动 configsvr
第一次启动 时,不要添加 --keyFile /usr/local/mongodb/keyfile 先初始化用户
docker run --name mongo-cs0 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/cs0:/data/configdb -v /usr/local/mongodb/data/cs0/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet cs --configsvr --port 27020 --bind_ip=localhost,192.168.1.9 docker run --name mongo-cs1 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/cs1:/data/configdb -v /usr/local/mongodb/data/cs1/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet cs --configsvr --port 27021 --bind_ip=localhost,192.168.1.9 docker run --name mongo-cs2 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/cs2:/data/configdb -v /usr/local/mongodb/data/cs2/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet cs --configsvr --port 27022 --bind_ip=localhost,192.168.1.9
登陆任意一个节点 初始化配置,以及创建用户 注意,如果初始化配置以后,use db 多几次,会自动切换到primary,需要在primary节点上才可以有创建动作,或者设置slaveok
use admin; var cnf={_id:"cs",members:[{_id:0,host:"192.168.1.9:27020"},{_id:1,host:"192.168.1.9:27021"},{_id:2,host:"192.168.1.9:27022"}]} rs.initiate(cnf) db.createUser( { user: "dba", pwd: "dba", roles: [ { role: "root", db: "admin" } ] } ) use testdb db.createUser( { user: "testuser", pwd: "testuser", roles: [ { role: "readWrite", db: "testdb" } ] } )
启动分片1
第一次启动 时,不要添加 --keyFile /usr/local/mongodb/keyfile 先初始化用户
docker run --name mongo-s100 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s100:/data/db -v /usr/local/mongodb/data/s100/log:/var/log/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet shard1 --shardsvr --port 27017 --bind_ip=localhost,192.168.1.9 docker run --name mongo-s101 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s101:/data/db -v /usr/local/mongodb/data/s101/log:/var/log/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet shard1 --shardsvr --port 27018 --bind_ip=localhost,192.168.1.9 docker run --name mongo-s102 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s102:/data/db -v /usr/local/mongodb/data/s102/log:/var/log/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet shard1 --shardsvr --port 27019 --bind_ip=localhost,192.168.1.9
登陆路任意一个分片1 注意,如果初始化配置以后,use db 多几次,会自动切换到primary,需要在primary节点上才可以有创建动作,或者设置slaveok
use admin; var cnf={_id:"shard1",members:[{_id:0,host:"192.168.1.9:27017"},{_id:1,host:"192.168.1.9:27018"}]} rs.initiate(cnf) rs.addArb("192.168.1.9:27019")#仲裁节点 db.createUser( { user: "dba", pwd: "dba", roles: [ { role: "root", db: "admin" } ] } ) use 目标数据库 db.createUser( { user: "testuser", pwd: "testuser", roles: [ { role: "readWrite", db: "testdb" } ] } )
启动分片2
第一次启动 时,不要添加 --keyFile /usr/local/mongodb/keyfile 先初始化用户
docker run --name mongo-s200 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s200:/data/db -v /usr/local/mongodb/data/s200/log:/var/log/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet shard2 --shardsvr --port 27024 --bind_ip=localhost,192.168.1.9 docker run --name mongo-s201 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s201:/data/db -v /usr/local/mongodb/data/s201/log:/var/log/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet shard2 --shardsvr --port 27025 --bind_ip=localhost,192.168.1.9 docker run --name mongo-s202 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s202:/data/db -v /usr/local/mongodb/data/s202/log:/var/log/mongodb -d d22888af0ce0 --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet shard2 --shardsvr --port 27026 --bind_ip=localhost,192.168.1.9
登陆任意一个分片2 初始化配置,并创建用户 注意,如果初始化配置以后,use db 多几次,会自动切换到primary,需要在primary节点上才可以有创建动作,或者设置slaveok
use admin; var cnf={_id:"shard2",members:[{_id:0,host:"192.168.1.9:27024"},{_id:1,host:"192.168.1.9:27025"}]} rs.initiate(cnf) rs.addArb("192.168.1.9:27026") #仲裁节点
db.createUser( { user: "dba", pwd: "dba", roles: [ { role: "root", db: "admin" } ] } ) use 目标数据库 db.createUser( { user: "testuser", pwd: "testuser", roles: [ { role: "readWrite", db: "testdb" } ] } )
启动路由
docker run -d --name mongos --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/mongos/log:/var/log/mongodb -d d22888af0ce0 mongos --port 27023 --configdb cs/192.168.1.9:27020,192.168.1.9:27021,192.168.1.9:27022 --bind_ip=192.168.1.9
登陆路由,添加分片
mongos> sh.addShard("shard1/192.168.1.9:27017")#只需要添加其中一个副本或者仲裁host即可 可选 192.168.1.9:27018 192.168.1.9:27019 { "shardAdded" : "shard1", "ok" : 1 } mongos> sh.addShard("shard2/192.168.1.9:27024")#同上规则 { "shardAdded" : "shard2", "ok" : 1 }
mongos> sh.enableSharding("testdb")
mongos> sh.shardCollection("testdb.orderInfo",{"_id":"hashed"}) Use testdb; db.createUser( { user: "testuser", pwd: "testuser", roles: [ { role: "readWrite", db: "testdb" } ] } )
关闭所有节点,重启加上鉴权
docker stop $(docker ps -a -q) docker rm $(docker ps -a -q)
重启configsvr
docker run --restart=always --name mongo-cs0 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/cs0:/data/configdb -v /usr/local/mongodb/data/cs0/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet cs --configsvr --port 27020 --bind_ip=localhost,192.168.1.9 docker run --restart=always --name mongo-cs1 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/cs1:/data/configdb -v /usr/local/mongodb/data/cs1/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet cs --configsvr --port 27021 --bind_ip=localhost,192.168.1.9 docker run --restart=always --name mongo-cs2 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/cs2:/data/configdb -v /usr/local/mongodb/data/cs2/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet cs --configsvr --port 27022 --bind_ip=localhost,192.168.1.9
重启分片1
docker run --restart=always --name mongo-s100 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s100:/data/db -v /usr/local/mongodb/data/s100/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet shard1 --shardsvr --port 27017 --bind_ip=localhost,192.168.1.9 docker run --restart=always --name mongo-s101 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s101:/data/db -v /usr/local/mongodb/data/s101/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet shard1 --shardsvr --port 27018 --bind_ip=localhost,192.168.1.9 docker run --restart=always --name mongo-s102 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s102:/data/db -v /usr/local/mongodb/data/s102/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet shard1 --shardsvr --port 27019 --bind_ip=localhost,192.168.1.9
重启分片2
docker run --restart=always --name mongo-s200 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s200:/data/db -v /usr/local/mongodb/data/s200/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet shard2 --shardsvr --port 27024 --bind_ip=localhost,192.168.1.9 docker run --restart=always --name mongo-s201 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s201:/data/db -v /usr/local/mongodb/data/s201/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet shard2 --shardsvr --port 27025 --bind_ip=localhost,192.168.1.9 docker run --restart=always --name mongo-s202 --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/s202:/data/db -v /usr/local/mongodb/data/s202/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 --keyFile /usr/local/mongodb/keyfile --directoryperdb --storageEngine=wiredTiger --smallfiles --oplogSize 128 --replSet shard2 --shardsvr --port 27026 --bind_ip=localhost,192.168.1.9
启动路由
docker run --restart=always -d --name mongos --net=host -v /etc/localtime:/etc/localtime:ro -v /usr/local/mongodb/data/mongos/log:/var/log/mongodb -v /usr/local/mongodb:/usr/local/mongodb -d d22888af0ce0 mongos --keyFile /usr/local/mongodb/keyfile --port 27023 --configdb cs/192.168.1.9:27020,192.168.1.9:27021,192.168.1.9:27022 --bind_ip=192.168.1.9
现在可以登陆路由进行测试了
以下测试,片键为公司编码以及用户名,复合键片,实现多热点
插入500W 单文档数据,不包含内嵌文档,速度非常快.只用了56秒
完成插入查看数据分布情况
分布不太均衡,等一会再看一次
非常均衡,因为我用了9个公司编码,14个随机用户名插入.但是只有两个分片,所以导致数据会分发倒第一次插入的分片中,导致数据需要频繁自动均衡.建议有条件的,初始化的时候,创建多个分片
