MongoDB 分片集群部署实验记录

======================================================== 环境说明 ========================================================

操作系统：Ubuntu 18.04.5 LTS
MongoDB版本：v4.4.4

======================================================== 角色规划 ========================================================

主机名     IP              角色
-------------------------------------------------------
mongodb-1  192.168.100.11  config-server,shard-1,mongos
mongodb-2  192.168.100.12  config-server,shard-2,mongos
mongodb-3  192.168.100.13  config-server,shard-3,mongos

======================================================== 端口规划 ========================================================

config-server: 27019
shard1: 27018
shard2: 27020
mongos: 27017

======================================================== systemd服务文件规划 ========================================================

注：这里使用systemd来管理mongodb的所有服务。

/etc/systemd/system/mongo-config-server.service

[Unit]
Description=MongoDB Database Config Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target

[Service]
User=mongodb
Group=mongodb
ExecStart=/usr/bin/mongod --config /etc/mongodb/config-server.conf
PIDFile=/var/run/mongodb/config-server.pid
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false

# Recommended limits for mongod as specified in
# https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings

[Install]
WantedBy=multi-user.target

/etc/systemd/system/mongo-shard@.service

[Unit]
Description=MongoDB Database Shard Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target
PartOf=mongo-shard.target

[Service]
User=mongodb
Group=mongodb
ExecStart=/usr/bin/mongod --config /etc/mongodb/shard%i.conf
PIDFile=/var/run/mongodb/shard%i.pid
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false

# Recommended limits for mongod as specified in
# https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings

[Install]
WantedBy=mongo-shard.target

/etc/systemd/system/mongo-shard.target

[Unit]
Description=mongo-shard target allowing to start/stop all mongo-shard@.service instances at once

[Install]
WantedBy=multi-user.target

/etc/systemd/system/mongos.service

[Unit]
Description=MongoDB Database Mongos Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target

[Service]
User=mongodb
Group=mongodb
ExecStart=/usr/bin/mongos --config /etc/mongodb/mongos.conf
PIDFile=/var/run/mongodb/mongos.pid
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false

# Recommended limits for mongos as specified in
# https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings

[Install]
WantedBy=multi-user.target

服务管理命令

# systemctl daemon-reload

# systemctl enable mongo-shard@1
# systemctl enable mongo-shard@2
# systemctl enable mongo-shard.target
# systemctl start mongo-shard.target

# systemctl enable mongo-config-server
# systemctl start mongo-config-server

# systemctl enable mongos
# systemctl start mongos

======================================================== 配置文件规划 ========================================================

/etc/mongodb/config-server.conf

storage:
  dbPath: /var/lib/mongodb/config-server
  journal:
    enabled: true
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/config-server.log
net:
  port: 27019
  bindIp: 0.0.0.0
processManagement:
  timeZoneInfo: /usr/share/zoneinfo
security:  # 启用访问控制
  authorization: enabled
  keyFile: /var/lib/mongodb/ussmongo.key
replication:
  replSetName: configrs
sharding:
  clusterRole: configsvr

/etc/mongodb/shard1.conf

storage:
  dbPath: /var/lib/mongodb/shard1
  journal:
    enabled: true
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/shard1.log
net:
  port: 27018
  bindIp: 0.0.0.0
processManagement:
  timeZoneInfo: /usr/share/zoneinfo
security:  # 启用访问控制
  authorization: enabled
  keyFile: /var/lib/mongodb/ussmongo.key
#replication:  # 生产环境建议启用副本集，这里为测试环境
#  replSetName: shardrs1
sharding:
  clusterRole: shardsvr

/etc/mongodb/mongos.conf

systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongos.log
net:
  port: 27017
  bindIp: 0.0.0.0
processManagement:
  timeZoneInfo: /usr/share/zoneinfo
security:  # 启用访问控制
  keyFile: /var/lib/mongodb/ussmongo.key
sharding:
  configDB: configrs/mongodb-1:27019,mongodb-2:27019

======================================================== 数据和日志目录规划 ========================================================

数据目录：

/var/lib/mongodb/config-server
/var/lib/mongodb/shard

日志目录：

/var/log/mongodb/config-server.log
/var/log/mongodb/shard.log
/var/log/mongodb/mongos.log

创建目录并赋予权限：

mkdir -p /var/lib/mongodb/{config-server,shard}
chown -R mongodb:mongodb /var/lib/mongodb/

======================================================== 初始化集群 ========================================================

在其中一个节点初始化config-server副本集

rs.initiate({_id: "configrs",configsvr: true,members: [{ _id : 0, host : "mongodb-1:27019" },{ _id : 1, host : "mongodb-2:27019" },{ _id : 2, host : "mongodb-3:27019" }]})

在所有shard节点查看分片实例是否运行（这里没有使用副本集）

mongo --port 27018

登录其中一个mongos节点添加所有分片

sh.addShard( "mongodb-1:27018")
sh.addShard( "mongodb-2:27018")
sh.addShard( "mongodb-3:27018")

另附副本集分片添加方法：sh.addShard( "/s1-mongo1.example.net:27018,s1-mongo2.example.net:27018,s1-mongo3.example.net:27018")

数据库和集合启用分片方法

sh.enableSharding("<database>")
sh.shardCollection("<database>.<collection>", { <shard key field> : "hashed" } )
sh.shardCollection("<database>.<collection>", { <shard key field> : 1, ... } )

集群状态

mongos> db.runCommand({ isdbgrid : 1})  # 查看当前config-server主服务器
{
        "isdbgrid" : 1,
        "hostname" : "mongodb-1",
        "ok" : 1,
        "operationTime" : Timestamp(1616050575, 2),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1616050575, 2),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}

mongos> use admin  # 列出分片
switched to db admin
mongos> db.runCommand({ listshards : 1})
{
        "shards" : [
                {
                        "_id" : "shard0000",
                        "host" : "mongodb-1:27018",
                        "state" : 1
                },
                {
                        "_id" : "shard0001",
                        "host" : "mongodb-2:27018",
                        "state" : 1
                },
                {
                        "_id" : "shard0002",
                        "host" : "mongodb-3:27018",
                        "state" : 1
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1616050609, 3),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1616050609, 3),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}

mongos> use config  # 列出启用了分片的数据库
switched to db config
mongos> db.databases.find( { "partitioned": true } )
mongos> db.collections.find()
{ "_id" : "config.system.sessions", "lastmodEpoch" : ObjectId("6052f8ba9666f28a1ccae9da"), "lastmod" : ISODate("1970-02-19T17:02:47.296Z"), "dropped" : false, "key" : { "_id" : 1 }, "unique" : false, "uuid" : UUID("7bdf89db-16cf-4edd-a853-e7cf35ba3eb0"), "distributionMode" : "sharded" }

mongos> db.printShardingStatus()  # 查看分片集群详细信息
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("6052ee3727ddc4b59c6704aa")
  }
  shards:
        {  "_id" : "shard0000",  "host" : "mongodb-1:27018",  "state" : 1 }
        {  "_id" : "shard0001",  "host" : "mongodb-2:27018",  "state" : 1 }
        {  "_id" : "shard0002",  "host" : "mongodb-3:27018",  "state" : 1 }
  active mongoses:
        "4.4.4" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours:
                23 : Failed with error 'aborted', from shard0000 to shard0002
                29 : Failed with error 'aborted', from shard0000 to shard0001
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       1024
                        too many chunks to print, use verbose if you want to force print
mongos> sh.status()  # 查看分片集群详细信息
...
mongos> sh.getBalancerState()  # 获取均衡状态
true
mongos> sh.isBalancerRunning()  # 均衡是否正在运行？
false

mongos> db.runCommand( { removeShard: "shard0002" } )  # 移除分片

======================================================== 重启分片集群 ========================================================

1. 禁用均衡

sh.stopBalancer()

2. 停止分片集群

2.1 在所有路由节点停止mongos服务

use admin
db.shutdownServer()

2.2 停止每一个分片复制集

在每个分片副本集成员上从管理数据库运行 db.shutdownServer() 以关闭其 mongod 进程。在关闭每个副本集中的主要成员之前关闭所有次要成员。

2.3 停止配置服务器

从每个配置服务器上的管理数据库运行 db.shutdownServer() 以关闭其 mongod 进程。在关闭主要成员之前关闭所有次要成员。

3. 启动分片集群

3.1 启动配置服务器

mongod --config <path-to-config-file>
mongod --configsvr --replSet <replica set name> --dbpath <path> --bind_ip localhost,<hostname(s)|ip address(es)>

启动所有配置服务器后，连接到主 mongod 并运行 rs.status() 以确认每个 CSRS 成员的健康状况和可用性。

3.2 启动每一个分片副本集

mongod --config <path-to-config-file>
mongod --shardsvr --replSet <replSetname> --dbpath <path> --bind_ip localhost,<hostname(s)|ip address(es)>

启动每个分片的所有成员后，连接到每个主 mongod 并运行 rs.status() 以确认每个成员的健康状况和可用性。

3.3 启动mongos服务

mongos --config <path-to-config>
mongos --configdb <configReplSetName>/cfg1.example.net:27019,cfg2.example.net:27019 --bind_ip localhost,<hostname(s)|ip address(es)>

4. 重新启用均衡

sh.startBalancer()
sh.getBalancerState() 
sh.isBalancerRunning()

从 MongoDB 4.2 开始， sh.startBalancer() 还为分片集群启用自动拆分。

5. 验证集群可访问性

Connect a mongo shell to one of the cluster’s mongos processes. Use sh.status() to check the overall cluster status.
将mongo shell连接到集群的mongos进程之一。使用sh.status（）检查整体群集状态。

To confirm that all shards are accessible and communicating, insert test data into a temporary sharded collection. Confirm that data is being split and migrated between each shard in your cluster. You can connect a mongo shell to each shard primary and use db.collection.find() to validate that the data was sharded as expected.
要确认所有分片均可以访问和通信，请将测试数据插入临时分片集合中。确认正在集群中的每个分片之间拆分和迁移数据。您可以将mongo shell连接到每个分片主数据库，并使用db.collection.find（）来验证是否按预期对数据进行了分片。

Important:
重要的：
To prevent potential data loss or reading stale data, do not start application reads and writes to the cluster until after confirming the cluster is healthy and accessible.
为防止潜在的数据丢失或读取过时的数据，请先确认集群运行状况良好且可访问，然后再开始对集群进行应用程序读写。

======================================================== 数据库和集合启用分片示例 ========================================================

基于范围分片

use admin
db.runCommand({ enablesharding: "test" })
db.runCommand({ shardcollection: "test.hu1", key: { id: 1 } })

插入数据脚本：

#!/usr/bin/python3

import pymongo

myclient = pymongo.MongoClient("mongodb://192.168.100.11:27017/?retryWrites=false")
mydb = myclient["test"]
mycol = mydb["hu1"]

for i in range(10, 1000000):
    mydict = {"id": i, "book": "chukiuxiang", "money": 320}
    x = mycol.insert_one(mydict)
    print(x)

myclient.close()

查看数据库状态：

use test
db.hu1.stats()
db.hu1.count()

基于哈希分片

use admin
db.runCommand({ enablesharding : "hu" })
db.runCommand({ shardcollection: "hu.hutab", key: { id: "hashed" } })

插入数据脚本：

#!/usr/bin/python3

import pymongo

myclient = pymongo.MongoClient("mongodb://192.168.100.11:27017/?retryWrites=false")
mydb = myclient["hu"]
mycol = mydb["hutab"]

for i in range(10, 100000):
    mydict = {"id": i, "book": "chukiuxiang", "money": 320}
    x = mycol.insert_one(mydict)
    print(x)

myclient.close()

在shard和mongos节点验证哈希分片

root@mongodb-1:~# mongo --port 27018
> use hu
switched to db hu
> db.hutab.count()
33753

root@mongodb-2:~# mongo --port 27018
> use hu
switched to db hu
> db.hutab.count()
33139

root@mongodb-3:~# mongo --port 27018
> use hu
switched to db hu
> db.hutab.count()
33098

root@mongodb-1:~# mongo --port 27017
> use hu
switched to db hu
> db.hutab.count()
99990

======================================================== 启用访问控制 ========================================================

1. 添加用户：

连接上mongos添加的用户会保存在config副本集中，但是不会保存到shard副本集，因此添加用户的操作需要分别在config、shard上执行。

1.1 config副本集

# mongo --port 27019
> use admin
> db.createUser(
...   {
...     user: "admin",
...     pwd: "admin",
...     roles: ["userAdminAnyDatabase", "dbAdminAnyDatabase", "readWriteAnyDatabase", "clusterAdmin"]
...   }
... )

1.2 shard副本集

# mongo --port 27018
> use admin
> db.createUser(
...   {
...     user: "admin",
...     pwd: "admin",
...     roles: ["userAdminAnyDatabase", "dbAdminAnyDatabase", "readWriteAnyDatabase", "clusterAdmin"]
...   }
... )

2. 创建秘钥文件

启用访问控制之后，外部访问MongoDB服务需要进行身份验证，而mongos访问config和shard服务则是通过配置的秘钥文件。

# openssl rand -base64 756 >/var/lib/mongodb/ussmongo.key
# chmod 0600 /var/lib/mongodb/ussmongo.key
# chown mongodb:mongodb /var/lib/mongodb/ussmongo.key

将密钥文件复制到所有节点上。

3. 添加security配置

3.1 mongos的配置文件添加如下配置

security:
  keyFile: /var/lib/mongodb/ussmongo.key

3.2 config和shard的配置文件添加如下配置

security:
  authorization: enabled
  keyFile: /var/lib/mongodb/ussmongo.key

4. 在所有节点上重启所有MongoDB服务

至此带访问控制的MongoDB分片集群就部署完成了。

posted @ 2021-09-28 10:39 Varden 阅读(235) 评论(0) 编辑收藏举报

刷新页面返回顶部

Varden