MongoDB高阶特性：分片

二、分片

一）简介

首先我们先来了解一下Mongo集群的概念，Mongo集群有3个主要组件

ConfigServer：在集群中扮演存储整个集群的配置信息，负责配置存储，如果需要高可用的ConfigServer那么需要3个节点。
Shard：分片，存储真实的数据，每一个Shard分片都负责存储集群中的数据，例如一个集群有3个分片，然后我们定义分片规则为哈希，那么整个集群的数据就会（分割）到3个分片中的某一个分片，那么分片是特别重要的，如果集群中的一个分片全部崩溃了那么集群将不可用，所以我们要保证集群的高可用，那么我们需要一个分片配置3个节点，2个副本集一个仲裁节点，仲裁节点类似于Redis的哨兵模式，如果发现主节点挂了那么让另一个副本集进行数据存储。
Mongos：Mongos我们可以理解为整个集群的入口，类似于Kafka的Broker代理，也就是客户端，我们通过客户端连接集群进行查询。

下面是MongoDB的官方集群架构图，我们看到Mongos是一个路由，他们的信息都存储在ConfigServer中，我们通过Mongos进行添加，然后根据条件将数据进行分片到分片的副本集中
图片描述

二）搭建

1、docker-compose-mongodb-shard.yml

version: '3'
services:
  shard1:
    image: mongo:5.0.8
    container_name: mongo_shard1
    command: mongod --replSet shard1 --port 27011 --keyFile /data/mongodb/keyFile --directoryperdb --shardsvr
    volumes:
      - /etc/localtime:/etc/localtime
      - ./mongodb-cluster/shard1:/data/db
      - ./mongodb-cluster/keyFile:/data/mongodb/keyFile
    networks:
      - weiji-net
  shard2:
    image: mongo:5.0.8
    container_name: mongo_shard2
    command: mongod --replSet shard2 --port 27012 --keyFile /data/mongodb/keyFile --directoryperdb --shardsvr
    volumes:
      - /etc/localtime:/etc/localtime
      - ./mongodb-cluster/shard2:/data/db
      - ./mongodb-cluster/keyFile:/data/mongodb/keyFile
    networks:
      - weiji-net
  shard3:
    image: mongo:5.0.8
    container_name: mongo_shard3
    command: mongod --replSet shard3 --port 27013 --keyFile /data/mongodb/keyFile --directoryperdb --shardsvr
    volumes:
      - /etc/localtime:/etc/localtime
      - ./mongodb-cluster/shard3:/data/db
      - ./mongodb-cluster/keyFile:/data/mongodb/keyFile
    networks:
      - weiji-net
  config1:
    image: mongo:5.0.8
    container_name: mongo_config1
    command: mongod --replSet fates-mongo-config --port 27021 --keyFile /data/mongodb/keyFile --directoryperdb --configsvr
    volumes:
      - /etc/localtime:/etc/localtime
      - ./mongodb-cluster/config1:/data/db
      - ./mongodb-cluster/keyFile:/data/mongodb/keyFile
    networks:
      - weiji-net
  config2:
    image: mongo:5.0.8
    container_name: mongo_config2
    command: mongod --replSet fates-mongo-config --port 27022 --keyFile /data/mongodb/keyFile --directoryperdb --configsvr
    volumes:
      - /etc/localtime:/etc/localtime
      - ./mongodb-cluster/config2:/data/db
      - ./mongodb-cluster/keyFile:/data/mongodb/keyFile
    networks:
      - weiji-net
  config3:
    image: mongo:5.0.8
    container_name: mongo_config3
    command: mongod --replSet fates-mongo-config --port 27023 --keyFile /data/mongodb/keyFile --directoryperdb --configsvr
    volumes:
      - /etc/localtime:/etc/localtime
      - ./mongodb-cluster/config3:/data/db
      - ./mongodb-cluster/keyFile:/data/mongodb/keyFile
    networks:
      - weiji-net
  mongos:
    image: mongo:5.0.8
    container_name: mongo_mongos
    command: mongos --configdb fates-mongo-config/config1:27021,config2:27022,config3:27023 --keyFile /data/mongodb/keyFile --bind_ip 0.0.0.0 --port 27017
    ports:
     - 27017:27017
    volumes:
      - /etc/localtime:/etc/localtime
      - ./mongodb-cluster/mongos:/data/db
      - ./mongodb-cluster/keyFile:/data/mongodb/keyFile
    depends_on:
      - config1
      - config2
      - config3
    networks:
      - weiji-net
networks:
  weiji-net:
    external: true
    name: app_net

2、准备keyFile

openssl rand -base64 128 > ./mongodb-cluster/keyFile
sudo chmod 600 keyFile
sudo chown 999:999 keyFile

3、配置config集群

docker-compose -f docker-compose-mongodb-shard.yml exec config1 bash -c "echo 'rs.initiate({_id: \"fates-mongo-config\",configsvr: true, members: [{ _id : 0, host : \"config1:27021\" },{ _id : 1, host : \"config2:27022\" }, { _id : 2, host : \"config3:27023\" }]})' | mongo --port 27021"

4、初始化3个分片服务器

docker-compose -f docker-compose-mongodb-shard.yml exec shard1 bash -c "echo 'rs.initiate({_id: \"shard1\",members: [{ _id : 0, host : \"shard1:27011\" }]})' | mongo --port 27011"
docker-compose -f docker-compose-mongodb-shard.yml exec shard2 bash -c "echo 'rs.initiate({_id: \"shard2\",members: [{ _id : 0, host : \"shard2:27012\" }]})' | mongo --port 27012"
docker-compose -f docker-compose-mongodb-shard.yml exec shard3 bash -c "echo 'rs.initiate({_id: \"shard3\",members: [{ _id : 0, host : \"shard3:27013\" }]})' | mongo --port 27013"

5、将mongo分片加入 mongos

docker-compose -f docker-compose-mongodb-shard.yml exec mongos bash -c "echo 'sh.addShard(\"shard1/shard1:27011\")' | mongo"
docker-compose -f docker-compose-mongodb-shard.yml exec mongos bash -c "echo 'sh.addShard(\"shard2/shard2:27012\")' | mongo"
docker-compose -f docker-compose-mongodb-shard.yml exec mongos bash -c "echo 'sh.addShard(\"shard3/shard3:27013\")' | mongo"

6、登陆 mongos ,创建用户

创建 admin用户，通过 admin 用户创建其他用户及数据库

# 进入容器
docker exec -it 18b325c28de7 /bin/bash
# 进入mongo命令模式
mongo
use admin
show collections
# 创建 admin用户
admin = db.getSiblingDB("admin")
admin.createUser({user: "admin",pwd: "111111", roles: [{role: "userAdminAnyDatabase", db: "admin"} ,{role: "clusterAdmin", db: "admin"}]});
# 切换为 admin 
db.auth("admin","111111")
show collections
# 创建不同数据库的用户
db.createUser({user:"testUser",pwd:"111111", roles:[{role:"readWrite",db:"account"}]});
db.createUser({user:"tom",pwd:"111111", roles:[{role:"readWrite",db:"video"}]});
# testUser 在account 库占增加 表 t_user
db.auth("testUser","111111")
use account
show collections
db.createCollection("t_user")
show collections
# 切换 成 tom 用户，但 tom 用户没有权限 查看 account库
use admin
db.auth("tom","111111")
use account
show collections
>> Warning: unable to run listCollections, attempting to approximate collection names by parsing connectionStatus

7、显示分片

mongo 分片键目前有两种:

范围
hash分片

操作

指定 video 为分片库
指定分片 collection 为 person，如果 collection 已经存在，则需要创建索引！！

# 切换为admin角色
use admin
db.auth("admin","111111")
# 指定 video 为分片库
sh.enableSharding("'video")
# 指定分片 collection 为 person， collection 不存在，不需要额外创建 索引,这里指定 hash 分片
sh.shardCollection("video.person",{id:"hashed"})
# 查看 MongoDB 分片集群的状态和配置信息
sh.status()
--- Sharding Status --- 
  shards:
        {  "_id" : "shard1",  "host" : "shard1/shard1:27018",  "state" : 1 }
        {  "_id" : "shard2",  "host" : "shard2/shard2:27018",  "state" : 1 }
        {  "_id" : "shard3",  "host" : "shard3/shard3:27018",  "state" : 1 }
  active mongoses:
        "4.2.21" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours: 
                682 : Success
  databases:
        {  "_id" : "'video",  "primary" : "shard3",  "partitioned" : true,  "version" : {  "uuid" : UUID("6d262f75-62e9-4b83-a2ae-d688bd59fba5"),  "lastMod" : 1 } }
        {  "_id" : "account",  "primary" : "shard2",  "partitioned" : true,  "version" : {  "uuid" : UUID("37b1fcf4-2923-461f-ac54-e2bfb312cdcd"),  "lastMod" : 1 } }
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
              
        {  "_id" : "video",  "primary" : "shard3",  "partitioned" : true,  "version" : {  "uuid" : UUID("71bc47c0-001b-466c-92ba-f9eee14af382"),  "lastMod" : 1 } }
                video.person
                        shard key: { "id" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                shard1  2
                                shard2  2
                                shard3  2
                        { "id" : { "$minKey" : 1 } } -->> { "id" : NumberLong("-6148914691236517204") } on : shard1 Timestamp(1, 0) 
                        { "id" : NumberLong("-6148914691236517204") } -->> { "id" : NumberLong("-3074457345618258602") } on : shard1 Timestamp(1, 1) 
                        { "id" : NumberLong("-3074457345618258602") } -->> { "id" : NumberLong(0) } on : shard2 Timestamp(1, 2) 
                        { "id" : NumberLong(0) } -->> { "id" : NumberLong("3074457345618258602") } on : shard2 Timestamp(1, 3) 
                        { "id" : NumberLong("3074457345618258602") } -->> { "id" : NumberLong("6148914691236517204") } on : shard3 Timestamp(1, 4) 
                        { "id" : NumberLong("6148914691236517204") } -->> { "id" : { "$maxKey" : 1 } } on : shard3 Timestamp(1, 5) 
 # 可以看见每个分片所管理的hash 范围

8、新增数据，观察分片

# 切换为tom
use admin
db.auth("tom","111111")
user video
# 初始分片数据
db.person.stats()
# 插入数据
var arr=[];
for(var i=0;i<300;i++){
  arr.push({"id":i,"name":"user"+i});
}
db.person.insertMany(arr);
# 查看当前 collection 分片情况
db.person.stats()
{
        "sharded" : true,
        "capped" : false,
        "nindexes" : 2,
        "nchunks" : 6,
        "shards" : {
                "shard1" : {
                        "ns" : "video.person",
                        "size" : 5473836,
                        "count" : 100205,
                        "avgObjSize" : 54,
                        "storageSize" : 1699840,
                        "capped" : false
                        "nindexes" : 2,
                        "indexBuilds" : [ ],
                        "totalIndexSize" : 4583424,
                },
                "shard2" : {
                        "ns" : "video.person",
                        "size" : 5451608,
                        "count" : 99791,
                        "avgObjSize" : 54,
                        "storageSize" : 1740800,
                        "capped" : false
                        "operationTime" : Timestamp(1664434149, 1)
                },
                "shard3" : {
                        "ns" : "video.person",
                        "size" : 5463446,
                        "count" : 100004,
                        "avgObjSize" : 54,
                        "storageSize" : 1703936,
                        "capped" : false
                }
        }
}

可以发现，hashed 分片键把文档都很均匀的分布在每个分片

9、一些其它命令

# 先让当前库支持分片
sh.enableSharding("'video")
# 当前集合加入分片，递增片键(timestamp已经作为索引了)
sh.shardCollection('video.person',{id:1})
# 刷新路由
db.adminCommand("flushRouterConfig")
# 让当前分片支持平衡
sh.enableBalancing("video.person")
# 开启平衡
sh.startBalancer() 
#查看详细分片信息
sh.status({"verbose":1})  
# 查看该表分片数据信息
db.getSiblingDB("video").t_video.getShardDistribution()

posted @ 2023-10-05 17:30 yifanSJ 阅读(65) 评论(0) 编辑收藏举报

刷新页面返回顶部

yifanSJ