redis 分片集群(cluster)的扩容、缩容、管理
##准备一个redis集群，集群的搭建参考我上一篇 https://www.cnblogs.com/lansetuerqi/p/15385933.html
这是一个3主3从的集群
192.168.2.64:6370(主) 192.168.2.24:6375(从) （0-5460）
192.168.2.64:6372(主) 192.168.2.24:6373(从) （5461-10922）
192.168.2.64:6371(主) 192.168.2.24:6374(从) （10923-16383）
[root@ora redis]# ./bin/redis-cli -h 192.168.2.64 -p 6370 -a 123456 -c cluster nodes Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. c4729c87272ebab033d1b78ca00b6c93a15c02eb 192.168.2.24:6375@16375 slave 05afba26af286f583204ed8db95cef91cf422d9f 0 1634092111839 1 connected 05afba26af286f583204ed8db95cef91cf422d9f 192.168.2.64:6370@16370 myself,master - 0 1634092107000 1 connected 0-5460 36f8084c986ce896593ea7cf00496fdcdef70a11 192.168.2.24:6374@16374 slave bd6c1bd4328c3a8f0b17987285f007e7a3733776 0 1634092112842 2 connected bd6c1bd4328c3a8f0b17987285f007e7a3733776 192.168.2.64:6371@16371 master - 0 1634092110835 2 connected 10923-16383 a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 192.168.2.64:6372@16372 master - 0 1634092110000 9 connected 5461-10922 088548d9ebb6e3b908bb74e0f8d8a397b2c53cd3 192.168.2.24:6373@16373 slave a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 0 1634092113845 9 connected
一、往集群中新增数据节点
1、现准备三个闲置redis节点，配置与上面一致，192.168.2.34 6376 6377 6378 加入集群。
##第一种方式 redis-cli --cluster add-node {new host}:{new port} {exist host}:{exist port}
##也可以一步到位，直接指定为master的slave
## ./bin/redis-cli -a 123456 --cluster add-node 172.17.0.9:6379 172.17.0.2:6379 --cluster-slave --cluster-master-id d1cfb89071861866baa7ba53f878905f15616445(master节点id)
[root@ora redis]# ./bin/redis-cli -a 123456 -c --cluster add-node 192.168.2.34:6376 192.168.2.64:637
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.>>> Adding node 192.168.2.34:6376 to cluster 192.168.2.64:6370
>>> Performing Cluster Check (using node 192.168.2.64:6370) M: 05afba26af286f583204ed8db95cef91cf422d9f 192.168.2.64:6370 slots:[0-5460] (5461 slots) master 1 additional replica(s) S: c4729c87272ebab033d1b78ca00b6c93a15c02eb 192.168.2.24:6375 slots: (0 slots) slave replicates 05afba26af286f583204ed8db95cef91cf422d9f S: 36f8084c986ce896593ea7cf00496fdcdef70a11 192.168.2.24:6374 slots: (0 slots) slave replicates bd6c1bd4328c3a8f0b17987285f007e7a3733776 M: bd6c1bd4328c3a8f0b17987285f007e7a3733776 192.168.2.64:6371 slots:[10923-16383] (5461 slots) master 1 additional replica(s) M: a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 192.168.2.64:6372 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 088548d9ebb6e3b908bb74e0f8d8a397b2c53cd3 192.168.2.24:6373 slots: (0 slots) slave replicates a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. >>> Send CLUSTER MEET to node 192.168.2.34:6376 to make it join the cluster. [OK] New node added correctly.
##已经以master形式加入，无slot
##Reids集群内有16384个slot，数据只会存储在slot上，故此节点无法存储数据，即使通过此节点登录集群存入数据，也只会存储到拥有数据槽（0-16383）的其他节点上
##如果某个slot没有归属于任意节点，那么此slot上的数据也是不可用的
##key通过CRC16算法取hash值然后对16384取余得出slot位置）
[root@ora redis]# ./bin/redis-cli -h 192.168.2.64 -p 6370 -c -a 123456 cluster nodes Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. c4729c87272ebab033d1b78ca00b6c93a15c02eb 192.168.2.24:6375@16375 slave 05afba26af286f583204ed8db95cef91cf422d9f 0 1634093065655 1 connected 05afba26af286f583204ed8db95cef91cf422d9f 192.168.2.64:6370@16370 myself,master - 0 1634093064000 1 connected 0-5460 36f8084c986ce896593ea7cf00496fdcdef70a11 192.168.2.24:6374@16374 slave bd6c1bd4328c3a8f0b17987285f007e7a3733776 0 1634093067662 2 connected bd6c1bd4328c3a8f0b17987285f007e7a3733776 192.168.2.64:6371@16371 master - 0 1634093068665 2 connected 10923-16383 a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 192.168.2.64:6372@16372 master - 0 1634093062545 9 connected 5461-10922 0936b30e022b7fad6ddb3ce6aee1b96b8c1d1c6b 192.168.2.34:6376@16376 master - 0 1634093066659 7 connected 088548d9ebb6e3b908bb74e0f8d8a397b2c53cd3 192.168.2.24:6373@16373 slave a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 0 1634093068000 9 connected
##第二种方式 ./bin/redis-cli -h 192.168.2.64 -p 6370 -c -a 123456 cluster meet 192.168.2.34 6377
[root@ora redis]# ./bin/redis-cli -h 192.168.2.64 -p 6370 -c -a 123456 cluster nodes Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. c4729c87272ebab033d1b78ca00b6c93a15c02eb 192.168.2.24:6375@16375 slave 05afba26af286f583204ed8db95cef91cf422d9f 0 1634093065655 1 connected 05afba26af286f583204ed8db95cef91cf422d9f 192.168.2.64:6370@16370 myself,master - 0 1634093064000 1 connected 0-5460 36f8084c986ce896593ea7cf00496fdcdef70a11 192.168.2.24:6374@16374 slave bd6c1bd4328c3a8f0b17987285f007e7a3733776 0 1634093067662 2 connected bd6c1bd4328c3a8f0b17987285f007e7a3733776 192.168.2.64:6371@16371 master - 0 1634093068665 2 connected 10923-16383 a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 192.168.2.64:6372@16372 master - 0 1634093062545 9 connected 5461-10922 0936b30e022b7fad6ddb3ce6aee1b96b8c1d1c6b 192.168.2.34:6376@16376 master - 0 1634093066659 7 connected 088548d9ebb6e3b908bb74e0f8d8a397b2c53cd3 192.168.2.24:6373@16373 slave a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 0 1634093068000 9 connected [root@ora redis]# ./bin/redis-cli -h 192.168.2.64 -p 6370 -c -a 123456 cluster meet 192.168.2.34 6377 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. OK [root@ora redis]# ./bin/redis-cli -h 192.168.2.64 -p 6370 -c -a 123456 cluster nodes Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. c4729c87272ebab033d1b78ca00b6c93a15c02eb 192.168.2.24:6375@16375 slave 05afba26af286f583204ed8db95cef91cf422d9f 0 1634093462000 1 connected 05afba26af286f583204ed8db95cef91cf422d9f 192.168.2.64:6370@16370 myself,master - 0 1634093461000 1 connected 0-5460 36f8084c986ce896593ea7cf00496fdcdef70a11 192.168.2.24:6374@16374 slave bd6c1bd4328c3a8f0b17987285f007e7a3733776 0 1634093464848 2 connected bd6c1bd4328c3a8f0b17987285f007e7a3733776 192.168.2.64:6371@16371 master - 0 1634093462842 2 connected 10923-16383 a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 192.168.2.64:6372@16372 master - 0 1634093461840 9 connected 5461-10922 0936b30e022b7fad6ddb3ce6aee1b96b8c1d1c6b 192.168.2.34:6376@16376 master - 0 1634093463845 7 connected e954db69f71beb9426e01325064de520d156a67a 192.168.2.34:6377@16377 master - 0 1634093461000 0 connected 088548d9ebb6e3b908bb74e0f8d8a397b2c53cd3 192.168.2.24:6373@16373 slave a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 0 1634093459000 9 connected
将6378节点也加入集群，此处不做展示
PS:以上两种增加方式都可往集群中增加节点，但推荐使用redis-cli add-node 将节点加入集群，该命令会对新节点的状态做检查，如果新节点已加入到其它集群或者包含数据，则会放弃加入集群，并报出如下异常（未验证）。
[ERR] Node 192.168.2.34:6376 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
而 cluster meet不会做检查操作，如果通过cluster meet加入的节点已经加入到其它集群，会出现被加入节点的集群合并到现有集群的现象，造成数据错乱，所以线上推荐使用redis-cli --cluster add-node方式将新节点加入到集群。
二、修改节点
1、将192.168.2.34 6376 修改为 192.168.2.64 6371 的 从节点，前面ip为需要更改的新增节点，后面id为master节点node_id
[root@ora redis]# ./bin/redis-cli -h 192.168.2.34 -p 6376 -a 123456 -c cluster replicate bd6c1bd4328c3a8f0b17987285f007e7a3733776 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. OK [root@ora redis]# ./bin/redis-cli -h 192.168.2.64 -p 6370 -c -a 123456 cluster nodes Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. c4729c87272ebab033d1b78ca00b6c93a15c02eb 192.168.2.24:6375@16375 slave 05afba26af286f583204ed8db95cef91cf422d9f 0 1634103157373 1 connected 05afba26af286f583204ed8db95cef91cf422d9f 192.168.2.64:6370@16370 myself,master - 0 1634103157000 1 connected 0-5460 36f8084c986ce896593ea7cf00496fdcdef70a11 192.168.2.24:6374@16374 slave bd6c1bd4328c3a8f0b17987285f007e7a3733776 0 1634103155366 2 connected 2f899821dbfe2e2d5aa2c22e9e023eb9a9f127f3 192.168.2.34:6378@16378 master - 0 1634103154363 8 connected bd6c1bd4328c3a8f0b17987285f007e7a3733776 192.168.2.64:6371@16371 master - 0 1634103158376 2 connected 10923-16383 a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 192.168.2.64:6372@16372 master - 0 1634103156370 9 connected 5461-10922 0936b30e022b7fad6ddb3ce6aee1b96b8c1d1c6b 192.168.2.34:6376@16376 slave bd6c1bd4328c3a8f0b17987285f007e7a3733776 0 1634103158376 2 connected e954db69f71beb9426e01325064de520d156a67a 192.168.2.34:6377@16377 master - 0 1634103154000 0 connected 088548d9ebb6e3b908bb74e0f8d8a397b2c53cd3 192.168.2.24:6373@16373 slave a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 0 1634103159379 9 connected
2、为新增master节点 6377 分配1025个slot
[root@project-deve redis]# ./bin/redis-cli -h 192.168.2.64 -p 6370 -a 123456 -c --cluster reshard 192.168.2.34:6377 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. >>> Performing Cluster Check (using node 192.168.2.34:6377) M: e954db69f71beb9426e01325064de520d156a67a 192.168.2.34:6377 slots: (0 slots) master S: c4729c87272ebab033d1b78ca00b6c93a15c02eb 192.168.2.24:6375 slots: (0 slots) slave replicates 05afba26af286f583204ed8db95cef91cf422d9f M: 05afba26af286f583204ed8db95cef91cf422d9f 192.168.2.64:6370 slots:[0-5460] (5461 slots) master 1 additional replica(s) M: a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 192.168.2.64:6372 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 36f8084c986ce896593ea7cf00496fdcdef70a11 192.168.2.24:6374 slots: (0 slots) slave replicates bd6c1bd4328c3a8f0b17987285f007e7a3733776 S: 088548d9ebb6e3b908bb74e0f8d8a397b2c53cd3 192.168.2.24:6373 slots: (0 slots) slave replicates a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 M: 2f899821dbfe2e2d5aa2c22e9e023eb9a9f127f3 192.168.2.34:6378 slots: (0 slots) master S: 0936b30e022b7fad6ddb3ce6aee1b96b8c1d1c6b 192.168.2.34:6376 slots: (0 slots) slave replicates bd6c1bd4328c3a8f0b17987285f007e7a3733776 M: bd6c1bd4328c3a8f0b17987285f007e7a3733776 192.168.2.64:6371 slots:[10923-16383] (5461 slots) master 2 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 1025 ##分配1025个slot What is the receiving node ID? e954db69f71beb9426e01325064de520d156a67a ##新节点node_id Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1: all ##输入all表示所有节点slot重新洗牌，输入单个node_id 或者多个node_id（空格分开）表示从输入的节点中抽取 Ready to move 1025 slots. Source nodes: M: 05afba26af286f583204ed8db95cef91cf422d9f 192.168.2.64:6370 slots:[0-5460] (5461 slots) master 1 additional replica(s) M: a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 192.168.2.64:6372 slots:[5461-10922] (5462 slots) master 1 additional replica(s) M: 2f899821dbfe2e2d5aa2c22e9e023eb9a9f127f3 192.168.2.34:6378 slots: (0 slots) master M: bd6c1bd4328c3a8f0b17987285f007e7a3733776 192.168.2.64:6371 slots:[10923-16383] (5461 slots) master 2 additional replica(s) Destination node: M: e954db69f71beb9426e01325064de520d156a67a 192.168.2.34:6377 slots: (0 slots) master Resharding plan: Moving slot 5461 from a0069cc8774b1139c4ff696c1c2dbfbe0b44d598
...
##再次查看node信息,6377已经有了slot，并且在多个区段内 [root@project-deve redis]# ./bin/redis-cli -h 192.168.2.64 -p 6370 -a 123456 -c cluster nodes Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. c4729c87272ebab033d1b78ca00b6c93a15c02eb 192.168.2.24:6375@16375 slave 05afba26af286f583204ed8db95cef91cf422d9f 0 1634110713692 1 connected 05afba26af286f583204ed8db95cef91cf422d9f 192.168.2.64:6370@16370 myself,master - 0 1634110708000 1 connected 342-5460 36f8084c986ce896593ea7cf00496fdcdef70a11 192.168.2.24:6374@16374 slave bd6c1bd4328c3a8f0b17987285f007e7a3733776 0 1634110712187 2 connected 2f899821dbfe2e2d5aa2c22e9e023eb9a9f127f3 192.168.2.34:6378@16378 master - 0 1634110711184 8 connected bd6c1bd4328c3a8f0b17987285f007e7a3733776 192.168.2.64:6371@16371 master - 0 1634110710000 2 connected 11264-16383 a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 192.168.2.64:6372@16372 master - 0 1634110714193 9 connected 5803-10922 0936b30e022b7fad6ddb3ce6aee1b96b8c1d1c6b 192.168.2.34:6376@16376 slave e954db69f71beb9426e01325064de520d156a67a 0 1634110710182 10 connected e954db69f71beb9426e01325064de520d156a67a 192.168.2.34:6377@16377 master - 0 1634110710000 10 connected 0-341 5461-5802 10923-11263 088548d9ebb6e3b908bb74e0f8d8a397b2c53cd3 192.168.2.24:6373@16373 slave a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 0 1634110713190 9 connected
三、删除节点
删除集群中6377节点，删除之前先把cluster分配的slot给重新分配一下（很重要）。
##转移6377的1025个slot，删除6377节点 [root@project-deve redis]# ./bin/redis-cli -h 192.168.2.64 -p 6370 -a 123456 -c --cluster reshard 192.168.2.34:6378 How many slots do you want to move (from 1 to 16384)? 1025 ##6377原有的1025个slot What is the receiving node ID? 2f899821dbfe2e2d5aa2c22e9e023eb9a9f127f3 #6378的node_id Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1: e954db69f71beb9426e01325064de520d156a67a ##6377的node_id Source node #2: done ##结束 ##查看slot已经全部分配给了6378，6377已经没有slot [root@project-deve redis]# ./bin/redis-cli -h 192.168.2.64 -p 6370 -a 123456 -c cluster nodes Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. c4729c87272ebab033d1b78ca00b6c93a15c02eb 192.168.2.24:6375@16375 slave 05afba26af286f583204ed8db95cef91cf422d9f 0 1634112240719 1 connected 05afba26af286f583204ed8db95cef91cf422d9f 192.168.2.64:6370@16370 myself,master - 0 1634112238000 1 connected 342-5460 36f8084c986ce896593ea7cf00496fdcdef70a11 192.168.2.24:6374@16374 slave bd6c1bd4328c3a8f0b17987285f007e7a3733776 0 1634112242707 2 connected 2f899821dbfe2e2d5aa2c22e9e023eb9a9f127f3 192.168.2.34:6378@16378 master - 0 1634112238000 13 connected 0-341 5461-5802 10923-11263 bd6c1bd4328c3a8f0b17987285f007e7a3733776 192.168.2.64:6371@16371 master - 0 1634112243710 2 connected 11264-16383 a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 192.168.2.64:6372@16372 master - 0 1634112240000 9 connected 5803-10922 0936b30e022b7fad6ddb3ce6aee1b96b8c1d1c6b 192.168.2.34:6376@16376 slave e954db69f71beb9426e01325064de520d156a67a 0 1634112241704 12 connected e954db69f71beb9426e01325064de520d156a67a 192.168.2.34:6377@16377 master - 0 1634112239089 12 connected 088548d9ebb6e3b908bb74e0f8d8a397b2c53cd3 192.168.2.24:6373@16373 slave a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 0 1634112242000 9 connected ##删除6377 ./bin/redis-cli -a {password} --cluster del-node {del_ip}:{del_port} {del_node_id} [root@project-deve redis]# ./bin/redis-cli -a 123456 --cluster del-node 192.168.2.34:6377 e954db69f71beb9426e01325064de520d156a67a Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. >>> Removing node e954db69f71beb9426e01325064de520d156a67a from cluster 192.168.2.34:6377 >>> Sending CLUSTER FORGET messages to the cluster... >>> Sending CLUSTER RESET SOFT to the deleted node. ##查看节点信息，node 6377已经被删除 [root@project-deve redis]# ./bin/redis-cli -h 192.168.2.64 -p 6370 -a 123456 -c cluster nodes Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. c4729c87272ebab033d1b78ca00b6c93a15c02eb 192.168.2.24:6375@16375 slave 05afba26af286f583204ed8db95cef91cf422d9f 0 1634112950788 1 connected 05afba26af286f583204ed8db95cef91cf422d9f 192.168.2.64:6370@16370 myself,master - 0 1634112948000 1 connected 342-5460 36f8084c986ce896593ea7cf00496fdcdef70a11 192.168.2.24:6374@16374 slave bd6c1bd4328c3a8f0b17987285f007e7a3733776 0 1634112953797 2 connected 2f899821dbfe2e2d5aa2c22e9e023eb9a9f127f3 192.168.2.34:6378@16378 master - 0 1634112951000 13 connected 0-341 5461-5802 10923-11263 bd6c1bd4328c3a8f0b17987285f007e7a3733776 192.168.2.64:6371@16371 master - 0 1634112947276 2 connected 11264-16383 a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 192.168.2.64:6372@16372 master - 0 1634112951790 9 connected 5803-10922 0936b30e022b7fad6ddb3ce6aee1b96b8c1d1c6b 192.168.2.34:6376@16376 slave 2f899821dbfe2e2d5aa2c22e9e023eb9a9f127f3 0 1634112950000 13 connected 088548d9ebb6e3b908bb74e0f8d8a397b2c53cd3 192.168.2.24:6373@16373 slave a0069cc8774b1139c4ff696c1c2dbfbe0b44d598 0 1634112952794 9 connected
备注：./bin/redis-cli -a 123456 --cluster del-node 192.168.2.34:6377 e954db69f71beb9426e01325064de520d156a67a
通过这种方式将节点从集群中移除，如果后续要将该节点再次加入集群，直接加入也会报上面这个异常，因为此时节点的nodes_cluster.conf配置文件中存在之前集群节点信息，需要将该文件删除再启动节点加入集群。
五、集群的启停
。。。
