Redis Cluster两slave节点处理

背景

系统：CentOS 7.6

Redis Cluster

IP	端口	角色
192.168.100.11	6380	Master
192.168.100.11	6381	slave
192.168.100.12	6380	Master
192.168.100.12	6381	Slave
192.168.100.13	6380	Master
192.168.100.13	6381	Slave

 

因为某种原因导致（192.168.100.12）操作系统宕机后无法修复。

 

查看集群节点信息，可以看到redis cluster的3个master节点均正常

[root]# redis-cli -p 6380 -a {redis密码} -c cluster nodes
4f61cdeb684fec56a82cc5f3cc72f0def5df05c3 :0@0 slave,fail,noaddr 297a72d200be658a37658bd0ed9bda01d96af43f 1567608093527 1567608091611 9 disconnected
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128078000 11 connected 10923-16383
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128077000 10 connected 0-5460
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 myself,master - 0 1684128076000 9 connected 5461-10922
3c2fdc0e8e7295b14aff8c6909b35c38f4939907 :0@0 slave,fail,noaddr 7d221e66c3bece129bbe013687feaea093cc3714 1567608092521 1567608091611 11 disconnected
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128079071 10 connected

处理过程

从集群中剔除ID为：“4f61cdeb684fec56a82cc5f3cc72f0def5df05c3”、“3c2fdc0e8e7295b14aff8c6909b35c38f4939907”的节点

[root]# redis-cli -p 6380 -a {redis密码} -c cluster forget 4f61cdeb684fec56a82cc5f3cc72f0def5df05c3
[root]# redis-cli -p 6380 -a {redis密码} -c cluster forget 3c2fdc0e8e7295b14aff8c6909b35c38f4939907

 

再次查看集群节点信息，可以看到异常的两个节点从redis cluster中剔除

[root]# redis-cli -p 6380 -a {redis密码} -c cluster nodes 
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128261000 11 connected 10923-16383
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128261513 10 connected 0-5460
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 myself,master - 0 1684128260000 9 connected 5461-10922
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128259508 10 connected

 

待192.168.100.12上的redis服务重建后，重新加入节点

[root]# redis-cli -p 6380 -a {redis密码} -c 
127.0.0.1:6380> cluster meet 192.168.100.12 6380
OK
127.0.0.1:6380> cluster meet 192.168.100.12 6381
OK

 

查看redis cluster节点信息，可以看到新增的两个节点是以master的角色加入到集群中的

10.12.31.79:6380> cluster nodes
c0f760cbfbc6314e7273bcb798d2621f932bd240 192.168.100.12:6381@16381 master - 0 1684128613906 0 connected
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 master - 0 1684128614907 9 connected 5461-10922
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128615909 10 connected 0-5460
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128615000 11 connected 10923-16383
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128612502 10 connected
84516161919c5b04a11b889ff8750725415902e3 192.168.100.12:6380@16380 myself,master - 0 1684128611000 12 connected

 

连接10.12.31.79:6380，配置为某个master的slave

10.12.31.79:6380> cluster replicate 297a72d200be658a37658bd0ed9bda01d96af43f
OK
10.12.31.79:6380> cluster nodes
c0f760cbfbc6314e7273bcb798d2621f932bd240 192.168.100.12:6381@16381 master - 0 1684128712119 0 connected
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 master - 0 1684128711118 9 connected 5461-10922
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128713121 10 connected 0-5460
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128711000 11 connected 10923-16383
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128711000 10 connected
84516161919c5b04a11b889ff8750725415902e3 192.168.100.12:6380@16380 myself,slave 297a72d200be658a37658bd0ed9bda01d96af43f 0 1684128709000 9 connected

 

分别连接10.12.31.79:6381，配置为某个master的slave

10.12.31.79:6381> cluster replicate 7d221e66c3bece129bbe013687feaea093cc3714
OK
10.12.31.79:6381> cluster nodes
c0f760cbfbc6314e7273bcb798d2621f932bd240 192.168.100.12:6381@16381 myself,slave 7d221e66c3bece129bbe013687feaea093cc3714 0 1684128760000 11 connected
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128762000 10 connected
84516161919c5b04a11b889ff8750725415902e3 192.168.100.12:6380@16380 slave 297a72d200be658a37658bd0ed9bda01d96af43f 0 1684128760000 9 connected
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 master - 0 1684128763000 9 connected 5461-10922
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128763957 10 connected 0-5460
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128763000 11 connected 10923-16383

 

后续根据节点分布可以重启master节点进行调整。