redis cluster-config-file文件丢失集群重建

环境:
OS:Centos 7
redis:4.0.14

1.新建集群模拟写入数据

[root@dev_test_redis conf]# /usr/local/services/redis/bin/redis-cli -c -h 192.168.1.84 -p 6379 -a hxl2018
Warning: Using a password with '-a' option on the command line interface may not be safe.
192.168.1.84:6379> set name hxl
-> Redirected to slot [5798] located at 192.168.1.84:6380
OK

 

2.停掉集群
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6379 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6380 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6381 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6382 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6383 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6384 shutdown

 

3.模拟删除一个节点的集群配置文件
我这里删除端口为6379的集群配置文件,该配置文件自动生成在dir参数指定的目录,名称由参数定义 cluster-config-file "6379.conf"
[root@dev_test_redis redis_6379]#cd /home/middle/redis_01/data/redis_6379
[root@dev_test_redis redis_6379]# mv 6379.conf mybak_6379.conf

 

4.同时修改配置改节点的配置文件
vi /home/middle/redis_01/conf/6379.conf
cluster-config-file "new_6379.conf"

 

5.启动集群
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6379.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6380.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6381.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6382.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6383.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6384.conf

 

这个时候6379节点会以新的名称生成文件
[root@dev_test_redis redis_6379]# ls
appendonly.aof dump.rdb mybak_6379.conf new_6379.conf

 

6.检查集群状态

/usr/local/services/redis/bin/redis-cli -c -h 192.168.1.84 -p 6379 -a hxl2018
192.168.1.84:6379> cluster info
cluster_state:fail
cluster_slots_assigned:1
cluster_slots_ok:1
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:1
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_pong_sent:5
cluster_stats_messages_sent:5
cluster_stats_messages_ping_received:5
cluster_stats_messages_received:5

这个时候改节点的集群状态是:fail
查看自动生成配置文件的内容:

[root@dev_test_redis redis_6379]# more new_6379.conf 
fc5c5353add6563a2ea9022d926cd0e7cd14c4e3 192.168.1.84:6379@16379 myself,master - 0 0 0 connected 741
vars currentEpoch 0 lastVoteEpoch 0
[root@dev_test_redis redis_6379]# 

 

其他节点的状态是ok的

[root@dev_test_redis redis_6379]# /usr/local/services/redis/bin/redis-cli -c -h 192.168.1.84 -p 6380 -a hxl2018
Warning: Using a password with '-a' option on the command line interface may not be safe.
192.168.1.84:6380> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:2
cluster_stats_messages_ping_sent:493
cluster_stats_messages_pong_sent:66
cluster_stats_messages_fail_sent:4
cluster_stats_messages_sent:563
cluster_stats_messages_ping_received:66
cluster_stats_messages_pong_received:60
cluster_stats_messages_fail_received:2
cluster_stats_messages_received:128

 

7.恢复1(原来的集群配置文件有备份)
停掉集群所有的节点

/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6379 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6380 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6381 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6382 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6383 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6384 shutdown

将原来备份的6379节点的集群配置文件恢复回来
cd /home/middle/redis_01/data/redis_6379
mv mybak_6379.conf 6379.conf

修改配置文件
vi /home/middle/redis_01/conf/
cluster-config-file "6379.conf"

重新启动集群
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6379.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6380.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6381.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6382.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6383.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6384.conf

 

这个时候集群状态就正常了

[root@dev_test_redis redis_6379]# /usr/local/services/redis/bin/redis-cli -c -h 192.168.1.84 -p 6379 -a hxl2018
Warning: Using a password with '-a' option on the command line interface may not be safe.
192.168.1.84:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:744
cluster_stats_messages_pong_sent:102
cluster_stats_messages_fail_sent:6
cluster_stats_messages_sent:852
cluster_stats_messages_ping_received:102
cluster_stats_messages_pong_received:99
cluster_stats_messages_fail_received:3
cluster_stats_messages_received:204

 

8.恢复1(原来的集群配置文件没有备份,需要重建集群)
a.停掉个实例
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6379 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6380 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6381 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6382 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6383 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6384 shutdown

b.修改配置文件将需要密码的注释掉
##masterauth "hxl2018"
##requirepass "hxl2018"

c.启动
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6379.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6380.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6381.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6382.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6383.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6384.conf

d.重新创建集群

/usr/local/services/redis/redis-trib.rb create --replicas 1 192.168.1.84:6379 192.168.1.84:6380 192.168.1.84:6381 192.168.1.84:6382 192.168.1.84:6383 192.168.1.84:6384

[root@dev_test_redis redis_6379]# /usr/local/services/redis/redis-trib.rb create --replicas 1 192.168.1.84:6379 192.168.1.84:6380 192.168.1.84:6381 192.168.1.84:6382 192.168.1.84:6383 192.168.1.84:6384
>>> Creating cluster
/opt/ruby/lib/ruby/gems/2.5.0/gems/redis-3.3.0/lib/redis/client.rb:459: warning: constant ::Fixnum is deprecated
[ERR] Node 192.168.1.84:6379 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

 

这里提示集群里存在数据

解决办法:
先停掉集群
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -p 6379 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -p 6380 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -p 6381 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -p 6382 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -p 6383 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -p 6384 shutdown

删除集群每个节点自动生成的配置文件
cd /home/middle/redis_01/data/redis_6379
rm 6379.conf
mv appendonly.aof bak_appendonly.aof
mv dump.rdb bak_dump.rdb

cd /home/middle/redis_01/data/redis_6380
rm 6380.conf
mv appendonly.aof bak_appendonly.aof
mv dump.rdb bak_dump.rdb

cd /home/middle/redis_01/data/redis_6381
rm 6381.conf
mv appendonly.aof bak_appendonly.aof
mv dump.rdb bak_dump.rdb

cd /home/middle/redis_01/data/redis_6382
rm 6382.conf
mv appendonly.aof bak_appendonly.aof
mv dump.rdb bak_dump.rdb

cd /home/middle/redis_01/data/redis_6383
rm 6383.conf
mv appendonly.aof bak_appendonly.aof
mv dump.rdb bak_dump.rdb

cd /home/middle/redis_01/data/redis_6384
rm 6384.conf
mv appendonly.aof bak_appendonly.aof
mv dump.rdb bak_dump.rdb

 

启动
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6379.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6380.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6381.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6382.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6383.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6384.conf

重新执行创建

/usr/local/services/redis/redis-trib.rb create --replicas 1 192.168.1.84:6379 192.168.1.84:6380 192.168.1.84:6381 192.168.1.84:6382 192.168.1.84:6383 192.168.1.84:6384

Can I set the above configuration? (type 'yes' to accept): yes
Traceback (most recent call last):
        11: from /usr/local/services/redis/redis-trib.rb:1830:in `<main>'
        10: from /usr/local/services/redis/redis-trib.rb:1426:in `create_cluster_cmd'
         9: from /usr/local/services/redis/redis-trib.rb:905:in `flush_nodes_config'
         8: from /usr/local/services/redis/redis-trib.rb:905:in `each'
         7: from /usr/local/services/redis/redis-trib.rb:906:in `block in flush_nodes_config'
         6: from /usr/local/services/redis/redis-trib.rb:212:in `flush_node_config'
         5: from /opt/ruby/lib/ruby/gems/2.5.0/gems/redis-3.3.0/lib/redis.rb:2699:in `method_missing'
         4: from /opt/ruby/lib/ruby/gems/2.5.0/gems/redis-3.3.0/lib/redis.rb:58:in `synchronize'
         3: from /opt/ruby/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
         2: from /opt/ruby/lib/ruby/gems/2.5.0/gems/redis-3.3.0/lib/redis.rb:58:in `block in synchronize'
         1: from /opt/ruby/lib/ruby/gems/2.5.0/gems/redis-3.3.0/lib/redis.rb:2700:in `block in method_missing'
/opt/ruby/lib/ruby/gems/2.5.0/gems/redis-3.3.0/lib/redis/client.rb:121:in `call': ERR Slot 741 is already busy (Redis::CommandError)

 

解决办法:
用redis-cli 登录到每个节点执行
flushall
cluster reset

/usr/local/services/redis/bin/redis-cli -c -h 192.168.1.84 -p 6379
/usr/local/services/redis/bin/redis-cli -c -h 192.168.1.84 -p 6380
/usr/local/services/redis/bin/redis-cli -c -h 192.168.1.84 -p 6381
/usr/local/services/redis/bin/redis-cli -c -h 192.168.1.84 -p 6382
/usr/local/services/redis/bin/redis-cli -c -h 192.168.1.84 -p 6383
/usr/local/services/redis/bin/redis-cli -c -h 192.168.1.84 -p 6384

停掉启用密码认证
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -p 6379 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -p 6380 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -p 6381 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -p 6382 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -p 6383 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -p 6384 shutdown

修改每个节点的配置文件,启用密码
masterauth "hxl2018"
requirepass "hxl2018"

启动
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6379.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6380.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6381.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6382.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6383.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6384.conf

这个时候就需要密码登录了
/usr/local/services/redis/bin/redis-cli -c -h 192.168.1.84 -p 6379 -a hxl2018
[root@dev_test_redis conf]# /usr/local/services/redis/bin/redis-cli -c -h 192.168.1.84 -p 6379 -a hxl2018
Warning: Using a password with '-a' option on the command line interface may not be safe.

恢复原来的数据
停掉每个实例
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6379 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6380 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6381 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6382 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6383 shutdown
/usr/local/services/redis/bin/redis-cli -h 192.168.1.84 -a hxl2018 -p 6384 shutdown

 

恢复原来的数据文件
cd /home/middle/redis_01/data/redis_6379
mv bak_dump.rdb dump.rdb
mv bak_appendonly.aof appendonly.aof

cd /home/middle/redis_01/data/redis_6380
mv bak_dump.rdb dump.rdb
mv bak_appendonly.aof appendonly.aof

cd /home/middle/redis_01/data/redis_6381
mv bak_dump.rdb dump.rdb
mv bak_appendonly.aof appendonly.aof

cd /home/middle/redis_01/data/redis_6382
mv bak_dump.rdb dump.rdb
mv bak_appendonly.aof appendonly.aof

cd /home/middle/redis_01/data/redis_6383
mv bak_dump.rdb dump.rdb
mv bak_appendonly.aof appendonly.aof

cd /home/middle/redis_01/data/redis_6384
mv bak_dump.rdb dump.rdb
mv bak_appendonly.aof appendonly.aof

启动
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6379.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6380.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6381.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6382.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6383.conf
/usr/local/services/redis/bin/redis-server /home/middle/redis_01/conf/6384.conf

 

登录查看数据
[root@dev_test_redis redis_6384]# /usr/local/services/redis/bin/redis-cli -c -h 192.168.1.84 -p 6379 -a hxl2018
Warning: Using a password with '-a' option on the command line interface may not be safe.
192.168.1.84:6379> get name
-> Redirected to slot [5798] located at 192.168.1.84:6380
"hxl"