redis迁移第三篇(cluster forget)

1、删除错误节点,带有 fail,noaddr , 这种需要用 cluster forget

redis集群迁移之后，由于之前的误操作，导致pod日志里面出现这样的错误，出现一会好一会不好的情况，就是由于这个错误节点导致
[uts-node-0] [12:22:48:102] [ERROR] - org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:113) - Unable to decode data. channel: [id: 0x2770e76d, L:/172.17.246.215:45564 - R:/10.129.51.30:7733] message: $879
f0e5d3d17babbce85a9290e679a626bc0bbacc93 10.129.51.30:7735@17735 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 0 1646366975000 63 connected
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 master - 0 1646366976611 63 connected 5461-10922
c3c7ba2d0709121e69c2881724a0c5be903a4a6a 10.129.51.30:7736@17736 slave d1c7d99e13a2d7317baf883dffa906470a606641 0 1646366975000 62 connected
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 myself,master - 0 1646366969000 62 connected 10923-16383
27f7e8dede5b68581486ce8cefdd656032baed70 :0@0 master,fail,noaddr - 1646366063257 1646366063257 4 disconnected
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734@17734 slave d46f032ea50763de8353fd530535412df6ffdc00 0 1646366976000 60 connected
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 master - 0 1646366977611 60 connected 0-5460

红色的一条就是我之前在迁移中失误操作造成的

pod启动后还会报错信息为：

java.lang.IllegalArgumentException: Illegal character in scheme name at index 0: []r[]e[]d[]i[]s[]:[]/[]/[]:[]0[]

红色的一条节点信息，是由于之前redis失误命令为：

[root@ht20 redis]#./redis-trib.rb del-node 10.129.51.30:7735 10.129.51.30

正确命令应该为：后面是 node_id
[root@ht20 redis]#./redis-trib.rb del-node 10.129.51.30:7735 10.129.51.30 32ee19af1f6a534c4014b9c41d387666f049354b

2、cluster forget命令操作,中间影响信息的部分处理掉了。

[root@ht20 data]# vi /data/redis_fpmai1/redis/data/nodes.conf   //查看master1的集群配置文件
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 slave   
c3c7ba2d0709121e69c2881724a0c5be903a4a6a 10.129.51.30:7736@17736 slave  
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 myself,master -  5461-10922
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master -  10923-16383
27f7e8dede5b68581486ce8cefdd656032baed70 :0@0 master,fail,noaddr - 1646368054015 1646368054015 4 disconnected
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734@17734 master -  0-5460
f0e5d3d17babbce85a9290e679a626bc0bbacc93 10.129.51.30:7735@17735 slave 
vars currentEpoch 64 lastVoteEpoch 64
 
[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7732 cluster forget 27f7e8dede5b68581486ce8cefdd656032baed70
OK
[root@ht20 data]# vi /data/redis_fpmai2/redis/data/nodes.conf   //查看master2的集群配置文件
c3c7ba2d0709121e69c2881724a0c5be903a4a6a 10.129.51.30:7736@17736 slave  
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734@17734 master  0-5460
f0e5d3d17babbce85a9290e679a626bc0bbacc93 10.129.51.30:7735@17735 slave  
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 slave 
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 myself,master  5461-10922
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master  10923-16383
vars currentEpoch 64 lastVoteEpoch 64
###就会发现没有这一条了   27f7e8dede5b68581486ce8cefdd656032baed70 :0@0 master,fail,noaddr - 1646368054015 1646368054015 4 disconnected

1、注意cluster forget 只针对当前的主和当前的从起作用。

2、我依次处理掉其他master ，一共3个master

//对三个master都要进行处理,否则会有残留信息。

[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7733 cluster forget 27f7e8dede5b68581486ce8cefdd656032baed70
OK

[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7732 cluster forget 27f7e8dede5b68581486ce8cefdd656032baed70

OK

[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7733 cluster forget 27f7e8dede5b68581486ce8cefdd656032baed70
OK

3、程序执行正常，但是会出现 No reachable node in cluster（这个意思很怪异，如果你刚接触redis，会蒙的，怎么会是不能到达节点，集群中的）
经过多方查找这个错误，最终确定是由于在程序中jredis没有改配置中心的配置文件的ip信息，开发人员只修改了redisson即连接池的信息，
但是没有修改jredis的ip端口信息导致jredis连接不上redis集群，报错为： No reachable node in cluster 详细说下排查过程
我们配置中心采用config-->eureka-->应用-->redis集群

[root@master-web-38 src]# kubectl exec -ti us-node-0 -- /bin/bash //进入master ping 外网也没有问题

[root@uts-node-0:/# kubectl cp us-node-0:/app.jar /usr/local/src //把程序包app.jar下载之后,改成app.zip，查看里面内容
里面lib下的java jar包进行破解，利用工具为 http://java-decompiler.github.io/#jd-gui-download
破解lib包里面的redission-redisson-3.6.5.jar和spring-data-redis-2.0.6.RELEASE.jar后跟踪到提示信息。

说明如下
1、程序端采用了redission连接池和spring支持的jredis客户端
2、redisson-3.6.5.jar //java客户端连接池,检查没有问题。
3、spring-data-redis-2.0.6.RELEASE.jar //客户端的jredis支持。

最终跟踪pod日志报错，发现提示set错误，跟踪连接池一直到jredis显示报错

public Jedis getConnection() {
List<JedisPool> pools = this.cache.getShuffledNodesPool();
for (JedisPool pool : pools) {
Jedis jedis = null;
try {
jedis = pool.getResource();
if (jedis == null)
continue;
String result = jedis.ping();
if (result.equalsIgnoreCase("pong"))
return jedis;
jedis.close();
} catch (JedisException ex) {
if (jedis != null)
jedis.close();
}
}
throw new JedisNoReachableClusterNodeException("No reachable node in cluster");
//查到这里基本问题就确定了，明显是连接jredis连接不上，从而确定问题来自 jredis，访问集群问题.
}

//这里jredis用了集群的ping命令

登录集群服务器客户端，执行ping,演示下jredis的机制。

[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7733
10.129.51.30:7733> ping
PONG

附日志错误

org.springframework.data.redis.RedisConnectionFailureException: No reachable node in cluster; nested exception is

 redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
     
    at redis.clients.jedis.BinaryJedisCluster.set(BinaryJedisCluster.java:77)
    at org.springframework.data.redis.connection.jedis.JedisClusterStringCommands.set(JedisClusterStringCommands.java:113)
    ... 80 more
org.springframework.data.redis.RedisConnectionFailureException: No reachable node in cluster; nested exception

is redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
    at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:67)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
    at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnection(JedisSlotBasedConnectionHandler.java:57)
 
    at org.springframework.data.redis.connection.jedis.JedisClusterStringCommands.set(JedisClusterStringCommands.java:113)
    ... 80 more
org.springframework.data.redis.RedisConnectionFailureException: No reachable node in cluster; nested exception is 
redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
    at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:67)
    at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:41)
    at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)

posted @ 2022-03-04 13:50 jinzi 阅读(1241) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部