etcd节点故障处理

问题:巡检发现k8s集群的etcd集群状态不对,其中有一个节点不健康,现象如下:

[root@k8s-master1 ~]# kubectl get cs
NAME                 STATUS      MESSAGE                                  ERROR
controller-manager   Healthy     ok                                       
scheduler            Healthy     ok                                       
etcd-1               Healthy     {"health":"true"}                        
etcd-0               Healthy     {"health":"true"}                        
etcd-2               Unhealthy   HTTP probe failed with statuscode: 503

而且查询etcd日志没有太多报错信息,时间和证书都是正常的,而且也没有防火墙问题,于是开始进行如下操作

1.将有故障的etcd节点remove出集群:

[root@k8s-master1 ~]# /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://172.16.23.120:2379,https://172.16.23.121:2379,https://172.16.23.122:2379" member list
20fd79755169a89, started, etcd-3, https://172.16.23.122:2380, https://172.16.23.122:2379, false
39356a19c9b19f6d, started, etcd-1, https://172.16.23.120:2380, https://172.16.23.120:2379, false
506e9a48a5c19ec3, started, etcd-2, https://172.16.23.121:2380, https://172.16.23.121:2379, false

由上面信息可知,有故障的etcd节点为etcd-2这个,对应etcd-3这个name也就是122这一台机器

[root@k8s-master1 ~]# /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://172.16.23.120:2379,https://172.16.23.121:2379,https://172.16.23.122:2379" member remove 20fd79755169a89
Member  20fd79755169a89 removed from cluster ad1f122f981ee2bf
[root@k8s-master1 ~]# /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://172.16.23.120:2379,https://172.16.23.121:2379,https://172.16.23.122:2379" member list
39356a19c9b19f6d, started, etcd-1, https://172.16.23.120:2380, https://172.16.23.120:2379, false
506e9a48a5c19ec3, started, etcd-2, https://172.16.23.121:2380, https://172.16.23.121:2379, false

2.第一步已经将有故障的etcd节点etcd-2剔除了集群,开始操作etcd-3这个节点,删除etcd数据,然后将etcd配置文件集群信息由new修改为existing

# rm -rf /var/lib/etcd/default.etcd/member/

修改etcd配置文件,将下面new修改为:

修改前:

ETCD_INITIAL_CLUSTER_STATE="new"

修改后:

ETCD_INITIAL_CLUSTER_STATE="existing"

3.然后将etcd-3节点加入到集群:

[root@k8s-master1 ~]# /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://172.16.23.120:2379,https://172.16.23.121:2379,https://172.16.23.122:2379" member add etcd-2 --peer-urls=https://172.16.23.122:2380
Member a98137c10970d43c added to cluster ad1f122f981ee2bf

然后查看集群列表:

[root@k8s-master1 ~]# /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://172.16.23.120:2379,https://172.16.23.121:2379,https://172.16.23.122:2379" member list
39356a19c9b19f6d, started, etcd-1, https://172.16.23.120:2380, https://172.16.23.120:2379, false
506e9a48a5c19ec3, started, etcd-2, https://172.16.23.121:2380, https://172.16.23.121:2379, false
a98137c10970d43c, unstarted, , https://172.16.23.122:2380, , false

4.重启etcd故障节点:

[root@k8s-master3 ~]# systemctl start etcd
[root@k8s-master3 ~]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since 日 2021-02-28 22:04:34 CST; 4s ago

最后查看k8s集群的etcd:

[root@k8s-master1 ~]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-2               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}