kubernetes集群断电后etcd启动失败之etcd备份方案
一.问题描述
二进制部署的单Master节点的v1.13.10版本的集群，etcd部署的是3.3.10版本，部署在master节点上。在异常断电后，kubernetes集群无法正常启动。这里通过查看kubernetes和etcd的服务日志信息，发现etcd服务异常，无法重新启动，具体日志信息如下：
Jun 29 09:39:37 k8s001 etcd[3348]: recovered store from snapshot at index 2600026
Jun 29 09:39:37 k8s001 etcd[3348]: recovering backend from snapshot error: database snapshot file path error: snap: snapshoJun 29 09:39:37 k8s001.wf etcd[3348]: panic: r
ecovering backend from snapshot error: database snapshot file path error: snap: Jun 29 09:39:37 k8s001 etcd[3348]: panic: runtime error: invalid memory address or nil pointer dereferenceJun 29 09:39:37 k8s001 etcd[3348]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xb8cb90]
Jun 29 09:39:37 k8s001 etcd[3348]: goroutine 1 [running]:
Jun 29 09:39:37 k8s001 etcd[3348]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.NewServer.func1(0xc4Jun 29 09:39:37 k8s001 etcd[3348]: /tmp/etc
d-release-3.3.10/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/Jun 29 09:39:37 k8s001.wf etcd[3348]: panic(0xde0ce0, 0xc4200b10a0)Jun 29 09:39:37 k8s001 etcd[3348]: /usr/local/go/src/runtime/panic.go:502 +0x229
Jun 29 09:39:37 k8s001 etcd[3348]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog.(*PackageLogger).PanicfJun 29 09:39:37 k8s001 etcd[3348]: /tmp/etc
d-release-3.3.10/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/Jun 29 09:39:37 k8s001.wf etcd[3348]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.NewServer(0x7ffe787eJun 29 09:39:37 k8s001.wf etcd[3348]: /tmp/etcd-release-3.3.10/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/Jun 29 09:39:37 k8s001 etcd[3348]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.StartEtcd(0xc42019d680, 0Jun 29 09:39:37 k8s001 etcd[3348]: /tmp/etcd-release-3.3.10/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/
通过查看异常日志来看，etcd执行了恢复操作，但是无法从现有的快照数据进行数据恢复。这里查看了资料，发现社区也有类似的问题，此问题暂未修复：
https://github.com/etcd-io/etcd/issues/11949
https://github.com/kubernetes/kubernetes/issues/88574
二.问题解决方案
对于单master节点的集群，master作为整个集群的核心，如果etcd服务挂掉，将影响我们整个集群的使用。因此这里对etcd做一个备份方案，以备不时之需。
2.1备份方案
这个我们采用kubernetes的CronJob来实现etcd数据的定时备份。也就是kubernetes集群正常时，CronJob执行定时备份任务，如果kubernetes集群异常，则CrobJob也将不会执行。
- 备份etcd数据的yaml文件
[root@k8s001 home]# cat etcd_cronjob.yaml
---
apiVersion: batch/v2alpha1
kind: CronJob
metadata:
name: etcd-backup
spec:
# 30分钟执行一次备份
schedule: "*/30 * * * *"
jobTemplate:
spec:
template:
metadata:
labels:
app: etcd-disaster-recovery
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/role
operator: In
values:
- master
containers:
- name: etcd
image: etcd:backup
command:
- sh
- -c
- "export ETCDCTL_API=3; \
# 备份前执行下清理的操作，最多保留6个快照
sh -x /usr/bin/delete_image_reserver_5.sh; \
etcdctl --endpoints $ENDPOINT snapshot save /snapshot/$(date +%Y%m%d_%H%M%S)_snapshot.db; \
echo etcd backup sucess"
env:
- name: ENDPOINT
value: "127.0.0.1:2379"
volumeMounts:
- mountPath: "/snapshot"
name: snapshot
subPath: etcd-snapshot
- mountPath: /etc/localtime
name: lt-config
- mountPath: /etc/timezone
name: tz-config
restartPolicy: OnFailure
volumes:
- name: snapshot
hostPath:
path: /var
- name: lt-config
hostPath:
path: /etc/localtime
- name: tz-config
hostPath:
path: /etc/timezone
hostNetwork: true
# 设置master节点可调度
[root@k8s001 ~]# kubectl uncordon ${masterip}
# 创建定时备份任务
# 创建etcd定时备份的job
[root@k8s001 home]# kubectl apply -f etcd_cronjob.yaml
# 查看备份的快照
[root@k8s001 ~]# ls /var/etcd-snapshot/ -alh
total 45M
drwxr-xr-x 2 root root 216 Jun 30 16:05 .
drwxr-xr-x. 20 root root 288 Jun 28 16:10 ..
-rw-r--r-- 1 root root 7.5M Jun 30 15:45 20200630_154509_snapshot.db
-rw-r--r-- 1 root root 7.5M Jun 30 15:50 20200630_155009_snapshot.db
-rw-r--r-- 1 root root 7.5M Jun 30 15:55 20200630_155510_snapshot.db
-rw-r--r-- 1 root root 7.5M Jun 30 16:00 20200630_160010_snapshot.db
-rw-r--r-- 1 root root 7.5M Jun 30 16:03 20200630_160357_snapshot.db
-rw-r--r-- 1 root root 7.5M Jun 30 16:05 20200630_160510_snapshot.d
# 监控任务执行情况
[root@k8s001 ~]# kubectl get job --watch
NAME COMPLETIONS DURATION AGE
etcd-backup-1593504000 1/1 1s 9m20s
2.2 验证
这里我们验证下创建的快照是否可以进行数据的恢复：
2.2.1 停止集群的etcd服务和删除etcd数据
# 停止etcd服务
[root@k8s001 ~]# systemctl stop etcd
# 删除etcd数据
[root@k8s001 ~]# rm -rf /var/lib/etcd
# 查看集群服务是否还正常
[root@k8s001 ~]# kubectl get pod
The connection to the server 172.16.33.5:6443 was refused - did you specify the right host or port?
2.2.2 基于etcd快照恢复数据目录
[root@k8s001 ~]# cd /var/etcd_backup/
# 这里选用最新的一个快照进行数据目录恢复
[root@k8s001 ~]# export ETCDCTL_API=3
[root@k8s001 ~]# etcdctl snapshot restore 20200630_161001_snapshot.db --data-dir /var/lib/etcd
2020-06-30 16:19:29.789757 I | mvcc: restore compact to 6142751
2020-06-30 16:19:29.807133 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
# 查看执行快照恢复后的数据目录
[root@k8s001 etcd-snapshot]# tree /var/lib/etcd/
/var/lib/etcd/
└── member
├── snap
│ ├── 0000000000000001-0000000000000001.snap
│ └── db
└── wal
└── 0000000000000000-0000000000000000.wal
# 启动etcd服务
[root@k8s001 etcd-snapshot]# systemctl restart etcd
[root@k8s001 etcd-snapshot]# systemctl status etcd
● etcd.service - Etcd Server
Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2020-06-30 16:20:27 CST; 6s ago
Docs: https://github.com/coreos
Main PID: 3069327 (etcd)
Tasks: 15
Memory: 17.5M
CGroup: /system.slice/etcd.service
└─3069327 /usr/bin/etcd --name=k8s001 --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem --peer-cert-file=/etc/etcd/ssl/etcd.pem --pe...
Jun 30 16:20:27 k8s001 etcd[3069327]: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2
Jun 30 16:20:27 k8s001 etcd[3069327]: setting up the initial cluster version to 3.3
Jun 30 16:20:27 k8s001 etcd[3069327]: set the initial cluster version to 3.3
Jun 30 16:20:27 k8s001 etcd[3069327]: published {Name:k8s001 ClientURLs:[https://172.16.33.5:2379]} to cluster cdf818194e3a8c32
Jun 30 16:20:27 k8s001 etcd[3069327]: enabled capabilities for version 3.3
Jun 30 16:20:27 k8s001 etcd[3069327]: ready to serve client requests
Jun 30 16:20:27 k8s001 etcd[3069327]: ready to serve client requests
Jun 30 16:20:27 k8s001 systemd[1]: Started Etcd Server.
Jun 30 16:20:27 k8s001 etcd[3069327]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
Jun 30 16:20:27 k8s001 etcd[3069327]: serving client requests on 172.16.33.5:2379
# 查看业务的服务是否丢失，从下面可知业务服务恢复正常
[root@k8s001 etcd-snapshot]# kubectl get pod -n business -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
redis-4ghyausyd-9hejh 1/1 Running 1 28d 172.20.0.30 172.16.33.5 <none> <none>
mysql-c6994b67c-jx9rb 1/1 Running 0 28d 172.20.0.233 172.16.33.5 <none> <none>
