etcd数据备份与恢复验证
一、单机
说明：执行etcd备份数据的恢复的机器必须和原先etcd所在机器一致
1、单机备份
1 etcdctl --endpoints="https://10.25.72.62:2379" \ 2 --cert=/etc/etcd/ssl/etcd.pem \ 3 --key=/etc/etcd/ssl/etcd-key.pem \ 4 --cacert=/etc/kubernetes/ssl/ca.pem \ 5 snapshot save snapshot.db
2、单机数据恢复
1 [root@SZD-L0105331 ~]# systemctl stop etcd # 停止etcd服务 2 [root@SZD-L0105331 ~]# export ETCDCTL_API=3; # 使用ETCDCTL API 3 3 [root@SZD-L0105331 ~]# etcdctl snapshot restore snapshot.db \ 4 --name=SZD-L0105331 \ 5 --endpoints=https://10.25.72.62:2379 \ 6 --cacert=/etc/kubernetes/ssl/ca.pem \ 7 --cert=/etc/etcd/ssl/etcd.pem \ 8 --key=/etc/etcd/ssl/etcd-key.pem \ 9 --initial-cluster=SZD-L0105331=https://10.25.72.62:2380 \ 10 --initial-advertise-peer-urls=https://10.25.72.62:2380 \ 11 --initial-cluster-token=etcd-cluster-0 \ 12 --data-dir=/var/lib/etcd4 # 注意--data-dir参数,以下步骤需要使用 13 14 [root@SZD-L0105331 ~]# # 修改etcd启动参数--data-dir指向上一步的数据回复目录，一般在/etc/etcd/etcd文件中 15 16 [root@SZD-L0105331 ~]# cat /etc/etcd/etcd # 完整参数文件如下 17 ETCD_OPTIONS="--name=SZD-L0105331 \ 18 --client-cert-auth=true \ 19 --cert-file=/etc/etcd/ssl/etcd.pem \ 20 --key-file=/etc/etcd/ssl/etcd-key.pem \ 21 --peer-cert-file=/etc/etcd/ssl/etcd.pem \ 22 --peer-key-file=/etc/etcd/ssl/etcd-key.pem \ 23 --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \ 24 --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \ 25 --initial-advertise-peer-urls=https://10.25.72.62:2380 \ 26 --listen-peer-urls=https://10.25.72.62:2380 \ 27 --listen-client-urls=https://10.25.72.62:2379,https://127.0.0.1:2379 \ 28 --advertise-client-urls=https://10.25.72.62:2379 \ 29 --initial-cluster-token=etcd-cluster-0 \ 30 --initial-cluster=SZD-L0105331=https://10.25.72.62:2380 \ 31 --initial-cluster-state=new \ 32 --data-dir=/var/lib/etcd" 33 34 [root@SZD-L0105331 ~]# systemctl start etcd # 启动etcd服务
二、集群
1、模拟写入数据到Etcd集群
1 # 使用API 3写入数据库 2 [root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=3 etcdctl --endpoints="https://10.25.84.251:2379,https://10.25.73.25:2379,https://10.25.73.150:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem put /name/1 zxg 3 # 读取数据 4 [root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=3 etcdctl --endpoints="https://10.25.84.251:2379,https://10.25.73.25:2379,https://10.25.73.150:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem get /name/1 5 2018-08-16 20:47:13.017586 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated 6 /name/1 7 zxg 8 9 # 使用API 2写入数据库 10 [root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=2 etcdctl --cert-file /root/etcd-cluster/etcd.pem --ca-file /root/etcd-cluster/ca.pem --key-file /root/etcd-cluster/etcd-key.pem --endpoints="https://10.25.73.25:2379,https://10.25.73.150:2379,https://10.25.84.251:2379" set /name1 zxg1 11 2018-08-16 18:27:35.060559 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated 12 zxg1 13 # 读取数据 14 [root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=2 etcdctl --cert-file /root/etcd-cluster/etcd.pem --ca-file /root/etcd-cluster/ca.pem --key-file /root/etcd-cluster/etcd-key.pem --endpoints="https://10.25.73.25:2379,https://10.25.73.150:2379,https://10.25.84.251:2379" get /name1 15 2018-08-16 18:27:47.884908 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated 16 zxg1
2、备份etcd数据
1 [root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=3 etcdctl --endpoints="https://10.25.84.251:2379,https://10.25.73.25:2379,https://10.25.73.150:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem snapshot save mysnapshot.db 2 2018-08-16 20:50:47.206334 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated 3 Snapshot saved at mysnapshot.db
3、停止etcd集群
停止方法：分别在3台etcd的宿主机上执行以下命令停止etcd服务
systemctl stop etcd
停掉Leader 10.25.73.25， 查看集群状况，重新选举出了leader，集群可正常使用
1 [root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=3 etcdctl --endpoints="https://10.25.84.251:2379,https://10.25.73.25:2379,https://10.25.73.150:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem endpoint status --write-out=table 2 2018-08-16 18:31:50.686352 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated 3 Failed to get the status of endpoint https://10.25.73.25:2379 (context deadline exceeded) 4 +---------------------------+------------------+---------+---------+-----------+-----------+------------+ 5 | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX | 6 +---------------------------+------------------+---------+---------+-----------+-----------+------------+ 7 | https://10.25.84.251:2379 | 2a53eb67dfa37cc0 | 3.1.10 | 25 kB | false | 14 | 19 | 8 | https://10.25.73.150:2379 | c267a0ca02c6bff7 | 3.1.10 | 25 kB | true | 14 | 19 | 9 +---------------------------+------------------+---------+---------+-----------+-----------+------------+
停掉10.25.73.150，查看集群状况，集群已经无法正常使用，说明3节点的Etcd容错为1
1 [root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=3 etcdctl --endpoints="https://10.25.84.251:2379,https://10.25.73.25:2379,https://10.25.73.150:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem endpoint status --write-out=table 2 2018-08-16 18:33:44.526382 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated 3 Failed to get the status of endpoint https://10.25.73.25:2379 (context deadline exceeded) 4 Failed to get the status of endpoint https://10.25.73.150:2379 (context deadline exceeded) 5 +---------------------------+------------------+---------+---------+-----------+-----------+------------+ 6 | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX | 7 +---------------------------+------------------+---------+---------+-----------+-----------+------------+ 8 | https://10.25.84.251:2379 | 2a53eb67dfa37cc0 | 3.1.10 | 25 kB | false | 14 | 19 | 9 +---------------------------+------------------+---------+---------+-----------+-----------+------------+
删除etcd数据，方法：登录etcd所在主机执行：(注意：危险操作，请谨慎操作，确保在有数据备份并且确定Etcd集群无法正常工作后操作)
rm -rf /var/lib/etcd
4、使用备份数据进行恢复
恢复10.25.84.251节点数据到/var/lib/etcd
[root@SZD-L0097856 etcd-cluster]# ETCDCTL_API=3 etcdctl --name=SZD-L0097856 --endpoints="https://10.25.84.251:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem --initial-cluster-token=etcd-cluster-0 --initial-advertise-peer-urls=https://10.25.84.251:2380 --initial-cluster=SZD-L0101798=https://10.25.73.25:2380,SZD-L0103739=https://10.25.73.150:2380,SZD-L0097856=https://10.25.84.251:2380 --data-dir=/varlib/etcd snapshot restore mysnapshot.db
2018-08-16 19:52:33.409406 I | etcdserver/membership: added member 2a53eb67dfa37cc0 [https://10.25.84.251:2380] to cluster 4741a9e2cf17e1fa
2018-08-16 19:52:33.409481 I | etcdserver/membership: added member 304bcbfa92e84c75 [https://10.25.73.25:2380] to cluster 4741a9e2cf17e1fa
2018-08-16 19:52:33.409499 I | etcdserver/membership: added member c267a0ca02c6bff7 [https://10.25.73.150:2380] to cluster 4741a9e2cf17e1fa
[root@SZD-L0097856 etcd-cluster]#
恢复10.25.73.25节点数据到/var/lib/etcd
[root@SZD-L0101798 etcd-cluster]# ETCDCTL_API=3 etcdctl --name=SZD-L0101798 --endpoints="https://10.25.73.25:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem --initial-cluster-token=etcd-cluster-0 --initial-advertise-peer-urls=https://10.25.73.25:2380 --initial-cluster=SZD-L0101798=https://10.25.73.25:2380,SZD-L0103739=https://10.25.73.150:2380,SZD-L0097856=https://10.25.84.251:2380 --data-dir=/varlib/etcd snapshot restore mysnapshot.db
2018-08-16 19:55:30.735518 I | etcdserver/membership: added member 2a53eb67dfa37cc0 [https://10.25.84.251:2380] to cluster 4741a9e2cf17e1fa
2018-08-16 19:55:30.735601 I | etcdserver/membership: added member 304bcbfa92e84c75 [https://10.25.73.25:2380] to cluster 4741a9e2cf17e1fa
2018-08-16 19:55:30.735617 I | etcdserver/membership: added member c267a0ca02c6bff7 [https://10.25.73.150:2380] to cluster 4741a9e2cf17e1fa
[root@SZD-L0101798 etcd-cluster]#
恢复10.25.73.150节点数据到/var/lib/etcd
[root@SZD-L0103739 etcd-cluster]# ETCDCTL_API=3 etcdctl --name=SZD-L0103739 --endpoints="https://10.25.73.150:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem --initial-cluster-token=etcd-cluster-0 --initial-advertise-peer-urls=https://10.25.73.150:2380 --initial-cluster=SZD-L0101798=https://10.25.73.25:2380,SZD-L0103739=https://10.25.73.150:2380,SZD-L0097856=https://10.25.84.251:2380 --data-dir=/varlib/etcd snapshot restore mysnapshot.db
2018-08-16 19:58:21.892719 I | etcdserver/membership: added member 2a53eb67dfa37cc0 [https://10.25.84.251:2380] to cluster 4741a9e2cf17e1fa
2018-08-16 19:58:21.892796 I | etcdserver/membership: added member 304bcbfa92e84c75 [https://10.25.73.25:2380] to cluster 4741a9e2cf17e1fa
2018-08-16 19:58:21.892812 I | etcdserver/membership: added member c267a0ca02c6bff7 [https://10.25.73.150:2380] to cluster 4741a9e2cf17e1fa
[root@SZD-L0103739 etcd-cluster]#
5、启动Etcd服务
分别在etcd所在主机执行如下命令：
systemctl start etcd
6、验证数据完整性
经过验证，使用ETCDCTL_API=2 存放的数据会丢失，使用ETCDCTL_API=3存放的数据能正常恢复
附录：
查询etcd API3的键
[root@SZD-L0101798 ~]# ETCDCTL_API=3 etcdctl --endpoints="https://10.25.84.251:2379,https://10.25.73.25:2379,https://10.25.73.150:2379" --cert=/root/etcd-cluster/etcd.pem --key=/root/etcd-cluster/etcd-key.pem --cacert=/root/etcd-cluster/ca.pem get / --prefix --keys-only
2018-08-16 20:38:06.954368 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
/name/1
/name/2
/name/3
/name/4
