etcd

etcd 是CoreOS开发的分布式k-v 数据库。通过raft算法选举主节点

第一节集群安装

主机名	主机ip	etcd节点名称	版本
etcd-1.k8s.com	10.4.7.200/24	etcd-1	v.13.5.0
etcd-2.k8s.com	10.4.7.201/24	etcd-2	v.13.5.0
etcd-3.k8s.com	10.4.7.202/24	etcd-3	v.13.5.0

第一步生成证书

请求文件

cat etcd.cnf 
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[ req_distinguished_name ]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation,digitalSignature,keyEncipherment
subjectAltName = @alt_names
[alt_names]
IP.1 = 10.4.7.200
IP.2 = 10.4.7.201
IP.3 = 10.4.7.202

ca证书

openssl genrsa -out ca.key 2048
openssl req -new -key ca.key -out ca.csr -subj "/CN=etcd"
openssl x509 -req -in ca.csr -out ca.crt -signkey ca.key -days 365

server证书

openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -subj "/CN=etcd-server" -config etcd.cnf
openssl x509 -req -in server.csr -out server.crt -signkey server.key -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 -extensions v3_req -extfile etcd.cnf
 

peer证书

openssl genrsa -out peer.key 2048
openssl req -new -key peer.key -out peer.csr -subj "/CN=etcd-peer" -config etcd.cnf
openssl x509 -req -in peer.csr -out peer.crt -signkey peer.key -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 -extensions v3_req -extfile etcd.cnf

第二步配置启动文件

注意

每个节点需要修改

--name
--listen-peer-urls
--initial-advertise-peer-urls
--listen-client-urls
--advertise-client-urls

 #!/bin/bash
 # v3.4+ 可以不指定ETCDCTL_API 
 export ETCDCTL_API=3
 /opt/etcd/etcd \
 --name etcd-3 \
 --listen-peer-urls 'https://10.4.7.202:2380' \
 --initial-advertise-peer-urls 'https://10.4.7.202:2380' \
 --listen-client-urls 'https://10.4.7.202:2379,http://127.0.0.1:2379' \
 --advertise-client-urls 'https://10.4.7.202:2379' \
 --initial-cluster-state 'new' \
 --initial-cluster-token 'etcd-cluster' \
 --initial-cluster 'etcd-1=https://10.4.7.200:2380,etcd-2=https://10.4.7.201:2380,etcd-3=https://10.4.7.202:2380' \
 --client-cert-auth --trusted-ca-file=./ca.crt \
 --cert-file=server.crt --key-file=server.key  \
 --peer-client-cert-auth --peer-trusted-ca-file=./ca.crt \
 --peer-cert-file=./peer.crt --peer-key-file=./peer.key

第三步启动etcd并检查

/opt/etcd/etcdctl \
--endpoints=https://10.4.7.200:2379,https://10.4.7.201:2379,https://10.4.7.202:2379 \
--cacert=./ca.crt \
--cert=peer.crt \
--key=peer.key \
member list -w table

---

第二节集群备份和恢复

COMMANDS:
        restore Restores an etcd member snapshot to an etcd directory
        save    Stores an etcd node backend snapshot to a given file
        status  [deprecated] Gets backend snapshot status of a given file

第一步生成测试内容

/opt/etcd/etcdctl --endpoints=https://10.4.7.200:2379 --cacert=./ca.crt --cert=peer.crt --key=peer.key put name 张三

第二步备份数据

连接到集群任意节点

/opt/etcd/etcdctl \
--endpoints=https://10.4.7.200:2379 \
--cacert=./ca.crt \
--cert=peer.crt \
--key=peer.key \
snapshot save /data/backup/2022-11-13.db

查看备份文件状态

/opt/etcd/etcdctl snapshot status /data/backup/2022-11-13.db -w table

第三步模拟破坏数据

/opt/etcd/etcdctl --endpoints=https://10.4.7.200:2379 --cacert=./ca.crt --cert=peer.crt --key=peer.key del name

第四步恢复数据

注意

• 停止集群所有节点etcd进程，备份原数据文件
• 停止所有kube-apiserver
• 集群所有节点执行恢复动作，注意修改 --name --initial-advertise-peer-urls --data-dir
• 执行启动命令

ETCDCTL_API=3 
/opt/etcd/etcdctl \
snapshot restore /data/backup/2022-11-13.db \
--name etcd-1 \
--initial-cluster "etcd-1=https://10.4.7.200:2380,etcd-2=https://10.4.7.201:2380,etcd-3=https://10.4.7.202:2380" \
--initial-cluster-token etcd-cluster \
--initial-advertise-peer-urls https://10.4.7.200:2380 \
--data-dir=./etcd-1.etcd

第五步启动集群

参考集群安装中启动脚本启动集群

验证恢复

[root@radius-db03 ~]# /opt/etcd/etcdctl --endpoints=https://10.4.7.200:2379 --cacert=./ca.crt --cert=peer.crt --key=peer.key get name
name
张三

---

第三节下线节点

第一步获取节点id

/opt/etcd/etcdctl \
--endpoints=https://10.4.7.200:2379,https://10.4.7.201:2379,https://10.4.7.202:2379 \
--cacert=./ca.crt \
--cert=peer.crt \
--key=peer.key  \
member list

第二步下线节点

/opt/etcd/etcdctl \
--endpoints=https://10.4.7.200:2379,https://10.4.7.201:2379,https://10.4.7.202:2379 \
--cacert=./ca.crt \
--cert=peer.crt \
--key=peer.key  \
member remove 19a4c376178d5b49

Member 19a4c376178d5b49 removed from cluster d5f1887e154dc473

---

第四节添加节点

第一步集群中执行添加

注意

示例中10.4.7.202 需要在签证书时已经存在在etcd.cnf 中

/opt/etcd/etcdctl \
--endpoints=https://10.4.7.200:2379,https://10.4.7.201:2379 \
--cacert=./ca.crt \
--cert=peer.crt \
--key=peer.key  \
member add etcd-2 \
--peer-urls=https://10.4.7.202:2380

执行命令后返回值

Member 8c993ee76df9da92 added to cluster d5f1887e154dc473
 
ETCD_NAME="etcd-2"
ETCD_INITIAL_CLUSTER="etcd-1=https://10.4.7.200:2380,etcd-2=https://10.4.7.201:2380,etcd-2=https://10.4.7.202:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.4.7.202:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

第二步启动新加入节点

参考集群安装中启动脚本启动集群，需要把--initial-cluster-state 'new' 修改为 --initial-cluster-state 'existing'

---

第五节其他指令

节点的健康状态 endpoint health

etcdctl --endpoints=10.4.7.250:2379 \
--cacert="/etc/kubernetes/ssl/ca.pem" \
--cert="/etc/kubernetes/ssl/etcd.pem" \
--key="/etc/kubernetes/ssl/etcd-key.pem" \
endpoint health --write-out=table
 
+-----------------+--------+-------------+-------+
|    ENDPOINT     | HEALTH |    TOOK     | ERROR |
+-----------------+--------+-------------+-------+
| 10.4.7.250:2379 |   true | 12.901855ms |       |
+-----------------+--------+-------------+-------+

集群成员 member list

etcdctl member list --write-out=table
+------------------+---------+-----------------+-------------------------+-------------------------+------------+
|        ID        | STATUS  |      NAME       |       PEER ADDRS        |      CLIENT ADDRS       | IS LEARNER |
+------------------+---------+-----------------+-------------------------+-------------------------+------------+
| 5c3282345e325481 | started | etcd-10.4.7.250 | https://10.4.7.250:2380 | https://10.4.7.250:2379 |      false |
+------------------+---------+-----------------+-------------------------+-------------------------+------------+

节点状态endpoint status

etcdctl --endpoints=10.4.7.250:2379 \
--cacert="/etc/kubernetes/ssl/ca.pem" \
--cert="/etc/kubernetes/ssl/etcd.pem" \
--key="/etc/kubernetes/ssl/etcd-key.pem" \
endpoint status --write-out=table
 
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT     |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.4.7.250:2379 | 5c3282345e325481 |   3.5.0 |  2.9 MB |      true |      false |         2 |     216020 |             216020 |        |
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

性能测试check perf

[root@etcd-1 ~]# etcdctl check perf
 60 / 60 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 1m0s
PASS: Throughput is 150 writes/s
PASS: Slowest request took 0.017941s
PASS: Stddev is 0.000807s
PASS

查询

查询key
1. 拿到所有key
etcdctl get / --prefix --keys-only
2. 查询指定key.值是被序列化过的，因此可能会乱码
etcdctl get /registry/storageclasses/csi-rbd-sc

增加

etcdctl put /name wang
OK

查询

etcdctl get /name
/name
wang
# watch 一直监听 /name 这个key的变动
etcdctl watch /name

删除

[root@ceph ~]# etcdctl del /name
1

参考

raft

https证书最佳实战目录 - _毛台 - 博客园 (cnblogs.com)

ETCD数据的备份与恢复 - taotaozh - 博客园 (cnblogs.com)