etcd
etcd 是CoreOS开发的分布式k-v 数据库。通过raft算法选举主节点
第一节集群安装
| 主机名 | 主机ip | etcd节点名称 | 版本 |
|---|---|---|---|
| etcd-1.k8s.com | 10.4.7.200/24 | etcd-1 | v.13.5.0 |
| etcd-2.k8s.com | 10.4.7.201/24 | etcd-2 | v.13.5.0 |
| etcd-3.k8s.com | 10.4.7.202/24 | etcd-3 | v.13.5.0 |
第一步生成证书
请求文件
cat etcd.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[ req_distinguished_name ]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation,digitalSignature,keyEncipherment
subjectAltName = @alt_names
[alt_names]
IP.1 = 10.4.7.200
IP.2 = 10.4.7.201
IP.3 = 10.4.7.202
ca证书
openssl genrsa -out ca.key 2048
openssl req -new -key ca.key -out ca.csr -subj "/CN=etcd"
openssl x509 -req -in ca.csr -out ca.crt -signkey ca.key -days 365
server证书
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -subj "/CN=etcd-server" -config etcd.cnf
openssl x509 -req -in server.csr -out server.crt -signkey server.key -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 -extensions v3_req -extfile etcd.cnf
peer证书
openssl genrsa -out peer.key 2048
openssl req -new -key peer.key -out peer.csr -subj "/CN=etcd-peer" -config etcd.cnf
openssl x509 -req -in peer.csr -out peer.crt -signkey peer.key -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 -extensions v3_req -extfile etcd.cnf
第二步配置启动文件
注意
每个节点需要修改
--name
--listen-peer-urls
--initial-advertise-peer-urls
--listen-client-urls
--advertise-client-urls
#!/bin/bash
# v3.4+ 可以不指定ETCDCTL_API
export ETCDCTL_API=3
/opt/etcd/etcd \
--name etcd-3 \
--listen-peer-urls 'https://10.4.7.202:2380' \
--initial-advertise-peer-urls 'https://10.4.7.202:2380' \
--listen-client-urls 'https://10.4.7.202:2379,http://127.0.0.1:2379' \
--advertise-client-urls 'https://10.4.7.202:2379' \
--initial-cluster-state 'new' \
--initial-cluster-token 'etcd-cluster' \
--initial-cluster 'etcd-1=https://10.4.7.200:2380,etcd-2=https://10.4.7.201:2380,etcd-3=https://10.4.7.202:2380' \
--client-cert-auth --trusted-ca-file=./ca.crt \
--cert-file=server.crt --key-file=server.key \
--peer-client-cert-auth --peer-trusted-ca-file=./ca.crt \
--peer-cert-file=./peer.crt --peer-key-file=./peer.key
第三步启动etcd并检查
/opt/etcd/etcdctl \
--endpoints=https://10.4.7.200:2379,https://10.4.7.201:2379,https://10.4.7.202:2379 \
--cacert=./ca.crt \
--cert=peer.crt \
--key=peer.key \
member list -w table
第二节集群备份和恢复
COMMANDS:
restore Restores an etcd member snapshot to an etcd directory
save Stores an etcd node backend snapshot to a given file
status [deprecated] Gets backend snapshot status of a given file
第一步生成测试内容
/opt/etcd/etcdctl --endpoints=https://10.4.7.200:2379 --cacert=./ca.crt --cert=peer.crt --key=peer.key put name 张三
第二步备份数据
连接到集群任意节点
/opt/etcd/etcdctl \
--endpoints=https://10.4.7.200:2379 \
--cacert=./ca.crt \
--cert=peer.crt \
--key=peer.key \
snapshot save /data/backup/2022-11-13.db
查看备份文件状态
/opt/etcd/etcdctl snapshot status /data/backup/2022-11-13.db -w table
第三步模拟破坏数据
/opt/etcd/etcdctl --endpoints=https://10.4.7.200:2379 --cacert=./ca.crt --cert=peer.crt --key=peer.key del name
第四步恢复数据
注意
- 停止集群所有节点etcd进程，备份原数据文件
- 停止所有kube-apiserver
- 集群所有节点执行恢复动作，注意修改
--name--initial-advertise-peer-urls--data-dir- 执行启动命令
ETCDCTL_API=3
/opt/etcd/etcdctl \
snapshot restore /data/backup/2022-11-13.db \
--name etcd-1 \
--initial-cluster "etcd-1=https://10.4.7.200:2380,etcd-2=https://10.4.7.201:2380,etcd-3=https://10.4.7.202:2380" \
--initial-cluster-token etcd-cluster \
--initial-advertise-peer-urls https://10.4.7.200:2380 \
--data-dir=./etcd-1.etcd
第五步启动集群
参考集群安装中启动脚本启动集群
验证恢复
[root@radius-db03 ~]# /opt/etcd/etcdctl --endpoints=https://10.4.7.200:2379 --cacert=./ca.crt --cert=peer.crt --key=peer.key get name
name
张三
第三节下线节点
第一步获取节点id
/opt/etcd/etcdctl \
--endpoints=https://10.4.7.200:2379,https://10.4.7.201:2379,https://10.4.7.202:2379 \
--cacert=./ca.crt \
--cert=peer.crt \
--key=peer.key \
member list
第二步下线节点
/opt/etcd/etcdctl \
--endpoints=https://10.4.7.200:2379,https://10.4.7.201:2379,https://10.4.7.202:2379 \
--cacert=./ca.crt \
--cert=peer.crt \
--key=peer.key \
member remove 19a4c376178d5b49
Member 19a4c376178d5b49 removed from cluster d5f1887e154dc473
第四节添加节点
第一步集群中执行添加
注意
示例中
10.4.7.202需要在签证书时已经存在在etcd.cnf中
/opt/etcd/etcdctl \
--endpoints=https://10.4.7.200:2379,https://10.4.7.201:2379 \
--cacert=./ca.crt \
--cert=peer.crt \
--key=peer.key \
member add etcd-2 \
--peer-urls=https://10.4.7.202:2380
执行命令后返回值
Member 8c993ee76df9da92 added to cluster d5f1887e154dc473
ETCD_NAME="etcd-2"
ETCD_INITIAL_CLUSTER="etcd-1=https://10.4.7.200:2380,etcd-2=https://10.4.7.201:2380,etcd-2=https://10.4.7.202:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.4.7.202:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
第二步启动新加入节点
参考集群安装中启动脚本启动集群，需要把--initial-cluster-state 'new' 修改为 --initial-cluster-state 'existing'
第五节其他指令
节点的健康状态 endpoint health
etcdctl --endpoints=10.4.7.250:2379 \
--cacert="/etc/kubernetes/ssl/ca.pem" \
--cert="/etc/kubernetes/ssl/etcd.pem" \
--key="/etc/kubernetes/ssl/etcd-key.pem" \
endpoint health --write-out=table
+-----------------+--------+-------------+-------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+-----------------+--------+-------------+-------+
| 10.4.7.250:2379 | true | 12.901855ms | |
+-----------------+--------+-------------+-------+
集群成员 member list
etcdctl member list --write-out=table
+------------------+---------+-----------------+-------------------------+-------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+-----------------+-------------------------+-------------------------+------------+
| 5c3282345e325481 | started | etcd-10.4.7.250 | https://10.4.7.250:2380 | https://10.4.7.250:2379 | false |
+------------------+---------+-----------------+-------------------------+-------------------------+------------+
节点状态endpoint status
etcdctl --endpoints=10.4.7.250:2379 \
--cacert="/etc/kubernetes/ssl/ca.pem" \
--cert="/etc/kubernetes/ssl/etcd.pem" \
--key="/etc/kubernetes/ssl/etcd-key.pem" \
endpoint status --write-out=table
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.4.7.250:2379 | 5c3282345e325481 | 3.5.0 | 2.9 MB | true | false | 2 | 216020 | 216020 | |
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
性能测试check perf
[root@etcd-1 ~]# etcdctl check perf
60 / 60 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 1m0s
PASS: Throughput is 150 writes/s
PASS: Slowest request took 0.017941s
PASS: Stddev is 0.000807s
PASS
查询
查询key
1. 拿到所有key
etcdctl get / --prefix --keys-only
2. 查询指定key.值是被序列化过的，因此可能会乱码
etcdctl get /registry/storageclasses/csi-rbd-sc
增加
etcdctl put /name wang
OK
查询
etcdctl get /name
/name
wang
# watch 一直监听 /name 这个key的变动
etcdctl watch /name
删除
[root@ceph ~]# etcdctl del /name
1
