如何在已有的 K8S 集群中使用 etcdctl 工具操作 etcd 集群

1、集群信息

K8S 集群信息，集群中有三个 master 节点

# kubectl get nodes
NAME               STATUS     ROLES    AGE   VERSION
k8s-m1             Ready      master   55d   v1.17.0
k8s-m2             Ready      master   55d   v1.17.0
k8s-m3             Ready      master   55d   v1.17.0

etcd 集群以 pod 方式运行在 K8S 集群之上

# kubectl get pods -n kube-system -o wide
NAME                                       READY   STATUS           RESTARTS   AGE   IP              NODE               NOMINATED NODE   READINESS GATES
etcd-k8s-m1                                1/1     Running          44         55d   172.0.2.139     k8s-m1             <none>           <none>
etcd-k8s-m2                                1/1     Running          2          26m   172.0.2.146     k8s-m2             <none>           <none>
etcd-k8s-m3                                1/1     Running          3779       55d   172.0.2.234     k8s-m3             <none>           <none>

2、下载 etcdctl

查看 etcd 版本，需要下载 3.4.3 版本的 etcdctl

[root@k8s-m1 member]# kubectl describe pods etcd-k8s-m1 -n kube-system
Name:                 etcd-k8s-m1
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 k8s-m1/172.0.2.139
Start Time:           Mon, 13 Apr 2020 02:28:39 -0400
Labels:               component=etcd
                      tier=control-plane
Annotations:          kubernetes.io/config.hash: 3d4819355a9752ba239aa13c1885dcc1
                      kubernetes.io/config.mirror: 3d4819355a9752ba239aa13c1885dcc1
                      kubernetes.io/config.seen: 2020-02-20T04:27:11.811231481-05:00
                      kubernetes.io/config.source: file
Status:               Running
IP:                   172.0.2.139
IPs:
  IP:           172.0.2.139
Controlled By:  Node/k8s-m1
Containers:
  etcd:
    Container ID:  docker://c8722c4def309777ca9be9fb7a273521f6fe3cb3195105a10121f22c24310fe6
    Image:         k8s.gcr.io/etcd:3.4.3-0

下载 etcd 版本，解压，copy etcdctl 到 k8s master 节点的 /usr/bin 目录下

# wget https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-amd64.tar.gz .

[root@k8s-m1 member]# ls -l /usr/bin/etcdctl
-rwxr-xr-x. 1 root root 17542688 Mar  4 03:09 /usr/bin/etcdctl
[root@k8s-m1 member]# etcdctl version
etcdctl version: 3.4.3
API version: 3.4

3、使用 etcdctl

3.1、获取 etcd 的 endpoint

endpoint 为 https://172.0.2.139:2379

# kubectl get pods etcd-k8s-m1 -o yaml -n kube-system
...
  containers:
  - command:
    - etcd
    - --advertise-client-urls=https://172.0.2.139:2379
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --client-cert-auth=true
    - --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://172.0.2.139:2380
    - --initial-cluster=k8s-m1=https://172.0.2.139:2380
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --listen-client-urls=https://127.0.0.1:2379,https://172.0.2.139:2379
    - --listen-metrics-urls=http://127.0.0.1:2381
    - --listen-peer-urls=https://172.0.2.139:2380
    - --name=k8s-m1
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --snapshot-count=10000
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    image: k8s.gcr.io/etcd:3.4.3-0
...

3.2、准备 key 和 cert

etcd 的 endpoint 为 https 方式，所以要为 etcdctl 命令准备 key 和 cert
在 3.1 节输出信息中：
key 使用 /etc/kubernetes/pki/etcd/peer.key
cert 使用 /etc/kubernetes/pki/etcd/peer.crt

3.3、执行 etcdctl 命令

# etcdctl --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key  --endpoints https://172.0.2.139:2379 --insecure-skip-tls-verify  member list
1e2fb9983e528532, started, k8s-m2, https://172.0.2.146:2380, https://172.0.2.146:2379, false
947c9889866d299a, started, k8s-m3, https://172.0.2.234:2380, https://172.0.2.234:2379, false
e97c0cc82d69a534, started, k8s-m1, https://172.0.2.139:2380, https://172.0.2.139:2379, false

注意：因为集群证书为自签发，所以这里需要加上–insecure-skip-tls-verify 参数，不然会报如下错误

# etcdctl --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key  --endpoints https://172.0.2.139:2379  member list
{"level":"warn","ts":"2020-04-16T05:00:52.085-0400","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-c086c9e1-cb96-4c26-890e-b311b761b2c3/172.0.2.139:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\""}
Error: context deadline exceeded