2-3.1 部署etcd集群

3.1.1 集群规划

主机名	角色	ip地址
k8s-master6021.k8s.host.com	etcd	10.20.60.21
k8s-master6022.k8s.host.com	etcd	10.20.60.22
k8s-master6023.k8s.host.com	etcd	10.20.60.23
注意：这里部署以 k8s-master6021.k8s.host.com 虚机为例，另外两台虚机安装部署方法类似。

3.1.2 创建生成证书签名请求（csr）的 json 配置文件

• 在 k8s-harbor60200.k8s.host.com 虚机实例上操作

shell> cat > etcd-peer-csr.json << EOF
{
    "CN": "k8s-etcd-peer",
    "hosts": [
        "127.0.0.1",
        "10.20.60.11",
        "10.20.60.12",
        "10.20.60.21",
        "10.20.60.22",
        "10.20.60.23"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "BeiJing",
            "L": "BeiJing",
            "O": "91donkey",
            "OU": "ops"
        }
    ]
}
EOF

3.1.3 生成etcd证书和私钥

shell> cd /root/certs/
shell> cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json | cfssl-json -bare etcd-peer
2020/05/07 20:08:17 [INFO] generate received request
2020/05/07 20:08:17 [INFO] received CSR
2020/05/07 20:08:17 [INFO] generating key: rsa-2048
2020/05/07 20:08:17 [INFO] encoded CSR
2020/05/07 20:08:17 [INFO] signed certificate with serial number 295564152097486943314443720870086989525801699218
2020/05/07 20:08:17 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

shell> ls etcd*
etcd-peer.csr etcd-peer-csr.json etcd-peer-key.pem etcd-peer.pem

3.1.4 分发生成的证书和私钥到各 etcd 节点

# 注意在每个 etcd 节点创建 /opt/etcd/pki 目录
shell> scp ca.pem etcd*.pem root@${etcd_ip}:/opt/etcd/pki/

3.1.5 下载并安装etcd软件

# GitHub地址：https://github.com/etcd-io/etcd
shell> cd /usr/local/src/
shell> wget http://dlsw.91donkey.com/software/source/k8s/etcd/etcd-v3.2.30-linux-amd64.tar.gz
shell> tar zxf etcd-v3.2.30-linux-amd64.tar.gz
shell> cd etcd-v3.2.30-linux-amd64
shell> cp etcd etcdctl /usr/local/bin/
shell> etcd --version
etcd Version: 3.2.30
Git SHA: b7644ae5f
Go Version: go1.12.17
Go OS/Arch: linux/amd64

3.1.6 创建 etcd 的 systemd unit 模板文件

在 pg60-21.k8s.host.com 虚机实例上操作

shell> cat > etcd.service.template << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/export/etcd/data
ExecStart=/opt/etcd/bin/etcd \\
  --name=etcd-server-60-21 \\
  --data-dir=/export/etcd/data \\
  --wal-dir=/export/etcd/wal \\
  --ca-file=/opt/etcd/pki/ca.pem \\
  --cert-file=/opt/etcd/pki/etcd-peer.pem \\
  --key-file=/opt/etcd/pki/etcd-peer-key.pem \\
  --trusted-ca-file=/opt/etcd/pki/ca.pem \\
  --peer-ca-file=/opt/etcd/pki/ca.pem \\
  --peer-cert-file=/opt/etcd/pki/etcd-peer.pem \\
  --peer-key-file=/opt/etcd/pki/etcd-peer-key.pem \\
  --peer-trusted-ca-file=/opt/etcd/pki/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth \\
  --listen-peer-urls=https://10.20.60.21:2380 \\
  --initial-advertise-peer-urls=https://10.20.60.21:2380 \\
  --listen-client-urls=https://10.20.60.21:2379,http://127.0.0.1:2379 \\
  --advertise-client-urls=https://10.20.60.21:2379,http://127.0.0.1:2379 \\
  --initial-cluster=etcd-server-60-21=https://10.20.60.21:2380,etcd-server-60-22=https://10.20.60.22:2380,etcd-server-60-23=https://10.20.60.23:2380 \\
  --initial-cluster-token=etcd-cluster \\
  --initial-cluster-state=new \\
# 新版 etcd 参数，该版本需要删除
#  --auto-compaction-mode=periodic \\
  --auto-compaction-retention=1 \\
  --max-request-bytes=33554432 \\
  --quota-backend-bytes=8589934592 \\
  --heartbeat-interval=250 \\
  --election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

shell> mv etcd.service.template /etc/systemd/system/etcd.service
shell> systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd

注意：etcd集群各主机的启动脚本略有不同，部署其他节点时注意修改。

3.1.7 检查etcd集群状态

• 确认三台etcd服务均启动后，再检查集群状态。

shell> etcdctl cluster-health
member ceb7db2d41998e0a is healthy: got healthy result from http://127.0.0.1:2379
member ded9eefca3d025f5 is healthy: got healthy result from http://127.0.0.1:2379
member f9e4c822e89b59be is healthy: got healthy result from http://127.0.0.1:2379
cluster is healthy

shell> etcdctl member list
ceb7db2d41998e0a: name=etcd-server-60-23 peerURLs=https://10.20.60.23:2380 clientURLs=http://127.0.0.1:2379,https://10.20.60.23:2379 isLeader=false
ded9eefca3d025f5: name=etcd-server-60-22 peerURLs=https://10.20.60.22:2380 clientURLs=http://127.0.0.1:2379,https://10.20.60.22:2379 isLeader=false
f9e4c822e89b59be: name=etcd-server-60-21 peerURLs=https://10.20.60.21:2380 clientURLs=http://127.0.0.1:2379,https://10.20.60.21:2379 isLeader=true