ETCD 集群部署

部署ETCD集群
Etcd 解析
- 方法一修改hosts文件
- 方法二增加bind解析
下载和分发etcd二进制文件
创建etcd证书和私钥
创建etcd的启动文件
测试 ETCD 集群状态

下面的操作依托于上一篇文章

部署前期准备工作

部署ETCD集群

这里使用的ETCD为三节点高可用集群，步骤如下

下载和分发etcd二进制文件
创建etcd集群各节点的x509证书，用于加密客户端(如kubectl)与etcd集群、etcd集群之间的数据流
创建etcd的system unit文件，配置服务参数
检查集群工作状态

注意: 没有特殊说明都在node01节点操作

Etcd 解析

本次使用etcd单独的域名解析

方法一修改hosts文件

在所有机器上操作， /etc/hosts 文件最后增加下面内容，也可以替换成自己的IP

10.0.20.11 etcd01 etcd01.k8s.com
10.0.20.12 etcd02 etcd02.k8s.com
10.0.20.13 etcd03 etcd03.k8s.com

方法二增加bind解析

如果使用的内网 DNS bind 做内网解析增加下面解析

etcd01  IN  A   10.0.20.11
etcd02  IN  A   10.0.20.12
etcd03  IN  A   10.0.20.13

下载和分发etcd二进制文件

二进制文件在部署前期准备工作文章中已经下载好，直接使用；

分发二进制文件到ETCD集群节点

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp etcd-v3.3.13-linux-amd64/etcd* root@${node_ip}:/opt/k8s/bin
    ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
  done

创建etcd证书和私钥

cd /opt/k8s/work
cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "10.0.20.11",
    "10.0.20.12",
    "10.0.20.13",
    "etcd01.k8s.com",
    "etcd02.k8s.com",
    "etcd03.k8s.com"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF


#host字段指定授权使用该证书的etcd节点IP或域名列表，需要将etcd集群的3个节点都添加其中

生成证书和私钥

cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \
    -ca-key=/opt/k8s/work/ca-key.pem \
    -config=/opt/k8s/work/ca-config.json \
    -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
ls etcd*pem

分发证书和私钥到etcd各个节点

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /etc/etcd/cert"
    scp etcd*.pem root@${node_ip}:/etc/etcd/cert/
  done

创建etcd的启动文件

这里相对应的etcd 的配置，就保存在启动文件中

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > etcd.service.template <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=${ETCD_DATA_DIR}
ExecStart=/opt/k8s/bin/etcd \\
  --data-dir=${ETCD_DATA_DIR} \\
  --wal-dir=${ETCD_WAL_DIR} \\
  --name=##NODE_NAME## \\
  --cert-file=/etc/etcd/cert/etcd.pem \\
  --key-file=/etc/etcd/cert/etcd-key.pem \\
  --trusted-ca-file=/etc/kubernetes/cert/ca.pem \\
  --peer-cert-file=/etc/etcd/cert/etcd.pem \\
  --peer-key-file=/etc/etcd/cert/etcd-key.pem \\
  --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth \\
  --listen-peer-urls=https://##NODE_IP##:2380 \\
  --initial-advertise-peer-urls=https://##NODE_IP##:2380 \\
  --listen-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \\
  --advertise-client-urls=https://##NODE_IP##:2379 \\
  --initial-cluster-token=etcd-cluster-0 \\
  --initial-cluster=${ETCD_NODES} \\
  --initial-cluster-state=new \\
  --auto-compaction-mode=periodic \\
  --auto-compaction-retention=1 \\
  --max-request-bytes=33554432 \\
  --quota-backend-bytes=6442450944 \\
  --heartbeat-interval=250 \\
  --election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

配置说明 (此处不需要修改任何配置)

WorkDirectory、–data-dir 指定etcd工作目录和数据存储为${ETCD_DATA_DIR},需要在启动前创建这个目录 (后面跟着我操作就可以，会有创建步骤)
–wal-dir 指定wal目录，为了提高性能，一般使用SSD和–data-dir不同的盘
–name 指定节点名称，当–initial-cluster-state值为new时，–name的参数值必须位于–initial-cluster列表中
–cert-file、–key-file ETCD server与client通信时使用的证书和私钥
–trusted-ca-file 签名client证书的CA证书，用于验证client证书
–peer-cert-file、–peer-key-file ETCD与peer通信使用的证书和私钥
–peer-trusted-ca-file 签名peer证书的CA证书，用于验证peer证书

拆分三个配置文件，并修改信息

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))
  do
    sed -e "s/##NODE_NAME##/${ETCD_NAMES[i]}/" -e "s/##NODE_IP##/${ETCD_IPS[i]}/" etcd.service.template > etcd-${ETCD_IPS[i]}.service 
  done

etcd*.service

分发生成的etcd启动文件到对应的服务器

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp etcd-${node_ip}.service root@${node_ip}:/etc/systemd/system/etcd.service
  done

启动etcd服务

etcd首次进程启动会等待其他节点加入etcd集群，执行启动命令会卡顿一会，为正常现象

远程创建对应 ETCD 的数据目录等

source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p ${ETCD_DATA_DIR} ${ETCD_WAL_DIR}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd " &
  done

测试 ETCD 集群状态

检查启动结果

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status etcd|grep Active"
  done

输出结果:

[root@node01 work]# for node_ip in ${MASTER_IPS[@]}
>   do
>     echo ">>> ${node_ip}"
>     ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
>     --endpoints=https://${node_ip}:2379 \
>     --cacert=/etc/kubernetes/cert/ca.pem \
>     --cert=/etc/etcd/cert/etcd.pem \
>     --key=/etc/etcd/cert/etcd-key.pem endpoint health
>   done
>>> 10.0.20.11
https://10.0.20.11:2379 is healthy: successfully committed proposal: took = 1.609991ms
>>> 10.0.20.12
https://10.0.20.12:2379 is healthy: successfully committed proposal: took = 1.117871ms
>>> 10.0.20.13
https://10.0.20.13:2379 is healthy: successfully committed proposal: took = 1.49139ms

通过下面命令查看当前etcd集群leader

source /opt/k8s/bin/environment.sh
ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
  -w table --cacert=/etc/kubernetes/cert/ca.pem \
  --cert=/etc/etcd/cert/etcd.pem \
  --key=/etc/etcd/cert/etcd-key.pem \
  --endpoints=${ETCD_ENDPOINTS} endpoint status

输出结果如下：

[root@node01 work]# source /opt/k8s/bin/environment.sh
[root@node01 work]# ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
>   -w table --cacert=/etc/kubernetes/cert/ca.pem \
>   --cert=/etc/etcd/cert/etcd.pem \
>   --key=/etc/etcd/cert/etcd-key.pem \
>   --endpoints=${ETCD_ENDPOINTS} endpoint status
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://etcd01.k8s.com:2379 | 6330dc0a28f62066 |  3.3.13 |   16 kB |     false |        35 |         14 |
| https://etcd02.k8s.com:2379 | 77bc4da10f4c40bb |  3.3.13 |   16 kB |      true |        35 |         14 |
| https://etcd03.k8s.com:2379 | d2573d5cc998d0f0 |  3.3.13 |   16 kB |     false |        35 |         14 |
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+

如果对ETCD集群安装不熟悉的，可以参考文章 CentOS 7 ETCD集群配置大全

posted @ 2019-12-05 21:39 司家勇阅读(4909) 评论(0) 编辑收藏举报

刷新页面返回顶部

winstom

ETCD 集群部署

部署ETCD集群

Etcd 解析

方法一修改hosts文件

方法二增加bind解析

下载和分发etcd二进制文件

创建etcd证书和私钥

创建etcd的启动文件

测试 ETCD 集群状态

公告

winstom

ETCD 集群部署

部署ETCD集群

Etcd 解析

方法一 修改hosts文件

方法二 增加bind解析

下载和分发etcd二进制文件

创建etcd证书和私钥

创建etcd的启动文件

测试 ETCD 集群状态

公告

方法一修改hosts文件

方法二增加bind解析