Kubernetes学习之路（三）之ETCD集群二进制部署
_____egon新书来袭请看：https://egonlin.com/book.html
ETCD集群部署
所有持久化的状态信息以KV的形式存储在ETCD中。类似zookeeper，提供分布式协调服务。之所以说kubenetes各个组件是无状态的，就是因为其中把数据都存放在ETCD中。由于ETCD支持集群，这里在三台主机master01、master02、master03上都部署上ETCD。
（1）在master01、master02、master03主机上下载安装etcd软件包
wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz tar xf etcd-v3.2.18-linux-amd64.tar.gz cd etcd-v3.2.18-linux-amd64
#有2个文件，etcdctl是操作etcd的命令
cp etcd etcdctl /opt/kubernetes/bin/
（2）在master01、master02、master03主机上均执行下述操作
或者在master01上操作完毕后将结果传送给master02和master03
生成的证书即用于别人来访问etcd（即etcd的服务端证书），又用于etc对等互联
#1、创建证书签名请求，etcd-csr.json文件中hosts指定的ip是etcd集群中各个节点的ip地址 mkdir /usr/local/src/ssl cd /usr/local/src/ssl cat > etcd-csr.json << EOF { "CN": "etcd", "hosts": [ "127.0.0.1", "10.1.1.100", "10.1.1.101", "10.1.1.102" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "ops" } ] } EOF #2、生成 etcd 证书和私钥 cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \ -ca-key=/opt/kubernetes/ssl/ca-key.pem \ -config=/opt/kubernetes/ssl/ca-config.json \ -profile=kubernetes etcd-csr.json | cfssljson -bare etcd 会生成以下证书文件 [root@linux-master01 ssl]# ls -l /usr/local/src/ssl/ 总用量 16 -rw-r--r-- 1 root root 1058 8月 20 08:55 etcd.csr -rw-r--r-- 1 root root 280 8月 20 08:55 etcd-csr.json -rw------- 1 root root 1679 8月 20 08:55 etcd-key.pem -rw-r--r-- 1 root root 1428 8月 20 08:55 etcd.pem # 3、将证书拷贝到/opt/kubernetes/ssl目录下 cp /usr/local/src/ssl/etcd*.pem /opt/kubernetes/ssl/ scp /usr/local/src/ssl/etcd*.pem root@master02:/opt/kubernetes/ssl/ scp /usr/local/src/ssl/etcd*.pem root@master03:/opt/kubernetes/ssl/
（4）在master01、master02、master03上配置ETCD配置文件
2379端口用于外部通信，2380用于内部通信
注意，中文注释不要加载配置项的末尾，加在头顶是可以的 # =================在master01主机上执行下述修改 cat > /opt/kubernetes/cfg/etcd.conf << EOF #[member] #ETCD节点名称修改，这个ETCD_NAME每个节点必须不同 ETCD_NAME="etcd01-master01" #ETCD数据目录 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_SNAPSHOT_COUNTER="10000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" #ETCD监听的URL，每个节点均不同、需要修改 ETCD_LISTEN_PEER_URLS="https://10.1.1.100:2380" #外部通信监听URL修改，每个节点均不同、需要修改 ETCD_LISTEN_CLIENT_URLS="https://10.1.1.100:2379,https://127.0.0.1:2379" #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" #ETCD_CORS="" #[cluster] # 每个节点均不同、需要修改 ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.1.1.100:2380" # if you use different ETCD_NAME (e.g. test), # set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..." #添加集群访问 ETCD_INITIAL_CLUSTER="etcd01-master01=https://10.1.1.100:2380,etcd02-master02=https://10.1.1.101:2380,etcd03-master03=https://10.1.1.102:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" # 每个节点均不同、需要修改 ETCD_ADVERTISE_CLIENT_URLS="https://10.1.1.100:2379" #[security] CLIENT_CERT_AUTH="true" ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" EOF # =================在master02主机上执行下述修改 cat > /opt/kubernetes/cfg/etcd.conf << EOF #[member] #ETCD节点名称修改，这个ETCD_NAME每个节点必须不同 ETCD_NAME="etcd02-master02" #ETCD数据目录 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_SNAPSHOT_COUNTER="10000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" #ETCD监听的URL，每个节点均不同、需要修改 ETCD_LISTEN_PEER_URLS="https://10.1.1.101:2380" #外部通信监听URL修改，每个节点均不同、需要修改 ETCD_LISTEN_CLIENT_URLS="https://10.1.1.101:2379,https://127.0.0.1:2379" #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" #ETCD_CORS="" #[cluster] # 每个节点均不同、需要修改 ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.1.1.101:2380" # if you use different ETCD_NAME (e.g. test), # set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..." # 添加集群访问 ETCD_INITIAL_CLUSTER="etcd01-master01=https://10.1.1.100:2380,etcd02-master02=https://10.1.1.101:2380,etcd03-master03=https://10.1.1.102:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" # 每个节点均不同、需要修改 ETCD_ADVERTISE_CLIENT_URLS="https://10.1.1.101:2379" #[security] CLIENT_CERT_AUTH="true" ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" EOF # =================在master03主机上执行下述修改 cat > /opt/kubernetes/cfg/etcd.conf << EOF #[member] #ETCD节点名称修改，这个ETCD_NAME每个节点必须不同 ETCD_NAME="etcd03-master03" #ETCD数据目录 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_SNAPSHOT_COUNTER="10000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" #ETCD监听的URL，每个节点均不同、需要修改 ETCD_LISTEN_PEER_URLS="https://10.1.1.102:2380" #外部通信监听URL修改，每个节点均不同、需要修改 ETCD_LISTEN_CLIENT_URLS="https://10.1.1.102:2379,https://127.0.0.1:2379" #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" #ETCD_CORS="" #[cluster] # 每个节点均不同、需要修改 ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.1.1.102:2380" # if you use different ETCD_NAME (e.g. test), # set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..." # 添加集群访问 ETCD_INITIAL_CLUSTER="etcd01-master01=https://10.1.1.100:2380,etcd02-master02=https://10.1.1.101:2380,etcd03-master03=https://10.1.1.102:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" # 每个节点均不同、需要修改 ETCD_ADVERTISE_CLIENT_URLS="https://10.1.1.102:2379" #[security] CLIENT_CERT_AUTH="true" ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" EOF
（5）加入systemd，并启动ETCD集群
#1、在master01、master02、master03上编写下述配置，配置均相同 cat > /etc/systemd/system/etcd.service << EOF [Unit] Description=Etcd Server After=network.target [Service] Type=simple WorkingDirectory=/var/lib/etcd EnvironmentFile=-/opt/kubernetes/cfg/etcd.conf # set GOMAXPROCS to number of processors ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/kubernetes/bin/etcd" Type=notify [Install] WantedBy=multi-user.target EOF #2、启动（注意，先自动后两个节点，即maseter02与master03，然后再启动第一个节点即master01，防止启动超时timeout） systemctl daemon-reload systemctl enable etcd # 默认不会创建etcd的数据存储目录，需要手动创建 mkdir /var/lib/etcd systemctl start etcd systemctl status etcd #在各节点上查看是否监听了2379和2380端口 netstat -tulnp |grep etcd
（6）验证ETCD集群
#验证集群状态，在master01、maser02、mater03任意一节点上执行即可，ip地址用任意一个都行 etcdctl --endpoints=https://10.1.1.100:2379 \ --ca-file=/opt/kubernetes/ssl/ca.pem \ --cert-file=/opt/kubernetes/ssl/etcd.pem \ --key-file=/opt/kubernetes/ssl/etcd-key.pem cluster-health cluster is healthy #表明ETCD集群是正常的！！！
