ETCD集群部署
一、准备ETCD软件包
wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz #之前的压缩包中存在此包，无需下载了
[root@linux-node1 src]# tar zxf etcd-v3.2.18-linux-amd64.tar.gz [root@linux-node1 src]# cd etcd-v3.2.18-linux-amd64 [root@linux-node1 etcd-v3.2.18-linux-amd64]# ll total 32368 drwxr-xr-x 11 478493 89939 4096 Mar 30 01:49 Documentation -rwxr-xr-x 1 478493 89939 17837888 Mar 30 01:49 etcd -rwxr-xr-x 1 478493 89939 15246720 Mar 30 01:49 etcdctl -rw-r--r-- 1 478493 89939 34246 Mar 30 01:49 README-etcdctl.md -rw-r--r-- 1 478493 89939 5801 Mar 30 01:49 README.md -rw-r--r-- 1 478493 89939 7855 Mar 30 01:49 READMEv2-etcdctl.md [root@linux-node1 etcd-v3.2.18-linux-amd64]# cp etcd etcdctl /opt/kubernetes/bin/ [root@linux-node1 etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 192.168.56.12:/opt/kubernetes/bin/ [root@linux-node1 etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 192.168.56.13:/opt/kubernetes/bin/
二、创建ETCD证书签名请求
[root@linux-node1 etcd-v3.2.18-linux-amd64]# cd /usr/local/src/ssl/ [root@linux-node1 ssl]# pwd /usr/local/src/ssl [root@linux-node1 ssl]# ll total 20 -rw-r--r-- 1 root root 290 May 30 20:55 ca-config.json -rw-r--r-- 1 root root 1001 May 30 20:57 ca.csr -rw-r--r-- 1 root root 208 May 30 20:56 ca-csr.json -rw------- 1 root root 1675 May 30 20:57 ca-key.pem -rw-r--r-- 1 root root 1359 May 30 20:57 ca.pem
[root@linux-node1 ssl]# vim etcd-csr.json { "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.56.11", "192.168.56.12", "192.168.56.13" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
三、生成ETCD证书和私钥
[root@linux-node1 ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \ -ca-key=/opt/kubernetes/ssl/ca-key.pem \ -config=/opt/kubernetes/ssl/ca-config.json \ -profile=kubernetes etcd-csr.json | cfssljson -bare etcd 2018/05/30 22:30:32 [INFO] generate received request 2018/05/30 22:30:32 [INFO] received CSR 2018/05/30 22:30:32 [INFO] generating key: rsa-2048 2018/05/30 22:30:32 [INFO] encoded CSR 2018/05/30 22:30:32 [INFO] signed certificate with serial number 357419029160249044617633366301914976457487157467 2018/05/30 22:30:32 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); 会生成以下证书文件 [root@linux-node1 ssl]# ls -l etcd* -rw-r--r-- 1 root root 1062 May 30 22:30 etcd.csr -rw-r--r-- 1 root root 287 May 30 22:27 etcd-csr.json -rw------- 1 root root 1675 May 30 22:30 etcd-key.pem -rw-r--r-- 1 root root 1436 May 30 22:30 etcd.pem
四、证书分发至/opt/kubernetes/ssl目录下
[root@linux-node1 ssl]# cp etcd*.pem /opt/kubernetes/ssl [root@linux-node1 ssl]# scp etcd*.pem 192.168.56.12:/opt/kubernetes/ssl [root@linux-node1 ssl]# scp etcd*.pem 192.168.56.13:/opt/kubernetes/ssl
五、设置ETCD配置文件
[root@linux-node1 ssl]# vim /opt/kubernetes/cfg/etcd.conf #[member] ETCD_NAME="etcd-node1" #每个节点要唯一 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_SNAPSHOT_COUNTER="10000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" ETCD_LISTEN_PEER_URLS="https://192.168.56.11:2380" #节点IP地址 ETCD_LISTEN_CLIENT_URLS="https://192.168.56.11:2379,https://127.0.0.1:2379" #节点IP地址 #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" #ETCD_CORS="" #[cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.56.11:2380" #节点IP地址 # if you use different ETCD_NAME (e.g. test), # set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..." ETCD_INITIAL_CLUSTER="etcd-node1=https://192.168.56.11:2380,etcd-node2=https://192.168.56.12:2380,etcd-node3=https://192.168.56.13:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.56.11:2379" #节点IP地址 #[security] CLIENT_CERT_AUTH="true" ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
六、创建ETCD系统服务
[root@linux-node1 ssl]# vim /etc/systemd/system/etcd.service [Unit] Description=Etcd Server After=network.target [Service] Type=simple WorkingDirectory=/var/lib/etcd EnvironmentFile=-/opt/kubernetes/cfg/etcd.conf # set GOMAXPROCS to number of processors ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/kubernetes/bin/etcd" Type=notify [Install] WantedBy=multi-user.target
七、重新加载系统服务
[root@linux-node1 ssl]# vim /etc/systemd/system/etcd.service [root@linux-node1 ssl]# systemctl daemon-reload [root@linux-node1 ssl]# [root@linux-node1 ssl]# systemctl enable etcd Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /etc/systemd/system/etcd.service. #提示先不要启动服务，把其它节点部署完成后，再同时启动。 [root@linux-node1 ssl]# scp /opt/kubernetes/cfg/etcd.conf 192.168.56.12:/opt/kubernetes/cfg/ [root@linux-node1 ssl]# scp /opt/kubernetes/cfg/etcd.conf 192.168.56.13:/opt/kubernetes/cfg/ [root@linux-node1 ssl]# scp /etc/systemd/system/etcd.service 192.168.56.12:/etc/systemd/system/ [root@linux-node1 ssl]# scp /etc/systemd/system/etcd.service 192.168.56.13:/etc/systemd/system/
其它节点配置ETCD
linux-node2节点配置ETCD
[root@linux-node2 ~]# vim /opt/kubernetes/cfg/etcd.conf #[member] ETCD_NAME="etcd-node2" #修改为自己的名称 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_SNAPSHOT_COUNTER="10000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" ETCD_LISTEN_PEER_URLS="https://192.168.56.12:2380" #修改为自己的IP ETCD_LISTEN_CLIENT_URLS="https://192.168.56.12:2379,https://127.0.0.1:2379" #修改为自己的IP #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" #ETCD_CORS="" #[cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.56.12:2380" #修改为自己的IP # if you use different ETCD_NAME (e.g. test), # set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..." ETCD_INITIAL_CLUSTER="etcd-node1=https://192.168.56.11:2380,etcd-node2=https://192.168.56.12:2380,etcd-node3=https://192.168.56.13:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.56.12:2379" #修改为自己的IP #[security] CLIENT_CERT_AUTH="true" ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" 在所有节点上创建etcd存储目录并启动etcd [root@linux-node2 ~]# mkdir /var/lib/etcd [root@linux-node2 ~]# systemctl daemon-reload [root@linux-node2 ~]# systemctl enable etcd Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /etc/systemd/system/etcd.service. #提示等其它节点部署完成后，再同时启动ETCD服务 [root@linux-node2 ~]# systemctl start etcd [root@linux-node2 ~]# systemctl status etcd ● etcd.service - Etcd Server Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-05-30 22:53:14 CST; 14s ago Main PID: 2166 (etcd) CGroup: /system.slice/etcd.service └─2166 /opt/kubernetes/bin/etcd May 30 22:53:15 linux-node2.example.com etcd[2166]: set the initial cluster version to 3.0 May 30 22:53:15 linux-node2.example.com etcd[2166]: enabled capabilities for version 3.0 May 30 22:53:16 linux-node2.example.com etcd[2166]: peer 6566e06d7343e1bb became active May 30 22:53:16 linux-node2.example.com etcd[2166]: established a TCP streaming connection with peer...er) May 30 22:53:16 linux-node2.example.com etcd[2166]: established a TCP streaming connection with peer...er) May 30 22:53:16 linux-node2.example.com etcd[2166]: established a TCP streaming connection with peer...er) May 30 22:53:16 linux-node2.example.com etcd[2166]: 435fb0a8da627a4c initialzed peer connection; fas...(s) May 30 22:53:16 linux-node2.example.com etcd[2166]: established a TCP streaming connection with peer...er) May 30 22:53:19 linux-node2.example.com etcd[2166]: updated the cluster version from 3.0 to 3.2 May 30 22:53:19 linux-node2.example.com etcd[2166]: enabled capabilities for version 3.2 Hint: Some lines were ellipsized, use -l to show in full. [root@linux-node2 ~]# netstat -ntlp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 192.168.56.12:2379 0.0.0.0:* LISTEN 2166/etcd tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 2166/etcd tcp 0 0 192.168.56.12:2380 0.0.0.0:* LISTEN 2166/etcd tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 657/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 908/master tcp6 0 0 :::22 :::* LISTEN 657/sshd tcp6 0 0 ::1:25 :::* LISTEN 908/master
linux-node3节点配置ETCD
[root@linux-node3 ~]# vim /opt/kubernetes/cfg/etcd.conf #[member] ETCD_NAME="etcd-node3" #修改为自己的名称 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_SNAPSHOT_COUNTER="10000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" ETCD_LISTEN_PEER_URLS="https://192.168.56.13:2380" #修改为自己的IP ETCD_LISTEN_CLIENT_URLS="https://192.168.56.13:2379,https://127.0.0.1:2379" #修改为自己的IP #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" #ETCD_CORS="" #[cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.56.13:2380" #修改为自己的IP # if you use different ETCD_NAME (e.g. test), # set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..." ETCD_INITIAL_CLUSTER="etcd-node1=https://192.168.56.11:2380,etcd-node2=https://192.168.56.12:2380,etcd-node3=https://192.168.56.13:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.56.13:2379" #修改为自己的IP #[security] CLIENT_CERT_AUTH="true" ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" 在所有节点上创建etcd存储目录并启动etcd [root@linux-node3 ~]# mkdir /var/lib/etcd [root@linux-node3 ~]# systemctl daemon-reload [root@linux-node3 ~]# systemctl enable etcd Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /etc/systemd/system/etcd.service. #其它节点部署完成后，同时启动此服务 [root@linux-node3 ~]# systemctl start etcd [root@linux-node3 ~]# systemctl status etcd ● etcd.service - Etcd Server Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-05-30 22:53:14 CST; 20s ago Main PID: 2190 (etcd) CGroup: /system.slice/etcd.service └─2190 /opt/kubernetes/bin/etcd May 30 22:53:16 linux-node3.example.com etcd[2190]: peer 6566e06d7343e1bb became active May 30 22:53:16 linux-node3.example.com etcd[2190]: established a TCP streaming connection with peer...er) May 30 22:53:16 linux-node3.example.com etcd[2190]: established a TCP streaming connection with peer...er) May 30 22:53:16 linux-node3.example.com etcd[2190]: established a TCP streaming connection with peer...er) May 30 22:53:16 linux-node3.example.com etcd[2190]: established a TCP streaming connection with peer...er) May 30 22:53:16 linux-node3.example.com etcd[2190]: ce7b884e428b6c8c initialzed peer connection; fas...(s) May 30 22:53:16 linux-node3.example.com etcd[2190]: health check for peer 6566e06d7343e1bb could not...sed May 30 22:53:19 linux-node3.example.com etcd[2190]: updating the cluster version from 3.0 to 3.2 May 30 22:53:19 linux-node3.example.com etcd[2190]: updated the cluster version from 3.0 to 3.2 May 30 22:53:19 linux-node3.example.com etcd[2190]: enabled capabilities for version 3.2 Hint: Some lines were ellipsized, use -l to show in full. [root@linux-node3 ~]# netstat -ntlp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 192.168.56.13:2379 0.0.0.0:* LISTEN 2190/etcd tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 2190/etcd tcp 0 0 192.168.56.13:2380 0.0.0.0:* LISTEN 2190/etcd tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 655/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 909/master tcp6 0 0 :::22 :::* LISTEN 655/sshd tcp6 0 0 ::1:25 :::* LISTEN 909/master
所有节点同时启动ETCD服务
#启动服务时会连接其它节点，如果其它节点为配置完成，此服务会处于一值连接状态
systemctl start etcd
systemctl status etcd
