k8s v1.9.9 二进制部署 （3）创建高可用etcd集群
1.etcd集群认证文件确认
#确认三个节点目录下都有下面文件。
$ll /etc/kubernetes/ssl/kubernetes*
-rw------- 1 root root 1675 Dec 28 12:24 /etc/kubernetes/ssl/kubernetes-key.pem
-rw-r--r-- 1 root root 1627 Dec 28 12:24 /etc/kubernetes/ssl/kubernetes.pem
2.安装Etcd
三个节点使用yum安装etcd服务。
#yum info etcd
Version : 3.3.11 我这里的版本。
$yum -y install etcd
3.创建etcd的systemd unit文件
注：
1、IP除了initial-cluster 配置项是配置集群内3个地址的IP外，其他IP均为本机的IP。
2、配置里--name必须与--initial-cluster的名称相对应。
3、通过不同方式安装的软件Execstart配置项下的程序启动命令路径注意修改。
4、WorkingDirectory工作目录需要实现创建，否则启动会报错，yum安装的方式是自动创建的。
3.1.master131节点:
$cat > /usr/lib/systemd/system/etcd.service <<eof
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd --name etcd1 --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --trusted-ca-file=/etc/kubernetes/ssl/ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem --initial-advertise-peer-urls https://192.168.7.131:2380 --listen-peer-urls https://192.168.7.131:2380 --listen-client-urls https://192.168.7.131:2379,http://127.0.0.1:2379 --advertise-client-urls https://192.168.7.131:2379 --initial-cluster-token etcd-cluster-0 --initial-cluster etcd1=https://192.168.7.131:2380,etcd2=https://192.168.7.132:2380,etcd3=https://192.168.7.133:2380 --initial-cluster-state new --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
eof
3.2 .node132节点
$cat > /usr/lib/systemd/system/etcd.service <<eof
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd --name etcd2 --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --trusted-ca-file=/etc/kubernetes/ssl/ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem --initial-advertise-peer-urls https://192.168.7.132:2380 --listen-peer-urls https://192.168.7.132:2380 --listen-client-urls https://192.168.7.132:2379,http://127.0.0.1:2379 --advertise-client-urls https://192.168.7.132:2379 --initial-cluster-token etcd-cluster-0 --initial-cluster etcd1=https://192.168.7.131:2380,etcd2=https://192.168.7.132:2380,etcd3=https://192.168.7.133:2380 --initial-cluster-state new --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
eof
3.3 .node133节点
$cat > /usr/lib/systemd/system/etcd.service <<eof
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd --name etcd3 --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --trusted-ca-file=/etc/kubernetes/ssl/ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem --initial-advertise-peer-urls https://192.168.7.133:2380 --listen-peer-urls https://192.168.7.133:2380 --listen-client-urls https://192.168.7.133:2379,http://127.0.0.1:2379 --advertise-client-urls https://192.168.7.133:2379 --initial-cluster-token etcd-cluster-0 --initial-cluster etcd1=https://192.168.7.131:2380,etcd2=https://192.168.7.132:2380,etcd3=https://192.168.7.133:2380 --initial-cluster-state new --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
eof
4.创建etcd配置文件
注：
1、IP地址替换为本机的即可。
2、ETCD_NAME按照etcd系统服务里面的配置一一对应。
4.1.master131节点:
$cat > /etc/etcd/etcd.conf <<eof
# [member]
ETCD_NAME=etcd1
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.7.131:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.7.131:2379"
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.7.131:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.7.131:2379"
eof
4.2 .node132节点：
$cat > /etc/etcd/etcd.conf <<eof
# [member]
ETCD_NAME=etcd2
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.7.132:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.7.132:2379"
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.7.132:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.7.132:2379"
eof
4.3 .node133节点：
$cat > /etc/etcd/etcd.conf <<eof
# [member]
ETCD_NAME=etcd3
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.7.133:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.7.133:2379"
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.7.133:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.7.133:2379"
eof
5.开机启动及启动etcd
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd
6.检测集群工作情况
在任意一个节点，master或者node都可以，执行以下命令
$etcdctl \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
cluster-health
如果输出类似如下如的情况，代表成功：
注：
1、建议所有节点都运行一次检测。
2、以后使用etcd查询数据都需要使用认证文件，即上述格式。
