K8s集群部署（一）------ETCD集群部署
环境说明
三台主机：
k8s-master 10.0.3.225
k8s-node1 10.0.3.226
k8s-node2 10.0.3.227
配置主机名解析
[root@k8s-master kubernetes]# vim /etc/hosts 10.0.3.225 k8s-master 10.0.3.226 k8s-node1 10.0.3.227 k8s-node2
关闭防火墙和SElinux
[root@k8s-master kubernetes]# systemctl stop firewalld [root@k8s-master kubernetes]# getenforce Disabled
设置环境变量
vim /root/.bash_profile PATH=$PATH:$HOME/bin:/opt/kubernetes/bin
source /root/.bash_profile
为了拷贝方便，设置SSH免密
[root@k8s-master src]# ssh-keygen [root@k8s-master src]# ssh-copy-id 10.0.3.225 [root@k8s-master src]# ssh-copy-id 10.0.3.226 [root@k8s-master src]# ssh-copy-id 10.0.3.227
系统环境初始化
1.安装Docker（三台操作）
第一步：使用国内Docker源
[root@linux-node1 ~]# cd /etc/yum.repos.d/ [root@linux-node1 yum.repos.d]# wget \ https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
第二步：Docker安装：
[root@linux-node1 ~]# yum install -y docker-ce
第三步：启动后台进程：
[root@linux-node1 ~]# systemctl start docker
2.准备部署目录（三台操作）
mkdir -p /opt/kubernetes/{cfg,bin,ssl,log}
3.准备软件包百度网盘下载地址：（在mastrt操作）
[https://pan.baidu.com/s/1zs8sCouDeCQJ9lghH1BPiw](https://pan.baidu.com/s/1zs8sCouDeCQJ9lghH1BPiw)
cd /usr/local/ unzip k8s-v1.10.1-manual.zip cd k8s-v1.10.1-manual/k8s-v1.10.1/ mv * /usr/local/src/
k8s官网地址：https://github.com/kubernetes
4.解压软件包（在mastrt操作）
cd /usr/local/src/ tar zxvf kubernetes.tar.gz && \ tar zxvf kubernetes-server-linux-amd64.tar.gz && \ tar zxvf kubernetes-client-linux-amd64.tar.gz && \ tar zxvf kubernetes-node-linux-amd64.tar.gz
所有软件包都在kubernetes目录下了
[root@k8s-master src]# ls cfssl-certinfo_linux-amd64 cfssl_linux-amd64 etcd-v3.2.18-linux-amd64.tar.gz kubernetes kubernetes-node-linux-amd64.tar.gz kubernetes.tar.gz cfssljson_linux-amd64 cni-plugins-amd64-v0.7.1.tgz flannel-v0.10.0-linux-amd64.tar.gz kubernetes-client-linux-amd64.tar.gz kubernetes-server-linux-amd64.tar.gz [root@k8s-master src]# cd kubernetes/ [root@k8s-master kubernetes]# ls addons client cluster docs examples hack kubernetes-src.tar.gz LICENSES node README.md server third_party version [root@k8s-master kubernetes]#
手动制作CA证书
1.安装 CFSSL
上面的安装包，已经下载好cfssl了。移动/opt/kubernetes/bin/即可。
cd /usr/local/src chmod +x cfssl* && \ mv cfssl-certinfo_linux-amd64 /opt/kubernetes/bin/cfssl-certinfo && \ mv cfssljson_linux-amd64 /opt/kubernetes/bin/cfssljson && \ mv cfssl_linux-amd64 /opt/kubernetes/bin/cfssl
复制cfssl命令文件到k8s-node1和k8s-node2节点。如果实际中多个节点，就都需要同步复制。
[root@k8s-master src]# scp /opt/kubernetes/bin/cfssl* 10.0.3.226:/opt/kubernetes/bin/ [root@k8s-master src]# scp /opt/kubernetes/bin/cfssl* 10.0.3.227:/opt/kubernetes/bin/
CFSSL下载地址：
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
2.初始化cfssl
[root@k8s-master src]# cd /usr/local/src/ [root@k8s-master src]# mkdir ssl && cd ssl [root@k8s-master ssl]# cfssl print-defaults config > config.json [root@k8s-master ssl]# cfssl print-defaults csr > csr.json
3.创建用来生成 CA 文件的 JSON 配置文件
[root@k8s-master ssl]# vim ca-config.json { "signing": { "default": { "expiry": "8760h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "8760h" } } } }
4.创建用来生成 CA 证书签名请求（CSR）的 JSON 配置文件
[root@k8s-master ssl]# vim ca-csr.json { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
5.生成CA证书（ca.pem）和密钥（ca-key.pem）
[root@k8s-master ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca 2018/11/13 16:12:11 [INFO] generating a new CA key and certificate from CSR 2018/11/13 16:12:11 [INFO] generate received request 2018/11/13 16:12:11 [INFO] received CSR 2018/11/13 16:12:11 [INFO] generating key: rsa-2048 2018/11/13 16:12:12 [INFO] encoded CSR 2018/11/13 16:12:12 [INFO] signed certificate with serial number 672169645465776774912654230107317469678160608506 [root@k8s-master ssl]# ls -l ca* -rw-r--r-- 1 root root 290 Nov 13 16:10 ca-config.json -rw-r--r-- 1 root root 1001 Nov 13 16:12 ca.csr -rw-r--r-- 1 root root 208 Nov 13 16:11 ca-csr.json -rw------- 1 root root 1675 Nov 13 16:12 ca-key.pem -rw-r--r-- 1 root root 1359 Nov 13 16:12 ca.pem
6.分发证书
[root@k8s-master ssl]# cp ca.csr ca.pem ca-key.pem ca-config.json /opt/kubernetes/ssl
#scp证书到k8s-node1和k8s-node2节点 [root@k8s-master ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 10.0.3.226:/opt/kubernetes/ssl ca.csr 100% 1001 1.0KB/s 00:00 ca.pem 100% 1359 1.3KB/s 00:00 ca-key.pem 100% 1675 1.6KB/s 00:00 ca-config.json 100% 290 0.3KB/s 00:00 [root@k8s-master ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 10.0.3.227:/opt/kubernetes/ssl ca.csr 100% 1001 1.0KB/s 00:00 ca.pem 100% 1359 1.3KB/s 00:00 ca-key.pem 100% 1675 1.6KB/s 00:00 ca-config.json 100% 290 0.3KB/s 00:00
部署ETCD集群
详细介绍查看etcd官网： https://github.com/coreos/etcd-operator
0.准备etcd软件包
[root@k8s-master src]# pwd /usr/local/src [root@k8s-master src]# tar zxvf etcd-v3.2.18-linux-amd64.tar.gz [root@k8s-master src]# cd etcd-v3.2.18-linux-amd64/ [root@k8s-master etcd-v3.2.18-linux-amd64]# ls Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md
[root@k8s-master etcd-v3.2.18-linux-amd64]# cp etcd etcdctl /opt/kubernetes/bin/ [root@k8s-master etcd-v3.2.18-linux-amd64]# cp etcd etcdctl 10.0.3.226:/opt/kubernetes/bin/ [root@k8s-master etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 10.0.3.227:/opt/kubernetes/bin/
下载地址：wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz
1.创建 etcd 证书签名请求：
[root@k8s-master ssl]# cd /usr/local/src/ssl/ [root@k8s-master ssl]# vim etcd-csr.json { "CN": "etcd", "hosts": [ "127.0.0.1", "10.0.3.225", #指定etcd节点的IP地址 "10.0.3.226", "10.0.3.227" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
2.生成 etcd 证书和私钥：
[root@k8s-master ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \ > -ca-key=/opt/kubernetes/ssl/ca-key.pem \ > -config=/opt/kubernetes/ssl/ca-config.json \ > -profile=kubernetes etcd-csr.json | cfssljson -bare etcd 会生成一下证书文件 [root@k8s-master ssl]# ls -l etcd* -rw-r--r-- 1 root root 1062 Nov 13 16:40 etcd.csr -rw-r--r-- 1 root root 278 Nov 13 16:39 etcd-csr.json -rw------- 1 root root 1675 Nov 13 16:40 etcd-key.pem -rw-r--r-- 1 root root 1436 Nov 13 16:40 etcd.pem
3.将证书移动到/opt/kubernetes/ssl目录下
[root@k8s-master ssl]# cp etcd*.pem /opt/kubernetes/ssl [root@k8s-master ssl]# scp etcd*.pem 10.0.3.226:/opt/kubernetes/ssl etcd-key.pem 100% 1675 1.6KB/s 00:00 etcd.pem 100% 1436 1.4KB/s 00:00 [root@k8s-master ssl]# scp etcd*.pem 10.0.3.227:/opt/kubernetes/ssl etcd-key.pem 100% 1675 1.6KB/s 00:00 etcd.pem
4.设置ETCD配置文件
[root@k8s-master ssl]# vim /opt/kubernetes/cfg/etcd.conf #[member] ETCD_NAME="etcd-node1" #唯一的，每个节点不能一样 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #数据存放目录 #ETCD_SNAPSHOT_COUNTER="10000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" ETCD_LISTEN_PEER_URLS="https://10.0.3.225:2380" #集群之间通信使用 ETCD_LISTEN_CLIENT_URLS="https://10.0.3.225:2379,https://127.0.0.1:2379" #是给客户端使用 #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" #ETCD_CORS="" #[cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.3.225:2380" # if you use different ETCD_NAME (e.g. test), # set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..." ETCD_INITIAL_CLUSTER="etcd-node1=https://10.0.3.225:2380,etcd-node2=https://10.0.3.226:2380,etcd-node3=https://10.0.3.227:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="https://10.0.3.225:2379" #[security] CLIENT_CERT_AUTH="true" ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
5.创建ETCD系统服务
[root@k8s-master ssl]# vim /etc/systemd/system/etcd.service [Unit] Description=Etcd Server After=network.target [Service] Type=simple WorkingDirectory=/var/lib/etcd EnvironmentFile=-/opt/kubernetes/cfg/etcd.conf # set GOMAXPROCS to number of processors ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/kubernetes/bin/etcd" Type=notify [Install] WantedBy=multi-user.target
6.重新加载系统服务
[root@k8s-master ssl]# systemctl daemon-reload
[root@k8s-master ssl]# systemctl enable etcd.service
etcd.conf和etcd.service拷贝到其他节点
[root@k8s-master ssl]# scp /opt/kubernetes/cfg/etcd.conf 10.0.3.226:/opt/kubernetes/cfg/ etcd.conf 100% 1148 1.1KB/s 00:00 [root@k8s-master ssl]# scp /opt/kubernetes/cfg/etcd.conf 10.0.3.227:/opt/kubernetes/cfg/ etcd.conf 100% 1148 1.1KB/s 00:00 [root@k8s-master ssl]# scp /etc/systemd/system/etcd.service 10.0.3.226:/opt/kubernetes/cfg/ etcd.service 100% 314 0.3KB/s 00:00 [root@k8s-master ssl]# scp /etc/systemd/system/etcd.service 10.0.3.227:/opt/kubernetes/cfg/ etcd.service
其他节点需要修改一下配置。以node1为例，node2一样要修改。
[root@k8s-node1 yum.repos.d]# vim /opt/kubernetes/cfg/etcd.conf 2 ETCD_NAME="etcd-node2" 7 ETCD_LISTEN_PEER_URLS="https://10.0.3.226:2380" 8 ETCD_LISTEN_CLIENT_URLS="https://10.0.3.226:2379,https://127.0.0.1:2379" 13 ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.3.226:2380" 19 ETCD_ADVERTISE_CLIENT_URLS="https://10.0.3.226:2379"
同样需要reload一下。systemctl daemon-reload
systemctl enable etcd
在所有节点上创建etcd存储目录并启动etcd
mkdir /var/lib/etcd systemctl start etcd systemctl status etcd
netstat -lntup|grep 23
需要在所有的 etcd 节点重复上面的步骤，直到所有机器的 etcd 服务都已启动。
7.验证集群
[root@k8s-node1 ~]# etcdctl --endpoints=https://10.0.3.225:2379 \ > --ca-file=/opt/kubernetes/ssl/ca.pem \ > --cert-file=/opt/kubernetes/ssl/etcd.pem \ > --key-file=/opt/kubernetes/ssl/etcd-key.pem cluster-health member 981424cb437f053 is healthy: got healthy result from https://10.0.3.226:2379 member 7a30be4a7549b699 is healthy: got healthy result from https://10.0.3.227:2379 member d12a89096ab8db09 is healthy: got healthy result from https://10.0.3.225:2379 cluster is healthy
