深入学习Kubernetes(二)：集群搭建
一、服务器环境及IP规划
1.1软硬件限制：
1）cpu和内存 master：至少1c2g，推荐2c4g；node：至少1c2g 2）linux系统 内核版本至少3.10，推荐CentOS7/RHEL7 3）docker 至少1.9版本，推荐1.12+ 4）etcd 至少2.0版本，推荐3.0+
1.2 高可用集群所需节点规划：
部署节点------x1 : 运行这份 ansible 脚本的节点 etcd节点------x3 : 注意etcd集群必须是1,3,5,7...奇数个节点 master节点----x2 : 根据实际集群规模可以增加节点数，需要额外规划一个master VIP(虚地址) lb节点--------x2 : 负载均衡节点两个，安装 haproxy+keepalived node节点------x3 : 真正应用负载的节点，根据需要提升机器配置和增加节点数
1.3 机器规划
1.4 设置hostname
#hostnamectl set-hostname master1 #hostnamectl set-hostname node1 #hostnamectl set-hostname node2 #hostnamectl set-hostname master2
1.5 四台服务器全部安装
#yum install -y epel-release;yum update -y;yum install python -y
1.6 deploy节点配置免密码登录
1.6.1 在master1上操作，生成密钥对
[root@master1 ~]# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:Sh7lTBOnvNCpQov6CpWrehTxDZtTIjCML0k1wv7neUA root@master1 The key's randomart image is: +---[RSA 2048]----+ |*o.o . . | |.=+ + .o = | |oo +.B. O | |o.o+=E.B o | | .=.+.+ S | | o.o * o | |o.. o = | |.o. o . | |=o. . | +----[SHA256]-----+
1.6.2将master1上的公钥同步到master1、node1、node2、master2
[root@master1 ~]# for ip in 110 111 112 113; do ssh-copy-id 192.168.0.$ip; done
1.7 deploy节点安装ansible
[root@master1 ~]# yum list|grep ansible [root@master1 ~]# yum install -y ansible [root@master1 ~]# rpm -qa ansible ansible-2.7.5-1.el7.noarch
二、部署kubernets
2.1 deploy上编排k8s
[root@master1 ~]# yum install -y git [root@master1 ~]# git clone https://github.com/gjmzj/kubeasz.git [root@master1 ~]# ls anaconda-ks.cfg kubeasz [root@master1 ~]# mv kubeasz/* /etc/ansible/
- 本文档参考 https://github.com/gjmzj/kubeasz,使用别人做好的kubernets部署项目
2.2 下载kubernets
[root@master1 ~]# ls anaconda-ks.cfg k8s.1-11-6.tar.gz kubeasz [root@master1 ~]# tar zxf k8s.1-11-6.tar.gz [root@master1 ~]# ls anaconda-ks.cfg bin k8s.1-11-6.tar.gz kubeasz [root@master1 ~]# mv bin/* /etc/ansible/bin/
- 从百度云网盘下载二进制文件 https://pan.baidu.com/s/1c4RFaA#list/path=%2F 可以根据自己所需版本，下载对应的tar包，这里我下载1.11 经过一番折腾，最终把k8s.1-11-2.tar.gz的tar包放到了depoly上
2.3 配置集群参数
[root@master1 ~]# cd /etc/ansible/
[root@master1 ansible]# cp example/hosts.m-masters.example hosts
[root@master1 ansible]# vim hosts
//根据实际情况修改IP地址：
# 集群部署节点：一般为运行ansible 脚本的节点
# 变量 NTP_ENABLED (=yes/no) 设置集群是否安装 chrony 时间同步
[deploy]
192.168.0.110 NTP_ENABLED=no
# etcd集群请提供如下NODE_NAME，注意etcd集群必须是1,3,5,7...奇数个节点
[etcd]
192.168.0.110 NODE_NAME=etcd1
192.168.0.111 NODE_NAME=etcd2
192.168.0.113 NODE_NAME=etcd3
[kube-master]
192.168.0.110
192.168.0.113
# 负载均衡(目前已支持多于2节点，一般2节点就够了) 安装 haproxy+keepalived
[lb]
192.168.0.110 LB_IF="ens33" LB_ROLE=backup
192.168.0.113 LB_IF="ens33" LB_ROLE=master
[kube-node]
192.168.0.111
192.168.0.112
#集群主版本号，目前支持: v1.8, v1.9, v1.10，v1.11, v1.12
K8S_VER="v1.11"
# 集群 MASTER IP即 LB节点VIP地址，为区别与默认apiserver端口，设置VIP监听的服务端口8443
MASTER_IP="192.168.0.115"
KUBE_APISERVER="https://{{ MASTER_IP }}:8443"
修改完hosts,测试
[root@master1 ansible]# ansible all -m ping
192.168.0.111 | SUCCESS => {
"changed": false,
"ping": "pong"
}
192.168.0.113 | SUCCESS => {
"changed": false,
"ping": "pong"
}
192.168.0.112 | SUCCESS => {
"changed": false,
"ping": "pong"
}
192.168.0.110 | SUCCESS => {
"changed": false,
"ping": "pong"
}
2.4 分步骤安装
2.4.1 创建证书和安装准备
[root@master1 ansible]# ansible-playbook 01.prepare.yml PLAY [all] ************************************************************************************************************** TASK [Gathering Facts] ************************************************************************************************** ok: [192.168.0.112] ok: [192.168.0.111] ok: [192.168.0.113] ok: [192.168.0.110] PLAY [deploy] *********************************************************************************************************** TASK [deploy : prepare some dirs] *************************************************************************************** changed: [192.168.0.110] => (item=/opt/kube/bin) changed: [192.168.0.110] => (item=/etc/kubernetes/ssl) ok: [192.168.0.110] => (item=/etc/ansible) ok: [192.168.0.110] => (item=/etc/kubernetes) TASK [deploy : 下载证书工具 CFSSL和 kubectl] *********************************************************************************** changed: [192.168.0.110] => (item=cfssl) changed: [192.168.0.110] => (item=cfssl-certinfo) changed: [192.168.0.110] => (item=cfssljson) changed: [192.168.0.110] => (item=kubectl) TASK [deploy : 读取ca证书stat信息] ******************************************************************************************** ok: [192.168.0.110] TASK [deploy : 准备CA配置文件] ************************************************************************************************ changed: [192.168.0.110] TASK [deploy : 准备CA签名请求] ************************************************************************************************ changed: [192.168.0.110] TASK [deploy : 生成 CA 证书和私钥] ********************************************************************************************* changed: [192.168.0.110] TASK [deploy : 准备kubectl使用的admin 证书签名请求] ******************************************************************************** changed: [192.168.0.110] TASK [deploy : 创建 admin证书与私钥] ******************************************************************************************* changed: [192.168.0.110] TASK [deploy : 设置集群参数] ************************************************************************************************** changed: [192.168.0.110] TASK [deploy : 设置客户端认证参数] *********************************************************************************************** changed: [192.168.0.110] TASK [deploy : 设置上下文参数] ************************************************************************************************* changed: [192.168.0.110] TASK [deploy : 选择默认上下文] ************************************************************************************************* changed: [192.168.0.110] TASK [deploy : 准备kube-proxy 证书签名请求] ************************************************************************************* changed: [192.168.0.110] TASK [deploy : 创建 kube-proxy证书与私钥] ************************************************************************************** changed: [192.168.0.110] TASK [deploy : 设置集群参数] ************************************************************************************************** changed: [192.168.0.110] TASK [deploy : 设置客户端认证参数] *********************************************************************************************** changed: [192.168.0.110] TASK [deploy : 设置上下文参数] ************************************************************************************************* changed: [192.168.0.110] TASK [deploy : 选择默认上下文] ************************************************************************************************* changed: [192.168.0.110] TASK [deploy : 移动 kube-proxy.kubeconfig] ******************************************************************************** changed: [192.168.0.110] PLAY [kube-master,kube-node,deploy,etcd,lb] ***************************************************************************** TASK [prepare : 删除centos/redhat默认安装] ************************************************************************************ changed: [192.168.0.110] changed: [192.168.0.111] changed: [192.168.0.112] changed: [192.168.0.113] TASK [prepare : 添加EPEL仓库] *********************************************************************************************** ok: [192.168.0.113] ok: [192.168.0.110] ok: [192.168.0.111] ok: [192.168.0.112] TASK [prepare : 安装基础软件包] ************************************************************************************************ changed: [192.168.0.111] changed: [192.168.0.110] changed: [192.168.0.113] changed: [192.168.0.112] TASK [prepare : 临时关闭 selinux] ******************************************************************************************* changed: [192.168.0.113] changed: [192.168.0.110] changed: [192.168.0.111] changed: [192.168.0.112] TASK [prepare : 永久关闭 selinux] ******************************************************************************************* ok: [192.168.0.113] ok: [192.168.0.112] ok: [192.168.0.111] ok: [192.168.0.110] TASK [prepare : 设置 Centos ulimits] ************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.111] changed: [192.168.0.112] changed: [192.168.0.110] TASK [prepare : prepare some dirs] ************************************************************************************** changed: [192.168.0.113] => (item=/opt/kube/bin) changed: [192.168.0.111] => (item=/opt/kube/bin) ok: [192.168.0.110] => (item=/opt/kube/bin) changed: [192.168.0.112] => (item=/opt/kube/bin) changed: [192.168.0.113] => (item=/etc/kubernetes/ssl) changed: [192.168.0.111] => (item=/etc/kubernetes/ssl) changed: [192.168.0.113] => (item=/root/.kube) changed: [192.168.0.111] => (item=/root/.kube) changed: [192.168.0.112] => (item=/etc/kubernetes/ssl) ok: [192.168.0.110] => (item=/etc/kubernetes/ssl) changed: [192.168.0.112] => (item=/root/.kube) ok: [192.168.0.110] => (item=/root/.kube) TASK [prepare : 分发证书工具 CFSSL] ******************************************************************************************* changed: [192.168.0.113] => (item=cfssl) ok: [192.168.0.110] => (item=cfssl) changed: [192.168.0.111] => (item=cfssl) changed: [192.168.0.112] => (item=cfssl) changed: [192.168.0.113] => (item=cfssl-certinfo) changed: [192.168.0.111] => (item=cfssl-certinfo) changed: [192.168.0.112] => (item=cfssl-certinfo) ok: [192.168.0.110] => (item=cfssl-certinfo) changed: [192.168.0.113] => (item=cfssljson) changed: [192.168.0.111] => (item=cfssljson) changed: [192.168.0.112] => (item=cfssljson) ok: [192.168.0.110] => (item=cfssljson) TASK [prepare : 分发 kubectl] ********************************************************************************************* ok: [192.168.0.110] changed: [192.168.0.111] changed: [192.168.0.113] changed: [192.168.0.112] TASK [prepare : 分发 kubeconfig配置文件] ************************************************************************************** ok: [192.168.0.110 -> 192.168.0.110] changed: [192.168.0.111 -> 192.168.0.110] changed: [192.168.0.113 -> 192.168.0.110] changed: [192.168.0.112 -> 192.168.0.110] TASK [prepare : 分发证书相关] ************************************************************************************************* ok: [192.168.0.110 -> 192.168.0.110] => (item=admin.pem) changed: [192.168.0.113 -> 192.168.0.110] => (item=admin.pem) changed: [192.168.0.111 -> 192.168.0.110] => (item=admin.pem) ok: [192.168.0.110 -> 192.168.0.110] => (item=admin-key.pem) changed: [192.168.0.112 -> 192.168.0.110] => (item=admin.pem) changed: [192.168.0.113 -> 192.168.0.110] => (item=admin-key.pem) changed: [192.168.0.111 -> 192.168.0.110] => (item=admin-key.pem) ok: [192.168.0.110 -> 192.168.0.110] => (item=ca.pem) changed: [192.168.0.112 -> 192.168.0.110] => (item=admin-key.pem) ok: [192.168.0.110 -> 192.168.0.110] => (item=ca-key.pem) changed: [192.168.0.113 -> 192.168.0.110] => (item=ca.pem) changed: [192.168.0.111 -> 192.168.0.110] => (item=ca.pem) changed: [192.168.0.112 -> 192.168.0.110] => (item=ca.pem) ok: [192.168.0.110 -> 192.168.0.110] => (item=ca.csr) changed: [192.168.0.113 -> 192.168.0.110] => (item=ca-key.pem) changed: [192.168.0.111 -> 192.168.0.110] => (item=ca-key.pem) changed: [192.168.0.112 -> 192.168.0.110] => (item=ca-key.pem) ok: [192.168.0.110 -> 192.168.0.110] => (item=ca-config.json) changed: [192.168.0.113 -> 192.168.0.110] => (item=ca.csr) changed: [192.168.0.111 -> 192.168.0.110] => (item=ca.csr) changed: [192.168.0.112 -> 192.168.0.110] => (item=ca.csr) changed: [192.168.0.113 -> 192.168.0.110] => (item=ca-config.json) changed: [192.168.0.111 -> 192.168.0.110] => (item=ca-config.json) changed: [192.168.0.112 -> 192.168.0.110] => (item=ca-config.json) TASK [prepare : 写入环境变量$PATH] ******************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.110] changed: [192.168.0.111] changed: [192.168.0.112] TASK [prepare : 添加 kubectl 命令自动补全] ************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.110] changed: [192.168.0.111] changed: [192.168.0.112] TASK [prepare : 禁用系统 swap] ********************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.110] changed: [192.168.0.111] changed: [192.168.0.112] TASK [prepare : 删除fstab swap 相关配置] ************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.111] changed: [192.168.0.110] changed: [192.168.0.112] TASK [prepare : 加载内核模块] ************************************************************************************************* changed: [192.168.0.112] => (item=br_netfilter) changed: [192.168.0.113] => (item=br_netfilter) changed: [192.168.0.111] => (item=br_netfilter) changed: [192.168.0.110] => (item=br_netfilter) changed: [192.168.0.113] => (item=ip_vs) changed: [192.168.0.112] => (item=ip_vs) changed: [192.168.0.111] => (item=ip_vs) changed: [192.168.0.110] => (item=ip_vs) changed: [192.168.0.113] => (item=ip_vs_rr) changed: [192.168.0.112] => (item=ip_vs_rr) changed: [192.168.0.111] => (item=ip_vs_rr) changed: [192.168.0.110] => (item=ip_vs_rr) changed: [192.168.0.113] => (item=ip_vs_wrr) changed: [192.168.0.112] => (item=ip_vs_wrr) changed: [192.168.0.111] => (item=ip_vs_wrr) changed: [192.168.0.110] => (item=ip_vs_wrr) changed: [192.168.0.112] => (item=ip_vs_sh) changed: [192.168.0.113] => (item=ip_vs_sh) changed: [192.168.0.111] => (item=ip_vs_sh) changed: [192.168.0.110] => (item=ip_vs_sh) changed: [192.168.0.112] => (item=nf_conntrack_ipv4) changed: [192.168.0.111] => (item=nf_conntrack_ipv4) changed: [192.168.0.113] => (item=nf_conntrack_ipv4) changed: [192.168.0.110] => (item=nf_conntrack_ipv4) TASK [prepare : 设置系统参数] ************************************************************************************************* changed: [192.168.0.113] changed: [192.168.0.112] changed: [192.168.0.111] changed: [192.168.0.110] TASK [prepare : 生效系统参数] ************************************************************************************************* changed: [192.168.0.113] changed: [192.168.0.111] changed: [192.168.0.110] changed: [192.168.0.112] PLAY [lb] *************************************************************************************************************** TASK [lb : 注册变量 LB_IF_TMP] ********************************************************************************************** changed: [192.168.0.110] changed: [192.168.0.113] TASK [lb : 设置变量 LB_IF] ************************************************************************************************** ok: [192.168.0.110] ok: [192.168.0.113] TASK [lb : yum安装 haproxy] *********************************************************************************************** changed: [192.168.0.110] changed: [192.168.0.113] TASK [lb : 创建haproxy配置目录] *********************************************************************************************** ok: [192.168.0.110] ok: [192.168.0.113] TASK [lb : 修改centos的haproxy.service] ************************************************************************************ changed: [192.168.0.113] changed: [192.168.0.110] TASK [lb : 配置 haproxy] ************************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.110] TASK [lb : yum安装 keepalived] ******************************************************************************************** changed: [192.168.0.110] changed: [192.168.0.113] TASK [lb : 创建keepalived配置目录] ******************************************************************************************** ok: [192.168.0.110] ok: [192.168.0.113] TASK [lb : 配置 keepalived 主节点] ******************************************************************************************* changed: [192.168.0.113] TASK [lb : 配置 keepalived 备节点] ******************************************************************************************* changed: [192.168.0.110] TASK [lb : daemon-reload for haproxy.service] *************************************************************************** changed: [192.168.0.113] changed: [192.168.0.110] TASK [lb : 开机启用haproxy服务] *********************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.110] TASK [lb : 重启haproxy服务] ************************************************************************************************* changed: [192.168.0.110] changed: [192.168.0.113] TASK [lb : 开机启用keepalived服务] ******************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.110] TASK [lb : 重启keepalived服务] ********************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.110] PLAY RECAP ************************************************************************************************************** 192.168.0.110 : ok=52 changed=40 unreachable=0 failed=0 192.168.0.111 : ok=19 changed=16 unreachable=0 failed=0 192.168.0.112 : ok=19 changed=16 unreachable=0 failed=0 192.168.0.113 : ok=33 changed=27 unreachable=0 failed=0
2.4.2 安装etcd集群
[root@master1 ansible]# ansible-playbook 02.etcd.yml PLAY [etcd] ************************************************************************************************************* TASK [Gathering Facts] ************************************************************************************************** ok: [192.168.0.111] ok: [192.168.0.113] ok: [192.168.0.110] TASK [etcd : 下载etcd二进制文件] *********************************************************************************************** changed: [192.168.0.111] => (item=etcd) changed: [192.168.0.113] => (item=etcd) changed: [192.168.0.110] => (item=etcd) changed: [192.168.0.113] => (item=etcdctl) changed: [192.168.0.111] => (item=etcdctl) changed: [192.168.0.110] => (item=etcdctl) TASK [etcd : 创建etcd证书目录] ************************************************************************************************ changed: [192.168.0.113] changed: [192.168.0.111] changed: [192.168.0.110] TASK [etcd : 读取etcd证书stat信息] ******************************************************************************************** ok: [192.168.0.111] ok: [192.168.0.113] ok: [192.168.0.110] TASK [etcd : 创建etcd证书请求] ************************************************************************************************ changed: [192.168.0.111] changed: [192.168.0.113] changed: [192.168.0.110] TASK [etcd : 创建 etcd证书和私钥] ********************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.111] changed: [192.168.0.110] TASK [etcd : 创建etcd工作目录] ************************************************************************************************ changed: [192.168.0.111] changed: [192.168.0.113] changed: [192.168.0.110] TASK [etcd : 创建etcd的systemd unit文件] ************************************************************************************* changed: [192.168.0.111] changed: [192.168.0.113] changed: [192.168.0.110] TASK [etcd : 开机启用etcd服务] ************************************************************************************************ changed: [192.168.0.111] changed: [192.168.0.110] changed: [192.168.0.113] TASK [etcd : 开启etcd服务] ************************************************************************************************** changed: [192.168.0.110] changed: [192.168.0.113] changed: [192.168.0.111] TASK [etcd : 以轮询的方式等待服务同步完成] ******************************************************************************************** changed: [192.168.0.111] changed: [192.168.0.110] changed: [192.168.0.113] PLAY RECAP ************************************************************************************************************** 192.168.0.110 : ok=11 changed=9 unreachable=0 failed=0 192.168.0.111 : ok=11 changed=9 unreachable=0 failed=0 192.168.0.113 : ok=11 changed=9 unreachable=0 failed=0
2.4.3检查etcd节点健康状况：
[root@master1 ansible]#bash [root@master1 ansible]# for ip in 110 111 113 ; do ETCDCTL_API=3 etcdctl --endpoints=https://192.168.0.$ip:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint healt; done https://192.168.0.110:2379 is healthy: successfully committed proposal: took = 3.224323ms https://192.168.0.111:2379 is healthy: successfully committed proposal: took = 3.186561ms https://192.168.0.113:2379 is healthy: successfully committed proposal: took = 4.731508ms
2.4.4安装docker
[root@master1 ansible]# ansible-playbook 03.docker.yml PLAY [kube-master,new-master,kube-node,new-node] ************************************************************************ TASK [Gathering Facts] ************************************************************************************************** ok: [192.168.0.112] ok: [192.168.0.111] ok: [192.168.0.113] ok: [192.168.0.110] TASK [docker : 准备docker相关目录] ******************************************************************************************** ok: [192.168.0.112] => (item=/opt/kube/bin) ok: [192.168.0.111] => (item=/opt/kube/bin) ok: [192.168.0.113] => (item=/opt/kube/bin) ok: [192.168.0.110] => (item=/opt/kube/bin) changed: [192.168.0.111] => (item=/etc/docker) changed: [192.168.0.112] => (item=/etc/docker) changed: [192.168.0.113] => (item=/etc/docker) changed: [192.168.0.110] => (item=/etc/docker) TASK [docker : 下载 docker 二进制文件] ***************************************************************************************** changed: [192.168.0.113] => (item=docker-containerd) changed: [192.168.0.111] => (item=docker-containerd) changed: [192.168.0.112] => (item=docker-containerd) changed: [192.168.0.110] => (item=docker-containerd) changed: [192.168.0.111] => (item=docker-containerd-shim) changed: [192.168.0.113] => (item=docker-containerd-shim) changed: [192.168.0.112] => (item=docker-containerd-shim) changed: [192.168.0.110] => (item=docker-containerd-shim) changed: [192.168.0.112] => (item=docker-init) changed: [192.168.0.113] => (item=docker-init) changed: [192.168.0.111] => (item=docker-init) changed: [192.168.0.110] => (item=docker-init) changed: [192.168.0.113] => (item=docker-runc) changed: [192.168.0.112] => (item=docker-runc) changed: [192.168.0.111] => (item=docker-runc) changed: [192.168.0.110] => (item=docker-runc) changed: [192.168.0.113] => (item=docker) changed: [192.168.0.112] => (item=docker) changed: [192.168.0.111] => (item=docker) changed: [192.168.0.110] => (item=docker) changed: [192.168.0.113] => (item=docker-containerd-ctr) changed: [192.168.0.112] => (item=docker-containerd-ctr) changed: [192.168.0.111] => (item=docker-containerd-ctr) changed: [192.168.0.110] => (item=docker-containerd-ctr) changed: [192.168.0.111] => (item=dockerd) changed: [192.168.0.110] => (item=dockerd) changed: [192.168.0.112] => (item=dockerd) changed: [192.168.0.113] => (item=dockerd) changed: [192.168.0.111] => (item=docker-proxy) changed: [192.168.0.112] => (item=docker-proxy) changed: [192.168.0.113] => (item=docker-proxy) changed: [192.168.0.110] => (item=docker-proxy) TASK [docker : docker命令自动补全] ******************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.112] changed: [192.168.0.111] changed: [192.168.0.110] TASK [docker : docker国内镜像加速] ******************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.112] changed: [192.168.0.111] changed: [192.168.0.110] TASK [docker : flush-iptables] ****************************************************************************************** changed: [192.168.0.112] changed: [192.168.0.110] changed: [192.168.0.111] changed: [192.168.0.113] TASK [docker : 创建docker的systemd unit文件] ********************************************************************************* changed: [192.168.0.113] changed: [192.168.0.111] changed: [192.168.0.112] changed: [192.168.0.110] TASK [docker : 开机启用docker 服务] ******************************************************************************************* changed: [192.168.0.113] changed: [192.168.0.111] changed: [192.168.0.112] changed: [192.168.0.110] TASK [docker : 开启docker 服务] ********************************************************************************************* changed: [192.168.0.113] changed: [192.168.0.110] changed: [192.168.0.111] changed: [192.168.0.112] TASK [docker : 下载 docker-tag] ******************************************************************************************* changed: [192.168.0.113] changed: [192.168.0.111] changed: [192.168.0.112] changed: [192.168.0.110] PLAY RECAP ************************************************************************************************************** 192.168.0.110 : ok=10 changed=9 unreachable=0 failed=0 192.168.0.111 : ok=10 changed=9 unreachable=0 failed=0 192.168.0.112 : ok=10 changed=9 unreachable=0 failed=0 192.168.0.113 : ok=10 changed=9 unreachable=0 failed=0
2.4.5 安装master节点
[root@master1 ansible]# ansible-playbook 04.kube-master.yml PLAY [kube-master] ****************************************************************************************************** TASK [Gathering Facts] ************************************************************************************************** ok: [192.168.0.110] ok: [192.168.0.113] TASK [kube-master : 下载 kube-master 二进制] ********************************************************************************* ok: [192.168.0.110] => (item=kube-apiserver) ok: [192.168.0.113] => (item=kube-apiserver) ok: [192.168.0.110] => (item=kube-controller-manager) ok: [192.168.0.110] => (item=kube-scheduler) ok: [192.168.0.113] => (item=kube-controller-manager) ok: [192.168.0.113] => (item=kube-scheduler) TASK [kube-master : 创建 kubernetes 证书签名请求] ******************************************************************************* ok: [192.168.0.110] TASK [kube-master : 创建 kubernetes 证书和私钥] ******************************************************************************** changed: [192.168.0.110] TASK [kube-master : 创建 aggregator proxy证书签名请求] ************************************************************************** ok: [192.168.0.110] TASK [kube-master : 创建 aggregator-proxy证书和私钥] *************************************************************************** changed: [192.168.0.110] TASK [kube-master : 创建 basic-auth.csv] ********************************************************************************** ok: [192.168.0.110] TASK [kube-master : 创建kube-apiserver的systemd unit文件] ******************************************************************** ok: [192.168.0.110] TASK [kube-master : 创建kube-controller-manager的systemd unit文件] *********************************************************** ok: [192.168.0.110] TASK [kube-master : 创建kube-scheduler的systemd unit文件] ******************************************************************** ok: [192.168.0.110] TASK [kube-master : enable master 服务] *********************************************************************************** changed: [192.168.0.110] TASK [kube-node : 下载 kubelet,kube-proxy 二进制和基础 cni plugins] ************************************************************* ok: [192.168.0.110] => (item=kubelet) ok: [192.168.0.110] => (item=kube-proxy) changed: [192.168.0.113] => (item=kubelet) ok: [192.168.0.110] => (item=bridge) ok: [192.168.0.110] => (item=host-local) ok: [192.168.0.110] => (item=loopback) changed: [192.168.0.113] => (item=kube-proxy) changed: [192.168.0.113] => (item=bridge) changed: [192.168.0.113] => (item=host-local) changed: [192.168.0.113] => (item=loopback) TASK [kube-node : 准备kubelet 证书签名请求] ************************************************************************************* ok: [192.168.0.110] changed: [192.168.0.113] TASK [kube-node : 创建 kubelet 证书与私钥] ************************************************************************************* changed: [192.168.0.113] changed: [192.168.0.110] TASK [kube-node : 设置集群参数] *********************************************************************************************** changed: [192.168.0.110] changed: [192.168.0.113] TASK [kube-node : 设置客户端认证参数] ******************************************************************************************** changed: [192.168.0.110] changed: [192.168.0.113] TASK [kube-node : 设置上下文参数] ********************************************************************************************** changed: [192.168.0.110] changed: [192.168.0.113] TASK [kube-node : 选择默认上下文] ********************************************************************************************** changed: [192.168.0.110] changed: [192.168.0.113] TASK [kube-node : 移动 kubelet.kubeconfig] ******************************************************************************** changed: [192.168.0.110] changed: [192.168.0.113] TASK [kube-node : 准备 cni配置文件] ******************************************************************************************* ok: [192.168.0.110] changed: [192.168.0.113] TASK [kube-node : 创建kubelet的systemd unit文件] ***************************************************************************** ok: [192.168.0.110] changed: [192.168.0.113] TASK [kube-node : 开机启用kubelet 服务] *************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.110] TASK [kube-node : 开启kubelet 服务] ***************************************************************************************** changed: [192.168.0.110] changed: [192.168.0.113] TASK [kube-node : 安装kube-proxy.kubeconfig配置文件] ************************************************************************** ok: [192.168.0.110 -> 192.168.0.110] changed: [192.168.0.113 -> 192.168.0.110] TASK [kube-node : 创建kube-proxy 服务文件] ************************************************************************************ ok: [192.168.0.110] changed: [192.168.0.113] TASK [kube-node : 开机启用kube-proxy 服务] ************************************************************************************ changed: [192.168.0.110] changed: [192.168.0.113] TASK [kube-node : 开启kube-proxy 服务] ************************************************************************************** changed: [192.168.0.110] changed: [192.168.0.113] TASK [kube-node : 轮询等待kubelet启动] **************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.110] FAILED - RETRYING: 轮询等待node达到Ready状态 (8 retries left). TASK [kube-node : 轮询等待node达到Ready状态] ************************************************************************************ changed: [192.168.0.110 -> 192.168.0.110] changed: [192.168.0.113 -> 192.168.0.110] TASK [kube-node : 设置node节点role] ***************************************************************************************** changed: [192.168.0.110 -> 192.168.0.110] changed: [192.168.0.113 -> 192.168.0.110] TASK [Making master nodes SchedulingDisabled] *************************************************************************** changed: [192.168.0.110 -> 192.168.0.110] changed: [192.168.0.113 -> 192.168.0.110] TASK [Setting master role name] ***************************************************************************************** changed: [192.168.0.110 -> 192.168.0.110] changed: [192.168.0.113 -> 192.168.0.110] PLAY RECAP ************************************************************************************************************** 192.168.0.110 : ok=35 changed=20 unreachable=0 failed=0 192.168.0.113 : ok=34 changed=26 unreachable=0 failed=0
查看集群的状态
[root@master1 ~]# kubectl get componentstatus
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
2.4.6 安装node节点
[root@master1 ansible]# ansible-playbook 05.kube-node.yml PLAY [kube-node] ******************************************************************************************************** TASK [Gathering Facts] ************************************************************************************************** ok: [192.168.0.112] ok: [192.168.0.111] TASK [kube-node : 创建kube-node 相关目录] ************************************************************************************* changed: [192.168.0.112] => (item=/var/lib/kubelet) changed: [192.168.0.111] => (item=/var/lib/kubelet) changed: [192.168.0.112] => (item=/var/lib/kube-proxy) changed: [192.168.0.111] => (item=/var/lib/kube-proxy) changed: [192.168.0.112] => (item=/etc/cni/net.d) changed: [192.168.0.111] => (item=/etc/cni/net.d) TASK [kube-node : 下载 kubelet,kube-proxy 二进制和基础 cni plugins] ************************************************************* changed: [192.168.0.112] => (item=kubelet) changed: [192.168.0.111] => (item=kubelet) changed: [192.168.0.112] => (item=kube-proxy) changed: [192.168.0.111] => (item=kube-proxy) changed: [192.168.0.112] => (item=bridge) changed: [192.168.0.111] => (item=bridge) changed: [192.168.0.112] => (item=host-local) changed: [192.168.0.111] => (item=host-local) changed: [192.168.0.112] => (item=loopback) changed: [192.168.0.111] => (item=loopback) TASK [kube-node : 准备kubelet 证书签名请求] ************************************************************************************* changed: [192.168.0.112] changed: [192.168.0.111] TASK [kube-node : 创建 kubelet 证书与私钥] ************************************************************************************* changed: [192.168.0.112] changed: [192.168.0.111] TASK [kube-node : 设置集群参数] *********************************************************************************************** changed: [192.168.0.111] changed: [192.168.0.112] TASK [kube-node : 设置客户端认证参数] ******************************************************************************************** changed: [192.168.0.112] changed: [192.168.0.111] TASK [kube-node : 设置上下文参数] ********************************************************************************************** changed: [192.168.0.112] changed: [192.168.0.111] TASK [kube-node : 选择默认上下文] ********************************************************************************************** changed: [192.168.0.111] changed: [192.168.0.112] TASK [kube-node : 移动 kubelet.kubeconfig] ******************************************************************************** changed: [192.168.0.111] changed: [192.168.0.112] TASK [kube-node : 准备 cni配置文件] ******************************************************************************************* changed: [192.168.0.112] changed: [192.168.0.111] TASK [kube-node : 创建kubelet的systemd unit文件] ***************************************************************************** changed: [192.168.0.112] changed: [192.168.0.111] TASK [kube-node : 开机启用kubelet 服务] *************************************************************************************** changed: [192.168.0.111] changed: [192.168.0.112] TASK [kube-node : 开启kubelet 服务] ***************************************************************************************** changed: [192.168.0.111] changed: [192.168.0.112] TASK [kube-node : 安装kube-proxy.kubeconfig配置文件] ************************************************************************** changed: [192.168.0.112 -> 192.168.0.110] changed: [192.168.0.111 -> 192.168.0.110] TASK [kube-node : 创建kube-proxy 服务文件] ************************************************************************************ changed: [192.168.0.112] changed: [192.168.0.111] TASK [kube-node : 开机启用kube-proxy 服务] ************************************************************************************ changed: [192.168.0.112] changed: [192.168.0.111] TASK [kube-node : 开启kube-proxy 服务] ************************************************************************************** changed: [192.168.0.111] changed: [192.168.0.112] TASK [kube-node : 轮询等待kubelet启动] **************************************************************************************** changed: [192.168.0.112] changed: [192.168.0.111] FAILED - RETRYING: 轮询等待node达到Ready状态 (8 retries left). TASK [kube-node : 轮询等待node达到Ready状态] ************************************************************************************ changed: [192.168.0.112 -> 192.168.0.110] changed: [192.168.0.111 -> 192.168.0.110] TASK [kube-node : 设置node节点role] ***************************************************************************************** changed: [192.168.0.112 -> 192.168.0.110] changed: [192.168.0.111 -> 192.168.0.110] PLAY RECAP ************************************************************************************************************** 192.168.0.111 : ok=21 changed=20 unreachable=0 failed=0 192.168.0.112 : ok=21 changed=20 unreachable=0 failed=0
查看node节点
[root@master1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.0.110 Ready,SchedulingDisabled master 2h v1.11.6 192.168.0.111 Ready node 2h v1.11.6 192.168.0.112 Ready node 2h v1.11.6 192.168.0.113 Ready,SchedulingDisabled master 2h v1.11.6
2.4.7部署集群网络
[root@master1 ansible]# ansible-playbook 06.network.yml PLAY [kube-master,kube-node] ******************************************************************************************** TASK [Gathering Facts] ************************************************************************************************** ok: [192.168.0.112] ok: [192.168.0.111] ok: [192.168.0.110] ok: [192.168.0.113] TASK [flannel : 在deploy 节点创建相关目录] *************************************************************************************** changed: [192.168.0.110 -> 192.168.0.110] TASK [flannel : 配置 flannel DaemonSet yaml文件] **************************************************************************** changed: [192.168.0.110 -> 192.168.0.110] TASK [flannel : 检查是否已下载离线flannel镜像] ************************************************************************************* changed: [192.168.0.110] TASK [flannel : 创建flannel cni 相关目录] ************************************************************************************* ok: [192.168.0.113] => (item=/etc/cni/net.d) ok: [192.168.0.110] => (item=/etc/cni/net.d) ok: [192.168.0.111] => (item=/etc/cni/net.d) ok: [192.168.0.112] => (item=/etc/cni/net.d) changed: [192.168.0.113] => (item=/opt/kube/images) changed: [192.168.0.112] => (item=/opt/kube/images) changed: [192.168.0.111] => (item=/opt/kube/images) changed: [192.168.0.110] => (item=/opt/kube/images) TASK [flannel : 下载flannel cni plugins] ********************************************************************************** ok: [192.168.0.113] => (item=bridge) ok: [192.168.0.111] => (item=bridge) ok: [192.168.0.112] => (item=bridge) ok: [192.168.0.110] => (item=bridge) changed: [192.168.0.112] => (item=flannel) changed: [192.168.0.111] => (item=flannel) changed: [192.168.0.113] => (item=flannel) changed: [192.168.0.110] => (item=flannel) ok: [192.168.0.112] => (item=host-local) ok: [192.168.0.111] => (item=host-local) ok: [192.168.0.113] => (item=host-local) ok: [192.168.0.110] => (item=host-local) ok: [192.168.0.112] => (item=loopback) ok: [192.168.0.111] => (item=loopback) ok: [192.168.0.113] => (item=loopback) ok: [192.168.0.110] => (item=loopback) changed: [192.168.0.112] => (item=portmap) changed: [192.168.0.111] => (item=portmap) changed: [192.168.0.113] => (item=portmap) changed: [192.168.0.110] => (item=portmap) TASK [flannel : 获取flannel离线镜像推送情况] ************************************************************************************** changed: [192.168.0.113] changed: [192.168.0.110] changed: [192.168.0.111] changed: [192.168.0.112] TASK [flannel : 运行 flannel网络] ******************************************************************************************* changed: [192.168.0.110 -> 192.168.0.110] TASK [flannel : 删除默认cni配置] ********************************************************************************************** changed: [192.168.0.110] changed: [192.168.0.113] changed: [192.168.0.112] changed: [192.168.0.111] FAILED - RETRYING: 轮询等待flannel 运行，视下载镜像速度而定 (15 retries left). FAILED - RETRYING: 轮询等待flannel 运行，视下载镜像速度而定 (15 retries left). FAILED - RETRYING: 轮询等待flannel 运行，视下载镜像速度而定 (15 retries left). FAILED - RETRYING: 轮询等待flannel 运行，视下载镜像速度而定 (15 retries left). FAILED - RETRYING: 轮询等待flannel 运行，视下载镜像速度而定 (14 retries left). FAILED - RETRYING: 轮询等待flannel 运行，视下载镜像速度而定 (14 retries left). FAILED - RETRYING: 轮询等待flannel 运行，视下载镜像速度而定 (14 retries left). FAILED - RETRYING: 轮询等待flannel 运行，视下载镜像速度而定 (14 retries left). TASK [flannel : 轮询等待flannel 运行，视下载镜像速度而定] ******************************************************************************* changed: [192.168.0.110 -> 192.168.0.110] changed: [192.168.0.113 -> 192.168.0.110] changed: [192.168.0.112 -> 192.168.0.110] FAILED - RETRYING: 轮询等待flannel 运行，视下载镜像速度而定 (13 retries left). changed: [192.168.0.111 -> 192.168.0.110] PLAY RECAP ************************************************************************************************************** 192.168.0.110 : ok=10 changed=9 unreachable=0 failed=0 192.168.0.111 : ok=6 changed=5 unreachable=0 failed=0 192.168.0.112 : ok=6 changed=5 unreachable=0 failed=0 192.168.0.113 : ok=6 changed=5 unreachable=0 failed=0
查看kube-system namespace上的pod，从中可以看到flannel相关的pod
[root@master1 ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-695f96dcd5-29z5n 1/1 Running 3 2h coredns-695f96dcd5-tn8fl 1/1 Running 1 2h kube-flannel-ds-g458b 1/1 Running 1 2h kube-flannel-ds-j8xs2 1/1 Running 1 2h kube-flannel-ds-w6sv6 1/1 Running 1 2h kube-flannel-ds-xzhjk 1/1 Running 1 2h metrics-server-75df6ff86f-nns2b 1/1 Running 22 2h
2.4.8安装集群插件(dns, dashboard)
[root@master1 ansible]# ansible-playbook 07.cluster-addon.yml PLAY [kube-node] ******************************************************************************************************** TASK [Gathering Facts] ************************************************************************************************** ok: [192.168.0.111] ok: [192.168.0.112] TASK [cluster-addon : 在deploy 节点创建相关目录] ********************************************************************************* ok: [192.168.0.111 -> 192.168.0.110] => (item=/opt/kube/kube-system/kubedns) ok: [192.168.0.111 -> 192.168.0.110] => (item=/opt/kube/kube-system/coredns) TASK [cluster-addon : 准备 DNS的部署文件] ************************************************************************************** ok: [192.168.0.111 -> 192.168.0.110] => (item=kubedns) ok: [192.168.0.111 -> 192.168.0.110] => (item=coredns) TASK [cluster-addon : 获取所有已经创建的POD信息] *********************************************************************************** changed: [192.168.0.111 -> 192.168.0.110] TASK [cluster-addon : 获取已下载离线镜像信息] ************************************************************************************** changed: [192.168.0.111] TASK [cluster-addon : 获取dashboard离线镜像推送情况] ****************************************************************************** changed: [192.168.0.111] changed: [192.168.0.112] TASK [cluster-addon : 创建 dashboard部署] *********************************************************************************** changed: [192.168.0.111 -> 192.168.0.110] PLAY RECAP ************************************************************************************************************** 192.168.0.111 : ok=7 changed=4 unreachable=0 failed=0 192.168.0.112 : ok=2 changed=1 unreachable=0 failed=0
查看kube-system namespace下的服务
[root@master1 ansible]# kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.68.0.2 <none> 53/UDP,53/TCP 2h kubernetes-dashboard NodePort 10.68.170.14 <none> 443:31875/TCP 2h metrics-server ClusterIP 10.68.157.131 <none> 443/TCP 2h上
上边的步骤可以一步安装;
[root@master1 ansible]# ansible-playbook 90.setup.yml
查看集群信息
[root@master1 ansible]# kubectl cluster-info Kubernetes master is running at https://192.168.0.115:8443 CoreDNS is running at https://192.168.0.115:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy kubernetes-dashboard is running at https://192.168.0.115:8443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy
查看node使用资源情况：
[root@master1 ansible]# kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 192.168.0.110 134m 13% 787Mi 71% 192.168.0.111 73m 7% 248Mi 67% 192.168.0.112 55m 5% 230Mi 62% 192.168.0.113 102m 10% 671Mi 76%
查看pod使用资源情况：
[root@master1 ansible]# kubectl top pod --all-namespaces NAMESPACE NAME CPU(cores) MEMORY(bytes) kube-system coredns-695f96dcd5-29z5n 4m 21Mi kube-system coredns-695f96dcd5-tn8fl 5m 19Mi kube-system kube-flannel-ds-g458b 5m 21Mi kube-system kube-flannel-ds-j8xs2 3m 19Mi kube-system kube-flannel-ds-w6sv6 5m 17Mi kube-system kube-flannel-ds-xzhjk 4m 20Mi kube-system kubernetes-dashboard-68bf55748d-7lbtj 1m 14Mi kube-system metrics-server-75df6ff86f-nns2b 2m 28Mi
三、测试DNS
a）创建nginx service
[root@master1 ansible]# kubectl run nginx --image=nginx --expose --port=80 service/nginx created deployment.apps/nginx created [root@master1 ansible]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.68.0.1 <none> 443/TCP 3h nginx ClusterIP 10.68.93.241 <none> 80/TCP 1m [root@master1 ansible]# kubectl get pod NAME READY STATUS RESTARTS AGE nginx-6f858d4d45-ngvtj 1/1 Running 0 1m
b）创建busybox 测试pod
[root@master1 ansible]# kubectl run busybox --rm -it --image=busybox /bin/sh //进入到busybox内部 If you don't see a command prompt, try pressing enter. / # nslookup nginx.default.svc.cluster.local //结果如下 Server: 10.68.0.2 Address: 10.68.0.2:53 Name: nginx.default.svc.cluster.local Address: 10.68.93.241
