kubeadm 部署kubernetes1.14

节点信息：

主机名	IP	角色
k8s-master	10.10.0.10	master节点
k8s-node01	10.10.0.11	集群worke节点
k8s-node02	10.10.0.12	集群worke节点

 一、初始化系统环境

系统初始化环境配置，需要在三台服务器都执行，以下命令请看对应的命令参数，在不同服务器执行。

1）主机名

分别在三台机器执行对应命令，设置主机名

[root@centos01 ~]# hostnamectl set-hostname k8s-master
[root@centos02 ~]# hostnamectl set-hostname k8s-node01
[root@centos03 ~]# hostnamectl set-hostname k8s-node02

2）关闭防火墙、selinux

[root@centos01 ~]# systemctl disable firewalld
[root@centos01 ~]# systemctl stop firewalld
[root@centos01 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

3）设置内核

[root@centos01 ~]# vim /etc/sysctl.d/k8s.conf
## 添加如下内容
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
## 执行命令生效
[root@centos01 ~]# modprobe br_netfilter
[root@centos01 ~]# sysctl -p /etc/sysctl.d/k8s.conf

4）免密钥

在master节点进行对2台node节点进行免密钥登陆

[root@k8s-master ~]# ssh-keygen
[root@k8s-master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.10.0.11
[root@k8s-master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.10.0.12

 二、部署kubernetes集群

1）配置yum源

yum源三台机器都需要配置，这里我们以master主机为例，node节点也按照此yum配置即可

[root@k8s-master ~]# yum -y install wget
[root@k8s-master ~]# cd /etc/yum.repos.d
## 配置docker-ce源
[root@k8s-master yum.repos.d]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
## 配置kubernetes源
[root@k8s-master yum.repos.d]# vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes Repo
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=1   
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
enabled=1
## 下载校验文件
[root@k8s-master ~]# wget https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
[root@k8s-master ~]# wget https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
## 导入校验文件
[root@k8s-master ~]# rpm --import rpm-package-key.gpg
[root@k8s-master ~]# rpm --import yum-key.gpg
[root@k8s-master ~]# yum clean all  && yum makecache fast

2）安装docker、kubelet、kubeadm等工具

[root@k8s-master ~]# yum  install kubelet-1.14.1 kubeadm-1.14.1 kubectl-1.14.1 docker-ce -y
## node节点无需配置kubctl组件
[root@k8s-node01 ~]# yum  install kubelet-1.14.1 kubeadm-1.14.1 docker-ce -y
[root@k8s-node02 ~]# yum  install kubelet-1.14.1 kubeadm-1.14.1 docker-ce -y

3）配置docker kubelet

2台node节点也需要修改对应配置

## 设置kubelet启动时忽略swap报错
[root@k8s-master ~]# vim /etc/sysconfig/kubelet 
 KUBELET_EXTRA_ARGS="--fail-swap-on=false"
## 设置开机自启动
[root@k8s-master ~]# systemctl daemon-reload 
[root@k8s-master ~]# systemctl enable docker && systemctl restart docker
[root@k8s-master ~]# systemctl enable kubelet && systemctl restart kubelet

4）初始化master节点

kubernetes从1.13版本开始，可以指定镜像仓库进行集群初始化操作，所以我们直接指定阿里云镜像仓库进行集群初始化，这样无需再关心国内网络环境是否可以下载到对应的官方原始pod镜像

[root@k8s-master ~]# swapoff -a
[root@k8s-master ~]# kubeadm init --apiserver-advertise-address=10.10.0.10 \
 --image-repository registry.aliyuncs.com/google_containers \
 --kubernetes-version v1.14.1 \
 --pod-network-cidr=10.244.0.0/16

参数解释：

• apiserver-advertise-address：apiserver地址及master节点地址
• image-repository：镜像仓库地址
• pod-network-cidr：pod网络

以上初始化，需要等待一段时间，因为需要下载对应组件镜像，master节点初始化完成后，可看到以下提示信息：

........
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.10.0.10:6443 --token 5ti5kd.o32bm9lofv6zej94 \
    --discovery-token-ca-cert-hash sha256:cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f

注意：

    token是node节点加入时需要用到的信息，需要记录下来，tocken值24小时后失效，若果超过24小时你再进行node节点加入集群，需要重新生成tocken。

5）配置集群环境变量

用户想要使用kubectl操作集群，则需要配置kubectl环境变量，这些命令也是上面kubeadm init后输出的内容

[root@k8s-master ~]# mkdir -p $HOME/.kube
[root@k8s-master ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master ~]# chown $(id -u):$(id -g) $HOME/.kube/config
## 查看集群信息
[root@k8s-master ~]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-0               Healthy   {"health":"true"}
## 查看node就绪状态
### 由于还没有安装网络插件，以及node节点未加入集群，所以只显示一个master节点信息
[root@k8s-master ~]# kubectl get nodes
NAME         STATUS     ROLES    AGE     VERSION
k8s-master   NotReady   master   7m33s   v1.14.1

 三、部署网络插件

集群的运行依赖于网络，k8s本身并不支持网络，需要额外部署对应的网络插件，才可实现集群的个组件网络通信。我们这里采用flannel作为集群网络插件。

项目地址：https://github.com/coreos/flannel

## 在线部署
[root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
## 或者把清单配置文件下载本地再执行清单文件应用
[root@k8s-master ~]# mkdir /opt/k8s/flannel
[root@k8s-master ~]# cd /opt/k8s/flannel
[root@k8s-master ~]# wget  https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
[root@k8s-master ~]# kubectl apply -f kube-flannel.yml

四、node节点加入集群

1）加入集群

node节点加入集群操作基本一致，这里以其中一台为例。kubeadm join命令为master节点初始化成功后显示的命令，上文已经提到过。

[root@k8s-node01 ~]# swapoff -a
[root@k8s-node01 ~]# kubeadm join 10.10.0.10:6443 --token 5ti5kd.o32bm9lofv6zej94 \
     --discovery-token-ca-cert-hash sha256:cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f

[preflight] Running pre-flight checks
 [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
 [WARNING Hostname]: hostname "k8s-node01" could not be reached
 [WARNING Hostname]: hostname "k8s-node01": lookup k8s-node01 on 114.114.114.114:53: no such host
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

注意：

　　node节点初始化加入集群，会看到对应的初始化成功信息，初始化时由于需要下载对应pod镜像，需要等待一段时间，node才会成功加入。

2）查看节点就绪状态

[root@k8s-master ~]# kubectl get nodes
NAME         STATUS   ROLES    AGE    VERSION
k8s-master   Ready    master   139m   v1.14.1
k8s-node01   Ready    <none>   2m8s   v1.14.1
k8s-node02   Ready    <none>   64s    v1.14.1

3）集群删除

以上就是kubeadm部署集群过程，过程其实并不复杂，如果在集群部署过程中，出现问题，想要删除重新初始化集群，使用以下命令：

kubeadm reset
ifconfig cni0 down && ip link delete cni0
ifconfig flannel.1 down && ip link delete flannel.1
rm -rf /var/lib/cni/

特别说明：

　　1）关于token失效

　　上面说过，master集群初始化后，token24小时后就会失效，如果到了token失效时间，node再加入集群，需要重新生产token：

## 查看token状态
### TTL值 就是token生于时间
[root@k8s-master ~]# kubeadm token list
TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
5ti5kd.o32bm9lofv6zej94   21h       2019-05-22T11:16:31+08:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token
## 重新生产token
[root@k8s-master ~]# kubeadm token create
　　hb0mhv.ckb79uumxh06br8e
## 获取--discovery-token-ca-cert-hash值
[root@k8s-master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
   openssl dgst -sha256 -hex | sed 's/^.* //'
cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f
## 加入集群命令
[root@k8s-master ~]# kubeadm join 10.10.0.10:6443 --token hb0mhv.ckb79uumxh06br8e \
    --discovery-token-ca-cert-hash sha256:cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f