kuadm搭建K8S集群

1.K8s 采用Kubeadm搭建

Kubeadm 是 Kubernetes 官方提供的快速安装和初始化 Kubernetes 集群的工具
阅读以下各组件与对象作用

https://kubernetes.io/docs/concepts/
https://kubernetes.io/docs/setup/independent/install-kubeadm/
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/

1.1环境要求

环境	硬件
centos7.6	CPU:2G 内存:2G

1.2环境角色

ip	角色	安装软件
10.0.0.134	k8s-Master	kube-apiserver kube-schduler kube-controller-manager docker flannel kubelet
10.0.0.135	k8s-node01	kubelet kube-proxy docker flannel

1.3环境初始化

所有机器关闭防火墙

systemctl stop firewalld && systemctl disable firewalld

所有机器关闭selinux

sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config  && setenforce 0

关闭swap分区

# 临时关闭
swapoff -a
# 永久关闭
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab


echo "vm.swappiness = 0">> /etc/sysctl.conf
sysctl -p

设置本机host

# master节点运行如下
hostnamectl set-hostname k8s-master
# node01节点运行如下
hostnamectl set-hostname k8s-node01

# 刷新
bash

master机器/etc/hosts 添加dnfs映射

cat >> /etc/hosts << EOF
10.0.0.137 k8s-master
10.0.0.138 k8s-node01
10.0.0.139 k8s-node02
EOF

将桥接的IPv4流量传递到iptables的链

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

sysctl --system

配置软件源

cd /etc/yum.repos.d
mv CentOS-Base.repo CentOS-Base.repo.bak
mv epel.repo  epel.repo.bak
curl https://mirrors.aliyun.com/repo/Centos-7.repo -o CentOS-Base.repo 
sed -i 's/gpgcheck=1/gpgcheck=0/g' /etc/yum.repos.d/CentOS-Base.repo
curl https://mirrors.aliyun.com/repo/epel-7.repo -o epel.repo

配置配置kubernetes源为阿里的yum源

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

更新缓存

yum clean all && yum makecache && yum repolist

时间同步

yum install ntpdate -y
ntpdate time.windows.com

1.4 安装docker

安装gcc

yum -y install gcc
yum -y install gcc-c++

卸载旧的版本

sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

安装需要软件包

yum install -y yum-utils device-mapper-persistent-data lvm2

master 和 node 安装docker

wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo


yum -y install docker-ce-18.06.1.ce-3.el7

yum 镜像加速

cat > /etc/docker/daemon.json << EOF
{
	"registry-mirrors":["https://4b6uops9.mirror.aliyuncs.com"]
}
EOF

master 和 node 节点启动docker 。并设置开机自启动

systemctl enable docker && systemctl start docker

检查

docker --version

1.5安装 kubeadm,kubelet和kubectl

这里拉取是1.15版本

yum install -y kubelet-1.15.0 kubeadm-1.15.0 kubectl-1.15.0

1.Kubelet：负责与其他节点集群通信，并进行本节点Pod和容器生命周期管理

2.Kubeadm是Kubernetes的自动化部署工具，降低了部署难度，提高效率

3.Kubectl是Kubernetes集群管理工具

启动kubelet

systemctl enable kubelet --now

查看一下kubeadm版本

kubeadm version

kubeadm version: &version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:30:03Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}

1.6K8s Master部署

kubeadm init \
--apiserver-advertise-address=10.0.0.134 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.15.0 \
--service-cidr=10.1.0.0/16 \
--pod-network-cidr=10.244.0.0/16

# 这里apiserve=10.0.0.134 是master节点ip


#master节点部署1.20.0

kubeadm init \
--apiserver-advertise-address=10.0.0.137 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.20.0 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16 \
--ignore-preflight-errors=all

# kubeadm init 创建master节点
# --apiserver-advertise-address 与node节点通信ip
# --image-repository 指定镜像仓库
# --kubernetes-version k8s的版本
# --service-cidr service网段 暴露pod虚拟ip的网段
# --pod-network-cidr pod配置的ip网段
# --ignore-preflight-errors 忽略与检查错误

# 这里apiserve=10.0.0.137 是master节点ip

kubeadm init 这里首先【preflight】做了下环境检查，检查完毕后从配置仓库地址拉取镜像，然后【certs】创建证书目录/etc/kubernetes/pki/，并生成证书。然后【kubeconfig】创建连接apiserver的配置文件目录在/etc/kubernetes, 【kube-start】生成kubelet配置文件并且启动，【control-plane】使用静态pod启动master组件/etc/kubernetes/mainfests, 【upload-config】,【upload-certs】,【kubelet 】使用ConfigMap存储kubelet配置。【mark-control-planne】给master节点添加标签。

通过【bootstrap-token 】kubelet自动申请证书，【addons】安装插件CoreDNS和kube-proxy
由于默认拉取镜像地址k8s.gcr.io国内无法访问，这里指定阿里云镜像仓库地址。
根据输出提示操作：

[root@k8s-master ~]# mkdir -p $HOME/.kube
[root@k8s-master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

1.7 K8s Node节点注册Master

上面根据输出提示输入指令会生成如下内容
kubeadm join 10.0.0.137:6443 --token s6y5ca.7z7v11jhxlp8xvin \
    --discovery-token-ca-cert-hash sha256:ad471efbaa32a0246099031e7439bee4e1b0ac29a811ada3d838c9b014e49460 


# node执行即可

默认token的有效期为24小时，当过期之后，该token就不可用了

如果提示错误：[kubeadm 报错 error execution phase preflight: couldn’t validate the identity of the API Server: abort connecting to API ...

# 在master重新生成token
[root@k8s-master ~]# kubeadm token create
4fa1sg.rtmdm0zvaz9xgvgi
[root@k8s-master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
ad471efbaa32a0246099031e7439bee4e1b0ac29a811ada3d838c9b014e49460


# node节点加入集群
kubeadm join 10.0.0.137:6443 --token 4fa1sg.rtmdm0zvaz9xgvgi \
    --discovery-token-ca-cert-hash sha256:ad471efbaa32a0246099031e7439bee4e1b0ac29a811ada3d838c9b014e49460

后续有nodes节点加入，解决方法如下：步骤：

# 0.重新生成新的token
	kubeadm token create

# 1.master节点查看token:
kubeadm token list

TOKEN		v2t93e.t8fnw16slsjtj5jc# 记住它
TTL			23h
EXPIRES	     2021-04-16T15:10:11+08:00  
USAGES		authentication,signing
DESCRIPTION  The default bootstrap token generated by 'kubeadm init'.                                         EXTRA GROUPS  system:bootstrappers:kubeadm:default-node-token

# 2.master节点生成密钥
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

# 此时会生成： 4fa53fd3b654db36c930152dd71704c1da8707f217a63801caebbb1a0af86022

# 3.node节点加入集群 10.0.0.134 为Master节点
# 先清理环境：
kubeadm reset
# 加入：
kubeadm join 10.0.0.134:6443 --token j3dhvb.oowj54et7l1tsruv --discovery-token-ca-cert-hash sha256:4fa53fd3b654db36c930152dd71704c1da8707f217a63801caebbb1a0af86022

这里记录几个node加入节点错误：

错误1：[ERROR Swap]: running with swap on is not supported. Please disable swap

# 关闭swapoff
swapoff -a
# 注释配置
vi /etc/fstab
# 注释掉 /dev/mapper/centos-swap
# 重启
init 6

错误2：error execution phase preflight: couldn't validate the identity of the API Server: encoding/hex: odd length hex string

# master重新生成密钥即可
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

错误3：[ERROR FileAvailable--etc-kubernetes-kubelet.conf]: ``/etc/kubernetes/kubelet``.conf already exists
```
rm -f /etc/kubernetes/kubelet.conf 
rm -f /etc/kubernetes/pki/ca.crt
# 然后重新join
```

master节点查看加入节点信息
```
kubectl get nodes
```

1.8安装网络插件

这里在master执行，获取kube-flannel.yml

wget https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

这里需要编辑kube-flannel.yml文件，原因是国内有可能访问不了quay.io这个registery
```
vim kube-flannel.yml
```
```
修改106 120行内容 
```
master加载kube-flannel.yml文件配置：
```
kubectl apply -f kube-flannel.yml
```

master查看flannel执行状态

[root@k8s-master ~]# ps -ef|grep flannel
root      26595  26580  1 16:07 ?        00:00:00 /opt/bin/flanneld --ip-masq --kube-subnet-mgr

节点信息查看

kubectl get nodes# 节点查看
kubectl get pod -n kube-system# 运行状态查看

如果在执行kubectl get pod -n kube-system 所有执行状态为1/1方位正常，如果存在0/1情况，应该如下重复操作

# 删除加载kube-flannel.yml
kubectl delete -f kube-flannel.yml
# 重新wget， 修改镜像配置
# 执行
kubectl apply -f kube-flannel.yml

1.9 验证

master 新增一个pod，对外暴漏80 端口

# 创建pod
kubectl create deployment nginx --image=nginx
# nginx暴漏80
kubectl expose deployment nginx --port=80 --type=NodePort
#

查看pods 和 service

[root@k8s-master ~]# kubectl get pods,svc
NAME                         READY   STATUS              RESTARTS   AGE
pod/nginx-554b9c67f9-vrjcf   0/1     ContainerCreating   0          57s

NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.1.0.1      <none>        443/TCP        63m
service/nginx        NodePort    10.1.23.175   <none>        80:31459/TCP   27s

访问http://10.0.0.134:31459/，显示如下成功：

1.10 Dashboard 可视化

获取kubernetes-dashboard.yaml配置文件

wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml

修改：

改为：lizhenliang/kubernetes-dashboard-amd64:v1.10.1

新增：

master执行：

kubectl apply -f kubernetes-dashboard.yaml

访问http://10.0.0.134:30001/，显示如下成功：谷歌因为受信任问题无法访问，用火狐访问

创建service account并绑定默认cluster-admin管理员集群角色

kubectl create serviceaccount cluster-admin -n kube-system
# 名字叫cluster-admin

将创建账号进行权限绑定

kubectl create clusterrolebinding cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:cluster-admin
# cluster-admin 具有超级管理员的权限

这里使用token认证，我们执行命令拿去token

kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/cluster-admin/{print $1}')

令牌登录

其他操作：删除签名

kubectl delete clusterrolebindings cluster-admin

其他浏览器访问不受信任问题解决

[root@k8s-master ~]# cd /etc/kubernetes/pki/
[root@k8s-master pki]# mkdir ui
[root@k8s-master pki]# cp apiserver.crt  ui/
[root@k8s-master pki]# cp apiserver.key  ui/
[root@k8s-master pki]# cd ui/
[root@k8s-master ui]# mv apiserver.crt dashboard.pem
[root@k8s-master ui]# mv  apiserver.key   dashboard-key.pem
[root@k8s-master ui]# kubectl delete secret kubernetes-dashboard-certs -n kube-system
[root@k8s-master ui]# kubectl create secret generic kubernetes-dashboard-certs --from-file=./ -n kube-system

[root@k8s-master]# vim kubernetes-dashboard.yaml #回到这个yaml的路径下修改
修改 dashboard-controller.yaml 文件，在args下面增加证书两行
          - --tls-key-file=dashboard-key.pem
          - --tls-cert-file=dashboard.pem
[root@k8s-master ~]kubectl apply -f kubernetes-dashboard.yaml
[root@k8s-master ~]# kubectl create serviceaccount dashboard-admin -n kube-system
serviceaccount/dashboard-admin created
[root@k8s-master ~]# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin
--serviceaccount=kube-system:dashboard-admin
[root@k8s-master ~]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
Name:

彻底卸载kubeadm

#!/bin/bash
kubeadm reset -f
modprobe -r ipip
lsmod
rm -rf ~/.kube/
rm -rf /etc/kubernetes/
rm -rf /etc/systemd/system/kubelet.service.d
rm -rf /etc/systemd/system/kubelet.service
rm -rf /usr/bin/kube*
rm -rf /etc/cni
rm -rf /opt/cni
rm -rf /var/lib/etcd
rm -rf /var/etcd
yum -y remove kubeadm-1.15.1 kubectl-1.15.1 kubelet-1.15.1