k8s高可用部署
采用3master节点和nginx+keepalived实现高可用
一、系统初始化（3个master节点均执行）
1、关闭swap swapoff -a 2、关闭防火墙 systemctl stop firewalld systemctl disable firewalld 3、关闭SELinux sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config setenforce 0 4、修改内核参数 cat > /etc/sysctl.d/k8s.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF sysctl -p /etc/sysctl.d/k8s.conf
二、配置yum源（3个master节点均执行）
cat >/etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF
三、时钟同步（3个master节点均执行）
yum install chrony -y -q systemctl start chronyd systemctl enable chronyd timedatectl set-timezone Asia/Shanghai timedatectl set-ntp yes
四、修改hostname添加host解析（3个master节点均执行）
分别修改3台master主机名 hostnamectl set-hostname k8s-master01 hostnamectl set-hostname k8s-master02 hostnamectl set-hostname k8s-master03 3台分别执行 cat >/etc/hosts << EOF 192.168.0.1 k8s-master01 192.168.0.2 k8s-master02 192.168.0.3 k8s-master03 EOF
五、docker安装（3个master节点均执行）
yum remove docker \ docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-engine yum install -y yum-utils yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo yum install docker-ce-20.10.8 docker-ce-cli-20.10.8 containerd.io # 修改Docker参数 vim /etc/docker/daemon.json { "data-root": "/data/docker/lib" # 需要替换成数据盘目录 } mkdir /data/docker/lib systemctl enable docker systemctl start docker
六、docker私有镜像仓库部署(在其中一个master节点部署即可)
mkdir -p /data/docker/registry docker run -d \ --restart=always \ -p 5000:5000 \ --restart=always \ --name registry \ -v /data/docker/registry:/var/lib/registry \ registry:2 /etc/docker/daemon.json 配置文件新增 #3个master均要配置 "insecure-registries" : ["k8s-registry.com"] { "data-root": "/data/docker/lib", "insecure-registries" : ["k8s-registry.com:5000"] } 在3个master添加hosts解析 部署仓库服务器ip k8s-registry.com systemctl restart docker 验证 docker pull busybox docker tag busybox k8s-registry.com:5000/busybox docker push k8s-registry.com:5000/busybox
七、keepalived+nginx
规划k8s-master-api的vip和域名
nginx使用tcp代理负载到3个master节点
keepalived使用标准的配置即可
upstream k8s_master{ server 192.168.0.1:6443; server 192.168.0.2:6443; server 192.168.0.3:6443; } server{ listen 16443; proxy_connect_timeout 20s; proxy_timeout 5m; proxy_pass k8s_master;
! Configuration File for keepalived global_defs { router_id k8s-001 script_user root enable_script_security } ## keepalived检测配置, 与track_script中的配置名保持一致 vrrp_script ha_switch { ## 检测脚本配置 script "/data/HA/sh/ha_switch.sh" interval 3 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { ## 指定keepalived的初始角色, MASTER为主服务器, BACKUP为备服务器 state BACKUP ## 配置是否为非抢占式, 需要将主备服务器的state都设置为BACKUP, 避免无谓的切换 nopreempt interface bond0 virtual_router_id 210 priority 100 advert_int 1 authentication { auth_type PASS ## 建议配置为浮动IP的后2位, 每一位长度用0补齐成3位 auth_pass 00100 } ## VRRP HA 虚拟地址, 如果有多个VIP, 继续换行填写 virtual_ipaddress { 192.168.0.100 } #当前节点成为主节时触发的脚本,通知 notify_master "/data/HA/sh/notify.py master" #当前节点转为备节点时触发的脚本,流通知 notify_backup "/data/HA/sh/notify.py backup" #当前节点转为失败状态时触发的脚本,流通知 notify_fault "/data/HA/sh/notify.py fault" ## 执行检测的服务 track_script { #引用VRRP脚本, 与vrrp_script指定的保持一致 ha_switch } }
八、安装k8s
1、安装依赖 yum install -y kubelet-1.21.0-0 kubeadm-1.21.0-0 kubectl-1.21.0-0 systemctl enable kubelet 2、下载k8s依赖的镜像 image.sh #!/bin/bash # 根据 kubeadm config images list 返回的结果，调整下面各组件的image tag # 比如 k8s.gcr.io/kube-apiserver:v1.21.7，则修改 kube-apiserver:v1.21.0 为 kube-apiserver:v1.21.7 images=( kube-scheduler:v1.21.1 kube-proxy:v1.21.1 kube-controller-manager:v1.21.1 etcd:3.4.13-0 kube-apiserver:v1.21.1 pause:3.4.1) for imageName in ${images[@]} ; do docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName done docker pull coredns/coredns:1.8.0 docker tag coredns/coredns:1.8.0 k8s.gcr.io/coredns/coredns:v1.8.0 docker rmi coredns/coredns:1.8.0 3、编辑初始化文件 cat kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta2 kind: InitConfiguration localAPIEndpoint: #master node ip advertiseAddress: 10.6.122.5 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock #master node hostname name: k8s-master01 --- apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration apiServer: #master vip and domain certSANs: - 10.6.122.210 - "k8s-master.com" clusterName: kubernetes controlPlaneEndpoint: "k8s-master.com:16443" controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /data/k8s-etcd #imageRepository: registry.aliyuncs.com/google_containers kubernetesVersion: v1.21.1 scheduler: {} 4、执行初始化 kubeadm init --config ./kubeadm-config.yaml --upload-certs 然后按照提执行并记录join命令 Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root: kubeadm join k8s-master.com:16443 --token 3qctog.36nbjvci60ib9vfm \ --discovery-token-ca-cert-hash sha256:b1038318d5f36e663d29239dd92c5d829fa21db332e0ea84d6349b41641dbaee \ --control-plane --certificate-key 18e365cba6d888e5ec064bf3cc86e347859b2c5e47c4afd144db0dcd99387eea Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join k8s-master.com:16443 --token 3qctog.36nbjvci60ib9vfm \ --discovery-token-ca-cert-hash sha256:b1038318d5f36e663d29239dd92c5d829fa21db332e0ea84d6349b41641dbaee 5、安装Calico 网络插件 curl https://docs.projectcalico.org/archive/v3.20/manifests/calico.yaml -O kubectl apply -f calico.yaml curl -o calicoctl -O -L "https://github.com/projectcalico/calicoctl/releases/download/v3.19.1/calicoctl" chmod +x calicoctl mv calicoctl /usr/local/bin 确定所有系统pod正常，在master02 master03节点执行join kubeadm join k8s-master.com:16443 完整命令在init输出命令可以看到
