编译安装Kubernetes 1.29 高可用集群(1)--系统初始配置
软件环境描述说明：
- OS：openEuler 22.03 LTS SP3
- kubernetes：1.29.6
- etcd：3.5.12
- helm：3.15.2
- cilium：1.15.6
- crictl：1.30.0
- cfssl/cfssljson/cfssl-certinfo：1.64
- haproxy：2.2.6
- keepalived：2.2.4
- metrics-server：0.7.1
- k8s-dashboard：7.5.0
- traefik：3.0.4
- harbor：2.10.3
网段规划：
物理主机：192.168.83.0/24 Service IP：10.66.0.0/16 Pod IP：172.31.0.0/16
服务器规划：
| 角色 | IP地址 | 组件 | 节点归属 |
|
k8s-haproxy01 |
192.168.83.201 VIP:192.168.83.200 |
haproxy/keepalived (Load Balancer kube-apiserver Master) | haproxy |
|
k8s-haproxy02 |
192.168.83.202 VIP:192.168.83.200 |
haproxy/keepalived (Load Balancer kube-apiserver Backup) | haproxy |
| k8s-master-etcd01 | 192.168.83.210 | kube-apiserver/kube-controller-manager/kube-scheduler/etcd | k8s和etcd |
| k8s-master-etcd02 | 192.168.83.211 | kube-apiserver/kube-controller-manager/kube-scheduler/etcd | k8s和etcd |
| k8s-etcd03 | 192.168.83.212 | etcd | etcd |
| k8s-harbor | 192.168.83.213 | docker/harbor | harbor |
| k8s-node01 | 192.168.83.220 | kubelet/kube-proxy/container | k8s |
| k8s-node02 | 192.168.83.221 | kubelet/kube-proxy/container | k8s |
1. 将所有节点openEuler系统显示时间修改为24小时制
cat >> /etc/profile << EOF
export LC_TIME=POSIX
EOF
source /etc/profile2. 所有节点关闭防火墙和selinux
systemctl stop firewalld && systemctl disable firewalld
sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
setenforce 0
sestatus注：如需开启防火墙，需开放端口可参考Ports and Protocols说明
3.1 所有k8s节点安装软件包
dnf -y install net-tools lrzsz nmap lsof bash-completion ipset ipvsadm chrony socat conntrack ebtables tar tree
cat >> ~/.bashrc << EOF
# enable bash completion in interactive shells
if [ -f /usr/share/bash-completion/bash_completion ]; then
. /usr/share/bash-completion/bash_completion
fi
EOF
source ~/.bashrc
或
source /etc/profile.d/bash_completion.sh3.2 所有haproxy和etcd及harbor节点安装软件包
dnf -y install net-tools lrzsz nmap lsof chrony tar tree bash-completion
source /etc/profile.d/bash_completion.sh4. 1 在所有k8s节点安装配置container
#添加docker安装源
dnf config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
#因docker源中没有openEuler的版本，需要将$releasever替换为openEuler兼容的CentOS8
sed -i s/\$releasever/8/g /etc/yum.repos.d/docker-ce.repo
#安装containerd
dnf -y install containerd.io
#生成container的config.toml
containerd config default > /etc/containerd/config.toml
#添加镜像加速源
mkdir -p /etc/containerd/certs.d/{docker.io,quay.io,harbor.test.local}
cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://docker.io"
[host."https://docker.1panel.live"]
capabilities = ["pull", "resolve"]
[host."https://docker.anyhub.us.kg"]
capabilities = ["pull", "resolve"]
[host."https://dockerpull.com"]
capabilities = ["pull", "resolve"]
EOF
cat > /etc/containerd/certs.d/quay.io/hosts.toml << EOF
server = "https://quay.io"
[host."https://docker.1panel.live"]
capabilities = ["pull", "resolve"]
[host."https://docker.anyhub.us.kg"]
capabilities = ["pull", "resolve"]
[host."https://dockerpull.com"]
capabilities = ["pull", "resolve"]
EOF
#修改修改Containerd的配置文件
sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml
sed -i "s#registry.k8s.io/pause:3.6#registry.aliyuncs.com/google_containers/pause:3.9#g" /etc/containerd/config.toml
sed -i "s#config_path\ \=\ \"\"#config_path\ \=\ \"/etc/containerd/certs.d\"#g" /etc/containerd/config.toml4.2 启动所有k8s节点container
systemctl enable --now containerd
container info4.3 在所有k8s节点安装配置crictl工具
#安装crictl
tar -zxvf crictl-v1.30.0-linux-amd64.tar.gz -C /usr/bin/
#创建crictl和container关联文件
cat > /etc/crictl.yaml << EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
#pull-image-on-create: false
#disable-pull-on-run: false
EOF
# crictl info
显示container配置文件内容即表示关联成功
# crictl version
Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: 1.6.32
RuntimeApiVersion: v15.1 将两个haproxy节点配置为NTP Server
vi /etc/chrony.conf
pool ntp.aliyun.com iburst
pool ntp.tencent.com iburst
allow 192.168.83.0/24
systemctl enable --now chronyd
#验证同步状态
chronyc sourcestats -v5.2 其它各节点与两个haproxy节点做时间同步
vi /etc/chrony.conf
pool 192.168.83.201 iburst
pool 192.168.83.202 iburst
systemctl enable --now chronyd
chronyc sourcestats -v6. 所有节点配置hosts
cat >> /etc/hosts << EOF
192.168.83.201 k8s-haproxy01
192.168.83.202 k8s-haproxy02
192.168.83.210 k8s-master-etcd01
192.168.83.211 k8s-master-etcd02
192.168.83.212 k8s-etcd03
192.168.83.213 k8s-harbor harbor.test.local
192.168.83.220 k8s-node01
192.168.83.221 k8s-node02
EOF7. 所有k8s节点设置ulimit进程资源限制参数
ulimit -SHn 65535
cat >> /etc/security/limits.conf << EOF
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* seft memlock unlimited
* hard memlock unlimitedd
EOF8.1 所有k8s节点加载containerd所需的br_netfilter和overlay模块
cat > /etc/modules-load.d/containerd.conf << EOF
overlay
br_netfilter
EOF
#加载模块
modprobe overlay && modprobe br_netfilter
#查看是否加载
lsmod | grep br_netfilter
lsmod | grep overlay8.2 所有k8s节点配置内核路由转发及网桥过滤并关闭swap
# vi /etc/sysctl.conf
将net.ipv4.ip_forward=0
改为
net.ipv4.ip_forward=1
#最后添加以下内容
###k8s Config###
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
vm.swappiness=0
user.max_user_namespaces=28633
sysctl -p
# swapoff -a
# sed -ri 's/.*swap.*/#&/' /etc/fstab9 .所有k8s节点配置ipvsadm模块加载
cat > /etc/sysconfig/modules/ipvs.modules << EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF
#授权并运行
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules
echo 'bash /etc/sysconfig/modules/ipvs.modules' >> /etc/rc.local && chmod a+x /etc/rc.d/rc.local
#查看是否加载
lsmod | grep -e ip_vs -e nf_conntrack10. 所有k8s节点关闭swap分区
vi /etc/fstab
......
#在加载swap的行前添加#号
#/dev/mapper/openeuler-swap none swap defaults 0 0
#重启后生效，不重启临时关闭命令
swapoff -11. 设置从k8s-master-etcd01节点免密登录到其他节点
[root@k8s-master-etcd01]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:D9YQOzG0M90OWqnQxZCMCTDd4zLVebxk5WEupwKUKjQ root@k8s-master-etcd01
The key's randomart image is:
+---[RSA 3072]----+
| oo.o.B**..+ |
| E...B.=O** . |
| . . +o.O+*.= |
| . + .o X.* |
| . o S o . |
| . + |
| . |
| |
| |
+----[SHA256]-----+
[root@k8s-master-etcd01]# ssh-copy-id root@k8s-master-etcd02
[root@k8s-master-etcd01]# ssh-copy-id root@k8s-etcd03
[root@k8s-master-etcd01]# ssh-copy-id root@k8s-node01
[root@k8s-master-etcd01]# ssh-copy-id root@k8s-node02
少壮不努力，老大干IT。
一入运维深似海，从此不见彼岸花。
