第六章 部署node运算节点服务
一、部署Kubelet
1.1 集群规划
主机名 角色 IP
hdss7-21 kubelet 10.4.7.21
hdss7-22 kubelet 10.4.7.22
注意:部署以10.4.7.21为例,10.4.7.22节点类似
1.2 签发kubelet证书
证书签发需要在10.4.7.200上操作
[root@hdss7-200 ~]# cd /opt/certs/
注意:将所有可能的kubelet服务器的IP都加进去,后期如果需要再加入其他IP节点的话就需要重新签发此证书,有计划的将证书替换成最新的,最好避免后期加入新的节点。
[root@hdss7-200 certs]# vim kubelet-csr.json
{
"CN": "k8s-kubelet",
"hosts": [
"127.0.0.1",
"10.4.7.10",
"10.4.7.21",
"10.4.7.22",
"10.4.7.23",
"10.4.7.24",
"10.4.7.25",
"10.4.7.26",
"10.4.7.27",
"10.4.7.28"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
生成证书
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kubelet-csr.json | cfssl-json -bare kubelet
注意私钥文件的属性权限是600
certs]# ll kubelet*
-rw-r--r-- 1 root root 1115 6月 10 00:04 kubelet.csr
-rw-r--r-- 1 root root 452 6月 10 00:03 kubelet-csr.json
-rw------- 1 root root 1675 6月 10 00:04 kubelet-key.pem
-rw-r--r-- 1 root root 1468 6月 10 00:04 kubelet.pem
分发证书
[root@hdss7-200 certs]# scp kubelet.pem kubelet-key.pem hdss7-21:/opt/kubernetes/server/bin/certs/
[root@hdss7-200 certs]# scp kubelet.pem kubelet-key.pem hdss7-22:/opt/kubernetes/server/bin/certs/
1.3 创建kubelet的配置
在10.4.7.21,22服务器上操作
1.3.1 set-cluster:创建需要连接的集群信息,可以创建多个k8s信息(会将ca.pem证书编码后嵌入到/opt/kubernetes/conf/kubelet.kubeconfig配置文件中)
注意:10.4.7.10是apiserver的VIP,之前我们在10.4.7.11/21上部署的nginx就是代理10.4.7.21/22的apiserver集群,部署的keepalived的VIP就是10.4.7.10
[root@hdss7-21 ~]# cd /opt/kubernetes/
[root@hdss7-21 conf]# kubectl config set-cluster myk8s \
--certificate-authority=/opt/kubernetes/server/bin/certs/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:7443 \
--kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
Cluster "myk8s" set.
[root@hdss7-21 conf]# ll /opt/kubernetes/conf/
总用量 8
-rw-r--r-- 1 root root 2223 6月 8 22:00 audit.yaml
-rw------- 1 root root 1986 6月 10 00:14 kubelet.kubeconfig
1.3.2 set-credentials:创建用户账号,即用户登录的客户端私有证书,可以创建多个证书(将client.pem证书和client-key.pem私钥编码后嵌入到kubelet.kubeconfig文件中)
[root@hdss7-21 conf]# kubectl config set-credentials k8s-node \
--client-certificate=/opt/kubernetes/server/bin/certs/client.pem \
--client-key=/opt/kubernetes/server/bin/certs/client-key.pem \
--embed-certs=true \
--kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
User "k8s-node" set.
1.3.3 set-context:设置context,即确定账号和集群对应关系
[root@hdss7-21 conf]# kubectl config set-context myk8s-context \
--cluster=myk8s \
--user=k8s-node \
--kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
Context "myk8s-context" created.
1.3.4 use-context:设置当前使用哪个context
[root@hdss7-21 conf]# kubectl config use-context myk8s-context --kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
Switched to context "myk8s-context".
把此配置传送给10.4.7.22,那么在22上就不需要重复操作以上4个步骤了
[root@hdss7-21 conf]# scp /opt/kubernetes/conf/kubelet.kubeconfig hdss7-22:/opt/kubernetes/conf/
1.4 授权k8s-node用户
此步骤只需要在一台master节点上操作就行(10.4.7.21)
授权k8s-node用户绑定集群角色system:node,让k8s-node拥有具备运算节点的权限
[root@hdss7-21 conf]# vim /opt/kubernetes/conf/k8s-node.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: k8s-node
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: k8s-node
[root@hdss7-21 conf]# kubectl create -f /opt/kubernetes/conf/k8s-node.yaml
clusterrolebinding.rbac.authorization.k8s.io/k8s-node created
创建资源(会存储到etcd中)
[root@hdss7-21 conf]# kubectl get clusterrolebinding k8s-node
NAME AGE
k8s-node 51s
注意:查看7443端口是否正常启动,非常重要,7443端口无法连接会导致node节点无法加入到master节点
~]# telnet 10.4.7.10 7443
Trying 10.4.7.10...
Connected to 10.4.7.10.
Escape character is '^]'.
^]
telnet> q
删除资源命令如下
[root@hdss7-21 conf]# kubectl delete -f k8s-node.yaml
clusterrolebinding.rbac.authorization.k8s.io "k8s-node" deleted
查看资源配置
[root@hdss7-21 conf]# kubectl get clusterrolebinding k8s-node -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: "2021-06-10T13:51:06Z"
name: k8s-node
resourceVersion: "12725"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/k8s-node
uid: e70f91af-c9f2-11eb-aaf3-000c29e396b1
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: k8s-node
1.5 准备pause基础镜像
因为kubelet在启动容器时需要有一个基础镜像初始化网络空间等来帮助我们去启动容器,从而让我们能够启动pod;
将pause镜像放入到harbor私有仓库中,仅在10.4.7.200上操作,确保harbor和docker运行正常,提前检查
下载镜像
[root@hdss7-200 ~]# docker image pull kubernetes/pause
打标签
[root@hdss7-200 ~]# docker image tag kubernetes/pause:latest harbor.od.com/public/pause:latest
登录harbor
[root@hdss7-200 ~]# docker login -u admin harbor.od.com
推送pause镜像到harbor私有仓库
[root@hdss7-200 ~]# docker image push harbor.od.com/public/pause:latest
1.6 创建kubelet启动脚本
在node节点创建启动脚本,并启动kubelet,在10.4.7.21/22上操作,以21为例
22上修改--hostname-override项
[root@hdss7-21 ~]# vim /opt/kubernetes/server/bin/kubelet-startup.sh
#!/bin/sh
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit
/opt/kubernetes/server/bin/kubelet \
--anonymous-auth=false \
--cgroup-driver systemd \
--cluster-dns 192.168.0.2 \
--cluster-domain cluster.local \
--runtime-cgroups=/systemd/system.slice \
--kubelet-cgroups=/systemd/system.slice \
--fail-swap-on="false" \
--client-ca-file ./certs/ca.pem \
--tls-cert-file ./certs/kubelet.pem \
--tls-private-key-file ./certs/kubelet-key.pem \
--hostname-override hdss7-21.host.com \
--image-gc-high-threshold 20 \
--image-gc-low-threshold 10 \
--kubeconfig ../../conf/kubelet.kubeconfig \
--log-dir /data/logs/kubernetes/kube-kubelet \
--pod-infra-container-image harbor.od.com/public/pause:latest \
--root-dir /data/kubelet
1.7 添加权限,创建目录
[root@hdss7-21 ~]# chmod +x /opt/kubernetes/server/bin/kubelet-startup.sh
[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet
1.8 配置supervisor配置
[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-kubelet.ini
[program:kube-kubelet-7-21]
command=/opt/kubernetes/server/bin/kubelet-startup.sh
numprocs=1
directory=/opt/kubernetes/server/bin
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
1.9 启动服务并检查
[root@hdss7-21 ~]# supervisorctl update
kube-kubelet-7-21: added process group
[root@hdss7-21 ~]# supervisorctl status
etcd-server-7-21 RUNNING pid 1172, uptime 1:06:46
kube-apiserver-7-21 RUNNING pid 1183, uptime 1:06:46
kube-controller-manager-7-21 RUNNING pid 1167, uptime 1:06:46
kube-kubelet-7-21 RUNNING pid 2280, uptime 0:01:44
kube-scheduler-7-21 RUNNING pid 1169, uptime 1:06:46
[root@hdss7-22 ~]# tail -100f /data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
I0713 21:44:08.453953 2265 kubelet_node_status.go:75] Successfully registered node hdss7-21.host.com
I0713 21:44:08.509328 2265 cpu_manager.go:155] [cpumanager] starting with none policy
I0713 21:44:08.509382 2265 cpu_manager.go:156] [cpumanager] reconciling every 10s
I0713 21:44:08.509441 2265 policy_none.go:42] [cpumanager] none policy: Start
W0713 21:44:08.644794 2265 manager.go:540] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0713 21:44:08.878478 2265 reconciler.go:154] Reconciler: start to sync state
出现如上表示正常启动
查看node节点是否加入到集群中
[root@hdss7-21 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready <none> 13s v1.14.10
hdss7-22.host.com NotReady <none> 0s v1.14.10
别急,需要加载一会
[root@hdss7-21 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready <none> 55m v1.14.10
hdss7-22.host.com Ready <none> 54m v1.14.10
1.10 修改节点角色
[root@hdss7-21 ~]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/node=
node/hdss7-21.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/master=
node/hdss7-21.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-22.host.com node-role.kubernetes.io/master=
node/hdss7-22.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-22.host.com node-role.kubernetes.io/node=
node/hdss7-22.host.com labeled
[root@hdss7-21 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready master,node 57m v1.14.10
hdss7-22.host.com Ready master,node 57m v1.14.10
1.11 安装部署其他节点
在10.4.7.22上同样操作
1.12 报错排查
在10.4.7.21(10.4.7.10)master节点上查看node,发现无任何资源可访问,如下
[root@hdss7-21 ~]# kubectl get node
No resources found.
查看kubectl日志
[root@hdss7-21 ~]# tail -100f /data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
第一种如下:
failed to ensure node lease exists connect: no route to host
原因:这是根本就没有10.4.7.10这个ip,或者无法连接到此ip,ping一下,检查是否可以连接,添加此虚IP重新,重新执行本章1.2步骤即可
第二种如下:
E0611 20:41:55.908234 1414 kubelet.go:2246] node "hdss7-22.host.com" not found
E0611 20:41:56.008667 1414 kubelet.go:2246] node "hdss7-22.host.com" not found
这个报错可以忽略
第三种如下:
E0611 20:41:55.838167 1414 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.RuntimeClass: Get https://10.4.7.10:7443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 10.4.7.10:7443: connect: connection refused
这就说明虽然找到了这个服务器,但是拒绝连接,测试一下端口的连通性,发现被拒绝,无此端口
[root@hdss7-21 ~]# telnet 10.4.7.10 7443
Trying 10.4.7.10...
telnet: connect to address 10.4.7.10: Connection refused
检查虚IP是否正确配置,检查1.2步骤是否正确执行,在本地telnet一下,是否正确启动7443端口,正常启动后重启一下kube-kubelet-7-21,kube-kubelet-7-22服务即可
正确的日志如下:
I0611 21:06:18.917499 9153 kubelet_node_status.go:72] Attempting to register node hdss7-22.host.com
I0611 21:06:18.947122 9153 kubelet_node_status.go:75] Successfully registered node hdss7-22.host.com
I0611 21:06:18.989477 9153 kubelet.go:1825] skipping pod synchronization - container runtime status check may not have completed yet.
I0611 21:06:19.015529 9153 cpu_manager.go:155] [cpumanager] starting with none policy
I0611 21:06:19.015565 9153 cpu_manager.go:156] [cpumanager] reconciling every 10s
二、部署kube-proxy
Kube-proxy实际上是维护了pod网络、节点(node)网络与cluster(service)网络三者之间的关系
2.1 集群规划
注意:部署以10.4.7.21为例,22节点部署类似
主机名 角色 IP
hdss7-21 kube-proxy 10.4.7.21
hdss7-22 kube-proxy 10.4.7.22
2.2 签发kube-proxy证书
在10.4.7.200证书签发服务器上操作
创建签发证书的请求文件
[root@hdss7-200 ~]# cd /opt/certs/
[root@hdss7-200 certs]# vim kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client kube-proxy-csr.json |cfssl-json -bare kube-proxy-client
certs]# ll kube-proxy*
-rw-r--r-- 1 root root 1005 6月 12 20:49 kube-proxy-client.csr
-rw------- 1 root root 1679 6月 12 20:49 kube-proxy-client-key.pem
-rw-r--r-- 1 root root 1375 6月 12 20:49 kube-proxy-client.pem
-rw-r--r-- 1 root root 267 6月 12 20:49 kube-proxy-csr.json
分发证书
[root@hdss7-200 certs]# scp kube-proxy-client-key.pem kube-proxy-client.pem hdss7-21:/opt/kubernetes/server/bin/certs/
[root@hdss7-200 certs]# scp kube-proxy-client-key.pem kube-proxy-client.pem hdss7-22:/opt/kubernetes/server/bin/certs/
2.3 创建kube-proxy配置
在所有node节点部署,涉及服务器10.4.7.21,22,步骤与创建kubelet相同
[root@hdss7-21 ~]# kubectl config set-cluster myk8s \
--certificate-authority=/opt/kubernetes/server/bin/certs/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:7443 \
--kubeconfig=/opt/kubernetes/conf/kube-proxy.kubeconfig
Cluster "myk8s" set.
[root@hdss7-21 ~]# kubectl config set-credentials kube-proxy \
--client-certificate=/opt/kubernetes/server/bin/certs/kube-proxy-client.pem \
--client-key=/opt/kubernetes/server/bin/certs/kube-proxy-client-key.pem \
--embed-certs=true \
--kubeconfig=/opt/kubernetes/conf/kube-proxy.kubeconfig
User "kube-proxy" set.
[root@hdss7-21 ~]# kubectl config set-context myk8s-context \
--cluster=myk8s \
--user=kube-proxy \
--kubeconfig=/opt/kubernetes/conf/kube-proxy.kubeconfig
Context "myk8s-context" created.
[root@hdss7-21 ~]# kubectl config use-context myk8s-context --kubeconfig=/opt/kubernetes/conf/kube-proxy.kubeconfig
Switched to context "myk8s-context".
传送配置,22上就不需要操作了
[root@hdss7-21 ~]# scp /opt/kubernetes/conf/kube-proxy.kubeconfig hdss7-22:/opt/kubernetes/conf/
2.4 加载IPvs模块
kube-proxy共有三种流量调度模式,分别是userspace,iptables和ipvs,目前ipvs是最匹配的
在21和22上操作
查看现有的ipvs模块
[root@hdss7-21 ~]# lsmod | grep ip_vs
加载ipvs模块
[root@hdss7-21 ~]# for i in $(ls /usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs|grep -o "^[^.]*");do echo $i; /sbin/modinfo -F filename $i >/dev/null 2>&1 && /sbin/modprobe $i;done
ip_vs_dh
ip_vs_ftp
ip_vs
ip_vs_lblc
ip_vs_lblcr
ip_vs_lc
ip_vs_nq
ip_vs_pe_sip
ip_vs_rr
ip_vs_sed
ip_vs_sh
ip_vs_wlc
ip_vs_wrr
查看
[root@hdss7-21 ~]# lsmod | grep ip_vs
ip_vs_wrr 12697 0
ip_vs_wlc 12519 0
ip_vs_sh 12688 0
ip_vs_sed 12519 0
ip_vs_rr 12600 0
ip_vs_pe_sip 12740 0
nf_conntrack_sip 33780 1 ip_vs_pe_sip
ip_vs_nq 12516 0
ip_vs_lc 12516 0
ip_vs_lblcr 12922 0
ip_vs_lblc 12819 0
ip_vs_ftp 13079 0
ip_vs_dh 12688 0
nf_nat 26583 3 ip_vs_ftp,nf_nat_ipv4,nf_nat_masquerade_ipv4
ip_vs 145497 24 ip_vs_dh,ip_vs_lc,ip_vs_nq,ip_vs_rr,ip_vs_sh,ip_vs_ftp,ip_vs_sed,ip_vs_wlc,ip_vs_wrr,ip_vs_pe_sip,ip_vs_lblcr,ip_vs_lblc
nf_conntrack 139224 8 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_sip,nf_conntrack_ipv4
libcrc32c 12644 3 ip_vs,nf_nat,nf_conntrack
2.5 创建kube-proxy启动脚本
此操作需要在21和22上进行操作
--hostname-override需要修改为本机的主机名
[root@hdss7-21 ~]# vim /opt/kubernetes/server/bin/kube-proxy-startup.sh
#!/bin/sh
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit
/opt/kubernetes/server/bin/kube-proxy \
--cluster-cidr 172.7.0.0/16 \
--hostname-override hdss7-21.host.com \
--proxy-mode=ipvs \
--ipvs-scheduler=nq \
--kubeconfig ../../conf/kube-proxy.kubeconfig
2.6 设置权限,创建目录
[root@hdss7-21 ~]# chmod +x /opt/kubernetes/server/bin/kube-proxy-startup.sh
[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-proxy
2.7 创建supervisor配置
[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-proxy.ini
[program:kube-proxy-7-21]
command=/opt/kubernetes/server/bin/kube-proxy-startup.sh
numprocs=1
directory=/opt/kubernetes/server/bin
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-proxy/proxy.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
2.8 启动服务并检查
[root@hdss7-21 ~]# supervisorctl update
kube-proxy-7-21: added process group
查看代理状态
[root@hdss7-21 ~]# supervisorctl status
etcd-server-7-21 RUNNING pid 1319, uptime 0:48:02
kube-apiserver-7-21 RUNNING pid 1328, uptime 0:48:02
kube-controller-manager-7-21 RUNNING pid 1305, uptime 0:48:02
kube-kubelet-7-21 RUNNING pid 1308, uptime 0:48:02
kube-proxy-7-21 RUNNING pid 11663, uptime 0:02:10
kube-scheduler-7-21 RUNNING pid 1316, uptime 0:48:02
[root@hdss7-21 ~]# yum -y install ipvsadm
查看ipvs代理信息,可以看到192.168.0.1:443端口代理了10.4.7.21/22:6443端口
[root@hdss7-21 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.1:443 nq
-> 10.4.7.21:6443 Masq 1 0 0
-> 10.4.7.22:6443 Masq 1 0 0
查看service的信息
[root@hdss7-21 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 192.168.0.1 <none> 443/TCP 3d23h
2.9 安装部署集群其他节点
部署集群其他节点