K8S二进制问题云集

一、故障描述

Error from server: Get "https://k8s-node01:10250/containerLogs/kube-system/calico-node-98cv6/calico-node": x509: certificate signed by unknown authority
# 解决方案
kubelet 启动参数未设置
rotateCertificates: true
serverTLSBootstrap: true

 

二、故障描述

[ERROR][10] startup/startup.go 154: failed to query kubeadm's config map error=Get "https://172.16.0.1:443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=2s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021-09-16 03:48:26.374 [WARNING][10] startup/utils.go 48: Terminating
# 解决方案
kind: DaemonSet
metadata:
name: calico-node
containers:
env:
# 增加如下配置
# kubernetes service host ip address
- name: KUBERNETES_SERVICE_HOST
value: "192.168.0.250"
- name: KUBERNETES_SERVICE_PORT
value: "6443"
- name: KUBERNETES_SERVICE_PORT_HTTPS
value: "6443"

三、故障描述

Error from server (BadRequest): container "calico-node" in pod "calico-node-gn4p7" is waiting to start: PodInitializing
# 故障原因排查
kubectl describe -n kube-system po calico-node-gn4p7
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned kube-system/calico-node-gn4p7 to 192.168.0.246
Warning FailedCreatePodSandBox 24s (x15 over 10m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "k8s.gcr.io/pause:3.2": failed to pull image "k8s.gcr.io/pause:3.2": failed to pull and unpack image "k8s.gcr.io/pause:3.2": failed to resolve reference "k8s.gcr.io/pause:3.2": failed to do request: Head "https://k8s.gcr.io/v2/pause/manifests/3.2": dial tcp 108.177.97.82:443: i/o timeout

# 解决方案
ctr -n k8s.io i pull docker.io/juestnow/pause-amd64:3.2
ctr -n k8s.io i tag docker.io/juestnow/pause-amd64:3.2 k8s.gcr.io/pause:3.2

四、故障描述

Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://172.16.0.1:443/api/v1/pods?limit=500&resourceVersion=0": dial tcp 172.16.0.1:443: i/o timeout
kind: Deployment
metadata:
name: calico-kube-controllers
containers:
env:
# 增加如下配置
# kubernetes service host ip address
- name: KUBERNETES_SERVICE_HOST
value: "192.168.0.250"
- name: KUBERNETES_SERVICE_PORT
value: "6443"
- name: KUBERNETES_SERVICE_PORT_HTTPS
value: "6443"

五、故障描述

Error while dialing dial tcp: address https://192.168.0.242:2379 : too many colons in address

# 解决方案
vim calico.yaml
etcd_endpoints: "https://192.168.0.240:2379,https://192.168.0.241:2379,https://192.168.0.242:2379"

-------不要怀疑，就是配置文件中的etcd_endpoint多了一个空格------

六、故障描述

Warning FailedCreatePodSandBox 2m6s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2ff4a45151e84e254bb11b96c02e68f3a0d610e75edc89337d262efa49ef073b": Get "https://[172.16.0.1]:443/api/v1/namespaces/default": dial tcp 172.16.0.1:443: i/o timeout

# 解决方案
vim /etc/cni/net.d/calico-kubeconfig
server: https://192.168.0.250:6443

七、 故障描述

kube-proxy问题
E1009 15:43:30.994633 32520 conntrack.go:115] failed to list mount points: wrong number of fields (expected 6, got 7)

解决方案
umount /data/k8s/kubelet /pods/f4e4118b-397e-4848-9b9c-5a3e5b54273e/volumes/kubernetes.io~secret/calico-node-token-zvwhb
umount /data/k8s/kubelet /pods/f4e4118b-397e-4848-9b9c-5a3e5b54273e/volumes/kubernetes.io~secret/etcd-certs

rm -fr /data/k8s/kubelet /pods/*

systemctl restart containerd
systemctl restart kubelet
systemctl restart kube-proxy