kube-apiserver起不来

1.三台二进制master的节点,总是有一台宕机

排查

[root@k8s-master01 kube-apiserver]# systemctl cat --no-pager kube-apiserver 
# /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=/opt/kubernetes/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver \
  --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \
  --anonymous-auth=false \
  --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --requestheader-allowed-names=aggregator \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --proxy-client-cert-file=/opt/kubernetes/ssl/metrics-server.pem \
  --proxy-client-key-file=/opt/kubernetes/ssl/metrics-server-key.pem \
  --enable-aggregator-routing=true \
  --experimental-encryption-provider-config=/opt/kubernetes/kube-apiserver/encryption-config.yaml \
  --advertise-address=10.88.33.218 \
  --insecure-bind-address=0.0.0.0  \
  --insecure-port=0 \
  --secure-port=6443 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth=true \
  --service-cluster-ip-range=10.0.0.0/16 \
  --service-node-port-range=30000-50000 \
  --tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \
  --tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \
  --client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --kubelet-certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/opt/kubernetes/ssl/kubernetes.pem \
  --kubelet-client-key=/opt/kubernetes/ssl/kubernetes-key.pem \
  --kubelet-https=true \
  --service-account-key-file=/opt/kubernetes/ssl/ca.pem \
  --etcd-cafile=/opt/kubernetes/ssl/ca.pem \
  --etcd-certfile=/opt/kubernetes/ssl/etcd.pem \
  --etcd-keyfile=/opt/kubernetes/ssl/etcd-key.pem \
  --etcd-servers=https://10.88.33.218:2379,https://10.88.33.219:2379,https://10.88.33.220:2379 \
  --enable-swagger-ui=true \
  --allow-privileged=true \
  --max-mutating-requests-inflight=2000 \
  --max-requests-inflight=4000 \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/opt/kubernetes/kube-apiserver/kube-apiserver.log \
  --audit-policy-file=/opt/kubernetes/cfg/audit-policy.yaml \
  --event-ttl=168h \
  --logtostderr=true \
  --v=4
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

通过前台运行的方式debug出问题,执行ExecStart部分,日志等级调为v=4

/opt/kubernetes/bin/kube-apiserver \
  --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \
  --anonymous-auth=false \
  --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --requestheader-allowed-names=aggregator \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --proxy-client-cert-file=/opt/kubernetes/ssl/metrics-server.pem \
  --proxy-client-key-file=/opt/kubernetes/ssl/metrics-server-key.pem \
  --enable-aggregator-routing=true \
  --experimental-encryption-provider-config=/opt/kubernetes/kube-apiserver/encryption-config.yaml \
  --advertise-address=10.88.33.218 \
  --insecure-bind-address=0.0.0.0  \
  --insecure-port=0 \
  --secure-port=6443 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth=true \
  --service-cluster-ip-range=10.0.0.0/16 \
  --service-node-port-range=30000-50000 \
  --tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \
  --tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \
  --client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --kubelet-certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/opt/kubernetes/ssl/kubernetes.pem \
  --kubelet-client-key=/opt/kubernetes/ssl/kubernetes-key.pem \
  --kubelet-https=true \
  --service-account-key-file=/opt/kubernetes/ssl/ca.pem \
  --etcd-cafile=/opt/kubernetes/ssl/ca.pem \
  --etcd-certfile=/opt/kubernetes/ssl/etcd.pem \
  --etcd-keyfile=/opt/kubernetes/ssl/etcd-key.pem \
  --etcd-servers=https://10.88.33.218:2379,https://10.88.33.219:2379,https://10.88.33.220:2379 \
  --enable-swagger-ui=true \
  --allow-privileged=true \
  --max-mutating-requests-inflight=2000 \
  --max-requests-inflight=4000 \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/opt/kubernetes/kube-apiserver/kube-apiserver.log \
  --audit-policy-file=/opt/kubernetes/cfg/audit-policy.yaml \
  --event-ttl=168h \
  --logtostderr=true \
  --v=4

执行完我们发现,hosts解析失败,于是我查看了本地的/etc/hosts发现没解析127.0.0.1

解决方案,添加解析

[root@k8s-master01 kube-apiserver]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
...

感觉张馆长的解答以及博客提供的思路

https://zhangguanzhang.github.io/