k8s里node 宕机后如何提高pod迁移速度

大概的配置参数：

 

 

node故障后，pod会迁移到 正常的node上， 迁移时间大概8分钟左右， 如果是微服务，注册到nacos，服务不受影响，但是对于其他的服务，
请求中会有大量失败。

 

需要几个流程：

kubelet 自身会定期更新状态到 apiserver，通过kubelet的参数 node-status-update-frequency 配置上报频率，默认 10s 上报一次。

kube-controller-manager定期去探测kubelet的运行状态，默认5s问kubelet，使用--node-monitor-grace-period参数

kube-controller-manager询问kubelet，如果5分钟没有回复，认为kubelet有问题，node-monitor-grace-period，是kubelet参数node-status-update-frequency的整数倍

如果有问题，故障node节点被设置成污点

• node.kubernetes.io/unreachable:NoExecute
• node.kubernetes.io/unreachable:NoSchedule

节点被设置了污点，pod节点上还是running，apiserver的参数--default-unreachable-toleration-seconds是驱逐掉时间。

 

vim /var/lib/kubelet/config.yaml

memorySwap: {}
#修改kubelet向apiserver 汇报时间，由10秒变成4秒
node-status-update-frequency: 4s
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

 

 vim /etc/kubernetes/manifests/kube-apiserver.yaml

spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.148.131
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    #当node节点为notready或者 unreachable时，5秒钟驱逐掉，默认300秒
    - --default-unreachable-toleration-seconds=5
    - --default-not-ready-toleration-seconds=5

 

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

spec:
  containers:
  - command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --bind-address=127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-cidr=10.244.0.0/16
    - --cluster-name=kubernetes
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --use-service-account-credentials=true
    - --v=5
    # kube-controller-manager 修改成2秒主动探测，默认值5秒
    - --node-monitor-period=2s
    # 将一个node标记为不健康之前允许其无响应的上线
    - --node-monitor-grace-period=12s

 

验证脚本，中间关闭node， 查看pod漂移时间

while true; do echo `date +%F-%T`; kubectl get node ;echo "####################################################################"; kubectl describe nodes node2 | grep -A5 Taint;echo "";echo "##############################################################"; kubectl get pod -o wide; sleep 5; done

 

注意：以前的pod需要重建。