I have an HA k8s cluster which has 3 control-plane nodes. When 2 of the 3 control-plane nodes are shut down, the cluster stops working.
When shut down zubt3:
scnzzh@zubt2:~$ kubectl get node
NAME STATUS ROLES AGE VERSION
zubt1 Ready control-plane,master 61m v1.20.2
zubt2 Ready control-plane,master 31m v1.20.2
zubt3 NotReady control-plane,master 13m v1.20.2
When also shut down zubt2:
scnzzh@zubt1:~$ kubectl get node
Error from server: etcdserver: request timed out
scnzzh@zubt1:~$ kubectl get node
Error from server: rpc error: code = Unknown desc = OK: HTTP status code 200; transport: missing content-type field
scnzzh@zubt1:~$ kubectl get node
Error from server (InternalError): an error on the server ("") has prevented the request from succeeding (get nodes)
scnzzh@zubt1:~$ kubectl get node
Error from server (InternalError): an error on the server ("") has prevented the request from succeeding (get nodes)
When restart up zubt3:
scnzzh@zubt1:~$ kubectl get node
NAME STATUS ROLES AGE VERSION
zubt1 Ready control-plane,master 68m v1.20.2
zubt2 NotReady control-plane,master 38m v1.20.2
zubt3 Ready control-plane,master 20m v1.20.2
When shut down zubt1:
scnzzh@zubt3:~$ kubectl get node
Error from server: etcdserver: request timed out
scnzzh@zubt3:~$ kubectl get node
Error from server: etcdserver: request timed out
scnzzh@zubt3:~$ kubectl get node
Unable to connect to the server: net/http: TLS handshake timeout
scnzzh@zubt3:~$ kubectl get node
Error from server (InternalError): an error on the server ("") has prevented the request from succeeding (get nodes)
When restart up zubt2:
scnzzh@zubt3:~$ kubectl get node
NAME STATUS ROLES AGE VERSION
zubt1 NotReady control-plane,master 75m v1.20.2
zubt2 Ready control-plane,master 45m v1.20.2
zubt3 Ready control-plane,master 26m v1.20.2
When restart up zubt1:
scnzzh@zubt1:~$ kubectl get node
NAME STATUS ROLES AGE VERSION
zubt1 Ready control-plane,master 88m v1.20.2
zubt2 Ready control-plane,master 58m v1.20.2
zubt3 Ready control-plane,master 40m v1.20.2
