k8s中访问coredns连接拒绝问题解决

问题现象

1、节点访问coredns连接拒绝
2、内部pod无法正常进行解析

问题解决思路

检查 CoreDNS Pod 状态是否正常

[root@k8s-master01 ~]# kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-7b8d6fc5d7-4cgcp   1/1     Running   1          16h

检查 CoreDNS 服务是否正常

[root@k8s-master01 ~]# kubectl get svc -n kube-system kube-dns
NAME       TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   115.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   55d

[root@k8s-master01 ~]# kubectl describe svc kube-dns -n kube-system
Name:              kube-dns
Namespace:         kube-system
Labels:            app.kubernetes.io/name=coredns
                   k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=CoreDNS
Annotations:       prometheus.io/port: 9153
                   prometheus.io/scrape: true
Selector:          app.kubernetes.io/name=coredns,k8s-app=kube-dns
Type:              ClusterIP
IP Families:       <none>
IP:                115.96.0.10
IPs:               115.96.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         <none>
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         <none>
Port:              metrics  9153/TCP
TargetPort:        9153/TCP
Endpoints:         <none>
Session Affinity:  None
Events:            <none>

[root@k8s-master01 ~]# kubectl get endpoints -n kube-system kube-dns
NAME       ENDPOINTS                                           AGE
kube-dns   <none>   55d
You have new mail in /var/spool/mail/root

问题发现

如上发现"ENDPOINTS "为"<none>"，意味着没有Pod与coreDNS服务关联

继续排查

查看coredns pod的标签发现没有app.kubernetes.io/name=coredns标签，说明 CoreDNS Pod 的标签与服务的选择器不匹配。这会导致 CoreDNS 服务无法关联到相应的 Pod，从而出现 Endpoints: <none> 的情况

[root@k8s-master01 ~]# kubectl get pods -n kube-system --show-labels | grep coredns
coredns-7b8d6fc5d7-4cgcp                   1/1     Running   1          16h   k8s-app=kube-dns,pod-template-hash=7b8d6fc5d7

解决办法

修改 CoreDNS Deployment 的标签

#找到spec-template-metadata-labels 部分在labels:新增app.kubernetes.io/name: coredns，如下
kubectl edit deployment coredns -n kube-system
labels:
  app.kubernetes.io/name: coredns
  k8s-app: kube-dns

保存后，Kubernetes 重新创建符合新标签的 Pod

问题验证

如下app.kubernetes.io/name: coredns已经加上，ENDPOINTS不为空，访问coreDNS正常，测试DNS 服务工作正常

[root@k8s-master01 ~]# kubectl get pods -n kube-system --show-labels | grep coredns
coredns-7b8d6fc5d7-4cgcp                   1/1     Running   1          16h   app.kubernetes.io/name=coredns,k8s-app=kube-dns,pod-template-hash=7b8d6fc5d7

[root@k8s-master01 ~]# kubectl get endpoints -n kube-system kube-dns
NAME       ENDPOINTS                                           AGE
kube-dns   172.25.92.87:53,172.25.92.87:53,172.25.92.87:9153   55d
You have new mail in /var/spool/mail/root


[root@k8s-master01 ~]# curl http://115.96.0.10:53 -k
curl: (52) Empty reply from server

[root@k8s-master01 ~]# kubectl exec -it  cluster-test-59f585c77c-tbd9b  -- nslookup kubernetes.default.svc.cluster.local
Server:		115.96.0.10
Address:	115.96.0.10#53

Name:	kubernetes.default.svc.cluster.local
Address: 115.96.0.1