k8s kube-prometheus controller-manager、scheduler 报错

笔记，解决kube-prometheus、controller-manager、scheduler红色问题
- 解决方案
- 测试
思路

笔记，解决kube-prometheus、controller-manager、scheduler红色问题

如果是kubeadm安装的话这里是正常的。

[root@master01 ~]# netstat -anptu | grep contro
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      1384/kube-controlle 
tcp6       0      0 :::10252                :::*                    LISTEN      1384/kube-controlle 

#因为我们是二进制安装的，这里监听的是127.0.0.1他没办法访问



#kube-rpom是怎么去找kube-controller-manager的呢
kubectl get servicemonitor -n monitoring 
NAME                      AGE
alertmanager-main         47h
blackbox-exporter         47h
coredns                   47h
grafana                   47h
kube-apiserver            47h
kube-controller-manager   47h
kube-scheduler            47h
kube-state-metrics        47h
kubelet                   47h
node-exporter             47h
prometheus-adapter        47h
prometheus-k8s            47h
prometheus-operator       47h

---
kubectl get servicemonitor -n monitoring -o yaml > 3
vi 3
391       port: https-metrics
392       scheme: https
393       tlsConfig:
394         insecureSkipVerify: true
395     jobLabel: app.kubernetes.io/name
396     namespaceSelector:
397       matchNames:
398       - kube-system
399     selector:
400       matchLabels:
401         app.kubernetes.io/name: kube-controller-manager

---


[root@master01 ~]# kubectl get svc -n monitoring -l app.kubernetes.io/name=kube-controller-manager
No resources found in monitoring namespace.

#可以看到他是通过标签app.kubernetes.io/name=kube-controller-manager来匹配controller-manager的当我们查看的时候，并没有符合这个标签的svc所以prometheus找不到controller-manager地址。


#我们需要做的是：
1.吧监听地址改成0.0.0.0
2.吧service创建上

解决方案

#1.修改监听地址
vi /etc/kubernetes/cfg/kube-controller-manager.conf
--bind-address=0.0.0.0 \
#从127.0.0.0 改成0.0.0.0

systemctl daemon-reload 
systemctl restart kube-controller-manager.service 

[root@master01 ~]# netstat -anptu |grep contro
tcp        0      0 172.16.1.33:38460       172.16.1.33:8443        ESTABLISHED 13932/kube-controll 
tcp6       0      0 :::10257                :::*                    LISTEN      13932/kube-controll 
tcp6       0      0 :::10252                :::*                    LISTEN      13932/kube-controll 
---

#监听地址修改完毕。也成功起来了
#有同学会想开0.0.0.0会不安全？
首先：
    1.k8s集群是在内网，
    2.k8s通信是需要双向证书的
所以不会像我们想的那样特别不安全，不可能随便泄漏证书，k8s集群也不可能随便暴露公网

---
2.创建service
首先创建一个endpoint，指向宿主机ip+10252，然后在创建一个同名的service，和上面查出来的标签

#关键信息：1.标签；2.端口；3端口名字；4.ip
1.标签：标签必须和上面我们查出来的保持一致，
2.端口：端口不能错
3.端口名字：端口名字是上面我们查出来的391行
4.ip：这里我写的是高可用集群ip
---
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    app.kubernetes.io/name: kube-controller-manager
  name: kube-controller-manager-monitoring
  namespace: kube-system
subsets:
- addresses:

- ip: 172.16.1.11
- ip: 172.16.1.12
- ip: 172.16.1.13

  ports:
  - name: https-metrics
    port: 10252
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: kube-controller-manager
  name: kube-controller-manager-monitoring
  namespace: kube-system
spec:
  ports:
  - name: https-metrics
    port: 10252
    protocol: TCP
    targetPort: 10252
  sessionAffinity: None
  type: ClusterIP
  
  
#注：kube-prometheus使用的是https、而暴露使用的是http，将https改成http
kubectl edit servicemonitor -n monitoring kube-controller-manager 
     60     scheme: http

测试

[root@master01 ~]# curl 172.16.1.33:10252
404 page not found
[root@master01 ~]# curl 10.96.170.38:10252
404 page not found

#访问service和访问高可用ip结果一致，service创建成功

scheduler

kubectl edit servicemonitor -n monitoring kube-scheduler
#scheme: https 改为scheme: http

#修改scheduler配置文件
--bind-address=127.0.0.1  #改为0.0.0.0


---
[root@master01 ~]# cat 3.yaml 
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: kube-scheduler
  name: scheduler
  namespace: kube-system
spec:
  ports:
  - name: https-metrics
    port: 10251
    protocol: TCP
    targetPort: 10251
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    app.kubernetes.io/name: kube-scheduler
  name: scheduler
  namespace: kube-system
subsets:
- addresses:
  - ip: 172.16.1.11
  - ip: 172.16.1.12
  - ip: 172.16.1.13
  ports:
  - name: https-metrics
    port: 10251
    protocol: TCP

至此 controller-manager、scheduler 已经起来

回到顶部

思路

controllers-manager和scheduler 监听修改127.0.0.1为0.0.0.0

#controllers-manager
--bind-address=127.0.0.1 \ #改为0.0.0.0

#scheduler
--bind-address=127.0.0.1  #改为0.0.0.0

重启服务。

然后手动curl http://172.16.1.33:端口/metrics 寻找暴露的端口
http、https都试试，本次实验，暴露的是http、官方写的是https；端口通过"netstat -anptu | grep contro"," netstat -anptu | grep schedu"过滤查看

#找到暴露的地址后，就添加service、Endpoints
#将刚刚找到的暴露信息地址、添加到server

注意添加service：1.标签；2.端口；3端口名字；4.ip