Prometheus Operator自定义监控项

1、查看etcd指标监听端口

cat /etc/kubernetes/manifests/etcd.yaml
	...
    - --listen-metrics-urls=http://0.0.0.0:2381

2、创建secret资源，其中包含以https访问etcd集群的证书和私钥

kubectl create secret generic etcd-certs \
--from-file=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
--from-file=/etc/kubernetes/pki/etcd/healthcheck-client.key \
--from-file=/etc/kubernetes/pki/etcd/ca.crt -n monitoring

3、修改prometheus的资源对象，将上步生成的secret挂载至prometheus中

kubectl edit prometheus k8s -n monitoring
spec:
  secrets:
  - etcd-certs
  ...

4、查看证书挂载情况

kubectl exec -it prometheus-k8s-0 -n monitoring -- ls /etc/prometheus/secrets/etcd-certs/

5、创建etcd的endpoints对象并关联至headless service，etcd-svc-endpoints.yaml

apiVersion: v1
kind: Service
metadata:
  name: etcd-k8s
  namespace: kube-system
  labels:
    k8s-app: etcd
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: port
    port: 2379
    protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
  name: etcd-k8s
  namespace: kube-system
  labels:
    k8s-app: etcd
subsets:
- addresses:
  - ip: 192.168.200.11
  - ip: 192.168.200.12
  - ip: 192.168.200.13
    nodeName: etcd-master
  ports:
  - name: port
    port: 2379
    protocol: TCP

6、创建etcd的servicemonitor资源对象并应用到集群，etcd-servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: etcd-k8s
  namespace: monitoring
  labels:
    k8s-app: etcd-k8s
spec:
  jobLabel: k8s-app
  endpoints:
  - port: port
    interval: 30s
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
      certFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.crt
      keyFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.key
      insecureSkipVerify: true
  selector:
    matchLabels:
      k8s-app: etcd
  namespaceSelector:
    matchNames:
    - kube-system