Prometheus Operator配置Kubernetes服务自动发现

1、编写prometheus-additional配置，prometheus-additional.yaml

- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name

2、创建secret资源

kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring

3、增加additionalScrapeConfigs配置段

kubectl edit prometheus k8s -n monitoring
spec:
  additionalScrapeConfigs:
    name: additional-configs
    key: prometheus-additional.yaml
    ...

4、修改prometheus-server的service account权限

kubectl edit clusterrole prometheus-k8s -n monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

5、验证。在需要监控的service上增加注解

kubectl edit service grafana -n monitoring
spec:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "3000"
    ...