Prometheus对接VictoriaMetrics

转载自博客：https://blog.csdn.net/alex_yangchuansheng/article/details/107852927

 

4. 实践
确定好了方案之后，下面来进行动手实践。

部署 VictoriaMetrics
首先部署一个单实例的 VictoriaMetrics，完整的 yaml 如下：

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: victoriametrics
  namespace: kube-system
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: victoriametrics
  name: victoriametrics
  namespace: kube-system
spec:
  serviceName: pvictoriametrics
  selector:
    matchLabels:
      app: victoriametrics
  replicas: 1
  template:
    metadata:
      labels:
        app: victoriametrics
    spec:
      nodeSelector:
        blog: "true"
      containers:    
      - args:
        - --storageDataPath=/storage
        - --httpListenAddr=:8428
        - --retentionPeriod=1
        image: victoriametrics/victoria-metrics
        imagePullPolicy: IfNotPresent
        name: victoriametrics
        ports:
        - containerPort: 8428
          protocol: TCP
        readinessProbe:
          httpGet:
            path: /health
            port: 8428
          initialDelaySeconds: 30
          timeoutSeconds: 30
        livenessProbe:
          httpGet:
            path: /health
            port: 8428
          initialDelaySeconds: 120
          timeoutSeconds: 30
        resources:
          limits:
            cpu: 2000m
            memory: 2000Mi
          requests:
            cpu: 2000m
            memory: 2000Mi
        volumeMounts:
        - mountPath: /storage
          name: storage-volume
      restartPolicy: Always
      priorityClassName: system-cluster-critical
      volumes:
      - name: storage-volume
        persistentVolumeClaim:
          claimName: victoriametrics
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: victoriametrics
  name: victoriametrics
  namespace: kube-system
spec:
  ports:
  - name: http
    port: 8428
    protocol: TCP
    targetPort: 8428
  selector:
    app: victoriametrics
  type: ClusterIP

有几个启动参数需要注意：

storageDataPath : 数据目录的路径。VictoriaMetrics 将所有数据存储在此目录中。

retentionPeriod : 数据的保留期限（以月为单位）。旧数据将自动删除。默认期限为1个月。

httpListenAddr : 用于监听 HTTP 请求的 TCP 地址。默认情况下，它在所有网络接口上监听端口 8428。

给 namespace 打标签
为了限定抓取 target 的 namespace，我们需要给 namespace 打上标签，使每个 Prometheus 实例只抓取特定 namespace 的指标。根据上文的方案，需要给 kube-system 打上标签 monitoring-role=system：

$ kubectl label ns kube-system monitoring-role=system
给其他的 namespace 打上标签 monitoring-role=others。例如：

$ kubectl label ns monitoring monitoring-role=others
$ kubectl label ns default monitoring-role=other

拆分 Prometheus
下一步是拆分 Prometheus 实例，根据上面的方案需要拆分成两个实例，一个用来监控 kube-system namespace，另一个用来监控其他 namespace：

# prometheus-prometheus-system.yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: system 
  name: system
  namespace: monitoring
spec:
  remoteWrite:
    - url: http://victoriametrics.kube-system.svc.cluster.local:8428/api/v1/write
      queueConfig:
        maxSamplesPerSend: 10000
  retention: 2h 
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  image: quay.io/prometheus/prometheus:v2.17.2
  nodeSelector:
    beta.kubernetes.io/os: linux
  podMonitorNamespaceSelector:
    matchLabels:
      monitoring-role: system 
  podMonitorSelector: {}
  replicas: 1 
  resources:
    requests:
      memory: 400Mi
    limits:
      memory: 2Gi
  ruleSelector:
    matchLabels:
      prometheus: system 
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: 
    matchLabels:
      monitoring-role: system 
  serviceMonitorSelector: {}
  version: v2.17.2
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: others
  name: others
  namespace: monitoring
spec:
  remoteWrite:
    - url: http://victoriametrics.kube-system.svc.cluster.local:8428/api/v1/write
      queueConfig:
        maxSamplesPerSend: 10000
  retention: 2h
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  image: quay.io/prometheus/prometheus:v2.17.2
  nodeSelector:
    beta.kubernetes.io/os: linux
  podMonitorNamespaceSelector: 
    matchLabels:
      monitoring-role: others 
  podMonitorSelector: {}
  replicas: 1
  resources:
    requests:
      memory: 400Mi
    limits:
      memory: 2Gi
  ruleSelector:
    matchLabels:
      prometheus: others 
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector:
    matchLabels:
      monitoring-role: others 
  serviceMonitorSelector: {}
  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional.yaml
  version: v2.17.2

需要注意的配置：

通过 remoteWrite 指定 remote write 写入的远程存储。

通过 ruleSelector 指定 PrometheusRule。

限制内存使用上限为 2Gi，可根据实际情况自行调整。

通过 retention 指定数据在本地磁盘的保存时间为 2 小时。因为指定了远程存储，本地不需要保存那么长时间，尽量缩短。

Prometheus 的自定义配置可以通过 additionalScrapeConfigs 在 others 实例中指定，当然你也可以继续拆分，放到其他实例中。

修改 Grafana 数据源
Prometheus 拆分成功之后，最后还要修改 Grafana 的数据源为 VictoriaMetrics 的地址，这样就可以在 Grafana 中查看全局视图，也能聚合查询。

打开 Grafana 的设置页面，将数据源修改为 http://victoriametrics.kube-system.svc.cluster.local:8428：

 

 

脚注
[1]

VictoriaMetrics: https://github.com/VictoriaMetrics/VictoriaMetrics

[2]
VictoriaMetrics: https://github.com/VictoriaMetrics/VictoriaMetrics

[3]
prometheus-rules-system.yaml: https://gist.github.com/yangchuansheng/4310ae9f41513899dc5f0176cdf804b1

[4]
prometheus-rules-others.yaml: https://gist.github.com/yangchuansheng/102595fc50436cf4a2ce18744467718c