监控-Prometheus09-监控Kubernetes
- 在过去的几年中,云计算已经成为及分布式计算最火热的技术之一,其中Docker、Kubernetes、Prometheus等开源软件的发展极大地推动了云计算的发展。
- Kubernetes使用Docker进行容器管理,如果说Docker和kubernetes的搭配是云原生时代的基石,那么Prometheus为云原生插上了飞翔的翅膀。随着云原生社区的不断壮大,应用场景越来越复杂,需要一套针对云原生环境的完善并且开放的监控平台。在这样的环境下,Prometheus应运而生,天然支持Kubernetes。
- 传统方式部署步骤相对复杂,随着Operator的日益成熟,推荐使用Operator方式部署Prometheus。通过Operator方式部署Prometheus,可将更多的操作集成到Operator中,简化了操作过程,也使部署更加简单。
1、Prometheus Operator介绍
- Kubernetes的Prometheus Operator为Kubernetes服务和Prometheus实例的部署和管理提供了简单的监控定义。
- Prometheus Operator(后面都简称Operater)提供如下功能:
- 创建/销毁:在Kubernetes namespace中更加容易地启动一个Prometheues实例,一个特定应用程序或者团队可以更容易使用Operator。
- 便捷配置:可以通过Kubernetes资源配置Prometheus的基本信息,比如版本、存储、保留策略和副本集等。
1.1、Prometheus Operator架构
- Prometheus Operator架构如图11-1所示:
- 架构中的各组以k8s自定义资源的方式运行在Kubernetes集群中,它们各自有不同的作用。
- Operator:根据自定义资源(Custom ResourceDefinition,CRD)来部署和管理Prometheus Server,同时监控这些自定义资源事件的变化来做相应的处理,是整个系统的控制中心。
- Prometheus资源:声明Prometheus statefulset控制器的期望状态,Prometheus Operator确保这个statefulset控制器运行时一直与定义保持一致。
- Prometheus Server:Operator根据Prometheus资源部署Prometheus Server集群,这些自定义资源可以看作是用来管理Prometheus Server集群的StatefulSets资源。
- Alertmanager资源:定义AlertManager statefulset控制器的期望状态,Prometheus Operator确保这个statefulset控制器运行时一直与定义保持一致。
- ServiceMonitor资源:声明Prometheus监控的target列表。该资源通过Labels来选取对应的Service Endpoint,让Prometheus Server通过选取的Service来获取Metrics信息。
- Service:简单的说就是Prometheus监控的对象。
2.2、Prometheus Operator的自定义资源
- Prometheus Operater有四种自定义资源:
- Prometheus
- ServiceMonitor
- Alertmanager
- PrometheusRule
- 查看名称空间中的所有资源
]# kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n monitoring
1、Prometheus资源
- Prometheus自定义资源(CRD):声明Prometheus statefulset控制器的期望状态,Prometheus Operator确保这个statefulset控制器运行时一直与定义保持一致。包含副本数量、持久化存储以及Prometheus实例发送警告到的Alertmanagers等配置选项。
- Prometheus Operator会根据Prometheus资源在相同namespace下生成一个StatefulSet控制器。Prometheus的Pod都会挂载一个名为<prometheus-name>的Secret,里面包含了Prometheus的配置。Prometheus Operator根据包含的ServiceMonitor生成配置,并且更新含有配置的Secret。无论是对ServiceMonitors或者Prometheus的修改,都会持续不断的被按照前面的步骤更新。
- 查看Prometheus资源(使用helm部署kube-Prometheus的内容)
]# kubectl edit -n monitoring prometheus.monitoring.coreos.com/kube-promet apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: annotations: meta.helm.sh/release-name: kube-prometheus meta.helm.sh/release-namespace: monitoring labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: kube-prometheus app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: kube-prometheus helm.sh/chart: kube-prometheus-8.1.11 name: kube-prometheus-prometheus namespace: monitoring spec: affinity: #定义亲和性 podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: kube-prometheus app.kubernetes.io/name: kube-prometheus namespaces: - monitoring topologyKey: kubernetes.io/hostname weight: 1 alerting: #定义Prometheus关联的Alertmanager alertmanagers: - name: kube-prometheus-alertmanager namespace: monitoring pathPrefix: / port: http containers: #定义容器 - name: prometheus #prometheus容器 livenessProbe: #容器存活性探针 failureThreshold: 10 httpGet: path: /-/healthy port: web scheme: HTTP initialDelaySeconds: 0 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 3 readinessProbe: #容器可用性探针 failureThreshold: 10 httpGet: path: /-/ready port: web scheme: HTTP initialDelaySeconds: 0 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 3 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: false runAsNonRoot: true startupProbe: #容器启动性探针 failureThreshold: 60 httpGet: path: /-/ready port: web scheme: HTTP initialDelaySeconds: 0 periodSeconds: 15 successThreshold: 1 timeoutSeconds: 3 - name: config-reloader #config-reloader容器 livenessProbe: failureThreshold: 6 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 tcpSocket: port: reloader-web timeoutSeconds: 5 readinessProbe: failureThreshold: 6 initialDelaySeconds: 15 periodSeconds: 20 successThreshold: 1 tcpSocket: port: reloader-web timeoutSeconds: 5 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: false runAsNonRoot: true enableAdminAPI: false evaluationInterval: 30s externalUrl: http://127.0.0.1:9090/ image: docker.io/bitnami/prometheus:2.39.1-debian-11-r1 #镜像 listenLocal: false logFormat: logfmt #日志风格 logLevel: info paused: false podMetadata: labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: kube-prometheus app.kubernetes.io/name: kube-prometheus podMonitorNamespaceSelector: {} podMonitorSelector: {} portName: web probeNamespaceSelector: {} probeSelector: {} replicas: 1 #定义Proemtheus“集群”有两个副本。说是集群,其实Prometheus自身不带集群功能,这里只是起两个完全一样的Prometheus来避免单点故障 retention: 10d routePrefix: / ruleNamespaceSelector: {} ruleSelector: {} #定义Prometheus使用哪些PrometheusRule,根据标签选择。 scrapeInterval: 30s securityContext: fsGroup: 1001 runAsUser: 1001 serviceAccountName: kube-prometheus-prometheus serviceMonitorNamespaceSelector: {} #定义Prometheus在哪些namespace中选择要被监控的ServiceMonitor,根据标签选择namespace。不声明则会全部选中 serviceMonitorSelector: {} #定义Prometheus选择哪些要被监控的ServiceMonitor,根据标签选择ServiceMonitor。不声明则会全部选中 shards: 1 storage: #定义存储卷 volumeClaimTemplate: spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi storageClassName: nfs-client status: availableReplicas: 1 conditions: - lastTransitionTime: "2022-10-22T21:24:42Z" status: "True" type: Available - lastTransitionTime: "2022-10-22T21:12:51Z" status: "True" type: Reconciled paused: false replicas: 1 shardStatuses: - availableReplicas: 1 replicas: 1 shardID: "0" unavailableReplicas: 0 updatedReplicas: 1 unavailableReplicas: 0 updatedReplicas: 1
- 查看Prometheus资源生成的StatefulSet控制器(使用helm部署kube-Prometheus的内容)
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
]# kubectl edit -n monitoring statefulset.apps/prometheus-kube-prometheus-prometheus apiVersion: apps/v1 kind: StatefulSet metadata: annotations: meta.helm.sh/release-name: kube-prometheus meta.helm.sh/release-namespace: monitoring generation: 1 labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: kube-prometheus app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: kube-prometheus helm.sh/chart: kube-prometheus-8.1.11 operator.prometheus.io/name: kube-prometheus-prometheus operator.prometheus.io/shard: "0" name: prometheus-kube-prometheus-prometheus namespace: monitoring ownerReferences: - apiVersion: monitoring.coreos.com/v1 blockOwnerDeletion: true controller: true kind: Prometheus name: kube-prometheus-prometheus spec: podManagementPolicy: Parallel replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/instance: kube-prometheus-prometheus app.kubernetes.io/managed-by: prometheus-operator app.kubernetes.io/name: prometheus operator.prometheus.io/name: kube-prometheus-prometheus operator.prometheus.io/shard: "0" prometheus: kube-prometheus-prometheus serviceName: prometheus-operated template: metadata: annotations: kubectl.kubernetes.io/default-container: prometheus creationTimestamp: null labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: kube-prometheus-prometheus app.kubernetes.io/managed-by: prometheus-operator app.kubernetes.io/name: prometheus app.kubernetes.io/version: 2.39.0 operator.prometheus.io/name: kube-prometheus-prometheus operator.prometheus.io/shard: "0" prometheus: kube-prometheus-prometheus spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: kube-prometheus app.kubernetes.io/name: kube-prometheus namespaces: - monitoring topologyKey: kubernetes.io/hostname weight: 1 automountServiceAccountToken: true containers: - args: - --web.console.templates=/etc/prometheus/consoles - --web.console.libraries=/etc/prometheus/console_libraries - --storage.tsdb.retention.time=10d - --config.file=/etc/prometheus/config_out/prometheus.env.yaml - --storage.tsdb.path=/prometheus - --web.enable-lifecycle - --web.external-url=http://127.0.0.1:9090/ - --web.route-prefix=/ - --web.config.file=/etc/prometheus/web_config/web-config.yaml image: docker.io/bitnami/prometheus:2.39.1-debian-11-r1 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 10 httpGet: path: /-/healthy port: web scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 3 name: prometheus ports: - containerPort: 9090 name: web protocol: TCP readinessProbe: failureThreshold: 10 httpGet: path: /-/ready port: web scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 3 resources: {} securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: false runAsNonRoot: true startupProbe: failureThreshold: 60 httpGet: path: /-/ready port: web scheme: HTTP periodSeconds: 15 successThreshold: 1 timeoutSeconds: 3 terminationMessagePath: /dev/termination-log terminationMessagePolicy: FallbackToLogsOnError volumeMounts: - mountPath: /etc/prometheus/config_out name: config-out readOnly: true - mountPath: /etc/prometheus/certs name: tls-assets readOnly: true - mountPath: /prometheus name: prometheus-kube-prometheus-prometheus-db subPath: prometheus-db - mountPath: /etc/prometheus/rules/prometheus-kube-prometheus-prometheus-rulefiles-0 name: prometheus-kube-prometheus-prometheus-rulefiles-0 - mountPath: /etc/prometheus/web_config/web-config.yaml name: web-config readOnly: true subPath: web-config.yaml - args: - --listen-address=:8080 - --reload-url=http://127.0.0.1:9090/-/reload - --config-file=/etc/prometheus/config/prometheus.yaml.gz - --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml - --watched-dir=/etc/prometheus/rules/prometheus-kube-prometheus-prometheus-rulefiles-0 command: - /bin/prometheus-config-reloader env: - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: SHARD value: "0" image: docker.io/bitnami/prometheus-operator:0.60.1-debian-11-r0 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 6 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 tcpSocket: port: reloader-web timeoutSeconds: 5 name: config-reloader ports: - containerPort: 8080 name: reloader-web protocol: TCP readinessProbe: failureThreshold: 6 initialDelaySeconds: 15 periodSeconds: 20 successThreshold: 1 tcpSocket: port: reloader-web timeoutSeconds: 5 resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: false runAsNonRoot: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: FallbackToLogsOnError volumeMounts: - mountPath: /etc/prometheus/config name: config - mountPath: /etc/prometheus/config_out name: config-out - mountPath: /etc/prometheus/rules/prometheus-kube-prometheus-prometheus-rulefiles-0 name: prometheus-kube-prometheus-prometheus-rulefiles-0 dnsPolicy: ClusterFirst initContainers: - args: - --watch-interval=0 - --listen-address=:8080 - --config-file=/etc/prometheus/config/prometheus.yaml.gz - --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml - --watched-dir=/etc/prometheus/rules/prometheus-kube-prometheus-prometheus-rulefiles-0 command: - /bin/prometheus-config-reloader env: - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: SHARD value: "0" image: docker.io/bitnami/prometheus-operator:0.60.1-debian-11-r0 imagePullPolicy: IfNotPresent name: init-config-reloader ports: - containerPort: 8080 name: reloader-web protocol: TCP resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: FallbackToLogsOnError volumeMounts: - mountPath: /etc/prometheus/config name: config - mountPath: /etc/prometheus/config_out name: config-out - mountPath: /etc/prometheus/rules/prometheus-kube-prometheus-prometheus-rulefiles-0 name: prometheus-kube-prometheus-prometheus-rulefiles-0 restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 1001 runAsUser: 1001 serviceAccount: kube-prometheus-prometheus serviceAccountName: kube-prometheus-prometheus terminationGracePeriodSeconds: 600 volumes: - name: config secret: defaultMode: 420 secretName: prometheus-kube-prometheus-prometheus - name: tls-assets projected: defaultMode: 420 sources: - secret: name: prometheus-kube-prometheus-prometheus-tls-assets-0 - emptyDir: {} name: config-out - configMap: defaultMode: 420 name: prometheus-kube-prometheus-prometheus-rulefiles-0 name: prometheus-kube-prometheus-prometheus-rulefiles-0 - name: web-config secret: defaultMode: 420 secretName: prometheus-kube-prometheus-prometheus-web-config updateStrategy: type: RollingUpdate volumeClaimTemplates: - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: null name: prometheus-kube-prometheus-prometheus-db spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi storageClassName: nfs-client volumeMode: Filesystem status: phase: Pending status: collisionCount: 0 currentReplicas: 1 currentRevision: prometheus-kube-prometheus-prometheus-8cbb4d97f observedGeneration: 1 readyReplicas: 1 replicas: 1 updateRevision: prometheus-kube-prometheus-prometheus-8cbb4d97f updatedReplicas: 1
2、ServiceMonitor资源
- ServiceMonitor自定义资源(CRD)能够声明如何监控一组动态的服务。它会使用标签选择一组需要被监控的服务(target)。
- Prometheus Operator想要监控Kubernetes集群中的应用时,它的Endpoints必须存在。
- Endpoints对象本质是一个IP地址列表。
- Endpoints对象由Service构建。Service对象通过对象选择器发现Pod并将它们添加到Endpoints对象中。
- 一个Service可以公开一个或多个端口,通常情况下,这些端口由指向一个Pod的多个Endpoints支持。
- Prometheus Operator引入ServiceMonitor对象,通过它发现Endpoints对象,然后让Prometheus去监控这些Pods。
- ServiceMonitor.Spec的endpoints部分用于配置需要收集metrics的Endpoints的端口和其他参数。在endpoints部分指定endpoint时,请严格使用。
- 注意:endpoints(小写)是ServiceMonitor CRD中的一个字段,而Endpoints(大写)是Kubernetes资源类型。
- ServiceMonitor和发现的目标可能来自任何namespace。这对于跨namespace的监控十分重要,比如monitoring。
- 使用Prometheus.Spec下ServiceMonitorNamespaceSelector,通过各自Prometheus server限制ServiceMonitors作用namespece。
- 使用ServiceMonitor.Spec下的namespaceSelector可以现在允许发现Endpoints对象的命名空间。要发现所有命名空间下的目标,namespaceSelector必须为空。
- 查看ServiceMonitor资源(使用helm部署kube-Prometheus的内容)
]# kubectl edit -n monitoring servicemonitor.monitoring.coreos.com/kube-prometheus-node-exporter apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: annotations: meta.helm.sh/release-name: kube-prometheus meta.helm.sh/release-namespace: monitoring generation: 1 labels: app.kubernetes.io/instance: kube-prometheus app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: node-exporter helm.sh/chart: node-exporter-3.2.1 name: kube-prometheus-node-exporter namespace: monitoring spec: endpoints: - port: metrics interval: 15s #抓取Endpoints的时间间隔 relabelings: #标签重写 - action: replace regex: (.*) replacement: $1 sourceLabels: - __meta_kubernetes_pod_node_name targetLabel: instance jobLabel: jobLabel namespaceSelector: #定义Prometheus在哪些namespace中选择要被监控的Endpoints,根据标签选择 matchNames: - monitoring selector: #定义Prometheus选择哪些要被监控的Endpoints,根据标签选择Endpoints matchLabels: app.kubernetes.io/instance: kube-prometheus app.kubernetes.io/name: node-exporter
3、PrometheusRule
- PrometheusRule CRD声明一个或多个Prometheus实例需要的Prometheus rule。
- Alerts和recording rules可以保存并应用为yaml文件,可以被动态加载而不需要重启。
- 获取PrometheusRule资源:https://github.com/prometheus-operator/kube-prometheus/tree/main/manifests
apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: app.kubernetes.io/component: exporter app.kubernetes.io/name: node-exporter app.kubernetes.io/part-of: kube-prometheus prometheus: k8s role: alert-rules name: node-exporter-rules namespace: monitoring spec: groups: - name: node-exporter rules: - alert: NodeFilesystemSpaceFillingUp annotations: description: Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available space left and is filling up. runbook_url: https://github.com/prometheus-operator/kube-prometheus/wiki/nodefilesystemspacefillingup summary: Filesystem is predicted to run out of space within the next 24 hours. expr: | ( node_filesystem_avail_bytes{job="node-exporter",fstype!=""} / node_filesystem_size_bytes{job="node-exporter",fstype!=""} * 100 < 40 and predict_linear(node_filesystem_avail_bytes{job="node-exporter",fstype!=""}[6h], 24*60*60) < 0 and node_filesystem_readonly{job="node-exporter",fstype!=""} == 0 ) for: 1h labels: severity: warning ...
4、Alertmanager
- Alertmanager资源:定义AlertManager statefulset控制器的期望状态,Prometheus Operator确保这个statefulset控制器运行时一直与定义保持一致。包含副本数量、持久化存储的选项。
- Prometheus Operator会根据Alertmanager资源在相同namespace下生成一个StatefulSet控制器。Alertmanager的Pod都会挂载一个名为<prometheus-name>的Secret。
- 当有两个或更多配置的副本时,Operator可以高可用性模式运行Alertmanager实例。
- 查看Alertmanager资源(使用helm部署kube-Prometheus的内容)
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
]# kubectl edit -n monitoring alertmanager.monitoring.coreos.com/kube-prometheus-alertmanager apiVersion: monitoring.coreos.com/v1 kind: Alertmanager metadata: annotations: meta.helm.sh/release-name: kube-prometheus meta.helm.sh/release-namespace: monitoring generation: 1 labels: app.kubernetes.io/component: alertmanager app.kubernetes.io/instance: kube-prometheus app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: kube-prometheus helm.sh/chart: kube-prometheus-8.1.11 name: kube-prometheus-alertmanager namespace: monitoring spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/component: alertmanager app.kubernetes.io/instance: kube-prometheus app.kubernetes.io/name: kube-prometheus namespaces: - monitoring topologyKey: kubernetes.io/hostname weight: 1 containers: - livenessProbe: failureThreshold: 120 httpGet: path: /-/healthy port: web scheme: HTTP initialDelaySeconds: 0 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 3 name: alertmanager readinessProbe: failureThreshold: 120 httpGet: path: /-/ready port: web scheme: HTTP initialDelaySeconds: 0 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 3 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: false runAsNonRoot: true - livenessProbe: failureThreshold: 6 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 tcpSocket: port: reloader-web timeoutSeconds: 5 name: config-reloader readinessProbe: failureThreshold: 6 initialDelaySeconds: 15 periodSeconds: 20 successThreshold: 1 tcpSocket: port: reloader-web timeoutSeconds: 5 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: false runAsNonRoot: true externalUrl: http://127.0.0.1:9093/ image: docker.io/bitnami/alertmanager:0.24.0-debian-11-r46 listenLocal: false logFormat: logfmt logLevel: info paused: false podMetadata: labels: app.kubernetes.io/component: alertmanager app.kubernetes.io/instance: kube-prometheus app.kubernetes.io/name: kube-prometheus portName: web replicas: 1 resources: {} retention: 120h routePrefix: / securityContext: fsGroup: 1001 runAsUser: 1001 serviceAccountName: kube-prometheus-alertmanager storage: #定义存储卷 volumeClaimTemplate: metadata: {} spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi storageClassName: nfs-client
- 查看Alertmanager资源生成的StatefulSet控制器(使用helm部署kube-Prometheus的内容)
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
]# kubectl edit -n monitoring statefulset.apps/alertmanager-kube-prometheus-alertmanager apiVersion: apps/v1 kind: StatefulSet metadata: annotations: meta.helm.sh/release-name: kube-prometheus meta.helm.sh/release-namespace: monitoring prometheus-operator-input-hash: "13509733468393518222" generation: 1 labels: app.kubernetes.io/component: alertmanager app.kubernetes.io/instance: kube-prometheus app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: kube-prometheus helm.sh/chart: kube-prometheus-8.1.11 name: alertmanager-kube-prometheus-alertmanager namespace: monitoring ownerReferences: - apiVersion: monitoring.coreos.com/v1 blockOwnerDeletion: true controller: true kind: Alertmanager name: kube-prometheus-alertmanager spec: podManagementPolicy: Parallel replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: alertmanager: kube-prometheus-alertmanager app.kubernetes.io/instance: kube-prometheus-alertmanager app.kubernetes.io/managed-by: prometheus-operator app.kubernetes.io/name: alertmanager serviceName: alertmanager-operated template: metadata: annotations: kubectl.kubernetes.io/default-container: alertmanager creationTimestamp: null labels: alertmanager: kube-prometheus-alertmanager app.kubernetes.io/component: alertmanager app.kubernetes.io/instance: kube-prometheus-alertmanager app.kubernetes.io/managed-by: prometheus-operator app.kubernetes.io/name: alertmanager app.kubernetes.io/version: 0.24.0 spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/component: alertmanager app.kubernetes.io/instance: kube-prometheus app.kubernetes.io/name: kube-prometheus namespaces: - monitoring topologyKey: kubernetes.io/hostname weight: 1 containers: - args: - --config.file=/etc/alertmanager/config_out/alertmanager.env.yaml - --storage.path=/alertmanager - --data.retention=120h - --cluster.listen-address= - --web.listen-address=:9093 - --web.external-url=http://127.0.0.1:9093/ - --web.route-prefix=/ - --cluster.peer=alertmanager-kube-prometheus-alertmanager-0.alertmanager-operated:9094 - --cluster.reconnect-timeout=5m - --web.config.file=/etc/alertmanager/web_config/web-config.yaml env: - name: POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP image: docker.io/bitnami/alertmanager:0.24.0-debian-11-r46 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 120 httpGet: path: /-/healthy port: web scheme: HTTP periodSeconds: 5 successThreshold: 1 timeoutSeconds: 3 name: alertmanager ports: - containerPort: 9093 name: web protocol: TCP - containerPort: 9094 name: mesh-tcp protocol: TCP - containerPort: 9094 name: mesh-udp protocol: UDP readinessProbe: failureThreshold: 120 httpGet: path: /-/ready port: web scheme: HTTP initialDelaySeconds: 3 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 3 resources: requests: memory: 200Mi securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: false runAsNonRoot: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: FallbackToLogsOnError volumeMounts: - mountPath: /etc/alertmanager/config name: config-volume - mountPath: /etc/alertmanager/config_out name: config-out readOnly: true - mountPath: /etc/alertmanager/certs name: tls-assets readOnly: true - mountPath: /alertmanager name: alertmanager-kube-prometheus-alertmanager-db subPath: alertmanager-db - mountPath: /etc/alertmanager/web_config/web-config.yaml name: web-config readOnly: true subPath: web-config.yaml - args: - --listen-address=:8080 - --reload-url=http://127.0.0.1:9093/-/reload - --config-file=/etc/alertmanager/config/alertmanager.yaml.gz - --config-envsubst-file=/etc/alertmanager/config_out/alertmanager.env.yaml command: - /bin/prometheus-config-reloader env: - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: SHARD value: "-1" image: docker.io/bitnami/prometheus-operator:0.60.1-debian-11-r0 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 6 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 tcpSocket: port: reloader-web timeoutSeconds: 5 name: config-reloader ports: - containerPort: 8080 name: reloader-web protocol: TCP readinessProbe: failureThreshold: 6 initialDelaySeconds: 15 periodSeconds: 20 successThreshold: 1 tcpSocket: port: reloader-web timeoutSeconds: 5 resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: false runAsNonRoot: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: FallbackToLogsOnError volumeMounts: - mountPath: /etc/alertmanager/config name: config-volume readOnly: true - mountPath: /etc/alertmanager/config_out name: config-out dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 1001 runAsUser: 1001 serviceAccount: kube-prometheus-alertmanager serviceAccountName: kube-prometheus-alertmanager terminationGracePeriodSeconds: 120 volumes: - name: config-volume secret: defaultMode: 420 secretName: alertmanager-kube-prometheus-alertmanager-generated - name: tls-assets projected: defaultMode: 420 sources: - secret: name: alertmanager-kube-prometheus-alertmanager-tls-assets-0 - emptyDir: {} name: config-out - name: web-config secret: defaultMode: 420 secretName: alertmanager-kube-prometheus-alertmanager-web-config updateStrategy: type: RollingUpdate volumeClaimTemplates: - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: null name: alertmanager-kube-prometheus-alertmanager-db spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi storageClassName: nfs-client volumeMode: Filesystem status: phase: Pending status: collisionCount: 0 currentReplicas: 1 currentRevision: alertmanager-kube-prometheus-alertmanager-b74c5965d observedGeneration: 1 readyReplicas: 1 replicas: 1 updateRevision: alertmanager-kube-prometheus-alertmanager-b74c5965d updatedReplicas: 1
2、使用helm部署kube-Prometheus
- Prometheus部署环境如下:
- Kubernetes版本为v1.20.14。
- helm版本为v3.8.2。
- kube-prometheus的版本bitnami/kube-prometheus:8.1.11。
2.1、创建动态存储卷
- 创建动态存储卷
- 参看:https://www.cnblogs.com/maiblogs/p/16392831.html的《6.2、动态存储卷》,只需创建到“创建NFS SotageClass”。
2.2、部署kube-prometheus
- kube-prometheus:8.1.11会自动安装如下组件:
- prometheus-operator
- prometheus
- state-metrics
- node-exporter
- blackbox-exporter
- alertmanager
1、创建名称空间
]# kubectl create namespace monitoring
2、下载kube-prometheus的chart
]# helm repo add bitnami https://charts.bitnami.com/bitnami ]# helm search repo prometheus ]# helm pull bitnami/kube-prometheus
3、修改values.yaml
//解压 ]# tar zvfx kube-prometheus-8.1.11.tgz //修改values.yaml ]# vim ./kube-prometheus/values.yaml prometheus: ingress: enabled: true hostname: annotations: {kubernetes.io/ingress.class: "nginx"} extraRules: - host: prometheus.local http: paths: - path: / pathType: Prefix backend: service: name: kube-prometheus-prometheus port: number: 9090 externalUrl: "http://127.0.0.1:9090/" persistence: enabled: true storageClass: "nfs-client" alertmanager: ingress: enabled: true hostname: annotations: {kubernetes.io/ingress.class: "nginx"} extraRules: - host: alertmanager.local http: paths: - path: / pathType: Prefix backend: service: name: kube-prometheus-alertmanager port: number: 9093 externalUrl: "http://127.0.0.1:9093/" persistence: enabled: true storageClass: "nfs-client"
- 修改后的values.yaml文件
4、应用kube-prometheus
]# helm install kube-prometheus kube-prometheus/ -n monitoring
5、访问prometheus和alertmanager
//修改hosts文件(C:\Windows\System32\drivers\etc) 10.1.1.11 prometheus.local alertmanager.local
- 使用http://prometheus.local:32080/访问prometheus。
- 使用http://alertmanager.local:32080/访问alertmanager。
2.3、实现告警
2.3.1、配置alertmanager
1、查看alertmanager配置文件
]# kubectl exec alertmanager-kube-prometheus-alertmanager-0 -n monitoring -- cat /etc/alertmanager/config_out/alertmanager.env.yaml ]# kubectl get secret alertmanager-kube-prometheus-alertmanager -n monitoring -o go-template='{{ index .data "alertmanager.yaml" }}' | base64 -d global: resolve_timeout: 5m receivers: - name: "null" route: group_by: - job group_interval: 5m group_wait: 30s receiver: "null" repeat_interval: 12h routes: - match: alertname: Watchdog receiver: "null"
2、修改alertmanager的配置文件
- 创建alertmanager.yaml
- 注意,这里的alertmanager.yaml顶层多了两级alertmanager和config。
- 注意,route如果没有子节点,就必须设置routes: []。
- 报错信息,level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="undefined receiver \"null\" used in route"
- 注意,pod将存储卷挂在到了/alertmanager/。
]# vim alertmanager.yaml alertmanager: config: global: resolve_timeout: 5m smtp_smarthost: 'smtp.qq.com:465' smtp_from: 'xxx@qq.com' smtp_auth_username: 'xxx@qq.com' smtp_auth_password: 'xxx' smtp_require_tls: false route: group_by: ['alertname'] group_wait: 10s group_interval: 10s repeat_interval: 10s receiver: 'email' routes: [] receivers: - name: 'email' email_configs: - to: 'xxx@xxx.com.cn' templates: - '/alertmanager/template.tmpl'
3、将告警模板template.tmpl放到存储卷上
]# vim /data1/monitoring-alertmanager-kube-prometheus-alertmanager-db-alertmanager-kube-prometheus-alertmanager-0-pvc-85d5342a-9f8c-41c3-95fe-f11c77579b0c/alertmanager-db/template.tmpl {{ define "__subject" }} {{ if gt (len .Alerts.Firing) 0 -}} {{ range .Alerts }} {{ .Labels.alertname }}{{ .Annotations.title }} {{ end }}{{ end }}{{ end }} {{ define "email.default.html" }} {{ range .Alerts }} 告警名称: {{ .Annotations.title }} <br> 告警级别: {{ .Labels.severity }} <br> 告警主机: {{ .Labels.instance }} <br> 告警信息: {{ .Annotations.description }} <br> 维护团队: {{ .Labels.team }} <br> 告警时间:{{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }} <br> {{ end }}{{ end }}
4、滚动更新kube-prometheus
- 更新alertmanager.yaml配置文件
]# helm upgrade kube-prometheus kube-prometheus/ --values=alertmanager.yaml -n monitoring
2.3.2、创建告警规则
1、创建PrometheusRule资源
]# vim node-exporter-rules.yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: app.kubernetes.io/component: exporter app.kubernetes.io/name: node-exporter app.kubernetes.io/part-of: kube-prometheus prometheus: k8s role: alert-rules name: node-exporter-rules namespace: monitoring spec: groups: - name: node-exporter rules: - alert: NodeFilesystemSpaceFillingUp expr: up == 3 for: 10s labels: severity: "告警级别critical" team: "维护团队OPS" annotations: title: "告警名称Instance {{ $labels.instance }} down" description: "告警信息{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 3 minutes."
2、应用PrometheusRule资源
]# kubectl apply -f node-exporter-rules.yaml //查看prometheusrule资源 ]# kubectl get prometheusrule -A NAMESPACE NAME AGE monitoring node-exporter-rules 39s
2.4、部署grafana
1、下载grafana的chart
]# helm repo add bitnami https://charts.bitnami.com/bitnami ]# helm search repo grafana ]# helm pull bitnami/grafana
2、修改values.yaml
//解压 ]# tar zvfx grafana-8.2.12.tgz //修改values.yaml ]# vim grafana/values.yaml admin: password: "admin" persistence: storageClass: "nfs-client" ingress: enabled: true hostname: annotations: {kubernetes.io/ingress.class: "nginx"} extraRules: - host: grafana.local http: paths: - path: / pathType: Prefix backend: service: name: grafana port: number: 3000
3、应用grafana
]# helm install grafana grafana/ -n monitoring
4、访问grafana
//修改hosts文件(C:\Windows\System32\drivers\etc) 10.1.1.11 prometheus.local alertmanager.local grafana.local
- 使用http://grafana.local:32080/访问grafana。
5、添加数据源
- 在Kubernetes中,集群内部的服务可用通过Kubernetes内部的域名相互访问,Kubernetes内部的域名是:Service_Name.Namespace_Name.svc.cluster.local。
- 同一个名称空间中的服务,可以直接通过Service_Name相互访问。
3、使用github部署kube-Prometheus
- 注意,这里只是快速入门安装,并没有使用持久卷。
- Prometheus部署环境如下:
- Kubernetes版本为v1.20.14。
- kube-prometheus的版本0.8.o。
- kube-prometheus和Kubernetes的兼容性
1、下载kube-Prometheus
]# wget https://github.com/prometheus-operator/kube-prometheus/archive/refs/tags/v0.8.0.tar.gz
2、快速部署kube-prometheus
]# tar zvfx v0.8.0.tar.gz ]# cd kube-prometheus-0.8.0/ //将k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.0.0替换为bitnami/kube-state-metrics:2.0.0 ]# vim ./manifests/kube-state-metrics-deployment.yaml //先创建名称空间、prometheus-operator等 ]# kubectl create -f manifests/setup //部署全部组件 ]# kubectl create -f manifests/
- 如果之前部署过prometheus,请先清除可能的残留
]# cd kube-prometheus-0.8.0/ ]# kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup
-
查看相关资源
//查看pod ]# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE monitoring alertmanager-main-0 2/2 Running 0 31s monitoring alertmanager-main-1 2/2 Running 0 31s monitoring alertmanager-main-2 2/2 Running 0 31s monitoring blackbox-exporter-55c457d5fb-4jvmm 3/3 Running 0 30s monitoring grafana-9df57cdc4-l7gxk 1/1 Running 0 29s monitoring kube-state-metrics-6cb48468f8-dbdnc 3/3 Running 0 29s monitoring node-exporter-6svtr 2/2 Running 0 29s monitoring node-exporter-hpfw9 2/2 Running 0 29s monitoring node-exporter-jksr2 2/2 Running 0 29s monitoring prometheus-adapter-59df95d9f5-rxzdg 1/1 Running 0 29s monitoring prometheus-adapter-59df95d9f5-zs46x 1/1 Running 0 29s monitoring prometheus-k8s-0 2/2 Running 1 29s monitoring prometheus-k8s-1 2/2 Running 1 29s monitoring prometheus-operator-7775c66ccf-bfxd9 2/2 Running 0 29m ... //查看service ]# kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE monitoring alertmanager-main ClusterIP 10.20.24.158 <none> 9093/TCP 64s monitoring alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 64s monitoring blackbox-exporter ClusterIP 10.20.248.189 <none> 9115/TCP,19115/TCP 64s monitoring grafana ClusterIP 10.20.214.103 <none> 3000/TCP 63s monitoring kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 63s monitoring node-exporter ClusterIP None <none> 9100/TCP 63s monitoring prometheus-adapter ClusterIP 10.20.74.223 <none> 443/TCP 63s monitoring prometheus-k8s ClusterIP 10.20.73.57 <none> 9090/TCP 63s monitoring prometheus-operated ClusterIP None <none> 9090/TCP 62s monitoring prometheus-operator ClusterIP None <none> 8443/TCP 30m ... //查看pod控制器 ]# kubectl get deployment -A NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE monitoring blackbox-exporter 1/1 1 1 109s monitoring grafana 1/1 1 1 108s monitoring kube-state-metrics 1/1 1 1 108s monitoring prometheus-adapter 2/2 2 2 108s monitoring prometheus-operator 1/1 1 1 30m ... //查看有状态的pod控制器 ]# kubectl get sts -A NAMESPACE NAME READY AGE monitoring alertmanager-main 3/3 2m1s monitoring prometheus-k8s 2/2 119s ...
3、访问服务
- 访问prometheus
- http://10.1.1.11:19090/
//监听10.1.1.11:19090,并将请求转发到service后面的pod的9090端口 ]# kubectl port-forward svc/prometheus-k8s --address=10.1.1.11 19090:9090 -n monitoring
- 访问grafana
- http://10.1.1.11:13000/ (admin:admin)
//监听10.1.1.11:13000,并将请求转发到service后面的pod的3000端口 ]# kubectl port-forward svc/grafana --address=10.1.1.11 13000:3000 -n monitoring
- 访问alertmanager
- http://10.1.1.11:19093/
//监听10.1.1.11:19093,并将请求转发到service后面的pod的9093端口 ]# kubectl port-forward svc/alertmanager-main --address=10.1.1.11 19093:9093 -n monitoring
4、创建ingress规则
]# vim prometheus-ingress apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: prometheus-ingress namespace: monitoring annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: prometheus.local http: paths: - path: / pathType: Prefix backend: service: name: prometheus-k8s port: number: 9090 - host: alertmanager.local http: paths: - path: / pathType: Prefix backend: service: name: alertmanager-main port: number: 9093 - host: grafana.local http: paths: - path: / pathType: Prefix backend: service: name: grafana port: number: 3000
1
# #