K8S集群监控
一、KubeStateMetrics简介
kube-state-metrics 是一个 Kubernetes 组件，它通过查询 Kubernetes 的 API 服务器，收集关于 Kubernetes 中各种资源（如节点、pod、服务等）的状态信息，并将这些信息转换成 Prometheus 可以使用的指标。
kube-state-metrics 主要功能：
- 节点状态信息，如节点 CPU 和内存的使用情况、节点状态、节点标签等。
- Pod 的状态信息，如 Pod 状态、容器状态、容器镜像信息、Pod 的标签和注释等。
- Deployment、Daemonset、Statefulset 和 ReplicaSet 等控制器的状态信息，如副本数、副本状态、创建时间等。
- Service 的状态信息，如服务类型、服务 IP 和端口等。
- 存储卷的状态信息，如存储卷类型、存储卷容量等。
- Kubernetes 的 API 服务器状态信息，如 API 服务器的状态、请求次数、响应时间等。
通过 kube-state-metrics 可以方便的对 Kubernetes 集群进行监控，发现问题，以及提前预警。
二、集群组件
2.0、部署 KubeStateMetrics
包含ServiceAccount、ClusterRole、ClusterRoleBinding、Deployment、ConfigMap、Service 六类YAML文件
2.0.1 部署RBAC
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-state-metrics
namespace: kube-system
labels:
k8s-app: kube-state-metrics
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kube-state-metrics
labels:
k8s-app: kube-state-metrics
rules:
- apiGroups: [""]
resources: ["configmaps","secrets","nodes","pods",
"services","resourcequotas",
"replicationcontrollers","limitranges",
"persistentvolumeclaims","persistentvolumes",
"namespaces","endpoints"]
verbs: ["list","watch"]
- apiGroups: ["extensions"]
resources: ["daemonsets","deployments","replicasets"]
verbs: ["list","watch"]
- apiGroups: ["apps"]
resources: ["statefulsets","daemonsets","deployments","replicasets"]
verbs: ["list","watch"]
- apiGroups: ["batch"]
resources: ["cronjobs","jobs"]
verbs: ["list","watch"]
- apiGroups: ["autoscaling"]
resources: ["horizontalpodautoscalers"]
verbs: ["list","watch"]
- apiGroups: ["authentication.k8s.io"]
resources: ["tokenreviews"]
verbs: ["create"]
- apiGroups: ["authorization.k8s.io"]
resources: ["subjectaccessreviews"]
verbs: ["create"]
- apiGroups: ["policy"]
resources: ["poddisruptionbudgets"]
verbs: ["list","watch"]
- apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests"]
verbs: ["list","watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses","volumeattachments"]
verbs: ["list","watch"]
- apiGroups: ["admissionregistration.k8s.io"]
resources: ["mutatingwebhookconfigurations","validatingwebhookconfigurations"]
verbs: ["list","watch"]
- apiGroups: ["networking.k8s.io"]
resources: ["networkpolicies","ingresses"]
verbs: ["list","watch"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kube-state-metrics
labels:
app: kube-state-metrics
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-state-metrics
subjects:
- kind: ServiceAccount
name: kube-state-metrics
namespace: kube-system
2.0.2 部署deploy、svc
apiVersion: v1
kind: Service
metadata:
name: kube-state-metrics
namespace: kube-system
labels:
k8s-app: kube-state-metrics
app.kubernetes.io/name: kube-state-metrics ##不能删除此注解,该注解用于Prometheus自动发现
spec:
type: ClusterIP
ports:
- name: http-metrics
port: 8080
targetPort: 8080
- name: telemetry
port: 8081
targetPort: 8081
selector:
k8s-app: kube-state-metrics
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kube-state-metrics
namespace: kube-system
labels:
k8s-app: kube-state-metrics
spec:
replicas: 1
selector:
matchLabels:
k8s-app: kube-state-metrics
template:
metadata:
labels:
k8s-app: kube-state-metrics
spec:
serviceAccountName: kube-state-metrics
containers:
- name: kube-state-metrics
image: bitnami/kube-state-metrics:2.0.0
#image: registry.cn-shenzhen.aliyuncs.com/starsl/kube-state-metrics:v2.3.0
securityContext:
runAsUser: 65534
ports:
- name: http-metrics ##用于公开kubernetes的指标数据的端口
containerPort: 8080
- name: telemetry ##用于公开自身kube-state-metrics的指标数据的端口
containerPort: 8081
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /
port: 8081
initialDelaySeconds: 5
timeoutSeconds: 5
2.0.3 确认验证
kubectl -n kube-system get pods | grep kube-state
curl -kL $(kubectl get service -n kube-system | grep kube-state-metrics |awk '{ print $3 }'):8080/metrics | tail -20
2.1、集群组件监控
在 prometheus-config.yaml 一次添加如下采集数据：
2.1.1、kube-apiserver
需要注意的是使用https访问时，需要tls相关配置，可以指定ca证书路径或者 insecure_skip_verify: true跳过证书验证。
除此之外，还要指定 bearer_token_file，否则会提示 server returned HTTP status 400 Bad Request;
- job_name: kube-apiserver
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name]
action: keep
regex: default;kubernetes
- source_labels: [__meta_kubernetes_endpoints_name]
action: replace
target_label: endpoint
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: service
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
更新配置
kubectl apply -f prometheus-cm.yaml
导入grafana模板
15761
2.1.2、controller-manager
- 查看controller-manager信息（名称可能不太一样，大家注意一下）
xxxxxxxxxx
# kubectl describe pod -n kube-system kube-controller-manager-master1
Name: kube-controller-manager-k8s-master
Namespace: kube-system
……
Labels: component=kube-controller-manager
tier=control-plane
……
Containers:
kube-controller-manager:
……
Command:
kube-controller-manager
--allocate-node-cidrs=true
--authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
--authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
--bind-address=127.0.0.1
……
由上可知，匹配pod对象，lable标签为component=kube-controller-manager即可，但需注意的是controller-manager默认只运行127.0.0.1访问，因此还需先修改所用master节点的controller-manager配置.
- 修改 /etc/kubernetes/manifests/kube-controller-manager.yaml
xxxxxxxxxx
# cat /etc/kubernetes/manifests/kube-controller-manager.yaml
……
- command:
- --bind-address=0.0.0.0 # 端口改为0.0.0.0
#- --port=0 # 注释0端口
……
- 编写prometheus配置文件，需要注意的是，他默认匹配到的是80端口，需要手动指定为10252端口;
- job_name: kube-controller-manager
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_component]
regex: kube-controller-manager
action: keep
- source_labels: [__meta_kubernetes_pod_ip]
regex: (.+)
target_label: __address__
replacement: ${1}:10252
- source_labels: [__meta_kubernetes_endpoints_name]
action: replace
target_label: endpoint
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: service
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
2.1.3、scheduler
xxxxxxxxxx
[root@tiaoban prometheus]# kubectl describe pod -n kube-system kube-scheduler-master1
Name: kube-scheduler-k8s-master
Namespace: kube-system
……
Labels: component=kube-scheduler
tier=control-plane
……
由上可知，匹配pod对象，lable标签为component=kube-scheduler即可scheduler和controller-manager一样，默认监听0端口，需要注释
修改所有master节点 /etc/kubernetes/manifests/kube-scheduler.yaml
xxxxxxxxxx
# cat /etc/kubernetes/manifests/kube-scheduler.yaml
……
- command:
- --bind-address=0.0.0.0 # 端口改为0.0.0.0
#- --port=0 # 注释0端口
……
- 编写prometheus配置文件，需要注意的是，他默认匹配到的是80端口，需要手动指定为10251端口,同时指定token，否则会提示 server returned HTTP status 400 Bad Request;
- job_name: kube-scheduler
kubernetes_sd_configs:
- role: pod
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_component]
regex: kube-scheduler
action: keep
- source_labels: [__meta_kubernetes_pod_ip]
regex: (.+)
target_label: __address__
replacement: ${1}:10251
- source_labels: [__meta_kubernetes_endpoints_name]
action: replace
target_label: endpoint
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: service
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
2.1.4、kube-state-metrics
- 编写prometheus配置文件，需要注意的是，他默认匹配到的是8080和801两个端口，需要手动指定为8080端口;
- job_name: "kube-state-metrics"
kubernetes_sd_configs:
- role: endpoints
namespaces:
names: ["kube-system"]
relabel_configs:
- action: keep
source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
regex: kube-state-metrics
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_pod_ip]
regex: (.+)
target_label: __address__
replacement: ${1}:8080
- action: replace
source_labels: [__meta_kubernetes_namespace]
target_label: k8s_namespace
- action: replace
source_labels: [__meta_kubernetes_service_name]
target_label: k8s_sname
2.1.5、coredns
- 编写prometheus配置文件，需要注意的是，他默认匹配到的是53端口，需要手动指定为9153端口
- job_name: coredns
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels:
- __meta_kubernetes_service_label_k8s_app
regex: kube-dns
action: keep
- source_labels: [__meta_kubernetes_pod_ip]
regex: (.+)
target_label: __address__
replacement: ${1}:9153
- source_labels: [__meta_kubernetes_endpoints_name]
action: replace
target_label: endpoint
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: service
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
2.1.6、etcd
# kubectl describe pod -n kube-system etcd-master1
Name: etcd-master1
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Node: master1/192.10.192.158
Start Time: Mon, 30 Jan 2023 15:06:35 +0800
Labels: component=etcd
tier=control-plane
···
Command:
etcd
--advertise-client-urls=https://192.10.192.158:2379
--cert-file=/etc/kubernetes/pki/etcd/server.crt
--client-cert-auth=true
--data-dir=/var/lib/etcd
--initial-advertise-peer-urls=https://192.10.192.158:2380
--initial-cluster=master1=https://192.10.192.158:2380
--key-file=/etc/kubernetes/pki/etcd/server.key
--listen-client-urls=https://127.0.0.1:2379,https://192.10.192.158:2379
--listen-metrics-urls=http://127.0.0.1:2381
--listen-peer-urls=https://192.10.192.158:2380
--name=master1
--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
--peer-client-cert-auth=true
--peer-key-file=/etc/kubernetes/pki/etcd/peer.key
--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
--snapshot-count=10000
--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
···
由上可知，启动参数里面有一个 --listen-metrics-urls=http://127.0.0.1:2381 的配置，该参数就是来指定 Metrics 接口运行在 2381 端口下面的，而且是 http 的协议，所以也不需要什么证书配置，这就比以前的版本要简单许多了，以前的版本需要用 https 协议访问，所以要配置对应的证书。但是还需修改配置文件，地址改为0.0.0.0
- 编写prometheus配置文件，需要注意的是，他默认匹配到的是2379端口，需要手动指定为2381端口
- job_name: etcd
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_label_component
regex: etcd
action: keep
- source_labels: [__meta_kubernetes_pod_ip]
regex: (.+)
target_label: __address__
replacement: ${1}:2381
- source_labels: [__meta_kubernetes_endpoints_name]
action: replace
target_label: endpoint
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
上面部分参数简介如下:
- kubernetes_sd_configs: 设置发现模式为 Kubernetes 动态服务发现
- kubernetes_sd_configs.role: 指定 Kubernetes 的服务发现模式，这里设置为 endpoints 的服务发现模式，该模式下会调用 kube-apiserver 中的接口获取指标数据。并且还限定只获取 kube-state-metrics 所在 - Namespace 的空间 kube-system 中的 Endpoints 信息
- kubernetes_sd_configs.namespace: 指定只在配置的 Namespace 中进行 endpoints 服务发现
- relabel_configs: 用于对采集的标签进行重新标记
热加载prometheus，使configmap配置文件生效（也可以等待prometheus的自动热加载）：
curl -XPOST http://prometheus.kubernets.cn/-/reload
旧版本监控etcd需要证书
-
通过secret创建证书
:::tips
kubectl create secret generic etcd-certs --from-file=/etc/kubernetes/pki/etcd/ca.crt --from-file=/etc/kubernetes/pki/etcd/server.crt --from-file=/etc/kubernetes/pki/etcd/server.key -n monitor
::: -
创建ServiceEndpoints资源
apiVersion: v1
kind: Service
metadata:
name: etcd-k8s
namespace: kube-system
labels:
k8s-app: etcd
app.kubernetes.io/name: etcd
spec:
type: ClusterIP
clusterIP: None
ports:
- name: port
port: 2379
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: etcd-k8s
namespace: kube-system
labels:
k8s-app: etcd
subsets:
- addresses:
- ip: 10.10.10.11
- ip: 10.10.10.12
- ip: 10.10.10.13
ports:
- port: 2379
- 修改prometheus配置
- job_name: "kubernetes-etcd"
scheme: https
tls_config:
ca_file: /certs/ca.crt
cert_file: /certs/server.crt
key_file: /certs/server.key
insecure_skip_verify: false
kubernetes_sd_configs:
- role: endpoints
namespaces:
names: ["kube-system"]
relabel_configs:
- action: keep
source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
regex: etcd
- 通过deploy挂载证书
- name: certs # 挂载证书配置
readOnly: true
mountPath: /certs
- name: certs
secret:
secretName: etcd-certs # 存储etcd证书Secret名称
kubectl apply -f prometheus_deploy.yaml
- 检查
导入模板grafana模板
2.1.7、kubelet
[root@master01 prometheus-all]# ss -tunlp | grep kubelet
tcp LISTEN 0 16384 127.0.0.1:26594 *:* users:(("kubelet",pid=1133,fd=15))
tcp LISTEN 0 16384 127.0.0.1:10248 *:* users:(("kubelet",pid=1133,fd=33))
tcp LISTEN 0 16384 [::]:10250 [::]:* users:(("kubelet",pid=1133,fd=28))
由上可知，kubelet在每个node节点都有运行，端口为10250，因此role使用node即可。
- 编写prometheus配置文件，需要注意的是，他的指标采集地址为/metrics/cadvisor，需要配置https访问，可以设置insecure_skip_verify: true跳过证书验证
添加配置
- job_name: kubelet
metrics_path: /metrics/cadvisor
scheme: https
tls_config:
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- source_labels: [__meta_kubernetes_endpoints_name]
action: replace
target_label: endpoint
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
或者，采用云原生grafna面版的配置
- job_name: 'k8s-kubelet'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
更新cm文件
kubectl apply -f prometheus-cm.yaml
kubectl apply -f prometheus-deploy.yaml
查看Prometheus配置
2.1.8、node
什么是node export
:::tips
Node Exporter是一个用于监控Linux系统性能指标的开源软件工具。它是Prometheus生态系统中的一部分，用于收集和暴露有关主机（通常是运行Linux操作系统的服务器）的各种性能指标和系统统计信息。Node Exporter可以帮助系统管理员、运维团队和开发人员实时监控服务器的性能，并在出现问题时提供警报和数据以进行故障排除。
Node Exporter可以收集的一些常见性能指标包括：
- CPU使用率：包括总体CPU使用率、每个核心的使用率等。
- 内存使用情况：包括可用内存、已使用内存、缓冲区和缓存等信息。
- 磁盘空间：监控硬盘分区的空闲空间、使用情况和I/O活动。
- 网络统计：包括网络接口的数据传输速率、错误率和丢包率等信息。
- 文件系统统计：关于文件系统的信息，如inode使用情况、文件句柄数等。
- 系统负载：系统的平均负载，用于衡量系统的忙碌程度。
- 进程信息：有关运行的进程的统计信息，如进程数、线程数等。
- 温度传感器数据：如果硬件支持，可以监控温度传感器的数据。
:::
DaemonSet运行node_exporter
[root@master01 prometheus-all]# cat node-exporter.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: node-exporter
namespace: kube-system
labels:
name: node-exporter
k8s-app: node-exporter
spec:
selector:
matchLabels:
name: node-exporter
template:
metadata:
labels:
name: node-exporter
app: node-exporter
spec:
hostPID: true
hostIPC: true
hostNetwork: true
containers:
- name: node-exporter
image: prom/node-exporter:v0.16.0
ports:
- containerPort: 9100
resources:
requests:
cpu: 0.15
securityContext:
privileged: true
args:
- --path.procfs
- /host/proc
- --path.sysfs
- /host/sys
- --collector.filesystem.ignored-mount-points
- '"^/(sys|proc|dev|host|etc)($|/)"'
volumeMounts:
- name: dev
mountPath: /host/dev
- name: proc
mountPath: /host/proc
- name: sys
mountPath: /host/sys
- name: rootfs
mountPath: /rootfs
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
volumes:
- name: proc
hostPath:
path: /proc
- name: dev
hostPath:
path: /dev
- name: sys
hostPath:
path: /sys
- name: rootfs
hostPath:
path: /
创建node
kubectl apply -f node-exporter.yaml
添加cm配置
- job_name: "kubernetes-nodes"
kubernetes_sd_configs:
- role: node
relabel_configs:
- source_labels: [__address__]
regex: '(.*):10250'
replacement: '${1}:9100'
target_label: __address__
action: replace
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- source_labels: [__meta_kubernetes_endpoints_name]
action: replace
target_label: endpoint
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
更新配置
:::tips
kubectl apply -f prometheus-cm.yaml
kubectl apply -f prometheus-deploy.yaml
:::
三、cAdvisor
cAdvisor 主要功能：
- 对容器资源的使用情况和性能进行监控。它以守护进程方式运行，用于收集、聚合、处理和导出正在运行容器的有关信息。
- cAdvisor 本身就对 Docker 容器支持，并且还对其它类型的容器尽可能的提供支持，力求兼容与适配所有类型的容器。
- Kubernetes 已经默认将其与 Kubelet 融合，所以我们无需再单独部署 cAdvisor 组件来暴露节点中容器运行的信息。
3.1、Prometheus 添加 cAdvisor 配置
由于 Kubelet 中已经默认集成 cAdvisor 组件，所以无需部署该组件。需要注意的是，他的指标采集地址为 /metrics/cadvisor，需要配置https访问，可以设置 insecure_skip_verify: true 跳过证书验证;
- job_name: 'k8s-kubelet'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
- job_name: 'k8s-cadvisor'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
metric_relabel_configs:
- source_labels: [instance]
separator: ;
regex: (.+)
target_label: node
replacement: $1
action: replace
热加载prometheus，使configmap配置文件生效：
curl -XPOST http://prometheus.kubernets.cn/-/reload
本文来自博客园，作者：&UnstopPable，转载请注明原文链接：https://www.cnblogs.com/Unstoppable9527/p/18327996
