Kubernetes Metrics Server：
-
Kubernetes Metrics Server 是 Cluster 的核心监控数据的聚合器，kubeadm 默认是不部署的。
-
Metrics Server 供 Dashboard 等其他组件使用，是一个扩展的 APIServer，依赖于 API Aggregator。所以，在安装 Metrics Server 之前需要先在 kube-apiserver 中开启 API Aggregator。
-
Metrics API 只可以查询当前的度量数据，并不保存历史数据。
-
Metrics API URI 为 /apis/metrics.k8s.io/，在 k8s.io/metrics 下维护。
-
必须部署 metrics-server 才能使用该 API，metrics-server 通过调用 kubelet Summary API 获取数据。
一、前提条件
注意：使用 Metrics Server 有必备两个条件：
1、API Server 启用 Aggregator Routing 支持。否则 API Server 不识别请求：
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)
2、API Server 能访问 Metrics Server Pod IP。否则 API Server 无法访问 Metrics Server：
E1223 07:23:04.330206 1 available_controller.go:420] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.171.248.214:4443/apis/metrics.k8s.io/v1beta1: Get https://10.171.248.214:4443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3、启用API Aggregator，API Aggregation 允许在不修改 Kubernetes 核心代码的同时扩展 Kubernetes API，即：将第三方服务注册到 Kubernetes API 中，这样就可以通过 Kubernetes API 来访问第三方服务了，例如：Metrics Server API。注：另外一种扩展 Kubernetes API 的方法是使用 CRD（Custom Resource Definition，自定义资源定义）。
二、修改配置
1、检查 API Server 是否开启了 Aggregator Routing：查看 API Server 是否具有 --enable-aggregator-routing=true 选项。
[root@master1 ~]# ps -ef | grep apiserver
root 12721 10738 0 20:46 pts/0 00:00:00 grep --color=auto apiserver
root 111575 111557 6 16:42 ? 00:16:51 kube-apiserver --advertise-address=192.168.200.3 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
2、修改每个 API Server 的 kube-apiserver.yaml 配置开启 Aggregator Routing：修改 manifests 配置后 API Server 会自动重启生效。
cat /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.200.3:6443
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
- --advertise-address=192.168.200.3
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
- --enable-aggregator-routing=true # 添加本行
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
- --etcd-servers=https://127.0.0.1:2379
- --insecure-port=0
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
- --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
- --requestheader-allowed-names=front-proxy-client
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --requestheader-group-headers=X-Remote-Group
- --requestheader-username-headers=X-Remote-User
- --secure-port=6443
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-cluster-ip-range=10.96.0.0/12
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
image: k8s.gcr.io/kube-apiserver:v1.18.2
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 192.168.200.3
path: /healthz
port: 6443
scheme: HTTPS
initialDelaySeconds: 15
timeoutSeconds: 15
name: kube-apiserver
resources:
requests:
cpu: 250m
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/pki
name: etc-pki
readOnly: true
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
hostNetwork: true
priorityClassName: system-cluster-critical
volumes:
- hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
name: ca-certs
- hostPath:
path: /etc/pki
type: DirectoryOrCreate
name: etc-pki
- hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
name: k8s-certs
status: {}
github地址：
https://github.com/kubernetes-sigs/metrics-server/releases
三、安装
1、下载yaml文件
wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.1/components.yaml
2、修改components.yaml文件
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-view: "true"
name: system:aggregated-metrics-reader
rules:
- apiGroups:
- metrics.k8s.io
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- pods
- nodes
- nodes/stats
- namespaces
- configmaps
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
ports:
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: metrics-server
strategy:
rollingUpdate:
maxUnavailable: 0
template:
metadata:
labels:
k8s-app: metrics-server
spec:
containers:
- args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP # 删掉 ExternalIP,Hostname这两个，这里已经改好了，你那边要自己核对一下
- --kubelet-use-node-status-port
- --kubelet-insecure-tls # 加上该启动参数
image: k8s.gcr.io/metrics-server/metrics-server:v0.4.1 # 镜像地址根据情况修改
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /livez
port: https
scheme: HTTPS
periodSeconds: 10
name: metrics-server
ports:
- containerPort: 4443
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz
port: https
scheme: HTTPS
periodSeconds: 10
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- mountPath: /tmp
name: tmp-dir
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: metrics-server
volumes:
- emptyDir: {}
name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
labels:
k8s-app: metrics-server
name: v1beta1.metrics.k8s.io
spec:
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100
3、安装
kubectl apply -f components.yaml
4、查看metrics-server服务状态
[root@master1 ~]# kubectl get pod -n kube-system | grep metrics-server
metrics-server-6f9f86ddf9-zphlw 1/1 Running 0 11s
5、检查 API Server 是否可以连通 Metrics Server
[root@master1 ~]# kubectl describe svc metrics-server -n kube-system
Name: metrics-server
Namespace: kube-system
Labels: kubernetes.io/cluster-service=true
kubernetes.io/name=Metrics-server
Annotations: Selector: k8s-app=metrics-server
Type: ClusterIP
IP: 10.107.86.150
Port: <unset> 443/TCP
TargetPort: main-port/TCP
Endpoints: 10.244.2.20:4443
Session Affinity: None
Events: <none>
# 在其他几个节点ping一下Endpoints的地址
[root@master3 ~]# ping 10.244.2.20
PING 10.244.2.20 (10.244.2.20) 56(84) bytes of data.
64 bytes from 10.244.2.20: icmp_seq=1 ttl=64 time=0.122 ms
64 bytes from 10.244.2.20: icmp_seq=2 ttl=64 time=0.032 ms
6、执行以下命令，检查节点占用性能情况。
[root@master1 ~]# kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
master1 223m 5% 1433Mi 52%
master2 148m 3% 972Mi 35%
master3 183m 4% 1123Mi 41%
node1 225m 5% 1389Mi 50%
[root@master1 ~]# kubectl top pods
NAME CPU(cores) MEMORY(bytes)
details-v1-6fc55d65c9-l97cs 4m 36Mi
nfs-client-provisioner-6d4469b5b5-ndg5x 4m 20Mi
productpage-v1-9cf6c76b5-hc2nr 10m 71Mi
ratings-v1-6f855c5fff-v9ht2 3m 28Mi
reviews-v1-697f45b7dd-w9fgc 7m 117Mi
reviews-v2-8c85b467b-jgx6w 6m 113Mi
reviews-v3-76ccf79776-74vz6 5m 117Mi
