k8s-资源指标API及自定义指标API-二十三
一、
原先版本是用heapster来收集资源指标才能看，但是现在heapster要废弃了。
从k8s v1.8开始后，引入了新的功能，即把资源指标引入api；
在使用heapster时，获取资源指标是由heapster自已获取的，heapster有自已的获取路径，没有通过apiserver，后来k8s引入了资源指标API(Metrics API)，于是资源指标的数据就从k8s的api中的直接获取，不必再通过其它途径。
metrics-server： 它也是一种API Server，提供了核心的Metrics API，就像k8s组件kube-apiserver提供了很多API群组一样，但它不是k8s组成部分，而是托管运行在k8s之上的Pod。
为了让用户无缝的使用metrics-server当中的API，还需要把这类自定义的API，通过聚合器聚合到核心API组里，
然后可以把此API当作是核心API的一部分，通过kubectl api-versions可直接查看。
metrics-server收集指标数据的方式是从各节点上kubelet提供的Summary API 即10250端口收集数据，收集Node和Pod核心资源指标数据，主要是内存和cpu方面的使用情况，并将收集的信息存储在内存中，所以当通过kubectl top不能查看资源数据的历史情况，其它资源指标数据则通过prometheus采集了。
k8s中很多组件是依赖于资源指标API的功能 ，比如kubectl top 、hpa，如果没有一个资源指标API接口，这些组件是没法运行的；
资源指标：metrics-server
自定义指标： prometheus, k8s-prometheus-adapter
新一代架构：
- 核心指标流水线：由kubelet、metrics-server以及由API server提供的api组成；cpu累计利用率、内存实时利用率、pod的资源占用率及容器的磁盘占用率；
- 监控流水线：用于从系统收集各种指标数据并提供终端用户、存储系统以及HPA，他们包含核心指标以及许多非核心指标。非核心指标不能被k8s所解析；
metrics-server是一个api server，收集cpu利用率、内存利用率等。
二、metrics
(1)卸载上一节heapster创建的资源；
[root@master metrics]# pwd /root/manifests/metrics [root@master metrics]# kubectl delete -f ./ deployment.apps "monitoring-grafana" deleted service "monitoring-grafana" deleted clusterrolebinding.rbac.authorization.k8s.io "heapster" deleted serviceaccount "heapster" deleted deployment.apps "heapster" deleted service "heapster" deleted deployment.apps "monitoring-influxdb" deleted service "monitoring-influxdb" deleted pod "pod-demo" deleted
metrics-server在GitHub上有单独的项目，在kubernetes的addons里面也有关于metrics-server插件的项目yaml文件；
我们这里使用kubernetes里面的yaml：
metrics-server on kubernetes：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/metrics-server
将以下几个文件下载出来：
[root@master metrics-server]# pwd /root/manifests/metrics/metrics-server [root@master metrics-server]# ls auth-delegator.yaml auth-reader.yaml metrics-apiservice.yaml metrics-server-deployment.yaml metrics-server-service.yaml resource-reader.yaml
需要修改一些内容：
目前metrics-server的镜像版本已经升级到metrics-server-amd64:v0.3.1了，此前的版本为v0.2.1，两者的启动的参数还是有所不同的。
[root@master metrics-server]# vim resource-reader.yaml ... ... rules: - apiGroups: - "" resources: - pods - nodes - namespaces - nodes/stats #添加此行 verbs: - get - list - watch - apiGroups: - "extensions" resources: - deployments ... ... [root@master metrics-server]# vim metrics-server-deployment.yaml ... ... containers: - name: metrics-server image: k8s.gcr.io/metrics-server-amd64:v0.3.1 #修改镜像(可以从阿里云上拉取，然后重新打标) command: - /metrics-server - --metric-resolution=30s - --kubelet-insecure-tls ##添加此行 - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP #添加此行 # These are needed for GKE, which doesn't support secure communication yet. # Remove these lines for non-GKE clusters, and when GKE supports token-based auth. #- --kubelet-port=10255 #- --deprecated-kubelet-completely-insecure=true ports: - containerPort: 443 name: https protocol: TCP - name: metrics-server-nanny image: k8s.gcr.io/addon-resizer:1.8.4 #修改镜像(可以从阿里云上拉取，然后重新打标) resources: limits: cpu: 100m memory: 300Mi requests: cpu: 5m memory: 50Mi ... ... # 修改containers，metrics-server-nanny 启动参数，修改好的如下： volumeMounts: - name: metrics-server-config-volume mountPath: /etc/config command: - /pod_nanny - --config-dir=/etc/config - --cpu=80m - --extra-cpu=0.5m - --memory=80Mi - --extra-memory=8Mi - --threshold=5 - --deployment=metrics-server-v0.3.1 - --container=metrics-server - --poll-period=300000 - --estimator=exponential # Specifies the smallest cluster (defined in number of nodes) # resources will be scaled to. #- --minClusterSize={{ metrics_server_min_cluster_size }} ... ... #创建 [root@master metrics-server]# kubectl apply -f ./ clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created serviceaccount/metrics-server created configmap/metrics-server-config created deployment.apps/metrics-server-v0.3.1 created service/metrics-server created #查看，pod已经起来了 [root@master metrics-server]# kubectl get pods -n kube-system |grep metrics-server metrics-server-v0.3.1-7d8bf87b66-8v2w9 2/2 Running 0 9m37s
[root@master ~]# kubectl api-versions |grep metrics metrics.k8s.io/v1beta1 [root@master ~]# kubectl top nodes Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io) #以下为pod中两个容器的日志 [root@master ~]# kubectl logs metrics-server-v0.3.1-7d8bf87b66-8v2w9 -c metrics-server -n kube-system I0327 07:06:47.082938 1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key) [restful] 2019/03/27 07:06:59 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi [restful] 2019/03/27 07:06:59 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/ I0327 07:06:59.783549 1 serve.go:96] Serving securely on [::]:443 [root@master ~]# kubectl logs metrics-server-v0.3.1-7d8bf87b66-8v2w9 -c metrics-server-nanny -n kube-system ERROR: logging before flag.Parse: I0327 07:06:40.684552 1 pod_nanny.go:65] Invoked by [/pod_nanny --config-dir=/etc/config --cpu=80m --extra-cpu=0.5m --memory=80Mi --extra-memory=8Mi --threshold=5 --deployment=metrics-server-v0.3.1 --container=metrics-server --poll-period=300000 --estimator=exponential] ERROR: logging before flag.Parse: I0327 07:06:40.684806 1 pod_nanny.go:81] Watching namespace: kube-system, pod: metrics-server-v0.3.1-7d8bf87b66-8v2w9, container: metrics-server. ERROR: logging before flag.Parse: I0327 07:06:40.684829 1 pod_nanny.go:82] storage: MISSING, extra_storage: 0Gi ERROR: logging before flag.Parse: I0327 07:06:40.689926 1 pod_nanny.go:109] cpu: 80m, extra_cpu: 0.5m, memory: 80Mi, extra_memory: 8Mi ERROR: logging before flag.Parse: I0327 07:06:40.689970 1 pod_nanny.go:138] Resources: [{Base:{i:{value:80 scale:-3} d:{Dec:<nil>} s:80m Format:DecimalSI} ExtraPerNode:{i:{value:5 scale:-4} d:{Dec:<nil>} s: Format:DecimalSI} Name:cpu} {Base:{i:{value:83886080 scale:0} d:{Dec:<nil>} s: Format:BinarySI} ExtraPerNode:{i:{value:8388608 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Name:memory}]
遗憾的是，pod虽然起来了，但是依然不能获取到资源指标；
由于初学，没有什么经验，网上查了一些资料，也没有解决；
上面贴出了日志，如果哪位大佬有此类经验，还望不吝赐教！
二、prometheus
metrics只能监控cpu和内存，对于其他指标如用户自定义的监控指标，metrics就无法监控到了。这时就需要另外一个组件叫prometheus；
node_exporter是agent;
PromQL相当于sql语句来查询数据;
k8s-prometheus-adapter：prometheus是不能直接解析k8s的指标的，需要借助k8s-prometheus-adapter转换成api；
kube-state-metrics是用来整合数据的；
kubernetes中prometheus的项目地址：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/prometheus
马哥的prometheus项目地址：https://github.com/ikubernetes/k8s-prom
1、部署node_exporter
[root@master metrics]# git clone https://github.com/iKubernetes/k8s-prom.git [root@master metrics]# cd k8s-prom/ [root@master k8s-prom]# ls k8s-prometheus-adapter kube-state-metrics namespace.yaml node_exporter podinfo prometheus README.md #创建一个叫prom的名称空间 [root@master k8s-prom]# kubectl apply -f namespace.yaml namespace/prom created #部署node_exporter [root@master k8s-prom]# cd node_exporter/ [root@master node_exporter]# ls node-exporter-ds.yaml node-exporter-svc.yaml [root@master node_exporter]# kubectl apply -f ./ daemonset.apps/prometheus-node-exporter created service/prometheus-node-exporter created [root@master ~]# kubectl get pods -n prom NAME READY STATUS RESTARTS AGE prometheus-node-exporter-5tfbz 1/1 Running 0 107s prometheus-node-exporter-6rl8k 1/1 Running 0 107s prometheus-node-exporter-rkx47 1/1 Running 0 107s
2、部署prometheus:
[root@master k8s-prom]# cd prometheus/ #prometheus-deploy.yaml文件中有限制使用内存的定义，如果内存不够用，可以将此规则删除； [root@master ~]# kubectl describe pods prometheus-server-76dc8df7b-75vbp -n prom 0/3 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 2 Insufficient memory. [root@master prometheus]# kubectl apply -f ./ configmap/prometheus-config created deployment.apps/prometheus-server created clusterrole.rbac.authorization.k8s.io/prometheus created serviceaccount/prometheus created clusterrolebinding.rbac.authorization.k8s.io/prometheus created service/prometheus created #查看prom名称空间下，所有资源信息 [root@master ~]# kubectl get all -n prom NAME READY STATUS RESTARTS AGE pod/prometheus-node-exporter-5tfbz 1/1 Running 0 15m pod/prometheus-node-exporter-6rl8k 1/1 Running 0 15m pod/prometheus-node-exporter-rkx47 1/1 Running 0 15m pod/prometheus-server-556b8896d6-cztlk 1/1 Running 0 3m5s #pod起来了 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/prometheus NodePort 10.99.240.192 <none> 9090:30090/TCP 9m55s service/prometheus-node-exporter ClusterIP None <none> 9100/TCP 15m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/prometheus-node-exporter 3 3 3 3 3 <none> 15m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/prometheus-server 1/1 1 1 3m5s NAME DESIRED CURRENT READY AGE replicaset.apps/prometheus-server-556b8896d6 1 1 1 3m5s
因为用的是NodePort，可以直接在集群外部访问：
浏览器输入：http://192.168.3.102:30090
192.168.3.102：为任意一个node节点的地址，不是master
生产环境应该使用pv+pvc的方式部署；
3、部署kube-state-metrics
[root@master k8s-prom]# cd kube-state-metrics/ [root@master kube-state-metrics]# ls kube-state-metrics-deploy.yaml kube-state-metrics-rbac.yaml kube-state-metrics-svc.yaml #创建，相关镜像可以去阿里云拉取，然后打标 [root@master kube-state-metrics]# kubectl apply -f ./ deployment.apps/kube-state-metrics created serviceaccount/kube-state-metrics created clusterrole.rbac.authorization.k8s.io/kube-state-metrics created clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created service/kube-state-metrics created [root@master ~]# kubectl get all -n prom NAME READY STATUS RESTARTS AGE pod/kube-state-metrics-5dbf8d5979-cc2pk 1/1 Running 0 20s pod/prometheus-node-exporter-5tfbz 1/1 Running 0 74m pod/prometheus-node-exporter-6rl8k 1/1 Running 0 74m pod/prometheus-node-exporter-rkx47 1/1 Running 0 74m pod/prometheus-server-556b8896d6-qk8jc 1/1 Running 0 48m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kube-state-metrics ClusterIP 10.98.0.63 <none> 8080/TCP 20s service/prometheus NodePort 10.111.85.219 <none> 9090:30090/TCP 48m service/prometheus-node-exporter ClusterIP None <none> 9100/TCP 74m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/prometheus-node-exporter 3 3 3 3 3 <none> 74m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kube-state-metrics 1/1 1 1 20s deployment.apps/prometheus-server 1/1 1 1 48m NAME DESIRED CURRENT READY AGE replicaset.apps/kube-state-metrics-5dbf8d5979 1 1 1 20s replicaset.apps/prometheus-server-556b8896d6 1 1 1 48m
4、部署k8s-prometheus-adapter
需要自制证书：
[root@master ~]# cd /etc/kubernetes/pki/ [root@master pki]# (umask 077; openssl genrsa -out serving.key 2048) Generating RSA private key, 2048 bit long modulus ................+++ ...+++ e is 65537 (0x10001)
创建：
#证书请求 [root@master pki]# openssl req -new -key serving.key -out serving.csr -subj "/CN=serving" #签证： [root@master pki]# openssl x509 -req -in serving.csr -CA ./ca.crt -CAkey ./ca.key -CAcreateserial -out serving.crt -days 3650 Signature ok subject=/CN=serving Getting CA Private Key [root@master k8s-prometheus-adapter]# pwd /root/manifests/metrics/k8s-prom/k8s-prometheus-adapter [root@master k8s-prometheus-adapter]# tail -n 4 custom-metrics-apiserver-deployment.yaml volumes: - name: volume-serving-cert secret: secretName: cm-adapter-serving-certs #此处写了secret的名字，所以下面创建的时候要和这里一致 #创建加密的配置文件： [root@master pki]# kubectl create secret generic cm-adapter-serving-certs --from-file=serving.crt=./serving.crt --from-file=serving.key=./serving.key -n prom secret/cm-adapter-serving-certs created [root@master pki]# kubectl get secrets -n prom NAME TYPE DATA AGE cm-adapter-serving-certs Opaque 2 23s default-token-4jlsz kubernetes.io/service-account-token 3 17h kube-state-metrics-token-klc7q kubernetes.io/service-account-token 3 16h prometheus-token-qv598 kubernetes.io/service-account-token 3 17h
部署k8s-prometheus-adapter：
#这里需要去下载最新的custom-metrics-apiserver-deployment.yaml和custom-metrics-config-map.yaml #先将现有目录中的文件移出去 [root@master k8s-prometheus-adapter]# mv custom-metrics-apiserver-deployment.yaml {,.bak} #拉取两个文件 [root@master k8s-prometheus-adapter]# wget https://raw.githubusercontent.com/DirectXMan12/k8s-prometheus-adapter/master/deploy/manifests/custom-metrics-apiserver-deployment.yaml [root@master k8s-prometheus-adapter]# wget https://raw.githubusercontent.com/DirectXMan12/k8s-prometheus-adapter/master/deploy/manifests/custom-metrics-config-map.yaml #把两个文件里面的namespace的字段值改成prom #创建 [root@master k8s-prometheus-adapter]# kubectl apply -f ./ clusterrolebinding.rbac.authorization.k8s.io/custom-metrics:system:auth-delegator created rolebinding.rbac.authorization.k8s.io/custom-metrics-auth-reader created deployment.apps/custom-metrics-apiserver created clusterrolebinding.rbac.authorization.k8s.io/custom-metrics-resource-reader created serviceaccount/custom-metrics-apiserver created service/custom-metrics-apiserver created apiservice.apiregistration.k8s.io/v1beta1.custom.metrics.k8s.io created clusterrole.rbac.authorization.k8s.io/custom-metrics-server-resources created configmap/adapter-config created clusterrole.rbac.authorization.k8s.io/custom-metrics-resource-reader created clusterrolebinding.rbac.authorization.k8s.io/hpa-controller-custom-metrics created #查看 [root@master ~]# kubectl get all -n prom NAME READY STATUS RESTARTS AGE pod/custom-metrics-apiserver-c86bfc77-6hgjh 1/1 Running 0 50s pod/kube-state-metrics-5dbf8d5979-cc2pk 1/1 Running 0 16h pod/prometheus-node-exporter-5tfbz 1/1 Running 0 18h pod/prometheus-node-exporter-6rl8k 1/1 Running 0 18h pod/prometheus-node-exporter-rkx47 1/1 Running 0 18h pod/prometheus-server-556b8896d6-qk8jc 1/1 Running 0 17h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/custom-metrics-apiserver ClusterIP 10.96.223.14 <none> 443/TCP 51s service/kube-state-metrics ClusterIP 10.98.0.63 <none> 8080/TCP 16h service/prometheus NodePort 10.111.85.219 <none> 9090:30090/TCP 17h service/prometheus-node-exporter ClusterIP None <none> 9100/TCP 18h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/prometheus-node-exporter 3 3 3 3 3 <none> 18h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/custom-metrics-apiserver 1/1 1 1 53s deployment.apps/kube-state-metrics 1/1 1 1 16h deployment.apps/prometheus-server 1/1 1 1 17h NAME DESIRED CURRENT READY AGE replicaset.apps/custom-metrics-apiserver-c86bfc77 1 1 1 52s replicaset.apps/kube-state-metrics-5dbf8d5979 1 1 1 16h replicaset.apps/prometheus-server-556b8896d6 1 1 1 17h [root@master ~]# kubectl get cm -n prom NAME DATA AGE adapter-config 1 60s prometheus-config 1 17h
可以看到资源都起来了；
#查看api [root@master ~]# kubectl api-versions |grep custom custom.metrics.k8s.io/v1beta1 #此项已经有了
5、prometheus和grafana整合
1、获取grafana.yaml
2、修改yaml文件
[root@master metrics]# vim grafana.yaml apiVersion: apps/v1 kind: Deployment metadata: name: monitoring-grafana namespace: prom #此处namespace改为prom spec: replicas: 1 selector: matchLabels: task: monitoring k8s-app: grafana template: metadata: labels: task: monitoring k8s-app: grafana spec: containers: - name: grafana image: angelnu/heapster-grafana:v5.0.4 ports: - containerPort: 3000 protocol: TCP volumeMounts: - mountPath: /etc/ssl/certs name: ca-certificates readOnly: true - mountPath: /var name: grafana-storage env: #- name: INFLUXDB_HOST #注释此行 # value: monitoring-influxdb #注释此行 - name: GF_SERVER_HTTP_PORT value: "3000" # The following env variables are required to make Grafana accessible via # the kubernetes api-server proxy. On production clusters, we recommend # removing these env variables, setup auth for grafana, and expose the grafana # service using a LoadBalancer or a public IP. - name: GF_AUTH_BASIC_ENABLED value: "false" - name: GF_AUTH_ANONYMOUS_ENABLED value: "true" - name: GF_AUTH_ANONYMOUS_ORG_ROLE value: Admin - name: GF_SERVER_ROOT_URL # If you're only using the API Server proxy, set this value instead: # value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy value: / volumes: - name: ca-certificates hostPath: path: /etc/ssl/certs - name: grafana-storage emptyDir: {} --- apiVersion: v1 kind: Service metadata: labels: # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons) # If you are NOT using this as an addon, you should comment out this line. kubernetes.io/cluster-service: 'true' kubernetes.io/name: monitoring-grafana name: monitoring-grafana namespace: prom #此处namespace改为prom spec: # In a production setup, we recommend accessing Grafana through an external Loadbalancer # or through a public IP. # type: LoadBalancer # You could also use NodePort to expose the service at a randomly-generated port # type: NodePort ports: - port: 80 targetPort: 3000 type: NodePort #添加此行 selector: k8s-app: grafana
3、创建grafana，整合Prometheus
[root@master metrics]# kubectl apply -f grafana.yaml deployment.apps/monitoring-grafana created service/monitoring-grafana created [root@master metrics]# kubectl get pods -n prom |grep grafana monitoring-grafana-8549b985b6-zghcj 1/1 Running 0 108s [root@master metrics]# kubectl get svc -n prom |grep grafana monitoring-grafana NodePort 10.101.124.148 <none> 80:31808/TCP 118s #此处为NodePort，外部直接访问31808端口
grafana运行在node02上了：
[root@master pki]# kubectl get pods -n prom -o wide |grep grafana monitoring-grafana-8549b985b6-zghcj 1/1 Running 0 27m 10.244.2.58 node02 <none> <none>
在外部浏览器打开：
[root@master ~]# kubectl get svc -n prom -o wide |grep prometheus prometheus NodePort 10.111.85.219 <none> 9090:30090/TCP 41h app=prometheus,component=server
然后修改框住的内容：
以上通过以后，点击“Dashboards”，将三个模板都导入；
如下图，已经有些监控数据了：
也可以去下载一些模板：
https://grafana.com/dashboards
https://grafana.com/dashboards?dataSource=prometheus&search=kubernetes
然后导入：
三、HPA（水平pod自动扩展）
(1)
Horizontal Pod Autoscaling可以根据CPU利用率(内存为不可压缩资源)自动伸缩一个Replication Controller、Deployment 或者Replica Set中的Pod数量；
目前HPA只支持两个版本，其中v1版本只支持核心指标的定义；
[root@master ~]# kubectl api-versions |grep autoscaling autoscaling/v1 autoscaling/v2beta1 autoscaling/v2beta2
(2)下面我们用命令行的方式创建一个带有资源限制的pod
[root@master ~]# kubectl run myapp --image=ikubernetes/myapp:v1 --replicas=1 --requests='cpu=50m,memory=256Mi' --limits='cpu=50m,memory=256Mi' --labels='app=myapp' --expose --port=80 kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. service/myapp created deployment.apps/myapp created [root@master ~]# kubectl get pods NAME READY STATUS RESTARTS AGE myapp-657fb86dd-nkhhx 1/1 Running 0 56s
(3)下面我们让myapp 这个pod能自动水平扩展，用kubectl autoscale，其实就是创建HPA控制器的；
#查看帮助 [root@master ~]# kubectl autoscale -h #创建 [root@master ~]# kubectl autoscale deployment myapp --min=1 --max=8 --cpu-percent=60 horizontalpodautoscaler.autoscaling/myapp autoscaled --min：表示最小扩展pod的个数 --max：表示最多扩展pod的个数 --cpu-percent：cpu利用率 #查看hpa [root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE myapp Deployment/myapp 0%/60% 1 8 1 64s [root@master ~]# kubectl get svc |grep myapp myapp ClusterIP 10.107.17.18 <none> 80/TCP 7m46s #把service改成NodePort的方式： [root@master ~]# kubectl patch svc myapp -p '{"spec":{"type": "NodePort"}}' service/myapp patched [root@master ~]# kubectl get svc |grep myapp myapp NodePort 10.107.17.18 <none> 80:31043/TCP 9m
接着可以对pod进行压测，看看pod会不会扩容：
#安装ab压测工具 [root@master ~]# yum -y install httpd-tools #压测 [root@master ~]# ab -c 1000 -n 50000000 http://192.168.3.100:31043/index.html #压测的同时，可以看到pods的cpu利用率为102%，需要扩展为2个pod了： [root@master ~]# kubectl describe hpa |grep -A 3 "resource cpu" resource cpu on pods (as a percentage of request): 102% (51m) / 60% Min replicas: 1 Max replicas: 8 Deployment pods: 1 current / 2 desired #已经扩展为两个pod了 [root@master ~]# kubectl get pods NAME READY STATUS RESTARTS AGE myapp-657fb86dd-k4jdg 1/1 Running 0 62s myapp-657fb86dd-nkhhx 1/1 Running 0 110m #等压测完，cpu使用率降下来，pod数量还会自动恢复为1个，如下 [root@master ~]# kubectl describe hpa |grep -A 3 "resource cpu" resource cpu on pods (as a percentage of request): 0% (0) / 60% Min replicas: 1 Max replicas: 8 Deployment pods: 1 current / 1 desired [root@master ~]# kubectl get pods NAME READY STATUS RESTARTS AGE myapp-657fb86dd-nkhhx 1/1 Running 0 116m #但是如果cpu使用率还是一直上升，pod数量会扩展的更多
(4)hpa v2
上面用的是hpav1来做的水平pod自动扩展的功能，hpa v1版本只能根据cpu利用率括水平自动扩展pod。
接下来我们看一下hpa v2的功能，它可以根据自定义指标利用率来水平扩展pod。
#删除刚才的hpa [root@master ~]# kubectl delete hpa myapp horizontalpodautoscaler.autoscaling "myapp" deleted #hpa-v2资源定义清单 [root@master hpav2]# vim hpa-v2-demo.yaml apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: myapp-hpa-v2 spec: scaleTargetRef: #根据什么指标来做评估压力 apiVersion: apps/v1 #对谁来做自动扩展 kind: Deployment name: myapp minReplicas: 1 #最少副本数量 maxReplicas: 10 #最多副本数量 metrics: #表示依据哪些指标来进行评估 - type: Resource #表示基于资源进行评估 resource: name: cpu target: type: Utilization averageUtilization: 50 #pod cpu使用率超过55%，就自动水平扩展pod个数 #创建 [root@master hpav2]# kubectl apply -f hpa-v2-demo.yaml horizontalpodautoscaler.autoscaling/myapp-hpa-v2 created [root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE myapp-hpa-v2 Deployment/myapp <unknown>/50% 1 10 0 9s
接着可以对pod进行压测，看看pod会不会扩容：
[root@master hpav2]# kubectl get pods NAME READY STATUS RESTARTS AGE myapp-657fb86dd-nkhhx 1/1 Running 0 3h16m #压测 [root@master ~]# ab -c 1000 -n 80000000 http://192.168.3.100:31043/index.html #看到cpu使用率已经到了100% [root@master ~]# kubectl describe hpa |grep -A 3 "resource cpu" resource cpu on pods (as a percentage of request): 100% (50m) / 50% Min replicas: 1 Max replicas: 10 Deployment pods: 1 current / 2 desired #pod已经自动扩容为两个了 [root@master hpav2]# kubectl get pods NAME READY STATUS RESTARTS AGE myapp-657fb86dd-fkdxq 1/1 Running 0 27s myapp-657fb86dd-nkhhx 1/1 Running 0 3h19m #等压测结束后，资源使用正常一段时间后，pod个数还会收缩为正常个数；
(5)hpa v2可以根据cpu和内存使用率进行伸缩Pod个数，还可以根据其他参数进行pod处理，如http并发量
[root@master hpa]# vimt hpa-v2-custom.yaml apiVersion: autoscaling/v2beta2 #从这可以看出是hpa v2版本 kind: HorizontalPodAutoscaler metadata: name: myapp-hpa-v2 spec: scaleTargetRef: #根据什么指标来做评估压力 apiVersion: apps/v1 #对谁来做自动扩展 kind: Deployment name: myapp minReplicas: 1 #最少副本数量 maxReplicas: 10 metrics: #表示依据哪些指标来进行评估 - type: Pods #表示基于资源进行评估 pods: metricName: http_requests #自定义的资源指标 targetAverageValue: 800m #m表示个数,表示并发数800
hpa-v2版本的，有需要以后可以深入学习一下；
