Kubernetes HPA控制器
一、pod扩容
1.1、pod自动升缩
- 根据当前pod的负载,动态调整 pod副本数量,业务高峰期自动扩容pod的副本数以尽快响应pod的请求。
- 在业务低峰期对pod进行缩容,实现降本增效的目的。
- 公有云支持node级别的弹性伸缩。
1.2、手动调整pod副本数
[root@easzlab-deploy ~]# kubectl get pod -n magedu NAME READY STATUS RESTARTS AGE magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 47h [root@easzlab-deploy ~]# [root@easzlab-deploy ~]# kubectl --help |grep scale #查看命令帮助 scale Set a new size for a deployment, replica set, or replication controller autoscale Auto-scale a deployment, replica set, stateful set, or replication controller [root@easzlab-deploy ~]# [root@easzlab-deploy ~]# kubectl get deployment -n magedu NAME READY UP-TO-DATE AVAILABLE AGE magedu-jenkins-deployment 1/1 1 1 47h [root@easzlab-deploy ~]# [root@easzlab-deploy ~]# kubectl scale deployment magedu-jenkins-deployment --replicas=2 -n magedu deployment.apps/magedu-jenkins-deployment scaled [root@easzlab-deploy ~]# [root@easzlab-deploy ~]# kubectl get deployment -n magedu NAME READY UP-TO-DATE AVAILABLE AGE magedu-jenkins-deployment 1/2 2 1 47h [root@easzlab-deploy ~]# kubectl get deployment -n magedu NAME READY UP-TO-DATE AVAILABLE AGE magedu-jenkins-deployment 2/2 2 2 47h [root@easzlab-deploy ~]# [root@easzlab-deploy ~]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 47h 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 92s 10.200.105.177 172.16.88.164 <none> <none> [root@easzlab-deploy ~]#
1.3、动态伸缩控制器类型
水平pod自动缩放器(HPA):
- 基于pod 资源利用率横向调整pod副本数量。
垂直pod自动缩放器(VPA):
- 基于pod资源利用率, 调整对单个pod的最大资源限制, 不能与HPA同时使用。
集群伸缩(Cluster Autoscaler,CA)
- 基于集群中node 资源使用情况, 动态伸缩node节点, 从而保证有CPU和内存资源用于创建pod。
二、HPA控制器
2.1、HPA控制器简介
Horizontal Pod Autoscaling (HPA)控制器, 根据预定义好的阈值及pod当前的资源利用率, 自动控制在k8s集群中运行的pod数量(自动弹性水平自动伸缩).
--horizontal-pod-autoscaler-sync-period #默认每隔15s(可以通过– horizontal-pod-autoscaler-sync-period修改)查询metrics的资源使用情况。
--horizontal-pod-autoscaler-downscale-stabilization #缩容间隔周期, 默认5分钟。
--horizontal-pod-autoscaler-sync-period #HPA控制器同步pod副本数的间隔周期
--horizontal-pod-autoscaler-cpu-initialization-period #初始化延迟时间, 在此时间内 pod的CPU 资源指标将不会生效, 默认为5分钟。
--horizontal-pod-autoscaler-initial-readiness-delay #用于设置 pod 准备时间, 在此时间内的 pod 统统被认为未就绪及不采集数据,默认为30秒。
--horizontal-pod-autoscaler-tolerance #HPA控制器能容忍的数据差异(浮点数, 默认为0.1), 即新的指标要与当前的阈值差异在0.1或以上,即要大于1+0.1=1.1, 或小于1-0.1=0.9, 比如阈值为CPU利用率50%,当前为80%,那么80/50=1.6 > 1.1则会触发扩容, 反之会缩容。即触发条件: avg(CurrentPodsConsumption) / Target >1.1 或 <0.9=把N个pod的数据相加后根据pod的数量计算出平均数除以阈值,大于1.1就扩容, 小于0.9就缩容。
计算公式: TargetNumOfPods = ceil(sum(CurrentPodsCPUUtilization) / Target) #ceil是一个向上取整的目的pod整数。
指标数据需要部署metrics-server, 即HPA使用metrics-server作为数据源。
https://github.com/kubernetes-sigs/metrics-server
在k8s 1.1引入HPA控制器, 早期使用Heapster组件采集pod指标数据, 在k8s 1.11版本开始使用Metrices Server完成数据采集, 然后将采集到的数据通过API(Aggregated API, 汇总API) , 例如metrics.k8s.io、 custom.metrics.k8s.io、 external.metrics.k8s.io, 然后再把数据提供给HPA控制器进行查询, 以实现基于某个资源利用率对pod进行扩缩容的目的。
2.2、metrics-server 部署
Metrics Server 是 Kubernetes 内置的容器资源指标来源。
Metrics Server 从node节点上的 Kubelet 收集资源指标,并通过Metrics API在 Kubernetes apiserver 中公开指标数据,以供Horizontal Pod Autoscaler和Vertical Pod Autoscaler使用,也可以通过访问kubectl top node/pod 查看指标数据。
[root@easzlab-deploy kube-state-metrics]# wget https://github.com/kubernetes-sigs/metrics-server/releases/download/metrics-server-helm-chart-3.8.2/components.yaml [root@easzlab-deploy kube-state-metrics]# kubectl apply -f components.yaml [root@easzlab-deploy kube-state-metrics]# kubectl get pod -A |grep metrics kube-system metrics-server-686ff776cf-dbgq6 1/1 Running 4 (5d16h ago) 6d16h kubernetes-dashboard dashboard-metrics-scraper-8c47d4b5d-5cxhb 1/1 Running 2 (5d16h ago) 6d18h [root@easzlab-deploy kube-state-metrics]# [root@easzlab-deploy kube-state-metrics]# kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 172.16.88.157 311m 15% 1825Mi 50% 172.16.88.158 341m 17% 1835Mi 50% 172.16.88.159 295m 14% 1689Mi 46% 172.16.88.160 242m 6% 1543Mi 20% 172.16.88.161 291m 7% 2164Mi 28% 172.16.88.162 329m 8% 1549Mi 20% 172.16.88.163 298m 7% 2087Mi 27% 172.16.88.164 352m 8% 2110Mi 27% 172.16.88.165 316m 7% 1677Mi 21% [root@easzlab-deploy kube-state-metrics]# kubectl top pod -n magedu NAME CPU(cores) MEMORY(bytes) magedu-jenkins-deployment-79bbd88cb7-84999 4m 521Mi magedu-jenkins-deployment-79bbd88cb7-pmd9z 4m 280Mi [root@easzlab-deploy kube-state-metrics]#
2.3、HPA控制器配置简要说明
apiVersion: autoscaling/v2beta1 #定义API版本 kind: HorizontalPodAutoscaler #对象类型 metadata: #定义对象元数据 namespace: linux36 #创建后隶属的namespace name: linux36-tomcat-app1-podautoscaler #对象名称 labels: 这样的label标签 app: linux36-tomcat-app1 #自定义的label名称 version: v2beta1 #自定义的api版本 spec: #定义对象具体信息 scaleTargetRef: #定义水平伸缩的目标对象, Deployment、 ReplicationController/ReplicaSet apiVersion: apps/v1 #API版本, HorizontalPodAutoscaler.spec.scaleTargetRef.apiVersion kind: Deployment #目标对象类型为deployment name: linux36-tomcat-app1-deployment #deployment 的具体名称 minReplicas: 2 #最小pod数 maxReplicas: 5 #最大pod数 metrics: #调用metrics数据定义 - type: Resource #类型为资源 resource: #定义资源 name: cpu #资源名称为cpu targetAverageUtilization: 80 #CPU使用率 - type: Resource #类型为资源 resource: #定义资源 name: memory #资源名称为memory targetAverageValue: 1024Mi #memory使用率
2.4、web服务与HPA控制器部署
创建tomcat部署文件
[root@easzlab-deploy tomcat-app1]# vi tomcat-app1.yaml [root@easzlab-deploy tomcat-app1]# cat tomcat-app1.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app1-deployment-label name: magedu-tomcat-app1-deployment namespace: magedu spec: replicas: 5 selector: matchLabels: app: magedu-tomcat-app1-selector template: metadata: labels: app: magedu-tomcat-app1-selector spec: containers: - name: magedu-tomcat-app1-container image: harbor.magedu.net/app2/tomcat-app1:v1 #command: ["/apps/tomcat/bin/run_tomcat.sh"] imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http env: - name: "password" value: "123456" - name: "age" value: "18" resources: limits: cpu: 1 memory: "512Mi" requests: cpu: 500m memory: "512Mi" volumeMounts: - name: magedu-images mountPath: /usr/local/nginx/html/webapp/images readOnly: false - name: magedu-static mountPath: /usr/local/nginx/html/webapp/static readOnly: false volumes: - name: magedu-images nfs: server: 172.16.88.169 path: /data/k8sdata/magedu/images - name: magedu-static nfs: server: 172.16.88.169 path: /data/k8sdata/magedu/static # nodeSelector: # project: magedu # app: tomcat --- kind: Service apiVersion: v1 metadata: labels: app: magedu-tomcat-app1-service-label name: magedu-tomcat-app1-service namespace: magedu spec: type: NodePort ports: - name: http port: 80 protocol: TCP targetPort: 8080 nodePort: 30092 selector: app: magedu-tomcat-app1-selector [root@easzlab-deploy tomcat-app1]#
部署业务镜像
[root@easzlab-deploy tomcat-app1]# kubectl apply -f tomcat-app1.yaml deployment.apps/magedu-tomcat-app1-deployment created service/magedu-tomcat-app1-service created [root@easzlab-deploy tomcat-app1]# vi hpa.yaml [root@easzlab-deploy tomcat-app1]# cat hpa.yaml #apiVersion: autoscaling/v2beta1 apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: namespace: magedu name: magedu-tomcat-app1-podautoscaler labels: app: magedu-tomcat-app1 version: v2beta1 spec: scaleTargetRef: apiVersion: apps/v1 #apiVersion: extensions/v1beta1 kind: Deployment name: magedu-tomcat-app1-deployment minReplicas: 2 maxReplicas: 20 targetCPUUtilizationPercentage: 60 #metrics: #- type: Resource # resource: # name: cpu # targetAverageUtilization: 60 #- type: Resource # resource: # name: memory [root@easzlab-deploy tomcat-app1]# kubectl apply -f hpa.yaml horizontalpodautoscaler.autoscaling/magedu-tomcat-app1-podautoscaler created [root@easzlab-deploy tomcat-app1]# [root@easzlab-deploy tomcat-app1]# kubectl describe hpa magedu-tomcat-app1-podautoscaler -n magedu Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler Name: magedu-tomcat-app1-podautoscaler Namespace: magedu Labels: app=magedu-tomcat-app1 version=v2beta1 Annotations: <none> CreationTimestamp: Tue, 25 Oct 2022 17:36:34 +0800 Reference: Deployment/magedu-tomcat-app1-deployment Metrics: ( current / target ) resource cpu on pods (as a percentage of request): 0% (4m) / 60% Min replicas: 2 Max replicas: 20 Deployment pods: 5 current / 5 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited False DesiredWithinRange the desired count is within the acceptable range Events: <none> [root@easzlab-deploy tomcat-app1]# kubectl get hpa -n magedu NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE magedu-tomcat-app1-podautoscaler Deployment/magedu-tomcat-app1-deployment 0%/60% 2 20 5 28s [root@easzlab-deploy tomcat-app1]#
三、HPA控制器弹性伸缩测试
3.1、自动缩容
[root@easzlab-deploy tomcat-app1]# kubectl describe deployments.apps magedu-tomcat-app1-deployment -n magedu Name: magedu-tomcat-app1-deployment Namespace: magedu CreationTimestamp: Tue, 25 Oct 2022 17:35:54 +0800 Labels: app=magedu-tomcat-app1-deployment-label Annotations: deployment.kubernetes.io/revision: 1 Selector: app=magedu-tomcat-app1-selector Replicas: 5 desired | 5 updated | 5 total | 5 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app=magedu-tomcat-app1-selector Containers: magedu-tomcat-app1-container: Image: harbor.magedu.net/app2/tomcat-app1:v1 Port: 8080/TCP Host Port: 0/TCP Limits: cpu: 1 memory: 512Mi Requests: cpu: 500m memory: 512Mi Environment: password: 123456 age: 18 Mounts: /usr/local/nginx/html/webapp/images from magedu-images (rw) /usr/local/nginx/html/webapp/static from magedu-static (rw) Volumes: magedu-images: Type: NFS (an NFS mount that lasts the lifetime of a pod) Server: 172.16.88.169 Path: /data/k8sdata/magedu/images ReadOnly: false magedu-static: Type: NFS (an NFS mount that lasts the lifetime of a pod) Server: 172.16.88.169 Path: /data/k8sdata/magedu/static ReadOnly: false Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: <none> NewReplicaSet: magedu-tomcat-app1-deployment-8545c4bd5d (5/5 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 2m41s deployment-controller Scaled up replica set magedu-tomcat-app1-deployment-8545c4bd5d to 5 [root@easzlab-deploy tomcat-app1]#
经观察,五分钟之后会自动缩容
describe deployment 可以看到详细过程
[root@easzlab-deploy RBAC]# kubectl describe deployment -n magedu magedu-tomcat-app1-deployment Name: magedu-tomcat-app1-deployment Namespace: magedu CreationTimestamp: Tue, 25 Oct 2022 17:35:54 +0800 Labels: app=magedu-tomcat-app1-deployment-label Annotations: deployment.kubernetes.io/revision: 1 Selector: app=magedu-tomcat-app1-selector Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app=magedu-tomcat-app1-selector Containers: magedu-tomcat-app1-container: Image: harbor.magedu.net/app2/tomcat-app1:v1 Port: 8080/TCP Host Port: 0/TCP Limits: cpu: 1 memory: 512Mi Requests: cpu: 500m memory: 512Mi Environment: password: 123456 age: 18 Mounts: /usr/local/nginx/html/webapp/images from magedu-images (rw) /usr/local/nginx/html/webapp/static from magedu-static (rw) Volumes: magedu-images: Type: NFS (an NFS mount that lasts the lifetime of a pod) Server: 172.16.88.169 Path: /data/k8sdata/magedu/images ReadOnly: false magedu-static: Type: NFS (an NFS mount that lasts the lifetime of a pod) Server: 172.16.88.169 Path: /data/k8sdata/magedu/static ReadOnly: false Conditions: Type Status Reason ---- ------ ------ Progressing True NewReplicaSetAvailable Available True MinimumReplicasAvailable OldReplicaSets: <none> NewReplicaSet: magedu-tomcat-app1-deployment-8545c4bd5d (2/2 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 10m (x2 over 70m) deployment-controller Scaled up replica set magedu-tomcat-app1-deployment-8545c4bd5d to 5 Normal ScalingReplicaSet 5m14s (x2 over 64m) deployment-controller Scaled down replica set magedu-tomcat-app1-deployment-8545c4bd5d to 2 [root@easzlab-deploy RBAC]#
3.2、自动扩容
[root@easzlab-deploy tomcat-app1]# vi tomcat-app1.yaml [root@easzlab-deploy tomcat-app1]# cat tomcat-app1.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app1-deployment-label name: magedu-tomcat-app1-deployment namespace: magedu spec: replicas: 2 selector: matchLabels: app: magedu-tomcat-app1-selector template: metadata: labels: app: magedu-tomcat-app1-selector spec: containers: - name: magedu-tomcat-app1-container #image: harbor.magedu.net/app2/tomcat-app1:v1 image: lorel/docker-stress-ng args: ["--vm","2","--vm-bytes","256M"] #command: ["/apps/tomcat/bin/run_tomcat.sh"] imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http env: - name: "password" value: "123456" - name: "age" value: "18" resources: limits: cpu: 1 memory: "512Mi" requests: cpu: 500m memory: "512Mi" volumeMounts: - name: magedu-images mountPath: /usr/local/nginx/html/webapp/images readOnly: false - name: magedu-static mountPath: /usr/local/nginx/html/webapp/static readOnly: false volumes: - name: magedu-images nfs: server: 172.16.88.169 path: /data/k8sdata/magedu/images - name: magedu-static nfs: server: 172.16.88.169 path: /data/k8sdata/magedu/static # nodeSelector: # project: magedu # app: tomcat --- kind: Service apiVersion: v1 metadata: labels: app: magedu-tomcat-app1-service-label name: magedu-tomcat-app1-service namespace: magedu spec: type: NodePort ports: - name: http port: 80 protocol: TCP targetPort: 8080 nodePort: 30092 selector: app: magedu-tomcat-app1-selector [root@easzlab-deploy tomcat-app1]#
重新部署加载
[root@easzlab-deploy tomcat-app1]# vi hpa.yaml [root@easzlab-deploy tomcat-app1]# [root@easzlab-deploy tomcat-app1]# cat hpa.yaml #apiVersion: autoscaling/v2beta1 apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: namespace: magedu name: magedu-tomcat-app1-podautoscaler labels: app: magedu-tomcat-app1 version: v2beta1 spec: scaleTargetRef: apiVersion: apps/v1 #apiVersion: extensions/v1beta1 kind: Deployment name: magedu-tomcat-app1-deployment minReplicas: 3 maxReplicas: 10 targetCPUUtilizationPercentage: 60 #metrics: #- type: Resource # resource: # name: cpu # targetAverageUtilization: 60 #- type: Resource # resource: # name: memory [root@easzlab-deploy tomcat-app1]# kubectl delete -f tomcat-app1.yaml [root@easzlab-deploy tomcat-app1]# kubectl get pod -n magedu NAME READY STATUS RESTARTS AGE magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 2d4h magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 4h39m [root@easzlab-deploy tomcat-app1]# [root@easzlab-deploy tomcat-app1]# kubectl apply -f tomcat-app1.yaml deployment.apps/magedu-tomcat-app1-deployment created service/magedu-tomcat-app1-service created [root@easzlab-deploy tomcat-app1]# kubectl apply -f hpa.yaml horizontalpodautoscaler.autoscaling/magedu-tomcat-app1-podautoscaler created [root@easzlab-deploy tomcat-app1]#
验证自动扩容
[root@easzlab-deploy tomcat-app1]# kubectl describe deployment -n magedu magedu-tomcat-app1-deployment Name: magedu-tomcat-app1-deployment Namespace: magedu CreationTimestamp: Tue, 25 Oct 2022 19:07:41 +0800 Labels: app=magedu-tomcat-app1-deployment-label Annotations: deployment.kubernetes.io/revision: 1 Selector: app=magedu-tomcat-app1-selector Replicas: 10 desired | 10 updated | 10 total | 10 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app=magedu-tomcat-app1-selector Containers: magedu-tomcat-app1-container: Image: lorel/docker-stress-ng Port: 8080/TCP Host Port: 0/TCP Args: --vm 2 --vm-bytes 256M Limits: cpu: 1 memory: 512Mi Requests: cpu: 500m memory: 512Mi Environment: password: 123456 age: 18 Mounts: /usr/local/nginx/html/webapp/images from magedu-images (rw) /usr/local/nginx/html/webapp/static from magedu-static (rw) Volumes: magedu-images: Type: NFS (an NFS mount that lasts the lifetime of a pod) Server: 172.16.88.169 Path: /data/k8sdata/magedu/images ReadOnly: false magedu-static: Type: NFS (an NFS mount that lasts the lifetime of a pod) Server: 172.16.88.169 Path: /data/k8sdata/magedu/static ReadOnly: false Conditions: Type Status Reason ---- ------ ------ Progressing True NewReplicaSetAvailable Available True MinimumReplicasAvailable OldReplicaSets: <none> NewReplicaSet: magedu-tomcat-app1-deployment-7dd7df4f4d (10/10 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 3m56s deployment-controller Scaled up replica set magedu-tomcat-app1-deployment-7dd7df4f4d to 2 Normal ScalingReplicaSet 3m33s deployment-controller Scaled up replica set magedu-tomcat-app1-deployment-7dd7df4f4d to 3 Normal ScalingReplicaSet 3m18s deployment-controller Scaled up replica set magedu-tomcat-app1-deployment-7dd7df4f4d to 4 Normal ScalingReplicaSet 3m2s deployment-controller Scaled up replica set magedu-tomcat-app1-deployment-7dd7df4f4d to 7 Normal ScalingReplicaSet 2m32s deployment-controller Scaled up replica set magedu-tomcat-app1-deployment-7dd7df4f4d to 10 [root@easzlab-deploy tomcat-app1]#