kubernetes(24)：基于metrics-server水平扩展-资源HPA
k8基于metrics server水平扩展-HPA
https://blog.51cto.com/ylw6006/2113848
https://blog.51cto.com/ylw6006/2114338
https://www.jianshu.com/p/9abb697cd833
1 HPA简介
HPA全称Horizontal Pod Autoscaling，即pod的水平自动扩展。
自动扩展主要分为两种，其一为水平扩展，针对于实例数目的增减；其二为垂直扩展，即单个实例可以使用的资源的增减。HPA属于前者。
云计算具有水平弹性的特性，这个是云计算区别于传统IT技术架构的主要特性。对于Kubernetes中的POD集群来说，HPA可以实现很多自动化功能，比如当POD中业务负载上升的时候，可以创建新的POD来保证业务系统稳定运行，当POD中业务负载下降的时候，可以销毁POD来提高资源利用率。
工作方式
HPA的操作对象是RC、RS或Deployment对应的Pod
根据观察到的CPU等实际使用量与用户的期望值进行比对，做出是否需要增减实例数量的决策。
Horizontal Pod Autoscaling可以根据CPU使用率或应用自定义metrics自动扩展Pod数量（支持replication controller、deployment和replica set）。
- 控制管理器每隔30s（可以通过–horizontal-pod-autoscaler-sync-period修改）查询metrics的资源使用情况
- 支持三种metrics类型
- 预定义metrics（比如Pod的CPU）以利用率的方式计算
- 自定义的Pod metrics，以原始值（raw value）的方式计算
- 自定义的object metrics
- 支持两种metrics查询方式：Heapster和自定义的REST API（Heapster 1.13版本开始废弃）
- 支持多metrics （建议metrics server）
客户端;
通过kubectl创建一个horizontalPodAutoscaler对象，并存储到etcd中
服务端：
api server:负责接受创建hpa对象，然后存入etcd
hpa controler和其他的controler类似，每30s同步一次，将已经创建的hpa进行一次管理（从heapster获取监控数据，查看是否需要scale, controler的store中就保存着从始至终创建出来的hpa，当做一个缓存），watch hpa有变化也会运行。从heapster中获取scale数据，和hpa对比，计算cup利用率等信息，然后重新调整scale。根据hpa.Spec.ScaleTargetRef.Kind（例如Deployment，然后deployment控制器在调整pod数量），调整其值，发送到apiserver存储到etcd，然后更新hpa到etcd.
2 HPA控制副本数量—Nginx
2.1 nginx-deployment.yaml
# cat nginx-deployment.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.15 ports: - containerPort: 80
2.2 nginx-svc.yaml
# cat nginx-svc.yaml apiVersion: v1 kind: Service metadata: name: nginx spec: type: NodePort ports: - port: 80 nodePort: 30023 selector: #标签选择器 app: nginx
2.3 启动Nginx服务
[root@k8s-master nginx]# kubectl create -f nginx-deployment.yaml deployment "nginx-deployment" created [root@k8s-master nginx]# kubectl create -f nginx-svc.yaml service "nginx" created [root@k8s-master nginx]#
2.4 查看启动状态
我们能看到3个pod节点
[root@k8s-master k8s]# kubectl get deployments.apps NAME READY UP-TO-DATE AVAILABLE AGE nginx-deployment 3/3 3 3 109s [root@k8s-master k8s]# [root@k8s-master k8s]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-deployment-6f6d9b887f-57dvw 1/1 Running 0 119s 10.254.1.15 k8s-node-1 <none> <none> nginx-deployment-6f6d9b887f-bk5nl 1/1 Running 0 119s 10.254.2.14 k8s-node-2 <none> <none> nginx-deployment-6f6d9b887f-lbr27 1/1 Running 0 119s 10.254.1.16 k8s-node-1 <none> <none> [root@k8s-master k8s]#
2.5 通过hpa管理副本数量
[root@k8s-master nginx]# kubectl autoscale deployment nginx-deployment --max=8 --min=2 --cpu-percent=80 deployment "nginx-deployment" autoscaled root@k8s-master k8s]# kubectl get all -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/nginx-deployment-6f6d9b887f-57dvw 1/1 Running 0 4m40s 10.254.1.15 k8s-node-1 <none> <none> pod/nginx-deployment-6f6d9b887f-bk5nl 1/1 Running 0 4m40s 10.254.2.14 k8s-node-2 <none> <none> pod/nginx-deployment-6f6d9b887f-lbr27 1/1 Running 0 4m40s 10.254.1.16 k8s-node-1 <none> <none> NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5h34m <none> service/nginx NodePort 10.105.15.44 <none> 80:30023/TCP 4m24s app=nginx NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/nginx-deployment 3/3 3 3 4m40s nginx nginx:1.15 app=nginx NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/nginx-deployment-6f6d9b887f 3 3 3 4m40s nginx nginx:1.15 app=nginx,pod-template-hash=6f6d9b887f NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE horizontalpodautoscaler.autoscaling/nginx-deployment Deployment/nginx-deployment <unknown>/80% 2 8 3 81s
#我们手动把pod数量设置为1，但HPA设置最少是2个，最大8个。HPA直接拉起副本
[root@k8s-master nginx]# kubectl scale deployment nginx-deployment --replicas=1 deployment "nginx-deployment" scaled [root@k8s-master nginx]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-deployment-4019830974-01j1n 1/1 Running 0 14m 172.16.5.2 k8s-node-1 nginx-deployment-4019830974-2x9j4 1/1 Terminating 0 14m 172.16.96.2 k8s-node-2 nginx-deployment-4019830974-pxvb9 1/1 Terminating 0 14m 172.16.96.3 k8s-node-2 [root@k8s-master nginx]# [root@k8s-master nginx]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-deployment-4019830974-01j1n 1/1 Running 0 14m 172.16.5.2 k8s-node-1 nginx-deployment-4019830974-2x9j4 1/1 Terminating 0 14m 172.16.96.2 k8s-node-2 nginx-deployment-4019830974-pxvb9 1/1 Terminating 0 14m 172.16.96.3 k8s-node-2 [root@k8s-master nginx]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-deployment-4019830974-01j1n 1/1 Running 0 14m 172.16.5.2 k8s-node-1 [root@k8s-master nginx]# [root@k8s-master nginx]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-deployment-4019830974-01j1n 1/1 Running 0 14m 172.16.5.2 k8s-node-1 nginx-deployment-4019830974-bg35z 0/1 ContainerCreating 0 2s <none> k8s-node-2 [root@k8s-master nginx]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-deployment-4019830974-01j1n 1/1 Running 0 14m 172.16.5.2 k8s-node-1 nginx-deployment-4019830974-bg35z 1/1 Running 0 7s 172.16.96.2 k8s-node-2
2.6 通过配置文件定义HPA
# cat nginx_hpa_cpu.yaml apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: nginx-hpa namespace: default spec: minReplicas: 2 maxReplicas: 6 scaleTargetRef: kind: Deployment name: nginx-deployment targetCPUUtilizationPercentage: 80
[root@k8s-master nginx]# kubectl delete hpa nginx-deployment horizontalpodautoscaler "nginx-deployment" deleted [root@k8s-master nginx]# kubectl get hpa No resources found. [root@k8s-master nginx]# kubectl create -f nginx_hpa_cpu.yaml horizontalpodautoscaler "nginx-hpa" created [root@k8s-master nginx]# kubectl get hpa NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE nginx-hpa Deployment/nginx-deployment 80% <waiting> 2 6 3s [root@k8s-master nginx]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-4019830974-01j1n 1/1 Running 0 1h nginx-deployment-4019830974-g81hm 1/1 Running 0 20m [root@k8s-master nginx]# kubectl scale deployment nginx-deployment --replicas=1 deployment "nginx-deployment" scaled [root@k8s-master nginx]# [root@k8s-master nginx]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-4019830974-01j1n 1/1 Running 0 1h [root@k8s-master nginx]# [root@k8s-master nginx]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-4019830974-01j1n 1/1 Running 0 1h nginx-deployment-4019830974-pf24j 1/1 Running 0 9s [root@k8s-master nginx]#
3 基于metrics server的HPA测试
heapster废弃不是主要原因，主要是镜像下载不了，啊哈哈哈哈！
3.1 删除heapster
避免干扰，没部署成功也就不存在了
3.2 部署metrics-server
参考metrics-server安装
kubernetes(23)：安装部署资源收集插件metrics-server
3.3 测试HPA
https://blog.csdn.net/chenleiking/article/details/80197975
创建一个关联资源nginx的HPA，最小的pod副本数为2，最大为10。HPA会根据设定的cpu使用率（10%）动态的增加或者减少pod数量，此地方用于测试，所以设定的伸缩阈值会比较小；
使用curl方式加大、减小nginx的负载
资源使用方面，请参考资源限制和QoS
# cat nginx-deployment.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.15 ports: - containerPort: 80 resources: requests: memory: "10Mi" cpu: "10m" limits: memory: "64Mi" cpu: "40m"
配置参考 kubernetes(22)：k8s资源限制和服务质量QoS
#limits.cpu <==> --cpu-quota # docker inspect中的CpuQuota值
##requests.cpu <==> --cpu-shares # docker inspect中的CpuShares值
##requests: cpu: "1"
##limits: cpu: "2"
## "CpuShares": 1024,
## "CpuQuota": 200000,
##limits.cpu: 2 对应docker中"CpuQuota": 200000, "CpuPeriod": 100000默认值
##requests.cpu：1对应docker中"CpuShares": 1024,
##limits.cpu：1并不一定表示容器只会占用1个CPU，而表示的是容器最多可以使用的CPU时间的比例。 例如我4核 设置成1 如果docker CPU跑满 那就是4 * 25%
# cat nginx-svc.yaml apiVersion: v1 kind: Service metadata: name: nginx spec: type: NodePort ports: - port: 80 nodePort: 30023 selector: #标签选择器 app: nginx
# cat hpa.yaml apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: nginx-hpa spec: scaleTargetRef: apiVersion: extensions/v1beta1 kind: Deployment name: nginx-deployment minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: memory targetAverageValue: 20Mi - type: Resource resource: name: cpu targetAverageUtilization: 10
初试状态
我们设置的 minReplicas: 2，现在没人访问kubectl就把副本调都2
[root@k8s-master k8s]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx-deployment 2818048/20Mi, 0%/10% 2 10 2 67m [root@k8s-master k8s]# kubectl top pod NAME CPU(cores) MEMORY(bytes) nginx-deployment-7bb78b8c59-jqwjh 0m 2Mi nginx-deployment-7bb78b8c59-v9986 0m 2Mi [root@k8s-master k8s]#
压测过程
for i in {1..10000}; do curl http://10.6.76.24:30023/ > /dev/null -s; done;
随着CPU和内存的使用提高，kubectl把副本数慢慢加上去，最后到10个
[root@k8s-master k8s]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx-deployment 2963456/20Mi, 0%/10% 2 10 2 72m [root@k8s-master k8s]# kubectl apply -f nginx-deployment.yaml deployment.extensions/nginx-deployment configured [root@k8s-master k8s]# kubectl apply -f hpa.yaml horizontalpodautoscaler.autoscaling/nginx-hpa unchanged [root@k8s-master k8s]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx-deployment 2963456/20Mi, 0%/10% 2 10 2 73m [root@k8s-master k8s]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx-deployment 2963456/20Mi, 0%/10% 2 10 2 73m [root@k8s-master k8s]# [root@k8s-master k8s]# [root@k8s-master k8s]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx-deployment 2994176/20Mi, 0%/10% 2 10 4 73m [root@k8s-master k8s]# kubectl top pod error: metrics not available yet [root@k8s-master k8s]# kubectl top pod NAME CPU(cores) MEMORY(bytes) nginx-deployment-59bf58fcb5-7tmst 24m 2Mi nginx-deployment-59bf58fcb5-hjq6c 25m 2Mi nginx-deployment-59bf58fcb5-tt5v6 24m 2Mi nginx-deployment-59bf58fcb5-xv8gl 25m 2Mi [root@k8s-master k8s]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx-deployment 2994176/20Mi, 0%/10% 2 10 4 73m [root@k8s-master k8s]# kubectl top pod NAME CPU(cores) MEMORY(bytes) nginx-deployment-59bf58fcb5-7tmst 24m 2Mi nginx-deployment-59bf58fcb5-hjq6c 25m 2Mi nginx-deployment-59bf58fcb5-tt5v6 24m 2Mi nginx-deployment-59bf58fcb5-xv8gl 25m 2Mi [root@k8s-master k8s]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx-deployment 2953216/20Mi, 245%/10% 2 10 8 74m [root@k8s-master k8s]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx-deployment 2953216/20Mi, 245%/10% 2 10 8 74m [root@k8s-master k8s]# kubectl top pod NAME CPU(cores) MEMORY(bytes) nginx-deployment-59bf58fcb5-7tmst 24m 2Mi nginx-deployment-59bf58fcb5-hjq6c 25m 2Mi nginx-deployment-59bf58fcb5-tt5v6 24m 2Mi nginx-deployment-59bf58fcb5-xv8gl 25m 2Mi [root@k8s-master k8s]# kubectl top pod NAME CPU(cores) MEMORY(bytes) nginx-deployment-59bf58fcb5-7tmst 24m 2Mi nginx-deployment-59bf58fcb5-hjq6c 25m 2Mi nginx-deployment-59bf58fcb5-tt5v6 24m 2Mi nginx-deployment-59bf58fcb5-xv8gl 25m 2Mi [root@k8s-master k8s]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-59bf58fcb5-7tmst 1/1 Running 0 95s nginx-deployment-59bf58fcb5-bzr4q 1/1 Running 0 39s nginx-deployment-59bf58fcb5-hjq6c 1/1 Running 0 95s nginx-deployment-59bf58fcb5-j2mgj 1/1 Running 0 24s nginx-deployment-59bf58fcb5-jgt7r 1/1 Running 0 24s nginx-deployment-59bf58fcb5-mdngg 1/1 Running 0 39s nginx-deployment-59bf58fcb5-n82jz 1/1 Running 0 39s nginx-deployment-59bf58fcb5-tlvw7 1/1 Running 0 39s nginx-deployment-59bf58fcb5-tt5v6 1/1 Running 0 85s nginx-deployment-59bf58fcb5-xv8gl 1/1 Running 0 89s [root@k8s-master k8s]# kubectl top pod NAME CPU(cores) MEMORY(bytes) nginx-deployment-59bf58fcb5-7tmst 24m 2Mi nginx-deployment-59bf58fcb5-hjq6c 25m 2Mi nginx-deployment-59bf58fcb5-tt5v6 24m 2Mi nginx-deployment-59bf58fcb5-xv8gl 25m 2Mi [root@k8s-master k8s]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx-deployment 2953216/20Mi, 245%/10% 2 10 10 74m [root@k8s-master k8s]# kubectl top pod NAME CPU(cores) MEMORY(bytes) nginx-deployment-59bf58fcb5-7tmst 0m 2Mi nginx-deployment-59bf58fcb5-bzr4q 0m 3Mi nginx-deployment-59bf58fcb5-hjq6c 0m 2Mi nginx-deployment-59bf58fcb5-j2mgj 0m 2Mi nginx-deployment-59bf58fcb5-jgt7r 0m 2Mi nginx-deployment-59bf58fcb5-mdngg 0m 2Mi nginx-deployment-59bf58fcb5-n82jz 0m 2Mi nginx-deployment-59bf58fcb5-tlvw7 0m 3Mi nginx-deployment-59bf58fcb5-tt5v6 0m 2Mi nginx-deployment-59bf58fcb5-xv8gl 0m 2Mi [root@k8s-master k8s]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx-deployment 2992128/20Mi, 0%/10% 2 10 10 74m [root@k8s-master k8s]#
压测结束
压测结束后3-5分钟，HPA回到原始副本数量
[root@k8s-master k8s]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx-deployment 3084288/20Mi, 0%/10% 2 10 2 80m [root@k8s-master k8s]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-59bf58fcb5-7tmst 1/1 Running 0 7m44s nginx-deployment-59bf58fcb5-hjq6c 1/1 Running 0 7m44s [root@k8s-master k8s]# kubectl top pod NAME CPU(cores) MEMORY(bytes) nginx-deployment-59bf58fcb5-7tmst 0m 2Mi nginx-deployment-59bf58fcb5-hjq6c 0m 2Mi [root@k8s-master k8s]#
3.4 总结
不是所有的系统都可以依靠CPU/内存使用指标单独满足SLA，CPU和内存的设置，非常麻烦和粗维度。大多数Web和移动后端需要以每秒请求处理任何突发流量进行自动缩放。
通过prometheus检测你应用程序的正确指，并为自动是很所提供正确指标，您可以微调您的应用程序更好地处理突发和确保高可用性。
