k8s Pod 扩容和缩容
在生产环境下,在面临服务需要扩容的场景时,可以使用Deployment/RC的Scale机制来实现。
Kubernetes支持对Pod的手动扩容和自动扩容。
手动扩容缩容
通过执行扩容命令,对某个deployment直接进行扩容:
# kubectl scale deployment nginx-deployment --replicas=4
当要缩容,减少副本数量即可:
# kubectl scale deployment nginx-deployment --replicas=2
自动扩容缩容
在使用自动化扩容和缩容,需要kubernetes安装heapster插件。
参考链接
按照说明,修改如下YAML文件中的image信息为国内可以访问的镜像仓库,然后执行:
[root@node-1 influxdb]# ll
total 16
-rw-r--r-- 1 root root 2288 Jun 21 20:23 grafana.yaml
-rw-r--r-- 1 root root 264 Jun 21 20:24 heapster-rbac.yaml # 来源于上一级rbac目录
-rw-r--r-- 1 root root 1112 Jun 21 20:24 heapster.yaml
-rw-r--r-- 1 root root 972 Jun 21 20:24 influxdb.yaml
[root@node-1 influxdb]# kubectl create -f ./
安装完成之后,使用如下链接可以访问Grafana的web界面,查看资源使用情况:
https://10.0.0.1:6443/api/v1/namespaces/kube-system/services/http:monitoring-grafana:/proxy/
在kubernetes 1.11以后的版本中将不再使用heapster插件来监控资源的使用状态,而是使用metric server。
创建php-apche的deployment的应用和Service,当访问此应用时,返回“OK!”:
# cat php-apache.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: php-apache
labels:
app: php-apache
spec:
replicas: 1
selector:
matchLabels:
app: php-apache
template:
metadata:
name: php-apache
labels:
app: php-apache
spec:
containers:
- name: php-apache
image: mirrorgooglecontainers/hpa-example
resources:
requests:
cpu: 200m
ports:
- containerPort: 80
# cat php-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: php-apache
spec:
ports:
- port: 80
selector:
app: php-apache
创建一个busybox的工具Pod,用于做压力测试:
# cat php-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: php-apache
spec:
ports:
- port: 80
selector:
app: php-apache
创建完以上对象,还需要创建一个HPA控制器,用于监控对象的资源使用率:
kubectl autoscale deployment php-apache --min=1 --max=6 --cpu-percent=50
# 对php-apache的deployment的对象创建HPA控制器,当CPU的使率超过50%时实现自动化扩容,支持1到6之前Pod副本数量,以使得Pod CPU使用率维持在50% 以内。
查看当前hpa的状态,如果状态显示<unknown>,查看下文中的故障排查:
[root@node-1 ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 0%/50% 1 8 1 39m
登录busybox,执行循环访问命令,对php-apache进行压测:
# kubectl exec -it busybox -- sh
/ # while true;do wget -q -O - http://php-apache > /dev/null ;done
# 此处如果DNS没有正常工作将无法正常解析URL地址。
查看资源监控信息:
[root@node-1 hpa-test]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 42%/50% 1 8 1 1h
[root@node-1 hpa-test]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 491%/50% 1 8 4 1h
查看对应的对象创建情况,正在陆续创建:
[root@node-1 hpa-test]# kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
php-apache 4 4 4 2 1h
[root@node-1 hpa-test]# kubectl get rs
NAME DESIRED CURRENT READY AGE
php-apache-56b5765b95 4 4 2 1h
[root@node-1 hpa-test]# kubectl get pod
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 36m
php-apache-56b5765b95-8pclp 1/1 Running 0 1h
php-apache-56b5765b95-gsvqk 1/1 Running 0 1m
php-apache-56b5765b95-nbwbk 1/1 Running 0 1m
php-apache-56b5765b95-zmc29 0/1 ContainerCreating 0 1m
当CPU使用率下降后,自动缩容:
[root@node-1 hpa-test]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 0%/50% 1 8 1 1h
[root@node-1 hpa-test]# kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
php-apache 1 1 1 1 1h
[root@node-1 hpa-test]# kubectl get rs
NAME DESIRED CURRENT READY AGE
php-apache-56b5765b95 1 1 1 1h
[root@node-1 hpa-test]# kubectl get pod
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 43m
php-apache-56b5765b95-8pclp 1/1 Running 0 1h
故障排查
如果设置hpa后,出现如下无法获取资源状态的情况:
[root@node-1 hpa-test]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache <unknown>/50% 1 8 1 9m
在master上修改kube-controller-manager的配置,添加如下参数:
cat /usr/lib/systemd/system/kube-controller-manager.service
...
--horizontal-pod-autoscaler-use-rest-clients=false \
...
实例配置:
[root@pressure-master03 ~]# kubectl autoscale deployment mall-audit-provider -n x****ssure --min=1 --max=8 --cpu-percent=50 horizontalpodautoscaler.autoscaling/mall-audit-provider autoscaled [root@pressure-master03 ~]# kubectl autoscale deployment mall-basic-provider -n x****sure --min=1 --max=8 --cpu-percent=50 horizontalpodautoscaler.autoscaling/mall-basic-provider autoscaled [root@pressure-master03 ~]# kubectl get hpa -n x*****ure NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE ma****-api Deployment/mall-app-api 1%/50% 1 8 2 4m29s mall****ovider Deployment/mall-audit-provider 1%/50% 1 8 1 4m10s