Kubernetes HPA AutoScale引起异常告警现象

事件回顾

---

 

告警事件，在这个时间段，已知的信息，无版本发布，及人为操作

 事件排查

---

1. 查看Kubernetes事件信息，事件信息给出的引导信息非常少，只是一些简单的调度操作，并没有说明问题的本质，其中从事件信息获取到二个Kill的操作，但是是正常操作（type）

   <root@PROD-K8S-CP1 ~># kubectl get events -A
   NAMESPACE   LAST SEEN   TYPE      REASON              OBJECT                                              MESSAGE
   prod        57m         Normal    Killing             pod/prod-common-trust-service-6b6b85d679-d5tdn      Stopping container prod-common-trust-service
   prod        57m         Warning   Unhealthy           pod/prod-common-trust-service-6b6b85d679-d5tdn      Readiness probe failed: HTTP probe failed with statuscode: 503
   prod        57m         Warning   Unhealthy           pod/prod-common-trust-service-6b6b85d679-d5tdn      Liveness probe failed: HTTP probe failed with statuscode: 503
   prod        57m         Warning   Unhealthy           pod/prod-common-trust-service-6b6b85d679-d5tdn      Readiness probe failed: Get http://172.21.4.49:9141/actuator/health: dial tcp 172.21.4.49:9141: connect: no route to host
   prod        <unknown>   Normal    Scheduled           pod/prod-common-trust-service-6b6b85d679-gnvtr      Successfully assigned prod/prod-common-trust-service-6b6b85d679-gnvtr to prod-be-k8s-wn12
   prod        55m         Normal    Pulling             pod/prod-common-trust-service-6b6b85d679-gnvtr      Pulling image "imagehub.qiangyun.com/common/prod-common-trust-service:develop-6"
   prod        55m         Normal    Pulled              pod/prod-common-trust-service-6b6b85d679-gnvtr      Successfully pulled image "imagehub.qiangyun.com/common/prod-common-trust-service:develop-6"
   prod        55m         Normal    Created             pod/prod-common-trust-service-6b6b85d679-gnvtr      Created container prod-common-trust-service
   prod        55m         Normal    Started             pod/prod-common-trust-service-6b6b85d679-gnvtr      Started container prod-common-trust-service
   prod        49m         Normal    Killing             pod/prod-common-trust-service-6b6b85d679-gnvtr      Stopping container prod-common-trust-service
   prod        49m         Warning   Unhealthy           pod/prod-common-trust-service-6b6b85d679-gnvtr      Readiness probe failed: HTTP probe failed with statuscode: 503
   prod        49m         Warning   Unhealthy           pod/prod-common-trust-service-6b6b85d679-gnvtr      Liveness probe failed: Get http://172.21.4.47:9141/actuator/health: dial tcp 172.21.4.47:9141: connect: no route to host
   prod        57m         Normal    SuccessfulDelete    replicaset/prod-common-trust-service-6b6b85d679     Deleted pod: prod-common-trust-service-6b6b85d679-d5tdn
   prod        55m         Normal    SuccessfulCreate    replicaset/prod-common-trust-service-6b6b85d679     Created pod: prod-common-trust-service-6b6b85d679-gnvtr
   prod        49m         Normal    SuccessfulDelete    replicaset/prod-common-trust-service-6b6b85d679     Deleted pod: prod-common-trust-service-6b6b85d679-gnvtr
   prod        55m         Normal    SuccessfulRescale   horizontalpodautoscaler/prod-common-trust-service   New size: 6; reason: cpu resource utilization (percentage of request) above target
   prod        55m         Normal    ScalingReplicaSet   deployment/prod-common-trust-service                Scaled up replica set prod-common-trust-service-6b6b85d679 to 6
   prod        49m         Normal    SuccessfulRescale   horizontalpodautoscaler/prod-common-trust-service   New size: 5; reason: All metrics below target
   prod        49m         Normal    ScalingReplicaSet   deployment/prod-common-trust-service                Scaled down replica set prod-common-trust-service-6b6b85d679 to 5

2.  查看阿里云Kubernetes事件中心

    

3. 查看Pod历史副本数