第八章 Health Check

8.1 默认的健康检查

　　每个容器启动时会执行一个进程，此进程由Dockerfile的CMD或ENTRYPOINT指定。如果进程退出时返回码非零，则认为容器发生故障，K8s就会根据restartPolicy.

　　如果容器进程返回非零，K8s就认为容器发生故障，需要重启。

　　可是有时候，容器发生了故障，但是容器并不退出。

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: healthcheck
  name: healthcheck
spec:
  restartPolicy: OnFailure
  containers:
  - name: healthcheck
    image: busybox
    args: 
    - /bin/sh
    - -c
    - sleep 10; exit 1　

8.2 Liveness探测

　　Liveness探测让用户可以自定义判断容器是否健康的条件。如果探测失败，K8是就会重启容器。

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness
spec:
  restartPolicy: OnFailure
  containers:
  - name: liveness
    image: busybox
    args:
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600         # 创建文件，30秒后删除
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy                           # 如果文件存在，则认为容器处于正常状态，反之则发生故障；
      initialDelaySeconds: 10                    # 容器启动10秒后开始探测             
      periodSeconds: 5                           # 每5秒执行一次探测

开始30秒，/tmp/healthy文件存在，cat 命令返回0，liveness探测成功，

           kubectl describe pod liveness:

Events:
  Type    Reason                 Age   From                         Message
  ----    ------                 ----  ----                         -------
  Normal  Scheduled              32s   default-scheduler            Successfully assigned liveness to galaxykubernetes03
  Normal  SuccessfulMountVolume  32s   kubelet, galaxykubernetes03  MountVolume.SetUp succeeded for volume "default-token-qjj5t"
  Normal  Pulling                22s   kubelet, galaxykubernetes03  pulling image "busybox"
  Normal  Pulled                 21s   kubelet, galaxykubernetes03  Successfully pulled image "busybox"
  Normal  Created                21s   kubelet, galaxykubernetes03  Created container
  Normal  Started                21s   kubelet, galaxykubernetes03  Started container

　　35秒之后，文件本删除，cat返回非零，Liveness 探测失败；

Events:
  Type     Reason                 Age                From                         Message
  ----     ------                 ----               ----                         -------
  Normal   Scheduled              1m                 default-scheduler            Successfully assigned liveness to galaxykubernetes03
  Normal   SuccessfulMountVolume  1m                 kubelet, galaxykubernetes03  MountVolume.SetUp succeeded for volume "default-token-qjj5t"
  Warning  Unhealthy              34s (x3 over 44s)  kubelet, galaxykubernetes03  Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
  Normal   Pulling                4s (x2 over 1m)    kubelet, galaxykubernetes03  pulling image "busybox"
  Normal   Killing                4s                 kubelet, galaxykubernetes03  Killing container with id docker://liveness:Container failed liveness probe.. Container will be killed and recreated.
  Normal   Pulled                 3s (x2 over 1m)    kubelet, galaxykubernetes03  Successfully pulled image "busybox"
  Normal   Created                3s (x2 over 1m)    kubelet, galaxykubernetes03  Created container
  Normal   Started                3s (x2 over 1m)    kubelet, galaxykubernetes03  Started container

kubeusr@GalaxyKubernetesMaster:~$ kubectl get pods
NAME       READY     STATUS    RESTARTS   AGE                     # 反复重启
liveness   1/1       Running   4          6m

8.3 Readiness探测

　　Readiness探测：告诉k8s什么时候可以将容器加入到Service负载均衡池中，对外提供服务。

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness
spec:
  restartPolicy: OnFailure
  containers:
  - name: liveness
    image: busybox
    args:
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
    readinessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 10
      periodSeconds: 5

 

kubectl describe pod liveness


Events:
  Type     Reason                 Age              From                         Message
  ----     ------                 ----             ----                         -------
  Normal   Scheduled              51s              default-scheduler            Successfully assigned liveness to galaxykubernetes03
  Normal   SuccessfulMountVolume  51s              kubelet, galaxykubernetes03  MountVolume.SetUp succeeded for volume "default-token-qjj5t"
  Normal   Pulling                41s              kubelet, galaxykubernetes03  pulling image "busybox"
  Normal   Pulled                 40s              kubelet, galaxykubernetes03  Successfully pulled image "busybox"
  Normal   Created                40s              kubelet, galaxykubernetes03  Created container
  Normal   Started                40s              kubelet, galaxykubernetes03  Started container
  Warning  Unhealthy              1s (x2 over 6s)  kubelet, galaxykubernetes03  Readiness probe failed: cat: can't open '/tmp/healthy': No such file or directory

 

liveness 和 Readiness 比较：

（1）默认情况二者都是看容器退出时返回值是否为0，来判断是否失败。

（2）二者的配置方法完全相同，不同的是探测失败后的处理：Liveness重启容器；Readiness将容器设置为不可用。

   (3) 二者独立执行，没有依赖。Liveness探测判断容器是否需要重启来自愈； Readiness探测容器是否已经准备好对外提供服务。

8.4 Health Check 在 Scale Up中的应用

　　当执行scale up(扩容)的时候，新的副本添加到service的负载均衡中，与已有副本一起处理客户请求。应用启动通常需要一个准备阶段，比如加载缓存、连接数据库等，从容器启动到真正提供服务需要一段时间。可以通过Readiness探测判断容器是否就绪，避免将请求发到没有准备好的backend.

　　httpGet:  探测成功的判断条件是http请求返回码在200 ~ 400之间。（支持HTTP 和 HTTPS）

        需要实现   http://[container_ip]:8080/healthy 接口。

readinessProbe:
  httpGet:
    scheme: HTTP                   
    path: /healthy                 # 访问路径
    port: 8080                     # 指定端口
    initialDealySeconds: 10        # 容器启动10秒后开始探测
    periodSeconds: 5               # 每隔5秒探测一次

8.5 Health check 在滚动更新中的应用 

kubeusr@GalaxyKubernetesMaster:~$ kubectl get deployment
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
httpd     2         2         2            0           6s

DESIRED:  期望有多少个副本
CURRENT: 当前副本数目，  旧的 +  新的
UP-TO-DATE: 已经完成更新的副本数
AVAILABLE: 状态是READY可以提供服务的副本数