K8S kubelet 资源垃圾回收机制
Kubelet 垃圾回收(kubelet-garbage-collection)负责自动清理节点上的无用镜像和容器。Kubelet 每隔 1 分钟进行一次容器清理,每隔 5 分钟进行一次镜像清理(截止到 v1.15 版本,垃圾回收间隔时间都是在源码中固化的,不可自定义配置)。如果节点上已经运行了 Kubelet,不建议再额外运行其它的垃圾回收工具,因为这些工具可能错误地清理掉 Kubelet 认为本应保留的镜像或容器,从而可能造成不可预知的问题。
1. 镜像回收
1.1 概述
Kubernetes 对节点上的所有镜像提供生命周期管理服务,这里的所有镜像是真正意义上的所有镜像,不仅仅是通过 Kubelet 拉取的镜像。当磁盘使用率超过设定上限 HighThresholdPercent
时,Kubelet 就会按照 LRU 清除策略逐个清理掉那些没有被任何 Pod 容器(包括已经死亡的容器)所使用的镜像,直到磁盘使用率降到设定下限 LowThresholdPercent
或没有空闲镜像可以清理。此外,在进行镜像清理时,会考虑镜像的生存年龄,对于年龄没有达到最短生存年龄 MinAge
要求的镜像,暂不予以清理。
镜像回收流程:
在磁盘使用率超出设定上限后:首先,通过 CRI 容器运行时接口读取节点上的所有镜像以及 Pod 容器;然后,根据现有容器列表过滤出那些已经不被任何容器所使用的镜像;接着,按照镜像最近被使用时间排序,越久被用到的镜像越会被排在前面,优先清理;最后,就按照排好的顺序逐个清理镜像,直到磁盘使用率降到设定下限(或者已经没有空闲镜像可以清理)。
需要注意的是,Kubelet 读取到的镜像列表是节点镜像列表,而读取到的容器列表却仅包括由其管理的容器(即 Pod 容器,包括 Pod 内的死亡容器)。因此,那些用户手动 docker run
起来的容器,对于 Kubelet 垃圾回收来说就是不可见的,也就不能阻止对相关镜像的垃圾回收。当然,Kubelet 的镜像回收不是 force 类型的回收,虽然会对用户手动下载的镜像进行回收动作,但如果确实有运行的(或者停止的任何)容器与该镜像关联的话,删除操作就会失败(被底层容器运行时阻止删除)。
影响镜像垃圾回收的关键参数有:
--image-gc-high-threshold:磁盘使用率上限,有效范围 [0-100],默认 85
--image-gc-low-threshold: 磁盘使用率下限,有效范围 [0-100],默认 80
--minimum-image-ttl-duration:镜像最短应该生存的年龄,默认 2 分钟
定义参数的配置文件:
# master 和 node 节点一样的,主要改工作节点
[root@master ~]#cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
1.2 实验
修改配置 Kubelet 启动参数,降低磁盘使用率上限,以便能够直接触发镜像回收
[root@node01/02 ~]#vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
......
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --image-gc-high-threshold=2 --image-gc-low-threshold=1
......
### 配置的回收参数很低,Kubelet 会一直忙于进行镜像回收
systemctl daemon-reload && systemctl restart kubelet
查看 worker 节点本地镜像
# 里面并没有 nginx 镜像
[root@node01/02 ~]#docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
lizhenliang/flannel v0.12.0-amd64 4e9f801d2217 2 years ago 52.8MB
registry.aliyuncs.com/google_containers/kube-proxy v1.17.0 7d54289267dc 2 years ago 116MB
registry.aliyuncs.com/google_containers/coredns 1.6.5 70f311871ae1 2 years ago 41.6MB
registry.aliyuncs.com/google_containers/pause 3.1 da86e6ba6ca1 4 years ago 742kB
运行一个 nginx 程序,让 Kubelet 自动拉取镜像
[root@master ~]#kubectl run nginx --image=nginx:1.14 --port=80 --replicas=3
......
[root@master ~]#kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-59d795d786-5ln28 1/1 Running 0 36s 10.244.1.3 node01 <none> <none>
nginx-59d795d786-xnfjq 0/1 ContainerCreating 0 36s <none> node01 <none> <none>
nginx-59d795d786-z86nn 1/1 Running 0 36s 10.244.2.3 node02 <none> <none>
# 在 node02 节点查看 nginx 镜像已经拉取
[14:50:07 root@node02~]#docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 605c77e624dd 5 months ago 141MB
lizhenliang/flannel v0.12.0-amd64 4e9f801d2217 2 years ago 52.8MB
registry.aliyuncs.com/google_containers/kube-proxy v1.17.0 7d54289267dc 2 years ago 116MB
registry.aliyuncs.com/google_containers/coredns 1.6.5 70f311871ae1 2 years ago 41.6MB
registry.aliyuncs.com/google_containers/pause 3.1 da86e6ba6ca1 4 years ago 742kB
删除 nginx deploy,过 5 分钟,查看 nginx 镜像是否会被删除
[root@master ~]#kubectl delete deploy nginx
deployment.apps "nginx" deleted
[root@master ~]#kubectl get pods,deploy -o wide
No resources found in default namespace.
# 查看 node 节点 nginx 镜像已经被删除
[root@node01 ~]#docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
lizhenliang/flannel v0.12.0-amd64 4e9f801d2217 2 years ago 52.8MB
registry.aliyuncs.com/google_containers/kube-proxy v1.17.0 7d54289267dc 2 years ago 116MB
registry.aliyuncs.com/google_containers/coredns 1.6.5 70f311871ae1 2 years ago 41.6MB
registry.aliyuncs.com/google_containers/pause 3.1 da86e6ba6ca1 4 years ago 742kB
[15:58:05 root@node02~]#docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
lizhenliang/flannel v0.12.0-amd64 4e9f801d2217 2 years ago 52.8MB
registry.aliyuncs.com/google_containers/kube-proxy v1.17.0 7d54289267dc 2 years ago 116MB
registry.aliyuncs.com/google_containers/coredns 1.6.5 70f311871ae1 2 years ago 41.6MB
registry.aliyuncs.com/google_containers/pause 3.1 da86e6ba6ca1 4 years ago 742kB
查看镜像垃圾回收日志
[root@node01 ~]#journalctl -u kubelet -o cat | grep imageGCManager
I0623 15:52:39.598609 12870 image_gc_manager.go:300] [imageGCManager]: Disk usage on image filesystem is at 6% which is over the high threshold (2%). Trying to free 4901549465 bytes down to the low threshold (1%).
I0623 15:52:39.604010 12870 image_gc_manager.go:371] [imageGCManager]: Removing image "sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369" to free 109129446 bytes
I0623 15:57:39.725290 12870 image_gc_manager.go:300] [imageGCManager]: Disk usage on image filesystem is at 6% which is over the high threshold (2%). Trying to free 4780021145 bytes down to the low threshold (1%).
测试手动拉取的镜像是否会被清理
[root@node01 ~]#docker run --name nginx -d nginx
......
# 查看镜像 GC 日志,会发现 GC 会尝试清理用户自己手动拉取的 nginx 镜像,但因为该镜像被使用中,所以这次删除操作不会成功
[root@node01 ~]#journalctl -u kubelet -o cat | grep imageGCManager
I0623 15:52:39.598609 12870 image_gc_manager.go:300] [imageGCManager]: Disk usage on image filesystem is at 6% which is over the high threshold (2%). Trying to free 4901549465 bytes down to the low threshold (1%).
I0623 15:52:39.604010 12870 image_gc_manager.go:371] [imageGCManager]: Removing image "sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369" to free 109129446 bytes
I0623 15:57:39.725290 12870 image_gc_manager.go:300] [imageGCManager]: Disk usage on image filesystem is at 6% which is over the high threshold (2%). Trying to free 4780021145 bytes down to the low threshold (1%).
I0623 16:02:39.731137 12870 image_gc_manager.go:300] [imageGCManager]: Disk usage on image filesystem is at 6% which is over the high threshold (2%). Trying to free 4937041305 bytes down to the low threshold (1%).
[root@node01 ~]#docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 605c77e624dd 5 months ago 141MB
......
# 将该容器停止,继续观察回收动作。可以看到,对于已经停止的容器,Kubelet 也是会尝试删除,但删除操作依然不会成功(存在死亡容器对该镜像的引用)
[root@node01 ~]#docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
63dbfc2ffa7e nginx "/docker-entrypoint.…" 9 minutes ago Up 9 minutes 80/tcp nginx
[root@node01 ~]#docker stop nginx
nginx
[root@node01 ~]#docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
63dbfc2ffa7e nginx "/docker-entrypoint.…" 9 minutes ago Exited (0) 2 seconds ago nginx
[root@node01 ~]#journalctl -u kubelet -o cat | grep imageGCManager
......
[root@node01 ~]#docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 605c77e624dd 5
......
# 彻底删除 nginx 容器,此时就没有任何容器继续使用该镜像,经过 1 次 GC 后,nginx 镜像就会被清理
[root@node01 ~]#docker rm 63dbfc2ffa7e
63dbfc2ffa7e
[root@node01 ~]#docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
caab076de9b2 registry.aliyuncs.com/google_containers/coredns "/coredns -conf /etc…" 6 days ago Up 6 days k8s_coredns_coredns-9d85f5447-k6qlt_kube-system_a14a6dfb-3121-4dfd-9710-1db4c2180d8c_0
44c962a17661 registry.aliyuncs.com/google_containers/pause:3.1 "/pause" 6 days ago Up 6 days k8s_POD_coredns-9d85f5447-k6qlt_kube-system_a14a6dfb-3121-4dfd-9710-1db4c2180d8c_0
32e2590eaa72 4e9f801d2217 "/opt/bin/flanneld -…" 6 days ago Up 6 days k8s_kube-flannel_kube-flannel-ds-amd64-qp9zj_kube-system_7601dba1-6cac-424b-ab4a-d9b6bdf974fd_0
f612bcd65b46 lizhenliang/flannel "cp -f /etc/kube-fla…" 6 days ago Exited (0) 6 days ago k8s_install-cni_kube-flannel-ds-amd64-qp9zj_kube-system_7601dba1-6cac-424b-ab4a-d9b6bdf974fd_0
3375f72a65d8 registry.aliyuncs.com/google_containers/pause:3.1 "/pause" 6 days ago Up 6 days k8s_POD_kube-flannel-ds-amd64-qp9zj_kube-system_7601dba1-6cac-424b-ab4a-d9b6bdf974fd_0
61130d150f47 registry.aliyuncs.com/google_containers/kube-proxy "/usr/local/bin/kube…" 6 days ago Up 6 days k8s_kube-proxy_kube-proxy-x28ph_kube-system_3ebe10a7-4428-41c2-a876-49dae2ebca9b_0
4682c653addf registry.aliyuncs.com/google_containers/pause:3.1 "/pause" 6 days ago Up 6 days k8s_POD_kube-proxy-x28ph_kube-system_3ebe10a7-4428-41c2-a876-49dae2ebca9b_0
[root@node01 ~]#docker images # 等 5 分钟
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 605c77e624dd 5 months ago 141MB
......
[root@node01 ~]#docker images # nginx 已被清理
REPOSITORY TAG IMAGE ID CREATED SIZE
lizhenliang/flannel v0.12.0-amd64 4e9f801d2217 2 years ago 52.8MB
registry.aliyuncs.com/google_containers/kube-proxy v1.17.0 7d54289267dc 2 years ago 116MB
registry.aliyuncs.com/google_containers/coredns 1.6.5 70f311871ae1 2 years ago 41.6MB
registry.aliyuncs.com/google_containers/pause 3.1 da86e6ba6ca1 4 years ago 742kB
2. 容器回收
2.1 概述
容器在停止运行(比如出错退出或者正常结束)后会残留一系列的垃圾文件,一方面会占据磁盘空间,另一方面也会影响系统运行速度。此时,就需要 Kubelet 容器回收了。要特别注意的是,Kubelet 回收的容器是指那些由其管理的的容器(也就是 Pod 容器),用户手动运行的容器不会被 Kubelet 进行垃圾回收。
容器回收主要针对三个目标资源:普通容器、sandbox 容器以及容器日志目录。
对于普通容器,主要根据 MaxPerPodContainer
与 MaxContainers
的设置,按照 LRU 策略,从 Pod 的死亡容器列表删除一定数量的容器,直到满足配置需求;对于 sandbox
容器,按照每个 Pod 保留一个的原则清理多余的死亡 sandbox
;对于日志目录,只要没有 Pod 与之关联了就将其删除。Kubelet 的容器垃圾回收只针对 Pod 容器,非 Kubelet Pod 容器(比如通过 docker run
启动的容器)不会被主动清理。
容器回收流程:
容器垃圾回收相关的控制参数主要有 3 个:
--minimum-container-ttl-duration:从容器停止运行时起经过设置时间后,该容器标记为已过期将来可以被回收(只是标记,不是回收),默认值为1m0s # 1.22.5 不支持
--maximum-dead-containers-per-container:每个 pod 上可以留下运行结束之后的容器的个数,默认值为 2
--maximum-dead-containers:节点可保留的死亡容器的最大数量,默认值是 -1,这意味着节点没有限制死亡容器数量
# 注意:当 maximum-dead-containers-per-container 与 maximum-dead-containers 发生冲突时,Kubelet 会自动调整 maximum-dead-containers-per-container 的取值以满足 maximum-dead-containers 要求。
# 参数示例
vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.1 --minimum-container-ttl-duration=5m --maximum-dead-containers-per-container=2 --maximum-dead-containers=10"
systemctl daemon-reload && systemctl restart kubelet
# 或者添加到 ExecStart
vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# 如果需要关闭容器的垃圾回收策略
将 --minimum-container-ttl-duration设为0(表示无限制),--maximum-dead-containers-per-container和--maximum-dead-containers设为负数。
2.2 实验
# 默认参数
# 启动一个 nginx 服务
[root@master ~]#kubectl run nginx --image=nginx:1.14 --port=80 --replicas=1
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/nginx created
[root@master ~]#kubectl get pods,deploy -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-59d795d786-l245m 1/1 Running 0 33s 10.244.2.4 node02 <none> <none>
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/nginx 1/1 1 1 33s nginx nginx:1.14 run=nginx
# 查看 node02 节点,Kubelet 启动了一个 sandbox 以及一个 nginx 实例
[16:41:44 root@node02~]#docker ps -a|grep nginx
318d5c56d726 nginx "nginx -g 'daemon of…" About a minute ago Up About a minute k8s_nginx_nginx-59d795d786-l245m_default_db5f8e0f-8263-4f82-99b7-c7ac2853187e_0
ee81ceddac55 registry.aliyuncs.com/google_containers/pause:3.1 "/pause" About a minute ago Up About a minute k8s_POD_nginx-59d795d786-l245m_default_db5f8e0f-8263-4f82-99b7-c7ac2853187e_0
# 手动杀死 nginx 实例,模拟容器异常退出
[16:41:48 root@node02~]#docker kill 318d5c56d726
318d5c56d726
# 可以看到 kubelet 拉起了一个新的 nginx 容器
[16:45:09 root@node02~]#docker ps -a|grep nginx
7c992886df4b 295c7be07902 "nginx -g 'daemon of…" 8 seconds ago Up 7 seconds k8s_nginx_nginx-59d795d786-l245m_default_db5f8e0f-8263-4f82-99b7-c7ac2853187e_1
318d5c56d726 nginx "nginx -g 'daemon of…" 4 minutes ago Exited (137) 8 seconds ago k8s_nginx_nginx-59d795d786-l245m_default_db5f8e0f-8263-4f82-99b7-c7ac2853187e_0
ee81ceddac55 registry.aliyuncs.com/google_containers/pause:3.1 "/pause" 4 minutes ago Up 4 minutes k8s_POD_nginx-59d795d786-l245m_default_db5f8e0f-8263-4f82-99b7-c7ac2853187e_0
# 等待几分钟,发现 Kubelet 并未清理异常退出的 nginx 容器(因为此时仅有一个 dead container)
# 继续杀死当前 nginx 实例
[16:47:22 root@node02~]#docker kill 7c992886df4b
7c992886df4b
[16:47:49 root@node02~]#docker ps -a|grep nginx
7c992886df4b 295c7be07902 "nginx -g 'daemon of…" 2 minutes ago Exited (137) 2 seconds ago k8s_nginx_nginx-59d795d786-l245m_default_db5f8e0f-8263-4f82-99b7-c7ac2853187e_1
ee81ceddac55 registry.aliyuncs.com/google_containers/pause:3.1 "/pause" 7 minutes ago Up 7 minutes k8s_POD_nginx-59d795d786-l245m_default_db5f8e0f-8263-4f82-99b7-c7ac2853187e_0
[16:47:52 root@node02~]#docker ps -a|grep nginx
7c992886df4b 295c7be07902 "nginx -g 'daemon of…" 2 minutes ago Exited (137) 7 seconds ago k8s_nginx_nginx-59d795d786-l245m_default_db5f8e0f-8263-4f82-99b7-c7ac2853187e_1
ee81ceddac55 registry.aliyuncs.com/google_containers/pause:3.1 "/pause" 7 minutes ago Up 7 minutes k8s_POD_nginx-59d795d786-l245m_default_db5f8e0f-8263-4f82-99b7-c7ac2853187e_0
[16:47:57 root@node02~]#docker ps -a|grep nginx
e20c07ae5c6c 295c7be07902 "nginx -g 'daemon of…" 2 seconds ago Up 2 seconds k8s_nginx_nginx-59d795d786-l245m_default_db5f8e0f-8263-4f82-99b7-c7ac2853187e_2
7c992886df4b 295c7be07902 "nginx -g 'daemon of…" 2 minutes ago Exited (137) 13 seconds ago k8s_nginx_nginx-59d795d786-l245m_default_db5f8e0f-8263-4f82-99b7-c7ac2853187e_1
ee81ceddac55 registry.aliyuncs.com/google_containers/pause:3.1 "/pause" 7 minutes ago Up 7 minutes k8s_POD_nginx-59d795d786-l245m_default_db5f8e0f-8263-4f82-99b7-c7ac2853187e_0
# 这下看到效果了,仍然只有一个退出的容器被保留,而且被清理掉的是最老的死亡容器
# 删除 nginx Deployment,会发现所有的 nginx 容器都会被清理
[root@master ~]#kubectl delete deploy nginx
deployment.apps "nginx" deleted
[16:50:02 root@node02~]#docker ps -a|grep nginx
ee81ceddac55 registry.aliyuncs.com/google_containers/pause:3.1 "/pause" 9 minutes ago Exited (0) 19 seconds ago k8s_POD_nginx-59d795d786-l245m_default_db5f8e0f-8263-4f82-99b7-c7ac2853187e_0
修改 Kubelet 参数,设置 --maximum-dead-containers=0,告诉 Kubelet 清理所有死亡容器。
[17:46:48 root@node02~]#vim /var/lib/kubelet/kubeadm-flags.env
[17:56:58 root@node02~]#systemctl daemon-reload && systemctl restart kubelet
[17:58:13 root@node02~]#docker ps -a | grep nginx
08bfe1ca6acc 295c7be07902 "nginx -g 'daemon of…" 24 seconds ago Up 23 seconds k8s_nginx_nginx-59d795d786-j58xv_default_13868010-4e4d-4422-a9c2-68501926cf83_2
7b7c6516f990 295c7be07902 "nginx -g 'daemon of…" 13 minutes ago Exited (137) 24 seconds ago k8s_nginx_nginx-59d795d786-j58xv_default_13868010-4e4d-4422-a9c2-68501926cf83_1
dfa8c3639983 registry.aliyuncs.com/google_containers/pause:3.1 "/pause" 14 minutes ago Up 14 minutes k8s_POD_nginx-59d795d786-j58xv_default_13868010-4e4d-4422-a9c2-68501926cf83_0
[17:58:13 root@node02~]#docker kill 08bfe1ca6acc
08bfe1ca6acc
[17:58:30 root@node02~]#docker ps -a | grep nginx
f7a8c0e7679c 295c7be07902 "nginx -g 'daemon of…" 6 seconds ago Up 6 seconds k8s_nginx_nginx-59d795d786-j58xv_default_13868010-4e4d-4422-a9c2-68501926cf83_0
dfa8c3639983 registry.aliyuncs.com/google_containers/pause:3.1 "/pause" 14 minutes ago Up 14 minutes k8s_POD_nginx-59d795d786-j58xv_default_13868010-4e4d-4422-a9c2-68501926cf83_0
对于手动运行的容器,例如我们通过 docker run 运行的 nginx 并不会被删除
[18:01:16 root@node02~]#docker run --name nginx-test -d nginx
......
[18:01:43 root@node02~]#docker ps -a|grep nginx-test
a8f53826b197 nginx "/docker-entrypoint.…" 27 seconds ago Up 26 seconds 80/tcp nginx-test
[18:02:10 root@node02~]#docker kill a8f53826b197
a8f53826b197
[18:02:23 root@node02~]#docker ps -a|grep nginx-test
a8f53826b197 nginx "/docker-entrypoint.…" 46 seconds ago Exited (137) 5 seconds ago nginx-test
[18:02:53 root@node02~]#docker ps -a|grep nginx-test
a8f53826b197 nginx "/docker-entrypoint.…" 2 minutes ago Exited (137) About a minute ago nginx-test
# 手动运行的容器并不会被删除
3. 总结
Kubelet 每 5 分钟进行一次镜像清理。当磁盘使用率超过上限阈值,Kubelet 会按照 LRU 策略逐一清理没有被任何容器所使用的镜像,直到磁盘使用率降到下限阈值或没有空闲镜像可以清理。Kubelet 认为镜像可被清理的标准是未被任何 Pod 容器(包括那些死亡了的容器)所引用,那些非 Pod 容器(如用户通过 docker run
启动的容器)是不会被用来计算镜像引用关系的。也就是说,即便用户运行的容器使用了 A 镜像,只要没有任何 Pod 容器使用到 A,那 A 镜像对于 Kubelet 而言就是可被回收的。但是我们无需担心手动运行容器使用的镜像会被意外回收,因为 Kubelet 的镜像删除是非 force 类型的,底层容器运行时会使存在容器关联的镜像删除操作失败(因为 Docker 会认为仍有容器使用着 A 镜像)。
Kubelet 每 1 分钟执行一次容器清理。根据启动配置参数,Kubelet 会按照 LRU 策略依次清理每个 Pod 内的死亡容器,直到达到死亡容器限制数要求,对于 sandbox
容器,Kubelet 仅会保留最新的(这不受 GC 策略的控制)。对于日志目录,只要已经没有 Pod 继续占用,就将其清理。对于非 Pod 容器(如用户通过 docker run
启动的容器)不会被 Kubelet 垃圾回收。