处理 K8S Orphaned pod found - but volume paths are still present on disk 孤儿pod

问题概述

查看kubelet或/var/log/messages日志一直包错，发现是孤儿pod，是由于其pod被删除后存储路径还保存在磁盘。

报错如下

[root@node5 ~]# journalctl -fu kubelet
-- Logs begin at Tue 2020-06-16 23:41:14 CST. --
Jun 19 17:25:12 node5 kubelet[4711]: E0619 17:25:12.038458    4711 kubelet_volumes.go:154] orphaned pod "27960f19-29f1-486a-9a9d-c6c9290d014a" found, but volume paths are still present on disk : There were a total of 2 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:14 node5 kubelet[4711]: E0619 17:25:14.071432    4711 kubelet_volumes.go:154] orphaned pod "27960f19-29f1-486a-9a9d-c6c9290d014a" found, but volume paths are still present on disk : There were a total of 2 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:16 node5 kubelet[4711]: E0619 17:25:16.037737    4711 kubelet_volumes.go:154] orphaned pod "27960f19-29f1-486a-9a9d-c6c9290d014a" found, but volume paths are still present on disk : There were a total of 2 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:18 node5 kubelet[4711]: E0619 17:25:18.070147    4711 kubelet_volumes.go:154] orphaned pod "62e47eeb-8de7-4d4e-9e0f-28503d63be6a" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:20 node5 kubelet[4711]: E0619 17:25:20.036447    4711 kubelet_volumes.go:154] orphaned pod "62e47eeb-8de7-4d4e-9e0f-28503d63be6a" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:22 node5 kubelet[4711]: E0619 17:25:22.069562    4711 kubelet_volumes.go:154] orphaned pod "62e47eeb-8de7-4d4e-9e0f-28503d63be6a" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:24 node5 kubelet[4711]: E0619 17:25:24.065490    4711 kubelet_volumes.go:154] orphaned pod "62e47eeb-8de7-4d4e-9e0f-28503d63be6a" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:26 node5 kubelet[4711]: E0619 17:25:26.073979    4711 kubelet_volumes.go:154] orphaned pod "62e47eeb-8de7-4d4e-9e0f-28503d63be6a" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:28 node5 kubelet[4711]: E0619 17:25:28.037987    4711 kubelet_volumes.go:154] orphaned pod "62e47eeb-8de7-4d4e-9e0f-28503d63be6a" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:30 node5 kubelet[4711]: E0619 17:25:30.068136    4711 kubelet_volumes.go:154] orphaned pod "62e47eeb-8de7-4d4e-9e0f-28503d63be6a" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.

解决办法

解决思路

• 根据/var/log/messages或kubelet报错，查找孤儿pod
• 通过mount命令查找孤儿pod的挂载
• 卸载孤儿pod的存储挂载
• 删除/var/lib/kubelet/pods下孤儿pod的存储路径

使用脚本处理，脚本内容如下

#!/bin/sh

orphanedPods=`cat /var/log/messages|grep 'orphaned pod'|awk -F '"' '{print $2}'|uniq`;
orphanedPodsNum=`echo $orphanedPods|awk -F ' ' '{print NF}'`;
echo -e "orphanedPods: $orphanedPodsNum \n$orphanedPods";

for i in $orphanedPods
do
echo "Deleting Orphaned pod id: $i";
rm -rf /var/lib/kubelet/pods/$i;
done

在报错的节点直接执行脚本即可

相关issue：

https://github.com/kubernetes/kubernetes/issues/60987
https://github.com/kubernetes/kubernetes/pull/68616