容器Merged聚合层丢失处理
服务器强制断电导致Docker容器 Merged聚合层 丢失
背景说明：物业转电，机房服务器强制断电，恢复供电后，服务器重新启动，发现容器无法启动
查看容器情况
# 发现容器挂了
[root@20-121 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
... ...
4e071c2b26cb zentao:v9.0.1 "/bin/sh /opt/run.sh" 2 years ago Exited (255) 3 days ago zentao
... ...
# 直接重启发现找不到 merged 层
[root@20-121 ~]# docker restart zentao
Error response from daemon: Cannot restart container zentao: error creating overlay mount to /data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385/merged: no such file or directory
# 确认merged目录存在
[root@20-121 ~]# ll /data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385/merged
ls: 无法访问/data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385/merged: 没有那个文件或目录
确认merged已丢失，这里猜测是因为机器强制断电，导致容器merged聚合层丢失
查看对应容器数据结构
docker inspect zentao
... ...
"GraphDriver": {
"Data": {
"LowerDir": "/data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385-init/diff:/data04/docker-data/overlay2/dd0313cf5d59560d7785a132205f622d30e741fd2362a9bf3d35111f68cdceae/diff:/data04/docker-data/overlay2/27b3c424054ca45020756b470b4564a44b0c0033fe72fe1403e44b9b4dfcc021/diff:/data04/docker-data/overlay2/c6f32c031dc415fafd99bcbb252f1d7b014c464f0d49406ceac0fc531768c17d/diff:/data04/docker-data/overlay2/1d6cdd959d67a7220de9132b613d43c1b47954ef7eefd6aefff2be8367e7e061/diff:/data04/docker-data/overlay2/a4beb7b1d27e0b19434d0458c7f921bc59512261e3f778eb244a27de1747f6b1/diff:/data04/docker-data/overlay2/25db583d880ae3d87502ad8ef3d55bc7cca306ec5a86fab1b34ac0eafbfcdd09/diff:/data04/docker-data/overlay2/1b6474e5dba65135459983c38d94ee82b0f9b8f6bafda3279e888334032f3bc7/diff:/data04/docker-data/overlay2/2674f00cb5aba0b1b7e67baba253e264eaf7052b34bb87d30a135f9cb1192292/diff:/data04/docker-data/overlay2/a4feaf6effd463ebaa608ce14913da91825c5ecacbe66baf3bb18039006e1050/diff:/data04/docker-data/overlay2/b69127f520a115053d477d95ba5dbfd74ad8006a80bc7c8008db7b646ad6bc84/diff",
"MergedDir": "/data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385/merged",
"UpperDir": "/data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385/diff",
"WorkDir": "/data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385/work"
},
"Name": "overlay2"
},
... ...
重新挂载Merged层
重新创建个空目录作为merged层的挂载点
[root@20-121 ~]# cd /data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385/
[root@20-121 d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385]# mkdir -p merged
手动重建merged层，示例用法：[mount -t overlay overlay -o lowerdir=...,upperdir=...,workdir=... /data/docker-data/xxxxx/merged]
# 以下是完整命令示例
[root@20-121 d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385]# mount -t overlay overlay -o lowerdir=/data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385-init/diff:/data04/docker-data/overlay2/dd0313cf5d59560d7785a132205f622d30e741fd2362a9bf3d35111f68cdceae/diff:/data04/docker-data/overlay2/27b3c424054ca45020756b470b4564a44b0c0033fe72fe1403e44b9b4dfcc021/diff:/data04/docker-data/overlay2/c6f32c031dc415fafd99bcbb252f1d7b014c464f0d49406ceac0fc531768c17d/diff:/data04/docker-data/overlay2/1d6cdd959d67a7220de9132b613d43c1b47954ef7eefd6aefff2be8367e7e061/diff:/data04/docker-data/overlay2/a4beb7b1d27e0b19434d0458c7f921bc59512261e3f778eb244a27de1747f6b1/diff:/data04/docker-data/overlay2/25db583d880ae3d87502ad8ef3d55bc7cca306ec5a86fab1b34ac0eafbfcdd09/diff:/data04/docker-data/overlay2/1b6474e5dba65135459983c38d94ee82b0f9b8f6bafda3279e888334032f3bc7/diff:/data04/docker-data/overlay2/2674f00cb5aba0b1b7e67baba253e264eaf7052b34bb87d30a135f9cb1192292/diff:/data04/docker-data/overlay2/a4feaf6effd463ebaa608ce14913da91825c5ecacbe66baf3bb18039006e1050/diff:/data04/docker-data/overlay2/b69127f520a115053d477d95ba5dbfd74ad8006a80bc7c8008db7b646ad6bc84/diff,upperdir=/data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385/diff,workdir=/data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385/work /data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385/merged
mount: 特殊设备 overlay 不存在
挂载报错，确认各镜像层lowerdir，可写层upperdir情况
[root@20-121 d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385]# ll /data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385-init/diff
... ...
[root@20-121 d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385]# ll /data04/docker-data/overlay2/dd0313cf5d59560d7785a132205f622d30e741fd2362a9bf3d35111f68cdceae/diff
ls: 无法访问/data04/docker-data/overlay2/dd0313cf5d59560d7785a132205f622d30e741fd2362a9bf3d35111f68cdceae/diff: 没有那个文件或目录
发现还有个diff目录丢失了，去掉已丢失的diff层，继续重构merged层
[root@20-121 d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385]# mount -t overlay overlay -o lowerdir=/data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385-init/diff:/data04/docker-data/overlay2/27b3c424054ca45020756b470b4564a44b0c0033fe72fe1403e44b9b4dfcc021/diff:/data04/docker-data/overlay2/c6f32c031dc415fafd99bcbb252f1d7b014c464f0d49406ceac0fc531768c17d/diff:/data04/docker-data/overlay2/1d6cdd959d67a7220de9132b613d43c1b47954ef7eefd6aefff2be8367e7e061/diff:/data04/docker-data/overlay2/a4beb7b1d27e0b19434d0458c7f921bc59512261e3f778eb244a27de1747f6b1/diff:/data04/docker-data/overlay2/25db583d880ae3d87502ad8ef3d55bc7cca306ec5a86fab1b34ac0eafbfcdd09/diff:/data04/docker-data/overlay2/1b6474e5dba65135459983c38d94ee82b0f9b8f6bafda3279e888334032f3bc7/diff:/data04/docker-data/overlay2/2674f00cb5aba0b1b7e67baba253e264eaf7052b34bb87d30a135f9cb1192292/diff:/data04/docker-data/overlay2/a4feaf6effd463ebaa608ce14913da91825c5ecacbe66baf3bb18039006e1050/diff:/data04/docker-data/overlay2/b69127f520a115053d477d95ba5dbfd74ad8006a80bc7c8008db7b646ad6bc84/diff,upperdir=/data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385/diff,workdir=/data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385/work /data04/docker-data/overlay2/d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385/merged
查看构建好的merged层，发现数据结构还是比较完整的
[root@20-121 d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385]# ll merged/
总用量 12
-rw-r--r-- 1 root root 12053 12月 4 2018 anaconda-post.log
lrwxrwxrwx 1 root root 7 12月 4 2018 bin -> usr/bin
drwxr-xr-x 1 root root 59 8月 15 2022 dev
drwxr-xr-x 1 root root 66 8月 15 2022 etc
drwxr-xr-x 2 root root 10 4月 11 2018 home
lrwxrwxrwx 1 root root 7 12月 4 2018 lib -> usr/lib
lrwxrwxrwx 1 root root 9 12月 4 2018 lib64 -> usr/lib64
drwxr-xr-x 2 root root 10 4月 11 2018 media
drwxr-xr-x 2 root root 10 4月 11 2018 mnt
drwxr-xr-x 1 root root 49 8月 15 2022 opt
drwxr-xr-x 2 root root 10 12月 4 2018 proc
dr-xr-x--- 1 root root 35 5月 11 2020 root
drwxr-xr-x 11 root root 192 12月 4 2018 run
lrwxrwxrwx 1 root root 8 12月 4 2018 sbin -> usr/sbin
drwxr-xr-x 2 root root 10 4月 11 2018 srv
drwxr-xr-x 2 root root 10 12月 4 2018 sys
drwxrwxrwt 1 root root 10 4月 7 13:56 tmp
drwxr-xr-x 13 root root 207 12月 4 2018 usr
drwxr-xr-x 18 root root 318 12月 4 2018 var
重新启动容器
[root@20-121 d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385]# docker restart zentao
zentao
[root@20-121 d20f59f0e47adad276352a859d27da19097410e8198c2e840681738e79b5b385]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4e071c2b26cb zentao:v9.0.1 "/bin/sh /opt/run.sh" 2 years ago Up 2 seconds 0.0.0.0:8060->80/tcp, :::8060->80/tcp, 0.0.0.0:3307->3306/tcp, :::3307->3306/tcp zentao
... ...
服务恢复完毕，验证即可
浙公网安备 33010602011771号