docker 配置 direct-lvm
当前需要设置的宿主机是环境是搭建在vbox虚拟机上的centos7系统。测试环境中出现过一次意外情况,当时为了测试docker日志文件限制,运行了一个docker容器,但是后面忘记停止了,几天后发现了,想去操作,发现容器命令都无效了,停止不了,也登陆不上去。不明原因,最终只能重启docker服务。
问题原因排查:
经过一番查证,发现可能与docker存储模式有关系。分别是docker默认的loop volume和Direct LVM。我们先看下loop volume和Direct LVM 之间的区别在哪里?
- 对于centos/rhel这类没有相关驱动的系统,一般使用devicemapper驱动利用LVM的一些机制来模拟分层存储。这样的做法除了性能比较差之外,稳定性一般也不好,而且配置相对复杂。Docker安装在CentOS/RHEL 上后,会默认选择 devicemapper,但是为了简化配置,其 devicemapper 是跑在一个稀疏文件模拟的块设备上,也被称为 loop-lvm。这样的选择是因为不需要额外配置就可以运行 Docker,这是自动配置唯一能做到的事情。但是 loop-lvm 的做法非常不好,其稳定性、性能更差,无论是日志还是 docker info 中都会看到警告信息。官方文档有明确的文章讲解了如何配置块设备给 devicemapper 驱动做存储层的做法,这类做法也被称为配置 direct-lvm。
- 除了前面说到的问题外,devicemapper + loop-lvm 还有一个缺陷,因为它是稀疏文件,所以它会不断增长。用户在使用过程中会注意到/var/lib/docker/devicemapper/devicemapper/data 不断增长,而且无法控制。很多人会希望删除镜像或者可以解决这个问题,结果发现效果并不明显。原因就是这个稀疏文件的空间释放后基本不进行垃圾回收的问题。因此往往会出现即使删除了文件内容,空间却无法回收,随着使用这个稀疏文件一直在不断增长。
- 对于 CentOS/RHEL 的用户来说,在没有办法使用 UnionFS 的情况下,一定要配置 direct-lvm 给 devicemapper,无论是为了性能、稳定性还是空间利用率
查找了网上的一些测试说明,得出如下的结论:
![](https://images2018.cnblogs.com/blog/996253/201804/996253-20180403181446374-1781922546.png)
其余略。
二、配置Direct LVM
- fdisk -l 查看磁盘信息
123456789101112131415161718192021222324252627282930[root@etcd1 dev]
# fdisk -l
Disk
/dev/sda
: 42.9 GB, 42949672960 bytes, 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
Disk label
type
: dos
Disk identifier: 0x000a5130
Device Boot Start End Blocks Id System
/dev/sda1
2048 4095 1024 83 Linux
/dev/sda2
* 4096 2101247 1048576 83 Linux
/dev/sda3
2101248 83886079 40892416 8e Linux LVM
Disk
/dev/sdb
: 8589 MB, 8589934592 bytes, 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
Disk
/dev/mapper/VolGroup00-LogVol00
: 40.2 GB, 40231763968 bytes, 78577664 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
Disk
/dev/mapper/VolGroup00-LogVol01
: 1610 MB, 1610612736 bytes, 3145728 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
- 创建pv
12
[root@etcd1 dev]
# pvcreate /dev/sdb
Physical volume
"/dev/sdb"
successfully created.
- 创建vg (从生成pv后的硬盘 创建类似一个虚拟硬盘的东西)
12
[root@etcd1 dev]
# vgcreate vgdocker /dev/sdb
Volume group
"vgdocker"
successfully created
vgdocker”为vg名,“/dev/sdb” 为组成VG的PV路径.
创建完成之后,执行 vgdisplay 进行查看:123456789101112131415161718192021222324252627282930313233343536373839404142[root@etcd1 dev]
# vgdisplay
--- Volume group ---
VG Name vgdocker
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 6
VG Access
read
/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size <8.00 GiB
PE Size 4.00 MiB
Total PE 2047
Alloc PE / Size 1984 / 7.75 GiB
Free PE / Size 63 / 252.00 MiB
VG UUID ydhjST-R1MI-iBBY-RIEm-YDmS-mVUr-GvB5SN
--- Volume group ---
VG Name VolGroup00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access
read
/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size <38.97 GiB
PE Size 32.00 MiB
Total PE 1247
Alloc PE / Size 1247 / <38.97 GiB
Free PE / Size 0 / 0
VG UUID Bmtp50-WfBl-wRER-UckX-ryOe-RMgL-osDsd4
- 创建能够组成 thin-pool 的两个LV
1234
[root@etcd1 dev]
# lvcreate --wipesignatures y -n thinpool -l 95%VG vgdocker
Logical volume
"thinpool"
created.
[root@etcd1 dev]
# lvcreate --wipesignatures y -n thinpoolmeta -l 1%VG vgdocker
Logical volume
"thinpoolmeta"
created.
-
根据上一步的两个LV创建thin-pool(thinpool 是docker storage能用的)
12345[root@etcd1 dev]
# lvconvert -y --zero n -c 512K --thinpool vgdocker/thinpool --poolmetadata vgdocker/thinpoolmeta
Thin pool volume with chunk size 512.00 KiB can address at most 126.50 TiB of data.
WARNING: Converting logical volume vgdocker
/thinpool
and vgdocker
/thinpoolmeta
to thin pool's data and metadata <br> volumes with metadata wiping.
THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Converted vgdocker
/thinpool_tdata
to thin pool.
- 配置thin-pool的自动扩展
修改lvm配置文件12345[root@etcd1 dev]
# vi /etc/lvm/profile/docker-thinpool.profile
activation {
thin_pool_autoextend_threshold=80
thin_pool_autoextend_percent=20
}
- 激活lv的配置文件LVM profile
12
[root@etcd1 dev]
# lvchange --metadataprofile docker-thinpool vgdocker/thinpool
Logical volume vgdocker
/thinpool
changed.
-
对主机上的逻辑卷启用监视
12345[root@etcd1 dev]
# lvs -o+seg_monitor
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Monitor
LogVol00 VolGroup00 -wi-ao---- <37.47g
LogVol01 VolGroup00 -wi-ao---- 1.50g
thinpool vgdocker twi-a-t--- 7.59g 0.00 0.05 monitored
如果没有这个步骤,即使在LVM配置文件出现时也不会发生自动扩展。
-
备份Docker原有数据
如果曾经在这个主机上运行过Docker,或者如果/var/ lib/docker存在,那么将它移出后,Docker就可以使用新的LVM池来存储镜像和容器的内容了。1[root@etcd1 dev]
# mkdir /var/lib/docker.bk[root@etcd1 dev]# cp /var/lib/docker/*/var/lib/docker.bk
- 编辑 /etc/docker/daemon.json
配置devicemapper存储驱动程序所需的选项。如果该文件以前是空的,那么现在应该包含以下内容:12345678{
"storage-driver"
:
"devicemapper"
,
"storage-opts"
: [
"dm.thinpooldev=/dev/mapper/docker-thinpool"
,
"dm.use_deferred_removal=true"
,
"dm.use_deferred_deletion=true"
]
}
“/dev/mapper/docker-thinpool”为上述步骤中产生的thin-pool
注意:延迟删除选项,dm.use_deferred_deletion=true,在使用默认内核版本3.18时还不支持RHEL、CentOS或Ubuntu 14.04。
- 启动Docker
12
[root@etcd1 dev]
# systemctl daemon-reload
[root@etcd1 dev]
# systemctl restart docker
- 查看Docker启动信息
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162
[root@etcd1 dev]
# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: devicemapper
Pool Name: vgdocker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data
file
:
Metadata
file
:
Data Space Used: 19.92 MB
Data Space Total: 8.154 GB
Data Space Available: 8.134 GB
Metadata Space Used: 45.06 kB
Metadata Space Total: 83.89 MB
Metadata Space Available: 83.84 MB
Thin Pool Minimum Free Space: 815.3 MB
Udev Sync Supported:
true
Deferred Removal Enabled:
true
Deferred Deletion Enabled:
true
Deferred Deleted Device Count: 0
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: json-
file
Cgroup Driver: cgroupfs
Plugins:
Volume:
local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-693.17.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 488.4 MiB
Name: etcd1.wae.haplat.net
ID: AR6D:ZTRJ:PESP:REXP:ZZ5G:7EBB:CQZQ:5KAR:JR4L:S4VO:6PCR:EFMY
Docker Root Dir:
/var/lib/docker
Debug Mode (client):
false
Debug Mode (server):
false
Username: hakimdstx
Registry: https:
//index
.docker.io
/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Experimental:
false
Insecure Registries:
127.0.0.0
/8
Registry Mirrors:
https:
//registry
.docker-cn.com
Live Restore Enabled:
false
- 验证配置正确后,删除/var/lib/docker.bk目录了
1
[root@etcd1 dev]
# rm -rf /var/lib/docker.bk
综上,完成
【引用】
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步