Ceph RBD的使用笔记
ceph的内容太多了，单独写一篇文章记录自己的一些RBD的学习笔记，其实简书和其他博客上已经记录的非常全面了，但是因为出处都比较分散，所以还是自己把自己的实验记录一下便于以后学习的查阅。也感谢各位大佬的无私分享。
1.RBD池的创建和enable
[cephadmin@ceph-node ~]$ ceph osd pool create rbd 32 32 pool 'rbd' created [cephadmin@ceph-node ~]$ ceph osd pool application enable rbd rbd enabled application 'rbd' on pool 'rbd' [cephadmin@ceph-node ~]$ ceph osd pool ls detail pool 1 'test-pool' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 43 flags hashpspool max_bytes 214748364800 max_objects 50000 stripe_width 0 application rgw pool 2 'images' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 41 flags hashpspool stripe_width 0 pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 53 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 56 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 58 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 6 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 60 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 7 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 70 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 8 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 73 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 9 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 85 flags hashpspool stripe_width 0 application rbd
2.快设备的操作
rbd create --size 10240 image01 [cephadmin@ceph-node ~]$ rbd ls image01 [cephadmin@ceph-node ~]$ rbd info image01 rbd image 'image01': size 10 GiB in 2560 objects order 22 (4 MiB objects) id: 1218f6b8b4567 block_name_prefix: rbd_data.1218f6b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Tue Aug 15 11:23:05 2023 [cephadmin@ceph-node ~]$ rbd feature disable image01 exclusive-lock, object-map, fast-diff, deep-flatten [root@ceph-node1 ~]# rbd map image01 /dev/rbd0 [root@ceph-node1 ~]# rbd showmapped id pool image snap device 0 rbd image01 - /dev/rbd0 [root@ceph-node1 ~]# mkfs.xfs /dev/rbd0 Discarding blocks...Done. meta-data=/dev/rbd0 isize=512 agcount=16, agsize=163840 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=2621440, imaxpct=25 = sunit=1024 swidth=1024 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=8 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@ceph-node1 ~]# mount /dev/rbd0 /mnt [root@ceph-node1 ~]# cd /mnt [root@ceph-node1 mnt]# ls [root@ceph-node1 mnt]# df -TH|grep rbd0 /dev/rbd0 xfs 11G 35M 11G 1% /mnt
我是在另外一个osd节点上操作的，需要把ceph.conf和ceph.client.admin.keyring拷贝到/etc/ceph目录下去。
如果是非ceph节点，需要安装ceph-common的包
3. 往块中写入一个文件
[root@ceph-node1 mnt]# dd if=/dev/zero of=/mnt/rbd_test bs=1M count=300 300+0 records in 300+0 records out 314572800 bytes (315 MB) copied, 1.54531 s, 204 MB/s [root@ceph-node1 mnt]# ll total 307200 -rw-r--r-- 1 root root 314572800 Aug 15 11:47 rbd_test [root@ceph-node1 mnt]# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 24 GiB 20 GiB 3.9 GiB 16.22 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS test-pool 1 124 MiB 1.30 9.2 GiB 104 images 2 0 B 0 9.2 GiB 0 .rgw.root 3 1.1 KiB 0 6.2 GiB 4 default.rgw.control 4 0 B 0 6.2 GiB 8 default.rgw.meta 5 1.3 KiB 0 6.2 GiB 8 default.rgw.log 6 0 B 0 6.2 GiB 175 default.rgw.buckets.index 7 0 B 0 6.2 GiB 1 default.rgw.buckets.data 8 3.1 KiB 0 6.2 GiB 1 rbd 9 314 MiB 3.21 6.2 GiB 99
发现一个比较奇怪的问题，我的每个osd总空间是8G,但是RBD建立块设备是10G,居然也能成功，应该是按照消耗量来的，消耗多少算多少。另外OSD空间不够支持扩展？
4.缩容和扩容
[root@ceph-node1 mnt]# rbd --image image01 resize --size 4096 --allow-shrink Resizing image: 100% complete...done. [root@ceph-node1 mnt]# rbd info image01 rbd image 'image01': size 4 GiB in 1024 objects order 22 (4 MiB objects) id: 1218f6b8b4567 block_name_prefix: rbd_data.1218f6b8b4567 format: 2 features: layering op_features: flags: create_timestamp: Tue Aug 15 11:23:05 2023 [root@ceph-node1 mnt]# df -Th Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 908M 0 908M 0% /dev tmpfs tmpfs 919M 0 919M 0% /dev/shm tmpfs tmpfs 919M 8.6M 911M 1% /run tmpfs tmpfs 919M 0 919M 0% /sys/fs/cgroup /dev/mapper/centos-root xfs 8.0G 1.8G 6.2G 23% / /dev/sda1 xfs 1014M 181M 834M 18% /boot tmpfs tmpfs 919M 28K 919M 1% /var/lib/ceph/osd/ceph-1 tmpfs tmpfs 184M 0 184M 0% /run/user/0 /dev/rbd0 xfs 10G 333M 9.7G 4% /mnt
df来看并没有什么变化，查阅了一些材料，基本上xfs不支持缩容
[root@ceph-node1 mnt]# rbd --image image01 resize --size 20480 Resizing image: 100% complete...done. [root@ceph-node1 mnt]# rbd info image01 rbd image 'image01': size 20 GiB in 5120 objects order 22 (4 MiB objects) id: 1218f6b8b4567 block_name_prefix: rbd_data.1218f6b8b4567 format: 2 features: layering op_features: flags: create_timestamp: Tue Aug 15 11:23:05 2023 [root@ceph-node1 mnt]# xfs_growfs /mnt meta-data=/dev/rbd0 isize=512 agcount=16, agsize=163840 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0 spinodes=0 data = bsize=4096 blocks=2621440, imaxpct=25 = sunit=1024 swidth=1024 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=8 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Invalid argument data blocks changed from 2621440 to 5242880 [root@ceph-node1 mnt]# df -Th Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 908M 0 908M 0% /dev tmpfs tmpfs 919M 0 919M 0% /dev/shm tmpfs tmpfs 919M 8.6M 911M 1% /run tmpfs tmpfs 919M 0 919M 0% /sys/fs/cgroup /dev/mapper/centos-root xfs 8.0G 1.9G 6.2G 23% / /dev/sda1 xfs 1014M 181M 834M 18% /boot tmpfs tmpfs 919M 28K 919M 1% /var/lib/ceph/osd/ceph-1 tmpfs tmpfs 184M 0 184M 0% /run/user/0 /dev/rbd0 xfs 20G 334M 20G 2% /mnt
问题解决
中间虚机出问题突然挂死，然后重启启动以后发现无法mount上了。
主要报错信息 mount: wrong fs type, bad option, bad superblock on /dev/rbd0, missing codepage or helper program, or other error ...... #dmesg一下 [ 230.385386] rbd: loaded (major 252) [ 230.390674] libceph: mon0 192.168.56.110:6789 session established [ 230.391311] libceph: client84118 fsid 90933ecf-4a54-475b-8222-b2c57868512a [ 230.471613] rbd: rbd0: capacity 21474836480 features 0x1 [ 259.268165] XFS (rbd0): Mounting V5 Filesystem [ 259.268885] XFS (rbd0): totally zeroed log [ 259.269379] XFS (rbd0): Corruption warning: Metadata has LSN (1:512) ahead of current LSN (1:0). Please unmount and run xfs_repair (>= v4.3) to resolve. [ 259.269382] XFS (rbd0): log mount/recovery failed: error -22 [ 259.269400] XFS (rbd0): log mount failed [ 295.912616] XFS (rbd0): Mounting V5 Filesystem [ 295.913160] XFS (rbd0): totally zeroed log [ 295.913576] XFS (rbd0): Corruption warning: Metadata has LSN (1:512) ahead of current LSN (1:0). Please unmount and run xfs_repair (>= v4.3) to resolve. [ 295.913578] XFS (rbd0): log mount/recovery failed: error -22 [ 295.913597] XFS (rbd0): log mount failed [ 504.223454] XFS (rbd0): Mounting V5 Filesystem [ 504.225363] XFS (rbd0): totally zeroed log [ 504.225880] XFS (rbd0): Corruption warning: Metadata has LSN (1:512) ahead of current LSN (1:0). Please unmount and run xfs_repair (>= v4.3) to resolve. [ 504.225883] XFS (rbd0): log mount/recovery failed: error -22 [ 504.225904] XFS (rbd0): log mount failed [ 519.475527] XFS (rbd0): Mounting V5 Filesystem [ 519.475956] XFS (rbd0): totally zeroed log [ 519.476219] XFS (rbd0): Corruption warning: Metadata has LSN (1:512) ahead of current LSN (1:0). Please unmount and run xfs_repair (>= v4.3) to resolve. [ 519.476235] XFS (rbd0): log mount/recovery failed: error -22 [ 519.476256] XFS (rbd0): log mount failed [root@ceph-node1 /]# xfs_repair /dev/rbd0 Phase 1 - find and verify superblock... - reporting progress in intervals of 15 minutes Phase 2 - using internal log - zero log... totally zeroed log - scan filesystem freespace and inode maps... bad magic number bad magic number bad magic number 。。。。 [root@ceph-node1 /]# mount /dev/rbd0 /mnt [root@ceph-node1 /]# cd /mnt [root@ceph-node1 mnt]# ls rbd_test
5.查看有没有客户端已经mount了这个块设备
[cephadmin@ceph-node ~]$ rbd info image01 rbd image 'image01': size 20 GiB in 5120 objects order 22 (4 MiB objects) id: 1218f6b8b4567 block_name_prefix: rbd_data.1218f6b8b4567 format: 2 features: layering op_features: flags: create_timestamp: Tue Aug 15 11:23:05 2023 [cephadmin@ceph-node ~]$ rados -p rbd listwatchers rbd_header.1218f6b8b4567 watcher=192.168.56.111:0/2196229061 client.84118 cookie=18446462598732840961
参考链接
https://www.jianshu.com/p/712e58d36a77
https://www.cnblogs.com/zhrx/p/16098075.html
https://zhuanlan.zhihu.com/p/343376229
