ceph的rbd块设备的使用指南
目录
一.rbd块设备的增删改查
推荐阅读:
https://docs.ceph.com/en/nautilus/rbd/rados-rbd-cmds/
1.创建存储池
[root@ceph141 ~]# ceph osd pool create yinzhengjie-rbd 16 16
pool 'yinzhengjie-rbd' created
[root@ceph141 ~]#
2.使用rbd工具初始化池以供RBD块设备使用
1.对存储池启用rbd功能.
[root@ceph141 ~]# ceph osd pool application get yinzhengjie-rbd
{}
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd pool application enable yinzhengjie-rbd rbd
enabled application 'rbd' on pool 'yinzhengjie-rbd'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd pool application get yinzhengjie-rbd
{
"rbd": {}
}
[root@ceph141 ~]#
2.对存储池进行环境初始化
[root@ceph141 ~]# rbd pool init yinzhengjie-rbd
[root@ceph141 ~]#
[root@ceph141 ~]# rbd pool stats yinzhengjie-rbd
Total Images: 0
Total Snapshots: 0
Provisioned Size: 0 B
[root@ceph141 ~]#
温馨提示:
- 1.如果rbd存储池不启用功能，则使用"ceph -s"命令查看时就会提示报错"1 pool(s) do not have an application enabled"信息；
- 2.可以使用"osd pool application enable <pool> <app>"来启用存储池的app功能，其中app的有效值为: "cephfs","rbd","rgw"；
3.创建一个2G的rbd设备
[root@ceph141 ~]# rbd create wordpress --size 2048 -p yinzhengjie-rbd
4.查看rbd块设备
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd
wordpress
[root@ceph141 ~]#
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME SIZE PARENT FMT PROT LOCK
wordpress 2 GiB 2
[root@ceph141 ~]#
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l --format json
[{"image":"wordpress","size":2147483648,"format":2}]
[root@ceph141 ~]#
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l --format json --pretty-format
[
{
"image": "wordpress",
"size": 2147483648,
"format": 2
}
]
[root@ceph141 ~]#
5. 查看镜像的详细信息
[root@ceph141 ~]# rbd info -p yinzhengjie-rbd wordpress
rbd image 'wordpress':
size 2 GiB in 512 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 12dc1b0f25c9
block_name_prefix: rbd_data.12dc1b0f25c9
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Thu Feb 1 11:42:11 2024
access_timestamp: Thu Feb 1 11:42:11 2024
modify_timestamp: Thu Feb 1 11:42:11 2024
[root@ceph141 ~]#
[root@ceph141 ~]# rbd info yinzhengjie-rbd/wordpress
rbd image 'wordpress':
size 2 GiB in 512 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 12dc1b0f25c9
block_name_prefix: rbd_data.12dc1b0f25c9
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Thu Feb 1 11:42:11 2024
access_timestamp: Thu Feb 1 11:42:11 2024
modify_timestamp: Thu Feb 1 11:42:11 2024
[root@ceph141 ~]#
常用的镜像features属性说明:
- layering:
分层克隆机制，磁盘的数据分层获取克隆机制。
- striping:
是否支持数据对象间的数据条带化。
- exclusive-lock:
排它锁的机制，磁盘应用于多路写机制场景，限制同时仅能有有一个客户端访问当前image。
- object-map:
对象位图机制，主要用于加速导入，导出及已用容量统计等操作，依赖于"exclusive-lock"特性。
- fast-diff:
快照定制机制，快速对比数据差异，便于做快照管理，依赖于object-map特性。
- deep-flatten:
数据处理机制，解除父子image及快照的依赖关系。
- journaling:
磁盘日志机制，将image的所有修改操作进行日志话，便于异地备份，依赖于"exclusive-lock"特性。
- data-pool:
是否支持将image的数据对象存储于纠删码存储池，主要用于将image的元数据于数据放置于不同的存储池。
6.修改rbd
1 修改rbd的名称
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME SIZE PARENT FMT PROT LOCK
wordpress 2 GiB 2
[root@ceph141 ~]#
[root@ceph141 ~]# rbd rename wordpress wp -p yinzhengjie-rbd
[root@ceph141 ~]#
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME SIZE PARENT FMT PROT LOCK
wp 2 GiB 2
[root@ceph141 ~]#
2 修改rbd的大小
[root@ceph141 ~]# rbd ls --pool yinzhengjie-rbd -l
NAME SIZE PARENT FMT PROT LOCK
wp 2 GiB 2
[root@ceph141 ~]#
[root@ceph141 ~]# rbd resize -s 4G yinzhengjie-rbd/wp
Resizing image: 100% complete...done.
[root@ceph141 ~]#
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME SIZE PARENT FMT PROT LOCK
wp 4 GiB 2
[root@ceph141 ~]#
[root@ceph141 ~]# rbd resize --size 10G --pool yinzhengjie-rbd --image wp
Resizing image: 100% complete...done.
[root@ceph141 ~]#
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME SIZE PARENT FMT PROT LOCK
wp 10 GiB 2
[root@ceph141 ~]#
7.删除块设备
基于回收站临时删除块设备：（回收站，推荐，【但并不释放空间】）
推荐阅读:
https://www.cnblogs.com/yinzhengjie/p/18383834
永久删除块设备
[root@ceph151 ~]# rbd ls --pool yinzhengjie-rbd -l
NAME SIZE PARENT FMT PROT LOCK
wp 10 GiB 2
[root@ceph151 ~]#
[root@ceph151 ~]# rbd rm yinzhengjie-rbd/wp
Removing image: 100% complete...done.
[root@ceph151 ~]#
[root@ceph151 ~]# rbd ls --pool yinzhengjie-rbd -l
[root@ceph151 ~]#
二.CentOS环境ceph客户端使用rbd实战
1.安装ceph的客户端模块
[root@harbor250 ~]# cat /etc/yum.repos.d/ceph.repo
[Ceph]
name=Ceph packages for
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/$basearch
gpgcheck=0
[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch
gpgcheck=0
[ceph-source]
name=Ceph source packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/SRPMS
gpgcheck=0
[root@harbor250 ~]#
[root@harbor250 ~]# yum -y install ceph-common
2.在RBD存储池创建块设备
1 创建镜像时可以指定镜像的特性
[root@ceph143 ~]# rbd create -s 2G yinzhengjie-rbd/wordpress
[root@ceph143 ~]#
[root@ceph143 ~]# rbd create -s 5G -p yinzhengjie-rbd --image mysqld --image-feature layering,exclusive-lock
[root@ceph143 ~]#
[root@ceph143 ~]# rbd ls -l -p yinzhengjie-rbd
NAME SIZE PARENT FMT PROT LOCK
mysqld 5 GiB 2
wordpress 2 GiB 2
[root@ceph143 ~]#
2 查看镜像的详细信息
[root@ceph142 ~]# rbd info -p yinzhengjie-rbd wordpress | grep '^[[:space:]]features'
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
[root@ceph142 ~]#
[root@ceph142 ~]# rbd info -p yinzhengjie-rbd mysqld | grep "\sfeatures"
features: layering, exclusive-lock
[root@ceph142 ~]#
3.修改RBD的镜像特性，由于centos 7 Linux内核版本过低(3.10.0-1127.el7.x86_64)，不支持更多的特性
[root@ceph141 ~]# rbd info -p yinzhengjie-rbd wordpress | grep '^[[:space:]]features'
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
[root@ceph141 ~]#
[root@ceph141 ~]# rbd feature disable yinzhengjie-rbd/wordpress object-map fast-diff deep-flatten
[root@ceph141 ~]#
[root@ceph141 ~]# rbd info -p yinzhengjie-rbd wordpress | grep '\sfeatures'
features: layering, exclusive-lock
[root@ceph141 ~]#
4.客户端映射镜像(块设备)
1 拷贝ceph的配置文件，主要用于识别ceph集群地址
[root@harbor250 ~]# scp ceph141:/etc/ceph/ceph.conf /etc/ceph/
ceph.conf 100% 264 113.6KB/s 00:00
[root@harbor250 ~]#
[root@harbor250 ~]# grep mon_host /etc/ceph/ceph.conf
mon_host = 10.0.0.141,10.0.0.142,10.0.0.143
[root@harbor250 ~]#
2 拷贝ceph的认证文件，主要用于ceph集群认证
[root@harbor250 ~]# scp ceph141:/etc/ceph/ceph.client.admin.keyring /etc/ceph/
ceph.client.admin.keyring 100% 151 111.1KB/s 00:00
[root@harbor250 ~]#
[root@harbor250 ~]# grep key /etc/ceph/ceph.client.admin.keyring
key = AQDjFrplyvFCDhAApJg111YMIGQ6/F/x/Y+qpQ==
[root@harbor250 ~]#
3 添加映射
[root@harbor250 ~]# ll /dev/rbd*
ls: cannot access /dev/rbd*: No such file or directory
[root@harbor250 ~]#
[root@harbor250 ~]# rbd -p yinzhengjie-rbd map mysqld
/dev/rbd0
[root@harbor250 ~]#
[root@harbor250 ~]# ll /dev/rbd*
brw-rw---- 1 root disk 252, 0 Feb 1 12:08 /dev/rbd0
/dev/rbd:
total 0
drwxr-xr-x 2 root root 60 Feb 1 12:08 yinzhengjie-rbd
[root@harbor250 ~]#
[root@harbor250 ~]# rbd -p yinzhengjie-rbd map wordpress
/dev/rbd1
[root@harbor250 ~]#
[root@harbor250 ~]# ll /dev/rbd*
brw-rw---- 1 root disk 252, 0 Feb 1 12:08 /dev/rbd0
brw-rw---- 1 root disk 252, 16 Feb 1 12:14 /dev/rbd1
/dev/rbd:
total 0
drwxr-xr-x 2 root root 80 Feb 1 12:14 yinzhengjie-rbd
[root@harbor250 ~]#
5.对块设备进行格式化
[root@harbor250 ~]# mkfs.xfs /dev/rbd0
meta-data=/dev/rbd0 isize=512 agcount=8, agsize=163840 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=1310720, imaxpct=25
= sunit=1024 swidth=1024 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@harbor250 ~]#
[root@harbor250 ~]# mkfs.ext4 /dev/rbd1
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1024 blocks, Stripe width=1024 blocks
131072 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=536870912
16 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
[root@harbor250 ~]#
6.创建挂载点
[root@harbor250 ~]# mkdir -pv /yinzhengjie/data/{wordpress,mysql}
mkdir: created directory ‘/yinzhengjie/data’
mkdir: created directory ‘/yinzhengjie/data/wordpress’
mkdir: created directory ‘/yinzhengjie/data/mysql’
[root@harbor250 ~]#
7.尝试手动挂载
[root@harbor250 ~]# mount /dev/rbd0 /yinzhengjie/data/mysql
[root@harbor250 ~]#
[root@harbor250 ~]# mount /dev/rbd1 /yinzhengjie/data/wordpress
[root@harbor250 ~]#
[root@harbor250 ~]# df -h | grep rbd
/dev/rbd0 5.0G 33M 5.0G 1% /yinzhengjie/data/mysql
/dev/rbd1 2.0G 6.0M 1.8G 1% /yinzhengjie/data/wordpress
[root@harbor250 ~]#
8.配置ceph开机自启动，(做完实验后可以注释掉，以免当你删除存储池后，无法挂载块设备的问题。)
[root@harbor250 ~]# vim /etc/rc.d/rc.local
...
rbd -p yinzhengjie-rbd map mysqld
rbd -p yinzhengjie-rbd map wordpress
mount /dev/rbd0 /yinzhengjie/data/mysql
mount /dev/rbd1 /yinzhengjie/data/wordpress
[root@harbor250 ~]#
[root@harbor250 ~]# ll /etc/rc.d/rc.local
-rw-r--r-- 1 root root 620 Feb 1 12:21 /etc/rc.d/rc.local
[root@harbor250 ~]#
[root@harbor250 ~]#
[root@harbor250 ~]# chmod +x /etc/rc.d/rc.local
[root@harbor250 ~]#
[root@harbor250 ~]# ll /etc/rc.d/rc.local
-rwxr-xr-x 1 root root 620 Feb 1 12:21 /etc/rc.d/rc.local
[root@harbor250 ~]#
[root@harbor250 ~]# df -h | grep rbd
/dev/rbd0 5.0G 33M 5.0G 1% /yinzhengjie/data/mysql
/dev/rbd1 2.0G 6.0M 1.8G 1% /yinzhengjie/data/wordpress
[root@harbor250 ~]#
[root@harbor250 ~]# umount /yinzhengjie/data/mysql
[root@harbor250 ~]#
[root@harbor250 ~]# df -h | grep rbd
/dev/rbd1 2.0G 6.0M 1.8G 1% /yinzhengjie/data/wordpress
[root@harbor250 ~]#
[root@harbor250 ~]#
[root@harbor250 ~]# umount /dev/rbd1
[root@harbor250 ~]#
[root@harbor250 ~]# df -h | grep rbd
[root@harbor250 ~]#
[root@harbor250 ~]# reboot
[root@harbor250 ~]#
[root@harbor250 ~]# df -h | grep yinzhengjie
/dev/rbd0 5.0G 33M 5.0G 1% /yinzhengjie/data/mysql
/dev/rbd1 2.0G 6.0M 1.8G 1% /yinzhengjie/data/wordpress
[root@harbor250 ~]#
9.测试RBD块设备的可用性
[root@harbor250 ~]# cp /etc/os-release /yinzhengjie/data/mysql/
[root@harbor250 ~]#
[root@harbor250 ~]# cp /etc/hosts /yinzhengjie/data/mysql/
[root@harbor250 ~]#
[root@harbor250 ~]# cp /etc/fstab /yinzhengjie/data/wordpress/
[root@harbor250 ~]#
[root@harbor250 ~]# ll /yinzhengjie/data/*
/yinzhengjie/data/mysql:
total 8
-rw-r--r-- 1 root root 288 Feb 1 12:24 hosts
-rw-r--r-- 1 root root 393 Feb 1 12:24 os-release
/yinzhengjie/data/wordpress:
total 20
-rw-r--r-- 1 root root 465 Feb 1 12:24 fstab
drwx------ 2 root root 16384 Feb 1 12:17 lost+found
[root@harbor250 ~]#
10.卸载设备
[root@harbor250 ~]# umount /yinzhengjie/data/mysql
[root@harbor250 ~]#
[root@harbor250 ~]# umount /yinzhengjie/data/wordpress
[root@harbor250 ~]#
[root@harbor250 ~]# df | grep yinzhengjie
[root@harbor250 ~]#
11.其他客户端重新挂载设备观察数据是否丢失
[root@k8s-worker05 ~]# cat /etc/yum.repos.d/ceph.repo
[Ceph]
name=Ceph packages for
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/$basearch
gpgcheck=0
[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch
gpgcheck=0
[ceph-source]
name=Ceph source packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/SRPMS
gpgcheck=0
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# yum -y install ceph-common
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# scp 10.0.0.141:/etc/ceph/ceph.client.admin.keyring /etc/ceph/
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# scp 10.0.0.141:/etc/ceph/ceph.conf /etc/ceph/
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# rbd map yinzhengjie-rbd/mysqld
/dev/rbd0
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# mount /dev/rbd0 /mnt/
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# ll /mnt/
total 8
-rw-r--r-- 1 root root 288 Feb 1 12:24 hosts
-rw-r--r-- 1 root root 393 Feb 1 12:24 os-release
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# mkdir /opt/yinzhengjie
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 0
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# rbd map yinzhengjie-rbd/wordpress
/dev/rbd1
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# mount /dev/rbd1 /opt/yinzhengjie/
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 20
-rw-r--r-- 1 root root 465 Feb 1 12:24 fstab
drwx------ 2 root root 16384 Feb 1 12:17 lost+found
[root@k8s-worker05 ~]#
12.两个节点不得使用同一个块设备，否则可能会出现数据不一致的问题！各节点应该单独使用块设备。
1.客户端1操作如下:
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 20
-rw-r--r-- 1 root root 465 Feb 1 12:24 fstab
drwx------ 2 root root 16384 Feb 1 12:17 lost+found
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# cp /etc/sysconfig/network-scripts/ifcfg-ens33 /opt/yinzhengjie/
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# cp /etc/resolv.conf /opt/yinzhengjie
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 28
-rw-r--r-- 1 root root 465 Feb 1 12:24 fstab
-rw-r--r-- 1 root root 163 Feb 1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb 1 12:17 lost+found
-rw-r--r-- 1 root root 72 Feb 1 12:34 resolv.conf
[root@k8s-worker05 ~]#
2.客户端2操作如下:
[root@harbor250 ~]# mount /dev/rbd0 /yinzhengjie/data/mysql/
[root@harbor250 ~]#
[root@harbor250 ~]# ll /yinzhengjie/data/mysql/
total 8
-rw-r--r-- 1 root root 288 Feb 1 12:24 hosts
-rw-r--r-- 1 root root 393 Feb 1 12:24 os-release
[root@harbor250 ~]#
[root@harbor250 ~]#
[root@harbor250 ~]# mount /dev/rbd1 /yinzhengjie/data/wordpress/
[root@harbor250 ~]#
[root@harbor250 ~]#
[root@harbor250 ~]# ll /yinzhengjie/data/wordpress/
total 28
-rw-r--r-- 1 root root 465 Feb 1 12:24 fstab
-rw-r--r-- 1 root root 163 Feb 1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb 1 12:17 lost+found
-rw-r--r-- 1 root root 72 Feb 1 12:34 resolv.conf
[root@harbor250 ~]#
3.客户端1继续操作如下:
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 28
-rw-r--r-- 1 root root 465 Feb 1 12:24 fstab
-rw-r--r-- 1 root root 163 Feb 1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb 1 12:17 lost+found
-rw-r--r-- 1 root root 72 Feb 1 12:34 resolv.conf
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# cp /etc/ceph/ceph.conf /opt/yinzhengjie
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# ll /opt/yinzhengjie
total 32
-rw-r--r-- 1 root root 264 Feb 1 12:36 ceph.conf
-rw-r--r-- 1 root root 465 Feb 1 12:24 fstab
-rw-r--r-- 1 root root 163 Feb 1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb 1 12:17 lost+found
-rw-r--r-- 1 root root 72 Feb 1 12:34 resolv.conf
[root@k8s-worker05 ~]#
4.客户端2再次验证，不难发现，两个节点数据已经开始不一致啦！为了避免这种情况，不建议两个节点使用同一个块设备。
[root@harbor250 ~]# ll /yinzhengjie/data -R
/yinzhengjie/data:
total 4
drwxr-xr-x 2 root root 37 Feb 1 12:24 mysql
drwxr-xr-x 3 root root 4096 Feb 1 12:34 wordpress
/yinzhengjie/data/mysql:
total 8
-rw-r--r-- 1 root root 288 Feb 1 12:24 hosts
-rw-r--r-- 1 root root 393 Feb 1 12:24 os-release
/yinzhengjie/data/wordpress:
total 28
-rw-r--r-- 1 root root 465 Feb 1 12:24 fstab
-rw-r--r-- 1 root root 163 Feb 1 12:34 ifcfg-ens33
drwx------ 2 root root 16384 Feb 1 12:17 lost+found
-rw-r--r-- 1 root root 72 Feb 1 12:34 resolv.conf
/yinzhengjie/data/wordpress/lost+found:
total 0
[root@harbor250 ~]#
13.温馨提示
[root@k8s-worker05 ~]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# uname -r
4.19.12-1.el7.elrepo.x86_64
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]#
[root@k8s-worker05 ~]# rbd map yinzhengjie-rbd/k8s # 尽管将内核升级为4系列的，发现依旧不好使，建议将内核升级为5系列的。
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable yinzhengjie-rbd/k8s object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
[root@k8s-worker05 ~]#
三. Ubuntu环境ceph客户端使用rbd实战
1.Ubuntu安装ceph环境
[root@prometheus-server32 ~]# wget -q -O- 'https://download.ceph.com/keys/release.asc' | apt-key add -
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# echo deb https://download.ceph.com/debian-reef/ $(lsb_release -sc) main | tee /etc/apt/sources.list.d/ceph.list
deb https://download.ceph.com/debian-reef/ jammy main
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# apt update
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# apt -y install ceph-common
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# ceph --version
ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
[root@prometheus-server32 ~]#
温馨提示:
但是官方对于Ceph的Nautilus(V14.2.22)版本而言，最高支持的Ubuntu系统是: Ubuntu 20.04 LTS Focal ，最高支持的CentOS 7。
由于我的系统是"Ubuntu 22.04 LTS Fammy"，不支持Nautilus(V14.2.22)版本安装。
因此我安装的Ceph reef(V18.2.1)版本，如果想要安装和我们自定义的Nautilus(V14.2.22)版本。需要降低Ubuntu系统的版本号并做如下调整。
[root@prometheus-server32 ~]# wget -q -O- 'https://download.ceph.com/keys/release.asc' | apt-key add -
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# echo deb https://download.ceph.com/debian-nautilus/ $(lsb_release -sc) main | tee /etc/apt/sources.list.d/ceph.list
deb https://download.ceph.com/debian-nautilus/ jammy main
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# cat /etc/apt/sources.list.d/ceph.list
deb https://download.ceph.com/debian-nautilus/ jammy main
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# apt update
如果网速慢的同学，可以将"download.ceph.com"更换为"mirrors.aliyun.com"以提高速度。
参考链接:
https://docs.ceph.com/en/latest/install/get-packages/#apt
https://docs.ceph.com/en/latest/install/get-packages/#debian-packages
2.Ubuntu拷贝ceph文件及证书
[root@prometheus-server32 ~]# scp 10.0.0.141:/etc/ceph/ceph.c* /etc/ceph/
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# ceph -s
cluster:
id: 5821e29c-326d-434d-a5b6-c492527eeaad
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 21h)
mgr: ceph141(active, since 21h), standbys: ceph143, ceph142
osd: 7 osds: 7 up (since 21h), 7 in (since 21h)
data:
pools: 2 pools, 32 pgs
objects: 43 objects, 86 MiB
usage: 7.7 GiB used, 1.9 TiB / 2.0 TiB avail
pgs: 32 active+clean
[root@prometheus-server32 ~]#
3.创建镜像设备
[root@prometheus-server32 ~]# rbd create -s 5G yinzhengjie-rbd/k8s
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# rbd ls -p yinzhengjie-rbd
k8s
mysqld
wordpress
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# rbd info yinzhengjie-rbd/k8s
rbd image 'k8s':
size 5 GiB in 1280 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 13fa4379e5c8
block_name_prefix: rbd_data.13fa4379e5c8
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Thu Feb 1 07:21:28 2024
access_timestamp: Thu Feb 1 07:21:28 2024
modify_timestamp: Thu Feb 1 07:21:28 2024
[root@prometheus-server32 ~]#
4.Ubuntu系统添加映射
[root@prometheus-server32 ~]# uname -r
5.15.0-92-generic
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# rbd info yinzhengjie-rbd/k8s | grep '\sfeatures'
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# rbd showmapped
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# rbd map yinzhengjie-rbd/k8s
/dev/rbd0
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# rbd showmapped # 查看本地的挂载信息。
id pool namespace image snap device
0 yinzhengjie-rbd k8s - /dev/rbd0
[root@prometheus-server32 ~]#
5.对设备进行格式化并挂载
[root@prometheus-server32 ~]# mkfs.xfs /dev/rbd0
meta-data=/dev/rbd0 isize=512 agcount=8, agsize=163840 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=1310720, imaxpct=25
= sunit=16 swidth=16 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Discarding blocks...Done.
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# mount /dev/rbd0 /mnt/
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# df -h | grep mnt
/dev/rbd0 5.0G 69M 5.0G 2% /mnt
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# cp /etc/os-release /etc/hosts /mnt/
[root@prometheus-server32 ~]#
[root@prometheus-server32 ~]# ll /mnt/
total 12
drwxr-xr-x 2 root root 37 Feb 1 07:25 ./
drwxr-xr-x 20 root root 4096 Jan 25 02:55 ../
-rw-r--r-- 1 root root 226 Feb 1 07:25 hosts
-rw-r--r-- 1 root root 386 Feb 1 07:25 os-release
[root@prometheus-server32 ~]#
6.卸载设备
[root@prometheus-server32 ~]# umount /mnt
7.centos系统尝试挂载设备并验证数据
[root@harbor250 ~]# uname -r
3.10.0-1127.el7.x86_64
[root@harbor250 ~]#
[root@harbor250 ~]#
[root@harbor250 ~]# rbd map yinzhengjie-rbd/k8s # 发现挂载会失败，报错所不支持特性
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable yinzhengjie-rbd/k8s object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
[root@harbor250 ~]#
[root@harbor250 ~]# rbd info yinzhengjie-rbd/k8s | grep '\sfeatures'
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
[root@harbor250 ~]#
[root@harbor250 ~]# rbd feature disable yinzhengjie-rbd/k8s object-map fast-diff deep-flatten # 根据上面的报错禁用特性
[root@harbor250 ~]#
[root@harbor250 ~]# rbd info yinzhengjie-rbd/k8s | grep '\sfeatures'
features: layering, exclusive-lock
[root@harbor250 ~]#
[root@harbor250 ~]# rbd map yinzhengjie-rbd/k8s # 禁用特性后，发现可以映射了，映射的设备类型为"/dev/rbd2"
/dev/rbd2
[root@harbor250 ~]#
[root@harbor250 ~]# mount /dev/rbd2 /mnt/ # 很明显，挂载失败了，说是不支持块设备。
mount: wrong fs type, bad option, bad superblock on /dev/rbd2,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
[root@harbor250 ~]#
[root@harbor250 ~]# mkfs.xfs /dev/rbd2 # 此处我重新格式化，发现说已经存在了xfs系统，因此需要强行格式化
mkfs.xfs: /dev/rbd2 appears to contain an existing filesystem (xfs).
mkfs.xfs: Use the -f option to force overwrite.
[root@harbor250 ~]#
[root@harbor250 ~]#
[root@harbor250 ~]# mkfs.xfs /dev/rbd2 -f # 强行格式化后，数据会丢失
meta-data=/dev/rbd2 isize=512 agcount=8, agsize=163840 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=1310720, imaxpct=25
= sunit=1024 swidth=1024 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@harbor250 ~]#
[root@harbor250 ~]# mount /dev/rbd2 /mnt/ # 格式化后再次挂载，发现数据丢失了。
[root@harbor250 ~]#
[root@harbor250 ~]# ll /mnt/
total 0
[root@harbor250 ~]#
[root@harbor250 ~]# cp /etc/sysconfig/network-scripts/ifcfg-ens33 /mnt/ # 添加数据
[root@harbor250 ~]#
[root@harbor250 ~]# ll /mnt/
total 4
-rw-r--r-- 1 root root 147 Feb 1 15:32 ifcfg-ens33
[root@harbor250 ~]#
[root@harbor250 ~]#
[root@harbor250 ~]# cat /mnt/ifcfg-ens33
TYPE="Ethernet"
BOOTPROTO="static"
NAME="ens33"
DEVICE="ens33"
ONBOOT="yes"
IPADDR="10.0.0.250"
PREFIX="24"
GATEWAY="10.0.0.254"
DNS1="10.0.0.254"
[root@harbor250 ~]#
[root@harbor250 ~]#
[root@harbor250 ~]# umount /mnt
[root@harbor250 ~]#
[root@harbor250 ~]# rbd unmap yinzhengjie-rbd/k8s # 取消映射
[root@harbor250 ~]#
四.ceph集群环境在线扩容案例xfs文件系统案例
1.实验环境
[root@harbor250 ~]# df -h | grep data
/dev/rbd0 10G 105M 9.9G 2% /data-rbd-xfs
/dev/rbd1 3.9G 28K 3.7G 1% /data-rbd-ext4
[root@harbor250 ~]#
2.扩容xfs系统
[root@ceph141 ~]# rbd -p yinzhengjie ls -l
NAME SIZE PARENT FMT PROT LOCK
mysql80 4 GiB 2 excl
wordpress 10 GiB 2 excl
[root@ceph141 ~]#
[root@ceph141 ~]#
[root@ceph141 ~]# rbd resize -s 20G yinzhengjie/wordpress
Resizing image: 100% complete...done.
[root@ceph141 ~]#
[root@ceph141 ~]# rbd -p yinzhengjie ls -l
NAME SIZE PARENT FMT PROT LOCK
mysql80 4 GiB 2 excl
wordpress 20 GiB 2 excl
[root@ceph141 ~]#
3.客户端验证
[root@harbor250 ~]# fdisk -l /dev/rbd0
Disk /dev/rbd0: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
[root@harbor250 ~]#
[root@harbor250 ~]# df -h | grep data
/dev/rbd0 10G 105M 9.9G 2% /data-rbd-xfs
/dev/rbd1 3.9G 28K 3.7G 1% /data-rbd-ext4
[root@harbor250 ~]#
4.自动扩展XFS文件系统到最大的可用大小
[root@harbor250 ~]# fdisk -l /dev/rbd0
Disk /dev/rbd0: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
[root@harbor250 ~]#
[root@harbor250 ~]# df -h | grep data
/dev/rbd0 10G 105M 9.9G 2% /data-rbd-xfs
/dev/rbd1 3.9G 28K 3.7G 1% /data-rbd-ext4
[root@harbor250 ~]#
[root@harbor250 ~]# xfs_growfs /data-rbd-xfs/
meta-data=/dev/rbd0 isize=512 agcount=16, agsize=163840 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=16 swidth=16 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 2621440 to 5242880
[root@harbor250 ~]#
[root@harbor250 ~]# df -h | grep data
/dev/rbd0 20G 177M 20G 1% /data-rbd-xfs
/dev/rbd1 3.9G 28K 3.7G 1% /data-rbd-ext4
[root@harbor250 ~]#
五.ceph集群环境在线扩容案例ext4文件系统案例
1.实验前环境准备
[root@harbor250 ~]# df -h | grep data
/dev/rbd0 20G 177M 20G 1% /data-rbd-xfs
/dev/rbd1 3.9G 28K 3.7G 1% /data-rbd-ext4
[root@harbor250 ~]#
2.扩容ext4系统
[root@ceph141 ~]# rbd -p yinzhengjie ls -l
NAME SIZE PARENT FMT PROT LOCK
mysql80 4 GiB 2 excl
wordpress 20 GiB 2 excl
[root@ceph141 ~]#
[root@ceph141 ~]#
[root@ceph141 ~]# rbd resize -s 50G yinzhengjie/mysql80
Resizing image: 100% complete...done.
[root@ceph141 ~]#
[root@ceph141 ~]# rbd -p yinzhengjie ls -l
NAME SIZE PARENT FMT PROT LOCK
mysql80 50 GiB 2 excl
wordpress 20 GiB 2 excl
[root@ceph141 ~]#
3.热加载设备文件
[root@harbor250 ~]# fdisk -l /dev/rbd1
Disk /dev/rbd1: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
[root@harbor250 ~]#
[root@harbor250 ~]# df -h | grep data
/dev/rbd0 20G 177M 20G 1% /data-rbd-xfs
/dev/rbd1 3.9G 28K 3.7G 1% /data-rbd-ext4
[root@harbor250 ~]#
[root@harbor250 ~]# resize2fs /dev/rbd1
resize2fs 1.46.5 (30-Dec-2021)
Filesystem at /dev/rbd1 is mounted on /data-rbd-ext4; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 7
The filesystem on /dev/rbd1 is now 13107200 (4k) blocks long.
[root@harbor250 ~]#
[root@harbor250 ~]# df -h | grep data
/dev/rbd0 20G 177M 20G 1% /data-rbd-xfs
/dev/rbd1 50G 28K 48G 1% /data-rbd-ext4
[root@harbor250 ~]#
本文来自博客园，作者：尹正杰，转载请注明原文链接：https://www.cnblogs.com/yinzhengjie/p/14249358.html，个人微信: "JasonYin2020"（添加时请备注来源及意图备注，有偿付费）
当你的才华还撑不起你的野心的时候，你就应该静下心来学习。当你的能力还驾驭不了你的目标的时候，你就应该沉下心来历练。问问自己，想要怎样的人生。
