cephfs文件系统场景
创建cephfs文件系统:
[cephfsd@ceph-admin ceph]$ cd /etc/ceph [cephfsd@ceph-admin ceph]$ ceph fs ls No filesystems enabled # 创建三个mds [cephfsd@ceph-admin ceph]$ ceph-deploy mds create ceph-node1 ceph-node2 ceph-node3 # cephfs需要两个pool: # 1、data pool:存放object # 2、meta data pool:存放元数据,可设置较高副本级别,也可调整pool的crush_ruleset,使其在ssd上存储,加快客户端响应速度,我这里直接使用默认crush_ruleset # 创建pool存储池,这里两个pg_num必须保持一致,否则后面无法挂载。 [cephfsd@ceph-admin ceph]$ ceph osd pool create cephfs_data 64 pool 'cephfs_data' created [cephfsd@ceph-admin ceph]$ ceph osd pool create cephfs_metadata 64 pool 'cephfs_metadata' created # 注: # 计算pg数量: # 1、集群pg 总数 = (OSD 总数* 100 )/最大副本数 # 2、每个pool中pg总数=(OSD总数*100)/ 最大副本数 )/ 池数 # 3、pg数需要是2的指数幂 # 创建fs,这里new后面的cephfs就是文件系统名,可自定义。后面接的cephfs_metadata cephfs_data 就是上面创建的pool。 [cephfsd@ceph-admin ceph]$ ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # 创建成功,查看创建后的fs [cephfsd@ceph-admin ceph]$ ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] # 查看mds节点状态,active是活跃的,另1个是处于热备份的状态 [cephfsd@ceph-admin ceph]$ ceph mds stat cephfs-1/1/1 up {0=ceph-node3=up:active}, 2 up:standby [cephfsd@ceph-admin ceph]$
挂载cephfs文件系统(在任意客户端,我们这里直接在ceph的admin主机上挂载了)
# 创建挂载点 [cephfsd@ceph-admin ceph]$ mkdir /opt # 去admin主机查看存储秘钥 [cephfsd@ceph-admin ceph]$ cat ceph.client.admin.keyring [client.admin] key = AQBIH+ld1okAJhAAmULVJM4zCCVAK/Vdi3Tz5Q== # 将key的值复制下来,保存到客户端,我们这里保存在了/etc/ceph/admin.secret [cephfsd@ceph-admin ceph]$ cat admin.secret AQBIH+ld1okAJhAAmULVJM4zCCVAK/Vdi3Tz5Q== # 挂载,有两种方式挂载,mount挂载和ceph-fuse挂载 # A.内核驱动挂载Ceph文件系统 [cephfsd@ceph-admin ceph]$ sudo mount.ceph 172.16.143.121:6789:/ /opt -o name=admin,secretfile=/etc/ceph/admin.secret [cephfsd@ceph-admin ceph]$ df -Th Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 484M 0 484M 0% /dev tmpfs tmpfs 496M 0 496M 0% /dev/shm tmpfs tmpfs 496M 26M 470M 6% /run tmpfs tmpfs 496M 0 496M 0% /sys/fs/cgroup /dev/mapper/centos-root xfs 46G 1.9G 44G 4% / /dev/sda1 xfs 497M 130M 368M 26% /boot tmpfs tmpfs 100M 0 100M 0% /run/user/0 172.16.143.121:6789:/ ceph 8.5G 0 8.5G 0% /opt # 注: # mount时,mon节点有几个写几个,如果配置了hosts解析,可以使用主机名代替IP。 # 还可以使用fuse挂载方式,fuse其实坑挺多的,能不用暂时不用 # 由于kernel挂载方式目前不支持quota的配置,如果需要quota的配置,那么只好使用fuse挂载 # 卸载 [cephfsd@ceph-admin ceph]$ umount /opt umount: /opt: umount failed: Operation not permitted [cephfsd@ceph-admin ceph]$ sudo umount /opt [cephfsd@ceph-admin ceph]$ df -Th Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 484M 0 484M 0% /dev tmpfs tmpfs 496M 0 496M 0% /dev/shm tmpfs tmpfs 496M 26M 470M 6% /run tmpfs tmpfs 496M 0 496M 0% /sys/fs/cgroup /dev/mapper/centos-root xfs 46G 1.9G 44G 4% / /dev/sda1 xfs 497M 130M 368M 26% /boot tmpfs tmpfs 100M 0 100M 0% /run/user/0 # 设置开机自动挂载/etc/fstab [cephfsd@ceph-admin ceph]$ sudo vim /etc/fstab 172.16.143.121:6789:/ /opt ceph name=admin,secretfile=/etc/ceph/admin.secret,noatime,_netdev 0 2 # 注意:检查是否启用cephx认证方法,如果值为none为禁用,cephx为启用 [root@node1 ceph]# cat /etc/ceph/ceph.conf | grep auth | grep required auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx # B.用户控件挂载Ceph文件系统 # 安装ceph-fuse [cephfsd@ceph-admin ceph]$ yum install -y ceph-fuse # 挂载,挂载需要sudo权限,否则会报错。 [cephfsd@ceph-admin ceph]$ sudo ceph-fuse -m 172.16.143.121:6789 /opt 2019-12-06 08:29:52.137542 7f0e2edb20c0 -1 init, newargv = 0x55c37f3887e0 newargc=9ceph-fuse[50371]: starting ceph client ceph-fuse[50371]: starting fuse [cephfsd@ceph-admin ceph]$ df -Th Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 484M 0 484M 0% /dev tmpfs tmpfs 496M 0 496M 0% /dev/shm tmpfs tmpfs 496M 26M 470M 6% /run tmpfs tmpfs 496M 0 496M 0% /sys/fs/cgroup /dev/mapper/centos-root xfs 46G 1.9G 44G 4% / /dev/sda1 xfs 497M 130M 368M 26% /boot tmpfs tmpfs 100M 0 100M 0% /run/user/0 ceph-fuse fuse.ceph-fuse 8.5G 0 8.5G 0% /opt # 卸载 [cephfsd@ceph-admin ceph]$ fusermount -u /opt fusermount: entry for /opt not found in /etc/mtab [cephfsd@ceph-admin ceph]$ sudo fusermount -u /opt [cephfsd@ceph-admin ceph]$ df -Th Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 484M 0 484M 0% /dev tmpfs tmpfs 496M 0 496M 0% /dev/shm tmpfs tmpfs 496M 26M 470M 6% /run tmpfs tmpfs 496M 0 496M 0% /sys/fs/cgroup /dev/mapper/centos-root xfs 46G 1.9G 44G 4% / /dev/sda1 xfs 497M 130M 368M 26% /boot tmpfs tmpfs 100M 0 100M 0% /run/user/0 [cephfsd@ceph-admin ceph]$ # 这里就是用ceph-fuse挂载时没有使用root权限,就挂载不上。 [cephfsd@ceph-admin ceph]$ ceph-fuse -m 172.16.143.121:6789 /opt 2019-12-06 08:29:28.606085 7f35fd5e60c0 -1 asok(0x5627d23481c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.asok': (13) Permission denied 2019-12-06 08:29:28.606957 7f35fd5e60c0 -1 init, newargv = 0x5627d24967e0 newargc=9 ceph-fuse[50325]: starting ceph client fusermount: user has no write access to mountpoint /opt ceph-fuse[50325]: fuse failed to start 2019-12-06 08:29:31.044358 7f35fd5e60c0 -1 fuse_mount(mountpoint=/opt) failed. [cephfsd@ceph-admin ceph]$ [cephfsd@ceph-admin ceph]$ df -Th Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 484M 0 484M 0% /dev tmpfs tmpfs 496M 0 496M 0% /dev/shm tmpfs tmpfs 496M 26M 470M 6% /run tmpfs tmpfs 496M 0 496M 0% /sys/fs/cgroup /dev/mapper/centos-root xfs 46G 1.9G 44G 4% / /dev/sda1 xfs 497M 130M 368M 26% /boot tmpfs tmpfs 100M 0 100M 0% /run/user/0 [cephfsd@ceph-admin ceph]$ # 设置开机自动挂载/etc/fstab [cephfsd@ceph-admin ceph]$ sudo vim /etc/fstab id=admin,conf=/etc/ceph/ceph.conf /mnt fuse.ceph defaults 0 0
删除cephfs,需要先将mds置为failed。
[cephfsd@ceph-admin ceph]$ ceph fs rm 128 Error EINVAL: all MDS daemons must be inactive before removing filesystem [cephfsd@ceph-admin ceph]$ ceph mds stat 128-1/1/1 up {0=ceph-node2=up:active(laggy or crashed)} [cephfsd@ceph-admin ceph]$ ceph mds fail 0 failed mds gid 4147 [cephfsd@ceph-admin ceph]$ ceph mds stat 128-0/1/1 up , 1 failed [cephfsd@ceph-admin ceph]$ ceph fs rm 128 Error EPERM: this is a DESTRUCTIVE operation and will make data in your filesystem permanently inaccessible. Add --yes-i-really-mean-it if you are sure you wish to continue. [cephfsd@ceph-admin ceph]$ ceph fs rm 128 --yes-i-really-mean-it [cephfsd@ceph-admin ceph]$ ceph fs ls No filesystems enabled [cephfsd@ceph-admin ceph]$
删除pg pool,需要先将使用该pool的cephfs删除。
[cephfsd@ceph-admin ceph]$ ceph osd pool rm cephfs_data2 cephfs_data2 --yes-i-really-mean-it Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool cephfs_data2. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. [cephfsd@ceph-admin ceph]$ ceph osd pool rm cephfs_data2 cephfs_data2 --yes-i-really-really-mean-it Error EBUSY: pool 'cephfs_data2' is in use by CephFS # 该pool被cephfs使用了,所以需要先删除该cephfs。 # 因为cephfs_data cephfs_metadata是挂载在128下的,cephfs_data2 cephfs_metadata2是挂载在cephfs下的,所以,删除cephfs # 接下来,删除cephfs [cephfsd@ceph-admin ceph]$ ceph fs rm cephfs Error EPERM: this is a DESTRUCTIVE operation and will make data in your filesystem permanently inaccessible. Add --yes-i-really-mean-it if you are sure you wish to continue. [cephfsd@ceph-admin ceph]$ ceph fs rm cephfs --yes-i-really-mean-it [cephfsd@ceph-admin ceph]$ ceph fs ls name: 128, metadata pool: cephfs_metadata, data pools: [cephfs_data ] [cephfsd@ceph-admin ceph]$ # cephfs删除了,接下来删除pool cephfs_data2 [cephfsd@ceph-admin ceph]$ ceph osd pool delete cephfs_data2 Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool cephfs_data2. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. [cephfsd@ceph-admin ceph]$ ceph osd pool delete cephfs_data2 cephfs_data2 --yes-i-really-really-mean-it Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool # 这里需要修改配置,增加mon_allow_pool_delete = true [cephfsd@ceph-admin ceph]$ vim ceph.conf [cephfsd@ceph-admin ceph]$ cat ceph.conf [global] fsid = 6d3fd8ed-d630-48f7-aa8d-ed79da7a69eb mon_initial_members = ceph-admin mon_host = 172.16.143.121 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx mon_allow_pool_delete = true osd_pool_default_size = 3 [mgr] mgr modules = dashboard # 然后把配置推送到mon节点。(这里其实不需要了,因为mon节点只有一个,部署在ceph-admin里) [cephfsd@ceph-admin ceph]$ ceph-deploy --overwrite-conf config push ceph-node1 ceph-node2 ceph-node3 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephfsd/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf config push ceph-node1 ceph-node2 ceph-node3 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] overwrite_conf : True [ceph_deploy.cli][INFO ] subcommand : push [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f946bf4fa28> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] client : ['ceph-node1', 'ceph-node2', 'ceph-node3'] [ceph_deploy.cli][INFO ] func : <function config at 0x7f946c18ac08> [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.config][DEBUG ] Pushing config to ceph-node1 [ceph-node1][DEBUG ] connection detected need for sudo [ceph-node1][DEBUG ] connected to host: ceph-node1 [ceph-node1][DEBUG ] detect platform information from remote host [ceph-node1][DEBUG ] detect machine type [ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph_deploy.config][DEBUG ] Pushing config to ceph-node2 [ceph-node2][DEBUG ] connection detected need for sudo [ceph-node2][DEBUG ] connected to host: ceph-node2 [ceph-node2][DEBUG ] detect platform information from remote host [ceph-node2][DEBUG ] detect machine type [ceph-node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph_deploy.config][DEBUG ] Pushing config to ceph-node3 [ceph-node3][DEBUG ] connection detected need for sudo [ceph-node3][DEBUG ] connected to host: ceph-node3 [ceph-node3][DEBUG ] detect platform information from remote host [ceph-node3][DEBUG ] detect machine type [ceph-node3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [cephfsd@ceph-admin ceph]$ ceph osd pool delete cephfs_data2 cephfs_data2 --yes-i-really-really-mean-it Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool # 依然报这个错,因为没有重启mon,所以这里需要重启一下mon服务。重启服务需要root权限。 [cephfsd@ceph-admin ceph]$ systemctl restart ceph-mon.target ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units === Authentication is required to manage system services or units. Authenticating as: root Password: ==== AUTHENTICATION COMPLETE === [cephfsd@ceph-admin ceph]$ # 接下来就可以删除pool了。 [cephfsd@ceph-admin ceph]$ ceph osd pool delete cephfs_data2 cephfs_data2 --yes-i-really-really-mean-it pool 'cephfs_data2' removed [cephfsd@ceph-admin ceph]$ ceph osd pool delete cephfs_metadata2 cephfs_metadata2 --yes-i-really-really-mean-it pool 'cephfs_metadata2' removed
挂载:
[cephfsd@ceph-admin ceph]$ !mount mount -t ceph 172.16.143.121:6789:/ /opt -o name=admin,secretfile=/etc/ceph/admin.secret mount: only root can use "--options" option [cephfsd@ceph-admin ceph]$ mount.ceph 172.16.143.121:6789:/ /opt -o name=admin,secretfile=/etc/ceph/admin.secret mount error 1 = Operation not permitted # 需要root权限执行mount
注意问题:
1、ceph pg_num设置太大。
ceph使用cephfs
[cephfsd@ceph-admin ceph]$ ceph osd pool create cephfs_data 128 pool 'cephfs_data' already exists [cephfsd@ceph-admin ceph]$ ceph osd pool create cephfs_metadata 128 Error ERANGE: pg_num 128 size 3 would mean 768 total pgs, which exceeds max 750 (mon_max_pg_per_osd 250 * num_in_osds 3) [cephfsd@ceph-admin ceph]$
类似这种,应该就是ceph pg_num设置太大了,设置小一点就好了,如下:
[cephfsd@ceph-admin ceph]$ ceph osd pool create cephfs_metadata 64 pool 'cephfs_metadata' created [cephfsd@ceph-admin ceph]$
2、 pg_num只能增加, 不能缩小.
[cephfsd@ceph-admin ceph]$ ceph osd pool set cephfs_data pg_num 64 Error EEXIST: specified pg_num 64 <= current 12
3、cephfs_data
与cephfs_metadata的pg_num值必须保持一致,否则,即使pool创建成功,也无法挂载,就像下面一样:
[cephfsd@ceph-admin ceph]$ sudo mount.ceph 172.16.143.121:6789:/ /opt -o name=admin,secretfile=/etc/ceph/admin.secret ^C^C ^C^C ^C ^CKilled [cephfsd@ceph-admin ceph]$ mount error 5 = Input/output error mount error 110 = Connection timed out
参考:http://www.strugglesquirrel.com/2019/04/23/centos7%E9%83%A8%E7%BD%B2ceph/