ceph集群用户管理实战指南
一.ceph集群用户管理
1.用户格式及权限说明
ceph的用户格式"TYPEID.USERID"
- TYPEID:
指定的是用户类型。
包括内置组件用户(mon,mds,rgw,osd,mgr)和普通用户(client)。
- USERID:
就是用户名，可以是数字，比如表示ods的第0块磁盘，对应的是"ods.0",
也可以是字符串，比如管理员用户，对应的是"client.admin"。
当然，用户可以自定义USERID，比如"client.jason","client.yinzhengjie"。
每个用户都可以授权，使用caps字段关联。授权的格式"allow 权限"
常用的权限有:
- r：
读权限
- w:
写权限
- x:
执行权限，可以调用方法(这些方法可能存在读写等操作)，还可以执行mon的auth等相关命令。
- *:
拥有rwx等权限。
- class-read:
拥有x能力的子集，授予用户调用类写入方法的能力。
- class-write:
拥有x能力的子集，授予用户调用类写入方法的能力。
- profile osd:
授予用户一某个OSD身份连接到其他OSD或监视器的权限，可以获取OSD的状态信息。
- profile mds:
授予用户以某个MDS身份连接到其他MDS或监视器的权限，可以获取mds的状态信息。
- profile bootstrap-osd:
授予用于引导OSD的权限，在部署时候产生。
- profile bootstrap-mds:
授予用于引导元数据服务器的权限，在部署时候产生。
举例说明：(通过上面的介绍，那就可以看懂下面的配置啦)
[root@ceph141 ~]# cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
key = AQDjFrplyvFCDhAApJg111YMIGQ6/F/x/Y+qpQ==
caps mds = "allow *"
caps mgr = "allow *"
caps mon = "allow *"
caps osd = "allow *"
[root@ceph141 ~]#
关于更多权限信息请参考官网:
https://docs.ceph.com/en/latest/rados/operations/user-management/#authorization-capabilities
https://docs.ceph.com/en/nautilus/rados/operations/user-management/
2.查看现有的用户
参考链接:
https://docs.ceph.com/en/nautilus/rados/operations/user-management/#get-a-user
1 查看指定用户
[root@ceph141 ~]# ceph auth get client.admin
[client.admin]
key = AQDjFrplyvFCDhAApJg111YMIGQ6/F/x/Y+qpQ==
caps mds = "allow *"
caps mgr = "allow *"
caps mon = "allow *"
caps osd = "allow *"
exported keyring for client.admin
[root@ceph141 ~]#
2 查看所有用户
[root@ceph141 ~]# ceph auth list # 和"ceph auth ls"等效
osd.0
key: AQBGG7pllktDHxAAt1KWf87MZAgaaP67aCeSiA==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
...
client.admin
key: AQDjFrplyvFCDhAApJg111YMIGQ6/F/x/Y+qpQ==
caps: [mds] allow *
caps: [mgr] allow *
caps: [mon] allow *
caps: [osd] allow *
...
mgr.ceph141
key: AQClHrpldmqBOxAAhVvr/j+7X0rC8dfc+1fgrQ==
caps: [mds] allow *
caps: [mon] allow profile mgr
caps: [osd] allow *
...
installed auth entries:
[root@ceph141 ~]#
3.三种方式自定义普通用户
参考链接:
https://docs.ceph.com/en/nautilus/rados/operations/user-management/#add-a-user
1 "ceph auth add" 创建用户
[root@ceph141 ~]# ceph auth add client.jasonyin2020 mon 'allow r' osd 'allow rwx pool=yinzhengjie-rbd'
added key for client.jasonyin2020
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.jasonyin2020
[client.jasonyin2020]
key = AQD3QLxldlFsFBAAXV+S18R6Y8Cel11QRzs5XA==
caps mon = "allow r"
caps osd = "allow rwx pool=yinzhengjie-rbd"
exported keyring for client.jasonyin2020
[root@ceph141 ~]#
2 "ceph auth get-or-create"创建用户
[root@ceph141 ~]# ceph auth get client.yinzhengjie # 查看用户不存在
Error ENOENT: failed to find client.yinzhengjie in keyring
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get-or-create client.yinzhengjie mon 'allow r' osd 'allow rwx' # 如果用户不存在则直接创建并返回认证信息
[client.yinzhengjie]
key = AQBzQbxleO+BChAALlM0Earyd2E+TwPs+G0KYw==
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.yinzhengjie # 再次查看用户，发现创建成功
[client.yinzhengjie]
key = AQBzQbxleO+BChAALlM0Earyd2E+TwPs+G0KYw==
caps mon = "allow r"
caps osd = "allow rwx"
exported keyring for client.yinzhengjie
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get-or-create client.yinzhengjie mon 'allow rwx' osd 'allow r' # 如果用户已存在，再次创建会报错
Error EINVAL: key for client.yinzhengjie exists but cap mon does not match
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.yinzhengjie # 很明显，上一条命令没有执行成功
[client.yinzhengjie]
key = AQBzQbxleO+BChAALlM0Earyd2E+TwPs+G0KYw==
caps mon = "allow r"
caps osd = "allow rwx"
exported keyring for client.yinzhengjie
[root@ceph141 ~]#
3 "ceph auth get-or-create-key"创建用户
[root@ceph141 ~]# ceph auth get client.k8s # 注意，用户是不存在的
Error ENOENT: failed to find client.k8s in keyring
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get-or-create-key client.k8s mon 'allow r' osd 'allow rwx' # 创建用户并返回KEY
AQBkQrxlR6aVGBAAerMOjQ5Nah/HYafJu+aTsg==
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.k8s # 再次查看用户信息
[client.k8s]
key = AQBkQrxlR6aVGBAAerMOjQ5Nah/HYafJu+aTsg==
caps mon = "allow r"
caps osd = "allow rwx"
exported keyring for client.k8s
[root@ceph141 ~]#
4 "ceph auth print-key"打印已经存在用户的KEY
[root@ceph141 ~]# ceph auth get client.jasonyin # 如果用户不存在则报错
Error ENOENT: failed to find client.jasonyin in keyring
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth print-key client.jasonyin2020 | more # 如果用户存在则打印该用户对应的KEY信息。
AQD3QLxldlFsFBAAXV+S18R6Y8Cel11QRzs5XA==
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.jasonyin2020
[client.jasonyin2020]
key = AQD3QLxldlFsFBAAXV+S18R6Y8Cel11QRzs5XA==
caps mon = "allow r"
caps osd = "allow rwx pool=yinzhengjie-rbd"
exported keyring for client.jasonyin2020
[root@ceph141 ~]#
4.修改用户权限，直接覆盖权限
参考链接:
https://docs.ceph.com/en/nautilus/rados/operations/user-management/#modify-user-capabilities
[root@ceph141 ~]# ceph auth get client.jasonyin2020
[client.jasonyin2020]
key = AQD3QLxldlFsFBAAXV+S18R6Y8Cel11QRzs5XA==
caps mon = "allow r"
caps osd = "allow rwx pool=yinzhengjie-rbd"
exported keyring for client.jasonyin2020
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth caps client.jasonyin2020 mon 'allow rx' osd 'allow r pool=yinzhengjie'
updated caps for client.jasonyin2020
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.jasonyin2020
[client.jasonyin2020]
key = AQD3QLxldlFsFBAAXV+S18R6Y8Cel11QRzs5XA==
caps mon = "allow rx"
caps osd = "allow r pool=yinzhengjie"
exported keyring for client.jasonyin2020
[root@ceph141 ~]#
5.删除用户
参考链接:
https://docs.ceph.com/en/nautilus/rados/operations/user-management/#delete-a-user
[root@ceph141 ~]# ceph auth get client.jasonyin2020
[client.jasonyin2020]
key = AQD3QLxldlFsFBAAXV+S18R6Y8Cel11QRzs5XA==
caps mon = "allow rx"
caps osd = "allow r pool=yinzhengjie"
exported keyring for client.jasonyin2020
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth del client.jasonyin2020 # 删除名为"jasonyin2020"的普通用户(client)。
updated
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.jasonyin2020
Error ENOENT: failed to find client.jasonyin2020 in keyring
[root@ceph141 ~]#
二.ceph用户的备份和恢复
参考链接:
https://docs.ceph.com/en/nautilus/rados/operations/user-management/#get-a-user
https://docs.ceph.com/en/nautilus/rados/operations/user-management/#import-a-user-s
1.创建测试用户
[root@ceph141 ~]# ceph auth add client.jasonyin2020 mon 'allow rwx' osd 'allow r pool=yinzhengjie-rbd'
added key for client.jasonyin2020
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.jasonyin2020
[client.jasonyin2020]
key = AQDtRLxl0V3wFRAA8Cz4Vaeey+k049B761iRZA==
caps mon = "allow rwx"
caps osd = "allow r pool=yinzhengjie-rbd"
exported keyring for client.jasonyin2020
[root@ceph141 ~]#
2.导出用户到文件，用于模拟备份
[root@ceph141 ~]# ceph-authtool --create-keyring ceph.client.jasonyin2020.keyring # 说白了，只是创建了一个普通文件。
creating ceph.client.jasonyin2020.keyring
[root@ceph141 ~]#
[root@ceph141 ~]# ll ceph.client.jasonyin2020.keyring
-rw------- 1 root root 0 Feb 2 09:28 ceph.client.jasonyin2020.keyring
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.jasonyin2020 -o ceph.client.jasonyin2020.keyring # 将内容导出到指定文件
exported keyring for client.jasonyin2020
[root@ceph141 ~]#
[root@ceph141 ~]# cat ceph.client.jasonyin2020.keyring
[client.jasonyin2020]
key = AQDtRLxl0V3wFRAA8Cz4Vaeey+k049B761iRZA==
caps mon = "allow rwx"
caps osd = "allow r pool=yinzhengjie-rbd"
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth export client.jasonyin2020 -o jasonyin2020.keyring # 也可以使用这种方式导入用户信息到文件
export auth(key=AQDtRLxl0V3wFRAA8Cz4Vaeey+k049B761iRZA==)
[root@ceph141 ~]#
[root@ceph141 ~]# cat jasonyin2020.keyring
[client.jasonyin2020]
key = AQDtRLxl0V3wFRAA8Cz4Vaeey+k049B761iRZA==
caps mon = "allow rwx"
caps osd = "allow r pool=yinzhengjie-rbd"
[root@ceph141 ~]#
温馨提示:
上述命令也可执行为:"ceph auth get client.jasonyin2020 > ceph.client.jasonyin2020.keyring"
3.删除用户
[root@ceph141 ~]# ceph auth get client.jasonyin2020
[client.jasonyin2020]
key = AQDtRLxl0V3wFRAA8Cz4Vaeey+k049B761iRZA==
caps mon = "allow rwx"
caps osd = "allow r pool=yinzhengjie-rbd"
exported keyring for client.jasonyin2020
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth del client.jasonyin2020
updated
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.jasonyin2020
Error ENOENT: failed to find client.jasonyin2020 in keyring
[root@ceph141 ~]#
4.导入用户，用于模拟恢复
[root@ceph141 ~]# cat ceph.client.jasonyin2020.keyring
[client.jasonyin2020]
key = AQDtRLxl0V3wFRAA8Cz4Vaeey+k049B761iRZA==
caps mon = "allow rwx"
caps osd = "allow r pool=yinzhengjie-rbd"
[root@ceph141 ~]#
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.jasonyin2020
Error ENOENT: failed to find client.jasonyin2020 in keyring
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth import -i ceph.client.jasonyin2020.keyring
imported keyring
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.jasonyin2020
[client.jasonyin2020]
key = AQDtRLxl0V3wFRAA8Cz4Vaeey+k049B761iRZA==
caps mon = "allow rwx"
caps osd = "allow r pool=yinzhengjie-rbd"
exported keyring for client.jasonyin2020
[root@ceph141 ~]#
5.再次删除用户，使用jasonyin2020.keyring文件进行恢复
[root@ceph141 ~]# ceph auth del client.jasonyin2020
updated
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.jasonyin2020
Error ENOENT: failed to find client.jasonyin2020 in keyring
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth import -i jasonyin2020.keyring
imported keyring
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.jasonyin2020
[client.jasonyin2020]
key = AQDtRLxl0V3wFRAA8Cz4Vaeey+k049B761iRZA==
caps mon = "allow rwx"
caps osd = "allow r pool=yinzhengjie-rbd"
exported keyring for client.jasonyin2020
[root@ceph141 ~]#
三.导出授权文件并验证用户权限
1.创建用户
[root@ceph141 ~]# ceph auth get-or-create client.k3s mon 'allow r' osd 'allow * pool=yinzhengjie-rdb'
[client.k3s]
key = AQDqR7xlyS+TIxAA9aW9AeUTdJcV/xoKmErBgw==
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.k3s
[client.k3s]
key = AQDqR7xlyS+TIxAA9aW9AeUTdJcV/xoKmErBgw==
caps mon = "allow r"
caps osd = "allow * pool=yinzhengjie-rdb"
exported keyring for client.k3s
[root@ceph141 ~]#
2.导出用户授权文件，钥匙环(keyring)
[root@ceph141 ~]# ceph auth export client.k3s -o ceph.client.k3s.keyring
export auth(key=AQDqR7xlyS+TIxAA9aW9AeUTdJcV/xoKmErBgw==)
[root@ceph141 ~]#
[root@ceph141 ~]# cat ceph.client.k3s.keyring
[client.k3s]
key = AQDqR7xlyS+TIxAA9aW9AeUTdJcV/xoKmErBgw==
caps mon = "allow r"
caps osd = "allow * pool=yinzhengjie-rdb"
[root@ceph141 ~]#
3.拷贝授权文件前，观察客户端是否有查看集群的权限
[root@ceph144 ~]# ll /etc/ceph/
total 8
-rw-r--r-- 1 root root 264 Feb 1 16:51 ceph.conf
-rw-r--r-- 1 root root 92 Jun 30 2021 rbdmap
-rw------- 1 root root 0 Feb 1 16:50 tmpEYwKWU
[root@ceph144 ~]#
[root@ceph144 ~]# cat /etc/ceph/ceph.conf
[global]
fsid = 5821e29c-326d-434d-a5b6-c492527eeaad
public_network = 10.0.0.0/24
mon_initial_members = ceph141, ceph142, ceph143
mon_host = 10.0.0.141,10.0.0.142,10.0.0.143
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
[root@ceph144 ~]#
[root@ceph144 ~]# ceph -s
2024-02-02 09:44:38.983 7f82e96cc700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2024-02-02 09:44:38.983 7f82e96cc700 -1 AuthRegistry(0x7f82e40662b8) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2024-02-02 09:44:39.010 7f82e96cc700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2024-02-02 09:44:39.010 7f82e96cc700 -1 AuthRegistry(0x7f82e40c7dc8) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2024-02-02 09:44:39.011 7f82e96cc700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2024-02-02 09:44:39.011 7f82e96cc700 -1 AuthRegistry(0x7f82e96cae78) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
[errno 2] error connecting to the cluster
[root@ceph144 ~]#
4.将授权文件拷贝到客户端
[root@ceph141 ~]# scp ceph.client.k3s.keyring ceph144:/etc/ceph/
5.验证权限
[root@ceph144 ~]# ll /etc/ceph/ceph.c*
-rw-r--r-- 1 root root 124 Feb 2 09:45 /etc/ceph/ceph.client.k3s.keyring
-rw-r--r-- 1 root root 264 Feb 1 16:51 /etc/ceph/ceph.conf
[root@ceph144 ~]#
[root@ceph144 ~]# ceph -s --user k3s
cluster:
id: 5821e29c-326d-434d-a5b6-c492527eeaad
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 40h)
mgr: ceph141(active, since 39h), standbys: ceph143, ceph142
osd: 7 osds: 7 up (since 16h), 7 in (since 16h)
data:
pools: 3 pools, 96 pgs
objects: 74 objects, 114 MiB
usage: 7.8 GiB used, 1.9 TiB / 2.0 TiB avail
pgs: 96 active+clean
[root@ceph144 ~]#
[root@ceph144 ~]# ceph --user k3s auth get client.k3s
Error EACCES: access denied
[root@ceph144 ~]#
5.服务端尝试修改k3s用户权限
[root@ceph141 ~]# ceph auth caps client.k3s mon 'allow rx'
updated caps for client.k3s
[root@ceph141 ~]#
温馨提示:
此处是不需要将授权文件导出并拷贝到144节点的，因为服务端权限已经修改过来了。
客户端在连接服务端时并不需要说明自己的权限，而是只需要说明自己是哪个用户，以及对应的KEY即可。
6.客户端再次验证权限
[root@ceph144 ~]# cat /etc/ceph/ceph.client.k3s.keyring # 此处我故意删除了caps相关字段，发现依旧是可以认证的。
[client.k3s]
key = AQDqR7xlyS+TIxAA9aW9AeUTdJcV/xoKmErBgw==
[root@ceph144 ~]#
[root@ceph144 ~]# ceph --user k3s auth get client.k3s # 很明显，客户端可以查看用户的权限信息啦。
[client.k3s]
key = AQDqR7xlyS+TIxAA9aW9AeUTdJcV/xoKmErBgw==
caps mon = "allow rx"
exported keyring for client.k3s
[root@ceph144 ~]#
[root@ceph144 ~]# rbd --user k3s ls -p yinzhengjie-rbd
2024-02-02 09:57:12.134 7fea59519c80 -1 librbd::api::Image: list_images: error listing v1 images: (1) Operation not permitted
rbd: listing images failed: (1) Operation not permitted
[root@ceph144 ~]#
7.服务端再次修改权限
[root@ceph141 ~]# ceph auth caps client.k3s mon 'allow *' osd 'allow *'
updated caps for client.k3s
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.k3s
[client.k3s]
key = AQDqR7xlyS+TIxAA9aW9AeUTdJcV/xoKmErBgw==
caps mon = "allow *"
caps osd = "allow *"
exported keyring for client.k3s
[root@ceph141 ~]#
8.客户端再次验证权限
[root@ceph144 ~]# rbd --user k3s ls -p yinzhengjie-rbd
k3s
k8s
mysqld
rbd-snap
wordpress
[root@ceph144 ~]#
[root@ceph144 ~]# cat /etc/ceph/ceph.client.k3s.keyring
[client.k3s]
key = AQDqR7xlyS+TIxAA9aW9AeUTdJcV/xoKmErBgw==
[root@ceph144 ~]#
9.ceph-authtool命令将用户的keyring合并至一个统一的keyring文件
[root@ceph141 ~]# ceph auth get osd.0 -o ./yinzhengjie.keyring
[root@ceph141 ~]#
[root@ceph141 ~]# cat ./yinzhengjie.keyring
[osd.0]
key = AQAp6MVmUxYtNhAAg/UbI5IrHoAlXgF5THKr8w==
caps mgr = "allow profile osd"
caps mon = "allow profile osd"
caps osd = "allow *"
[root@ceph141 ~]#
[root@ceph141 ~]# ceph-authtool --import-keyring /etc/ceph/ceph.client.admin.keyring ./yinzhengjie.keyring
importing contents of /etc/ceph/ceph.client.admin.keyring into ./yinzhengjie.keyring
[root@ceph141 ~]#
[root@ceph141 ~]# cat ./yinzhengjie.keyring
[osd.0]
key = AQAp6MVmUxYtNhAAg/UbI5IrHoAlXgF5THKr8w==
caps mgr = "allow profile osd"
caps mon = "allow profile osd"
caps osd = "allow *"
[client.admin]
key = AQD348VmF2HCJRAAuwU/bXJR2WeYJaoaIMz0ag==
caps mds = "allow *"
caps mgr = "allow *"
caps mon = "allow *"
caps osd = "allow *"
[root@ceph141 ~]#
10 导出授权文件并验证用户权限总结
1 如果使用"--user k3s"指定用户，则默认去找以下文件，找不到就报错:
- /etc/ceph/ceph.client.k3s.keyring
- /etc/ceph/ceph.keyring
- /etc/ceph/keyring
- /etc/ceph/keyring.bin
2 如果不使用"--user"选项，咱们可以立即为默认为"--user amdin"
- /etc/ceph/ceph.client.admin.keyring
- /etc/ceph/ceph.keyring
- /etc/ceph/keyring
- /etc/ceph/keyring.bin
3 对于认证文件不能随便起名字。
而是需要遵循上述2条的规范文件命名，否则ceph不识别用户的配置文件
4 客户端在连接ceph集群时，仅需要读取keyring文件中的KEY值。
其他caps字段会被忽视。也就是说，对于文件中只要保留key值依旧是有效的。
本文来自博客园，作者：尹正杰，转载请注明原文链接：https://www.cnblogs.com/yinzhengjie/p/14275022.html，个人微信: "JasonYin2020"（添加时请备注来源及意图备注，有偿付费）
当你的才华还撑不起你的野心的时候，你就应该静下心来学习。当你的能力还驾驭不了你的目标的时候，你就应该沉下心来历练。问问自己，想要怎样的人生。
