Ceph radosgw的基本使用
RadosGW 是对象存储(OSS,Object Storage Service)的一种访问实现方式，RADOS 网关也称为 Ceph 对象网关、RadosGW、RGW，是一种服务，使客户端能够利用标准对象存储API 来访问 Ceph 集群，它支持 AWS S3 和 Swift API，在 ceph 0.8 版本之后使用 Civetweb(https://github.com/civetweb/civetweb) 的 web 服务器来响应 api 请求，客户端使用http/https 协议通过 RESTful API 与 RGW 通信，而 RGW 则通过 librados 与 ceph 集群通信，RGW 客户端通过 s3 或者 swift api 使用 RGW 用户进行身份验证，然后 RGW 网关代表用户利用 cephx 与 ceph 存储进行身份验证。
S3 由 Amazon 于 2006 年推出，全称为 Simple Storage Service,S3 定义了对象存储，是对象存储事实上的标准，从某种意义上说，S3 就是对象存储，对象存储就是 S3,它是对象存储市场的霸主，后续的对象存储都是对 S3 的模仿。
部署 RadosGW 服务：
将 ceph-mgr1、ceph-mgr2 服务器部署为高可用的 radosGW 服务
添加ceph仓库源并安装radosgw
| #支持 https 镜像仓库源： | |
| apt install -y apt-transport-https ca-certificates curl software-properties-common | |
| #导入 key： | |
| wget -q -O- 'https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc' | sudo apt-key add - | |
| apt-add-repository 'deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific/ bionic main' | |
| root@ceph-mgr1:/etc/apt# apt update | |
| root@ceph-mgr1:~# apt-cache madison radosgw #搜索radosgw | |
| radosgw | 16.2.10-1bionic | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main amd64 Packages | |
| radosgw | 12.2.13-0ubuntu0.18.04.10 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 Packages | |
| radosgw | 12.2.13-0ubuntu0.18.04.10 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security/main amd64 Packages | |
| radosgw | 12.2.4-0ubuntu1 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/main amd64 Packages |
| #mg1和mgr2安装radosgw | |
| root@ceph-mgr1:/etc/apt# apt install radosgw | |
| root@ceph-mgr1:~# radosgw -v | |
| ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) | |
| root@ceph-mgr2:~# radosgw -v | |
| ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) |
ceph-deploy节点调度 mgr1 和mg2 部署rgw服务
| root@ceph-deploy:~# su - cephadmin | |
| cephadmin@ceph-deploy:~$ cd ceph-cluster/ | |
| cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy --overwrite-conf rgw create ceph-mgr1 | |
| cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy --overwrite-conf rgw create ceph-mgr2 |
--overwrite-conf 参数含义：以当前ceph-deploy的ceph.conf配置为准，替换掉mgr1节点上的/etc/ceph.conf
RGW部署完成后，会在mgr1、mgr2节点启动ceph-radosgw@rgw.ceph-mgr1、2服务，端口监听在tcp 7480
mgr1节点验证radosgw服务
| #检查radosgw服务 | |
| root@ceph-mgr1:~# systemctl status ceph-radosgw@rgw.ceph-mgr1.service | |
| root@ceph-mgr2:~# systemctl status ceph-radosgw@rgw.ceph-mgr2.service | |
| #检查服务端口 | |
| root@ceph-mgr1:~# ss -lntup|grep 7480 | |
| root@ceph-mgr2:~# ss -lntup|grep 7480 |
使用web http方式访问mgr1、mgr2 ip:7480服务
从ceph状态验证查看rgw服务部署
Radosgw默认存储池
初始化完成 radosgw 之后，会初始化默认的存储池如下：
名称以 default.rgw.* 为前缀和 .rgw.root的存储池
| cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool ls | |
| device_health_metrics | |
| rbd-data | |
| default.rgw.log | |
| .rgw.root | |
| default.rgw.control | |
| default.rgw.meta | |
| cephfs-metadata | |
| cephfs-data |
验证radosgw服务进程
| root@ceph-mgr1:~# ps -ef|grep radosgw | |
| ceph 1302 1 0 14:58 ? 00:00:04 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr1 --setuser ceph --setgroup ceph | |
| root 3562 3492 0 15:32 pts/0 00:00:00 grep --color=auto radosgw | |
| root@ceph-mgr2:~# ps -ef|grep radosgw | |
| ceph 19646 1 0 15:22 ? 00:00:01 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr2 --setuser ceph --setgroup ceph | |
| root 20332 2930 0 15:33 pts/0 00:00:00 grep --color=auto radosgw |
radosgw 的存储池类型：
| cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool ls | |
| device_health_metrics | |
| rbd-data | |
| default.rgw.log | |
| .rgw.root | |
| default.rgw.control | |
| default.rgw.meta | |
| cephfs-metadata | |
| cephfs-data |
查看默认 radosgw 的存储池信息：
| cephadmin@ceph-deploy:~/ceph-cluster$ radosgw-admin zone get --rgw-zone=default --rgw-zonegroup=default | |
| { | |
| "id": "638985bc-6486-4a1a-8012-a619266611ef", | |
| "name": "default", | |
| "domain_root": "default.rgw.meta:root", | |
| "control_pool": "default.rgw.control", | |
| "gc_pool": "default.rgw.log:gc", | |
| "lc_pool": "default.rgw.log:lc", | |
| "log_pool": "default.rgw.log", | |
| "intent_log_pool": "default.rgw.log:intent", | |
| "usage_log_pool": "default.rgw.log:usage", | |
| "roles_pool": "default.rgw.meta:roles", | |
| "reshard_pool": "default.rgw.log:reshard", | |
| "user_keys_pool": "default.rgw.meta:users.keys", | |
| "user_email_pool": "default.rgw.meta:users.email", | |
| "user_swift_pool": "default.rgw.meta:users.swift", | |
| "user_uid_pool": "default.rgw.meta:users.uid", | |
| "otp_pool": "default.rgw.otp", | |
| "system_key": { | |
| "access_key": "", | |
| "secret_key": "" | |
| }, | |
| "placement_pools": [ | |
| { | |
| "key": "default-placement", | |
| "val": { | |
| "index_pool": "default.rgw.buckets.index", | |
| "storage_classes": { | |
| "STANDARD": { | |
| "data_pool": "default.rgw.buckets.data" | |
| } | |
| }, | |
| "data_extra_pool": "default.rgw.buckets.non-ec", | |
| "index_type": 0 | |
| } | |
| } | |
| ], | |
| "realm_id": "", | |
| "notif_pool": "default.rgw.log:notif" | |
| } |
rgw.root： 包含 realm(领域信息)，比如 zone 和 zonegroup
default.rgw.log： 存储日志信息，用于记录各种 log 信息。
default.rgw.control： 系统控制池，在有数据更新时，通知其它 RGW 更新缓存。
default.rgw.meta： 元数据存储池，通过不同的名称空间分别存储不同的 rados 对象，这些名称空间包括⽤⼾UID 及其 bucket 映射信息的名称空间 users.uid、⽤⼾的密钥名称空间users.keys、⽤⼾的 email 名称空间 users.email、⽤⼾的 subuser 的名称空间 users.swift，以及 bucket 的名称空间 root 等。
default.rgw.buckets.index： 存放 bucket 到 object 的索引信息。
default.rgw.buckets.data： 存放对象的数据。
default.rgw.buckets.non-ec： 数据的额外信息存储池
default.rgw.users.uid： 存放用户信息的存储池。
default.rgw.data.root： 存放 bucket 的元数据，结构体对应 RGWBucketInfo，比如存放桶名、桶 ID、data_pool 等。
查看对象存储池的存储策略、副本数量、pgp和pg的数量
| cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool get default.rgw.meta crush_rule | |
| crush_rule: replicated_rule | |
| cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool get default.rgw.meta size | |
| size: 3 | |
| cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool get default.rgw.meta pgp_num | |
| pgp_num: 8 | |
| cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool get default.rgw.meta pg_num | |
| pg_num: 8 |
radosgw http 服务高可用配置
自定义 http 端口
配置文件可以在 ceph deploy 服务器修改然后统一推送，或者单独修改每个 radosgw 服务器的配置为统一配置，然后重启 RGW 服务。
https://docs.ceph.com/en/latest/radosgw/frontends/
在ceph.conf最后面添加针对当前节点的自定义配置如下
| root@ceph-mgr1:~# vim /etc/ceph/ceph.conf | |
| [client.rgw.ceph-mgr1] | |
| rgw_host = ceph-mgr1 | |
| rgw_frontends = civetweb port=9900 |
重启节点 mgr1 的 radosgw 服务
| root@ceph-mgr1:~# systemctl restart ceph-radosgw@rgw.ceph-mgr1.service | |
| root@ceph-mgr1:~# systemctl status ceph-radosgw@rgw.ceph-mgr1.service | |
| ceph-radosgw@rgw.ceph-mgr1.service - Ceph rados gateway | |
| Loaded: loaded (/lib/systemd/system/ceph-radosgw@.service; indirect; vendor preset: enabled) | |
| Active: active (running) since Wed 2022-12-14 11:44:11 CST; 6s ago | |
| Main PID: 4196 (radosgw) | |
| Tasks: 603 | |
| CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw@rgw.ceph-mgr1.service | |
| ©¸©¤4196 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr1 --setuser ceph --setgroup ceph | |
| Dec 14 11:44:11 ceph-mgr1 systemd[1]: Started Ceph rados gateway. | |
| Dec 14 11:44:11 ceph-mgr1 radosgw[4196]: 2022-12-14T11:44:11.494+0800 7f76c28843c0 -1 IMPORTANT: the civetweb frontend is | |
| root@ceph-mgr1:~# ss -lntup|grep 9900 | |
| tcp LISTEN 0 128 0.0.0.0:9900 0.0.0.0:* users:(("radosgw",pid=4196,fd=75)) |
实现高可用
安装haproxy并配置反向代理：
配置haproxy，反向代理 ceph-mgr1 和 mgr2 的radosgw服务tcp网络端口，mgr2的端口此时还为默认的7480端口.
| root@haproxyA:~# vim /etc/haproxy/haproxy.cfg | |
| listen ceph-radosgw-8090 | |
| bind :8090 | |
| mode tcp | |
| server ceph-mgr1 192.168.100.38:9900 check inter 3s fall 3 rise 2 | |
| server ceph-mgr2 192.168.100.39:7480 check inter 3s fall 3 rise 2 | |
| root@haproxyA:~# systemctl restart haproxy | |
| root@haproxyA:~# systemctl status haproxy | |
| ● haproxy.service - HAProxy Load Balancer | |
| Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled) | |
| Active: active (running) since Wed 2022-12-14 12:00:17 CST; 4s ago | |
| Docs: man:haproxy(1) | |
| file:/usr/share/doc/haproxy/configuration.txt.gz | |
| Process: 1401 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS) | |
| Main PID: 1413 (haproxy) | |
| Tasks: 2 (limit: 2236) | |
| Memory: 2.2M | |
| CGroup: /system.slice/haproxy.service | |
| ├─1413 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock | |
| └─1417 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock | |
| Dec 14 12:00:17 haproxyA systemd[1]: Starting HAProxy Load Balancer... | |
| Dec 14 12:00:17 haproxyA haproxy[1413]: Proxy ceph-radosgw-8090 started. | |
| Dec 14 12:00:17 haproxyA haproxy[1413]: Proxy ceph-radosgw-8090 started. | |
| Dec 14 12:00:17 haproxyA haproxy[1413]: Proxy statistics started. | |
| Dec 14 12:00:17 haproxyA haproxy[1413]: Proxy statistics started. | |
| Dec 14 12:00:17 haproxyA haproxy[1413]: [NOTICE] 347/120017 (1413) : New worker #1 (1417) forked | |
| Dec 14 12:00:17 haproxyA systemd[1]: Started HAProxy Load Balancer. | |
| root@haproxyA:~# ss -lntup|grep 8090 | |
| tcp LISTEN 0 3000 0.0.0.0:8090 0.0.0.0:* users:(("haproxy",pid=1417,fd=7) |
浏览器访问haproxy代理地址 192.168.100.20:8090
查看haproxy的服务代理日志，能看到将客户端的请求均衡代理到后端实际的ceph-mgr节点 endpoint
日志及其它优化配置
创建日志目录
| root@ceph-mgr2:~# mkdir /var/log/radosgw | |
| root@ceph-mgr2:~# chown ceph.ceph /var/log/radosgw | |
| root@ceph-mgr2:~# vim /etc/ceph/ceph.conf | |
| [client.rgw.ceph-mgr1] | |
| rgw_host = ceph-mgr1 | |
| rgw_frontends = "civetweb port=9900+9443s ssl_certificate=/etc/ceph/certs/cephrgw.pem error_log_file=/var/log/radosgw/radosgw.error.log access_log_file=/var/log/radosgw/radosgw.access.log request_timeout_ms=30000 num_threads=200" | |
| [client.rgw.ceph-mgr2] | |
| rgw_host = ceph-mgr2 | |
| rgw_frontends = "civetweb port=9900+9443s ssl_certificate=/etc/ceph/certs/cephrgw.pem error_log_file=/var/log/radosgw/radosgw.error.log access_log_file=/var/log/radosgw/radosgw.access.log request_timeout_ms=30000 num_threads=200" |
error_log_file： 指定radosgw错误日志路径
access_log_file： 指定radosgw访问日志路径
request_timeout_ms：指定radosgw访问超时时间
num_threads： 指定radosgw运行线程数量，默认线程数是100，https://docs.ceph.com/en/mimic/radosgw/config-ref/
重启radosgw
| root@ceph-mgr2:/etc/ceph# systemctl restart ceph-radosgw@rgw.ceph-mgr2.service | |
| root@ceph-mgr2:/etc/ceph# systemctl status ceph-radosgw@rgw.ceph-mgr2.service |
验证日志
创建 RGW 账户
在ceph管理节点创建对象用户
--uid 指定用户ID
--display-name 指定显示用户名称
| cephadmin@ceph-deploy:~/ceph-cluster$ radosgw-admin user create --uid="user1" --display-name="user1" | |
| { | |
| "user_id": "user1", | |
| "display_name": "user1", | |
| "email": "", | |
| "suspended": 0, | |
| "max_buckets": 1000, | |
| "subusers": [], | |
| "keys": [ | |
| { | |
| "user": "user1", | |
| "access_key": "45CMIRWTFQY9DGJX7W1Z", | |
| "secret_key": "EyFmlD51WWfCGbtxFYZcygwDc48QWMYyKs13nuDD" | |
| } | |
| ], | |
| "swift_keys": [], | |
| "caps": [], | |
| "op_mask": "read, write, delete", | |
| "default_placement": "", | |
| "default_storage_class": "", | |
| "placement_tags": [], | |
| "bucket_quota": { | |
| "enabled": false, | |
| "check_on_raw": false, | |
| "max_size": -1, | |
| "max_size_kb": 0, | |
| "max_objects": -1 | |
| }, | |
| "user_quota": { | |
| "enabled": false, | |
| "check_on_raw": false, | |
| "max_size": -1, | |
| "max_size_kb": 0, | |
| "max_objects": -1 | |
| }, | |
| "temp_url_keys": [], | |
| "type": "rgw", | |
| "mfa_ids": [] | |
| } |
注意保存对象用户的 access_key 和 secret_key
查看用户信息
| root@ceph-mgr1:/var/log/ceph# radosgw-admin user --uid="user1" info | |
| { | |
| "user_id": "user1", | |
| "display_name": "user1", | |
| "email": "", | |
| "suspended": 0, | |
| "max_buckets": 1000, | |
| "subusers": [], | |
| "keys": [ | |
| { | |
| "user": "user1", | |
| "access_key": "45CMIRWTFQY9DGJX7W1Z", | |
| "secret_key": "EyFmlD51WWfCGbtxFYZcygwDc48QWMYyKs13nuDD" | |
| } | |
| ], | |
| "swift_keys": [], | |
| "caps": [], | |
| "op_mask": "read, write, delete", | |
| "default_placement": "", | |
| "default_storage_class": "", | |
| "placement_tags": [], | |
| "bucket_quota": { | |
| "enabled": false, | |
| "check_on_raw": false, | |
| "max_size": -1, | |
| "max_size_kb": 0, | |
| "max_objects": -1 | |
| }, | |
| "user_quota": { | |
| "enabled": false, | |
| "check_on_raw": false, | |
| "max_size": -1, | |
| "max_size_kb": 0, | |
| "max_objects": -1 | |
| }, | |
| "temp_url_keys": [], | |
| "type": "rgw", | |
| "mfa_ids": [] | |
| } |
查询所有用户
| root@ceph-mgr1:/var/log/ceph# radosgw-admin metadata list user | |
| [ | |
| "user1" | |
| ] |
RGW 账户权限控制
参考aws 官网文档介绍：https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/example-bucket-policies.html
账户权限介绍
1、授权简介和预览
Resources： 授权的目的 Buckets、objects等资源，必须指定。
Actions：要授予的动作，CreateBucket、DeleteObject、GetObject、PubObject。必须指定
Effect：要授予的操作效果是允许（allow）还是拒绝（deny），默认为拒绝访问所有的资源，必须指定。
Principal: 要授权的目的账号，必须指定
Condition：授权策略生效的条件，比如访问TLS版本等，非必须，可不写。
| { | |
| “Condition”: { | |
| “NumericLessThan”: { | |
| “s3:TlsVersion”: 1.2 | |
| } | |
| } | |
| } |
2、权限集合
https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/API/API_Operations.html
权限配置
1、授予匿名用户对 bucket01 的 GetObject权限，仅可以查看桶内的文件。
创建权限json文件
| [root@ansible ~]# vim bucket01-policy.json | |
| { | |
| "Version": "2012-10-17", | |
| "Statement": [ | |
| { | |
| "Effect": "Allow", | |
| "Principal": "*", | |
| "Action": "s3:GetObject", | |
| "Resource": [ | |
| "arn:aws:s3:::bucket01/*" | |
| ] | |
| } | |
| ] | |
| } |
进行授权
| [root@ansible ~]# s3cmd setpolicy bucket01-policy.json s3://bucket01 | |
| s3://bucket01/: Policy updated |
验证权限，客户端浏览器访问 http://rgw.cncf.net/bucket01/<文件名>
