Ceph radosgw的基本使用

RadosGW 对象存储网关简介

RadosGW 是对象存储(OSS,Object Storage Service)的一种访问实现方式，RADOS 网关也称为 Ceph 对象网关、RadosGW、RGW，是一种服务，使客户端能够利用标准对象存储API 来访问 Ceph 集群，它支持 AWS S3 和 Swift API，在 ceph 0.8 版本之后使用 Civetweb(https://github.com/civetweb/civetweb) 的 web 服务器来响应 api 请求，客户端使用http/https 协议通过 RESTful API 与 RGW 通信，而 RGW 则通过 librados 与 ceph 集群通信，RGW 客户端通过 s3 或者 swift api 使用 RGW 用户进行身份验证，然后 RGW 网关代表用户利用 cephx 与 ceph 存储进行身份验证。

S3 由 Amazon 于 2006 年推出，全称为 Simple Storage Service,S3 定义了对象存储，是对象存储事实上的标准，从某种意义上说，S3 就是对象存储，对象存储就是 S3,它是对象存储市场的霸主，后续的对象存储都是对 S3 的模仿。

部署 RadosGW 服务：

将 ceph-mgr1、ceph-mgr2 服务器部署为高可用的 radosGW 服务

添加ceph仓库源并安装radosgw

#支持 https 镜像仓库源：
apt install -y apt-transport-https ca-certificates curl software-properties-common
#导入 key：
wget -q -O- 'https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc' | sudo apt-key add -

apt-add-repository 'deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific/ bionic main'

root@ceph-mgr1:/etc/apt# apt update

root@ceph-mgr1:~# apt-cache madison radosgw   #搜索radosgw
   radosgw | 16.2.10-1bionic | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main amd64 Packages
   radosgw | 12.2.13-0ubuntu0.18.04.10 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 Packages
   radosgw | 12.2.13-0ubuntu0.18.04.10 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security/main amd64 Packages
   radosgw | 12.2.4-0ubuntu1 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/main amd64 Packages

#mg1和mgr2安装radosgw
root@ceph-mgr1:/etc/apt# apt install radosgw
root@ceph-mgr1:~# radosgw -v
ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)

root@ceph-mgr2:~# radosgw -v
ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)

ceph-deploy节点调度 mgr1 和mg2 部署rgw服务

root@ceph-deploy:~# su - cephadmin
cephadmin@ceph-deploy:~$ cd ceph-cluster/
cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy --overwrite-conf rgw create ceph-mgr1
cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy --overwrite-conf rgw create ceph-mgr2

--overwrite-conf 参数含义：以当前ceph-deploy的ceph.conf配置为准，替换掉mgr1节点上的/etc/ceph.conf

RGW部署完成后，会在mgr1、mgr2节点启动ceph-radosgw@rgw.ceph-mgr1、2服务，端口监听在tcp 7480

mgr1节点验证radosgw服务

#检查radosgw服务
root@ceph-mgr1:~# systemctl status ceph-radosgw@rgw.ceph-mgr1.service
root@ceph-mgr2:~# systemctl status ceph-radosgw@rgw.ceph-mgr2.service

#检查服务端口
root@ceph-mgr1:~# ss -lntup|grep 7480
root@ceph-mgr2:~# ss -lntup|grep 7480

使用web http方式访问mgr1、mgr2 ip:7480服务

从ceph状态验证查看rgw服务部署

Radosgw默认存储池

初始化完成 radosgw 之后，会初始化默认的存储池如下：

名称以 default.rgw.* 为前缀和 .rgw.root的存储池

cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool ls
device_health_metrics
rbd-data
default.rgw.log
.rgw.root
default.rgw.control
default.rgw.meta
cephfs-metadata
cephfs-data

验证radosgw服务进程

root@ceph-mgr1:~# ps -ef|grep radosgw
ceph       1302      1  0 14:58 ?        00:00:04 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr1 --setuser ceph --setgroup ceph
root       3562   3492  0 15:32 pts/0    00:00:00 grep --color=auto radosgw


root@ceph-mgr2:~#  ps -ef|grep radosgw
ceph      19646      1  0 15:22 ?        00:00:01 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr2 --setuser ceph --setgroup ceph
root      20332   2930  0 15:33 pts/0    00:00:00 grep --color=auto radosgw

radosgw 的存储池类型：

cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool ls
device_health_metrics
rbd-data
default.rgw.log
.rgw.root
default.rgw.control
default.rgw.meta
cephfs-metadata
cephfs-data

查看默认 radosgw 的存储池信息：

cephadmin@ceph-deploy:~/ceph-cluster$ radosgw-admin zone get --rgw-zone=default --rgw-zonegroup=default
{
    "id": "638985bc-6486-4a1a-8012-a619266611ef",
    "name": "default",
    "domain_root": "default.rgw.meta:root",
    "control_pool": "default.rgw.control",
    "gc_pool": "default.rgw.log:gc",
    "lc_pool": "default.rgw.log:lc",
    "log_pool": "default.rgw.log",
    "intent_log_pool": "default.rgw.log:intent",
    "usage_log_pool": "default.rgw.log:usage",
    "roles_pool": "default.rgw.meta:roles",
    "reshard_pool": "default.rgw.log:reshard",
    "user_keys_pool": "default.rgw.meta:users.keys",
    "user_email_pool": "default.rgw.meta:users.email",
    "user_swift_pool": "default.rgw.meta:users.swift",
    "user_uid_pool": "default.rgw.meta:users.uid",
    "otp_pool": "default.rgw.otp",
    "system_key": {
        "access_key": "",
        "secret_key": ""
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",
                "storage_classes": {
                    "STANDARD": {
                        "data_pool": "default.rgw.buckets.data"
                    }
                },
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0
            }
        }
    ],
    "realm_id": "",
    "notif_pool": "default.rgw.log:notif"
}

rgw.root：包含 realm(领域信息)，比如 zone 和 zonegroup

default.rgw.log：存储日志信息，用于记录各种 log 信息。

default.rgw.control：系统控制池，在有数据更新时，通知其它 RGW 更新缓存。

default.rgw.meta：元数据存储池，通过不同的名称空间分别存储不同的 rados 对象，这些名称空间包括⽤⼾UID 及其 bucket 映射信息的名称空间 users.uid、⽤⼾的密钥名称空间users.keys、⽤⼾的 email 名称空间 users.email、⽤⼾的 subuser 的名称空间 users.swift，以及 bucket 的名称空间 root 等。

default.rgw.buckets.index：存放 bucket 到 object 的索引信息。

default.rgw.buckets.data：存放对象的数据。

default.rgw.buckets.non-ec：数据的额外信息存储池

default.rgw.users.uid：存放用户信息的存储池。

default.rgw.data.root：存放 bucket 的元数据，结构体对应 RGWBucketInfo，比如存放桶名、桶 ID、data_pool 等。

查看对象存储池的存储策略、副本数量、pgp和pg的数量

cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool get default.rgw.meta crush_rule
crush_rule: replicated_rule

cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool get default.rgw.meta size
size: 3

cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool get default.rgw.meta pgp_num
pgp_num: 8

cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool get default.rgw.meta pg_num
pg_num: 8

radosgw http 服务高可用配置

自定义 http 端口

配置文件可以在 ceph deploy 服务器修改然后统一推送，或者单独修改每个 radosgw 服务器的配置为统一配置，然后重启 RGW 服务。

https://docs.ceph.com/en/latest/radosgw/frontends/

在ceph.conf最后面添加针对当前节点的自定义配置如下

root@ceph-mgr1:~# vim /etc/ceph/ceph.conf
[client.rgw.ceph-mgr1]
rgw_host = ceph-mgr1
rgw_frontends = civetweb port=9900

重启节点 mgr1 的 radosgw 服务

root@ceph-mgr1:~# systemctl restart ceph-radosgw@rgw.ceph-mgr1.service 
root@ceph-mgr1:~# systemctl status ceph-radosgw@rgw.ceph-mgr1.service 
 ceph-radosgw@rgw.ceph-mgr1.service - Ceph rados gateway
   Loaded: loaded (/lib/systemd/system/ceph-radosgw@.service; indirect; vendor preset: enabled)
   Active: active (running) since Wed 2022-12-14 11:44:11 CST; 6s ago
 Main PID: 4196 (radosgw)
    Tasks: 603
   CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw@rgw.ceph-mgr1.service
           ©¸©¤4196 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr1 --setuser ceph --setgroup ceph

Dec 14 11:44:11 ceph-mgr1 systemd[1]: Started Ceph rados gateway.
Dec 14 11:44:11 ceph-mgr1 radosgw[4196]: 2022-12-14T11:44:11.494+0800 7f76c28843c0 -1 IMPORTANT: the civetweb frontend is 

root@ceph-mgr1:~# ss -lntup|grep 9900
tcp   LISTEN  0       128                  0.0.0.0:9900           0.0.0.0:*      users:(("radosgw",pid=4196,fd=75))

实现高可用

安装haproxy并配置反向代理：

配置haproxy，反向代理 ceph-mgr1 和 mgr2 的radosgw服务tcp网络端口，mgr2的端口此时还为默认的7480端口.

root@haproxyA:~# vim /etc/haproxy/haproxy.cfg 
listen ceph-radosgw-8090
   bind :8090
   mode tcp
   server ceph-mgr1 192.168.100.38:9900 check inter 3s fall 3 rise 2
   server ceph-mgr2 192.168.100.39:7480 check inter 3s fall 3 rise 2


root@haproxyA:~# systemctl restart haproxy
root@haproxyA:~# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
     Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-12-14 12:00:17 CST; 4s ago
       Docs: man:haproxy(1)
             file:/usr/share/doc/haproxy/configuration.txt.gz
    Process: 1401 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
   Main PID: 1413 (haproxy)
      Tasks: 2 (limit: 2236)
     Memory: 2.2M
     CGroup: /system.slice/haproxy.service
             ├─1413 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
             └─1417 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock

Dec 14 12:00:17 haproxyA systemd[1]: Starting HAProxy Load Balancer...
Dec 14 12:00:17 haproxyA haproxy[1413]: Proxy ceph-radosgw-8090 started.
Dec 14 12:00:17 haproxyA haproxy[1413]: Proxy ceph-radosgw-8090 started.
Dec 14 12:00:17 haproxyA haproxy[1413]: Proxy statistics started.
Dec 14 12:00:17 haproxyA haproxy[1413]: Proxy statistics started.
Dec 14 12:00:17 haproxyA haproxy[1413]: [NOTICE] 347/120017 (1413) : New worker #1 (1417) forked
Dec 14 12:00:17 haproxyA systemd[1]: Started HAProxy Load Balancer.

root@haproxyA:~# ss -lntup|grep 8090
tcp     LISTEN   0        3000             0.0.0.0:8090           0.0.0.0:*      users:(("haproxy",pid=1417,fd=7)

浏览器访问haproxy代理地址 192.168.100.20:8090

查看haproxy的服务代理日志，能看到将客户端的请求均衡代理到后端实际的ceph-mgr节点 endpoint

日志及其它优化配置

创建日志目录

root@ceph-mgr2:~# mkdir /var/log/radosgw
root@ceph-mgr2:~# chown ceph.ceph /var/log/radosgw
root@ceph-mgr2:~# vim /etc/ceph/ceph.conf 
[client.rgw.ceph-mgr1]
rgw_host = ceph-mgr1
rgw_frontends = "civetweb port=9900+9443s ssl_certificate=/etc/ceph/certs/cephrgw.pem error_log_file=/var/log/radosgw/radosgw.error.log access_log_file=/var/log/radosgw/radosgw.access.log request_timeout_ms=30000 num_threads=200"

[client.rgw.ceph-mgr2]
rgw_host = ceph-mgr2
rgw_frontends = "civetweb port=9900+9443s ssl_certificate=/etc/ceph/certs/cephrgw.pem error_log_file=/var/log/radosgw/radosgw.error.log access_log_file=/var/log/radosgw/radosgw.access.log request_timeout_ms=30000 num_threads=200"

error_log_file：指定radosgw错误日志路径

access_log_file：指定radosgw访问日志路径

request_timeout_ms：指定radosgw访问超时时间

num_threads：指定radosgw运行线程数量，默认线程数是100，https://docs.ceph.com/en/mimic/radosgw/config-ref/

重启radosgw

root@ceph-mgr2:/etc/ceph# systemctl restart ceph-radosgw@rgw.ceph-mgr2.service 
root@ceph-mgr2:/etc/ceph# systemctl status ceph-radosgw@rgw.ceph-mgr2.service

验证日志

创建 RGW 账户

在ceph管理节点创建对象用户

--uid 指定用户ID

--display-name 指定显示用户名称

cephadmin@ceph-deploy:~/ceph-cluster$ radosgw-admin user create --uid="user1" --display-name="user1"
{
    "user_id": "user1",
    "display_name": "user1",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "user1",
            "access_key": "45CMIRWTFQY9DGJX7W1Z",
            "secret_key": "EyFmlD51WWfCGbtxFYZcygwDc48QWMYyKs13nuDD"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

注意保存对象用户的 access_key 和 secret_key

查看用户信息

root@ceph-mgr1:/var/log/ceph# radosgw-admin user --uid="user1" info
{
    "user_id": "user1",
    "display_name": "user1",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "user1",
            "access_key": "45CMIRWTFQY9DGJX7W1Z",
            "secret_key": "EyFmlD51WWfCGbtxFYZcygwDc48QWMYyKs13nuDD"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

查询所有用户

root@ceph-mgr1:/var/log/ceph# radosgw-admin metadata list user
[
    "user1"
]

RGW 账户权限控制

参考aws 官网文档介绍：https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/example-bucket-policies.html

账户权限介绍

1、授权简介和预览

Resources： 授权的目的 Buckets、objects等资源，必须指定。

Actions：要授予的动作，CreateBucket、DeleteObject、GetObject、PubObject。必须指定

Effect：要授予的操作效果是允许（allow）还是拒绝（deny），默认为拒绝访问所有的资源，必须指定。

Principal: 要授权的目的账号，必须指定

Condition：授权策略生效的条件，比如访问TLS版本等，非必须，可不写。

{
 “Condition”: {
   “NumericLessThan”: {
     “s3:TlsVersion”: 1.2
    }
  }
}

2、权限集合

https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/API/API_Operations.html

权限配置

1、授予匿名用户对 bucket01 的 GetObject权限，仅可以查看桶内的文件。

创建权限json文件

[root@ansible ~]# vim bucket01-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": [
                "arn:aws:s3:::bucket01/*"
            ]
        }
    ]
}

进行授权

[root@ansible ~]# s3cmd setpolicy bucket01-policy.json s3://bucket01
s3://bucket01/: Policy updated

验证权限，客户端浏览器访问 http://rgw.cncf.net/bucket01/<文件名>

posted @ 2023-01-29 15:11 PunchLinux 阅读(940) 评论(0) 编辑收藏举报

刷新页面返回顶部

puchlinux