ceph(六)对象存储网关(RadosGW)总结及s3cmd使用

一、radosgw的基本使用总结

https://docs.ceph.com/en/latest/radosgw/

1.1 RadosGW对象存储网关简介

RadosGW是对象存储(OSS,Object Storage Service)的一种访问实现方式,RADOS网关也称为Ceph对象网关、RadosGW、RGW,是一种服务,使客户端能够利用标准对象存储API来访问Ceph集群,它支持AWS S3和Swift API,在 ceph 0.8版本之后使用Civetweb(https:/lgithub.com/civetweb/civetweb)的 web服务器来响应api请求,客户端使用http/https协议通过RESTful API与RGW通信,而RGW则通过librados与ceph集群通信,RGW客户端通过s3或者swift api使用RGW用户进行身份验证,然后RGW网关代表用户利用cephx与ceph存储进行身份验证。
说明:S3由Amazon于2006年推出,全称为Simple Storage Service,S3定义了对象存储,是对象存储事实上的标准,从某种意义上说,S3就是对象存储,对象存储就是S3,它是对象存储市场的霸主,后续的对象存储都是对S3的模仿。

1.2 RadosGW存储特点

  1. 通过对象存储网关将数据存储为对象,每个对象除了包含数据,还包含数据自身的元数据。
  2. 对象通过Object ID来检索,不是通过普通文件系统的挂载方式,而是通过文件路径加文件名称操作来直接访问对象,只能通过API来访问,或者第三方客户端(实际上也是对API的封装)来访问。
  3. 对象的存储不是垂直的目录树结构,而是存储在扁平的命名空间中,Amazon S3将这个扁平命名空间称为bucket,而swift则将其称为容器。无论是bucket还是容器,都不能再嵌套(bucket不能再包含 bucket)。
  4. bucket需要被授权才能访问到,一个帐户可以对多个bucket 授权,而权限可以不同。
  5. 方便横向扩展、快速检索数据。
  6. 不支持客户端挂载,且需要客户端在访问的时候指定文件名称。
  7. 不是很适用于文件过于频繁修改及删除的场景。

ceph 使用bucket作为存储桶(存储空间),实现对象数据的存储和多用户隔离,数据存储在bucket 中,用户的权限也是针对bucket进行授权,可以设置用户对不同的bucket拥有不同的权限,以实现权限管理。

bucket特性:

  1. 存储空间(bucket)是用于存储对象(Object)的容器,所有的对象都必须隶属于某个存储空间,可以设置和修改存储空间属性用来控制地域、访问权限、生命周期等,这些属性设置直接作用于该存储空间内所有对象,因此可以通过灵活创建不同的存储空间来完成不同的管理功能
  2. 同一个存储空间的内部是扁平的,没有文件系统的目录等概念,所有的对象都直接隶属于其对应的存储空间
  3. 每个用户可以拥有多个存储空间
  4. 存储空间的名称在OSS范围内必须是全局唯一的,一旦创建之后无法修改名称
  5. 存储空间内部的对象数目没有限制

bucket命名规范:

https://docs.amazonaws.cn/AmazonS3/latest/userguide/bucketnamingrules.html

  1. 只能包括小写字母、数字和短横线(-)
  2. 必须以小写字母或者数字开头和结尾
  3. 长度必须在3-63字节之间
  4. 存储桶名称不能使用用IP地址格式
  5. Bucket名称必须全局唯一

1.3 部署RadosGW服务

radosgw架构图

radosgw逻辑图

​​

1.3.1 安装radosgw服务

将ceph-mgr1、ceph-mgr2服务器部署为radosGW高可用服务

# 安装radosgw服务
[root@ceph-mgr1 ~]#apt install radosgw
[root@ceph-mgr2 ~]#apt install radosgw
# 在deploy服务器将rgw服务的ceph-mgr1、ceph-mgr2节点添加至集群
cephadmin@ceph-deploy:/data/ceph-cluster$ ceph-deploy rgw create ceph-mgr1
cephadmin@ceph-deploy:/data/ceph-cluster$ ceph-deploy rgw create ceph-mgr2
# 添加ceph-mgr2的服务过程
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.1.0): /usr/local/bin/ceph-deploy rgw create ceph-mgr2
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf object at 0x7f578fb45d60>
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] func : <function rgw at 0x7f578fc36f70>
[ceph_deploy.cli][INFO ] rgw : [('ceph-mgr2', 'rgw.ceph-mgr2')]
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts ceph-mgr2:rgw.ceph-mgr2
[ceph-mgr2][DEBUG ] connection detected need for sudo
[ceph-mgr2][DEBUG ] connected to host: ceph-mgr2
[ceph_deploy.rgw][INFO ] Distro info: ubuntu 20.04 focal
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-mgr2
[ceph-mgr2][WARNIN] rgw keyring does not exist yet, creating one
[ceph-mgr2][INFO ] Running command: sudo ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-mgr2 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-mgr2/keyring
[ceph-mgr2][INFO ] Running command: sudo systemctl enable ceph-radosgw@rgw.ceph-mgr2
[ceph-mgr2][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-mgr2.service → /lib/systemd/system/ceph-radosgw@.service.
[ceph-mgr2][INFO ] Running command: sudo systemctl start ceph-radosgw@rgw.ceph-mgr2
[ceph-mgr2][INFO ] Running command: sudo systemctl enable ceph.target
[ceph_deploy.rgw][INFO ] The Ceph Object Gateway (RGW) is now running on host ceph-mgr2 and default port 7480

1.3.2 验证radosgw服务状态

cephadmin@ceph-deploy:/data/ceph-cluster$ ceph -s
cluster:
id: 28820ae5-8747-4c53-827b-219361781ada
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 29h)
mgr: ceph-mgr2(active, since 29h), standbys: ceph-mgr1
mds: 2/2 daemons up, 2 standby
osd: 20 osds: 20 up (since 28h), 20 in (since 4d)
rgw: 2 daemons active (2 hosts, 1 zones) # rgw服务:2个活跃节点
data:
volumes: 1/1 healthy
pools: 10 pools, 321 pgs
objects: 392 objects, 243 MiB
usage: 6.5 GiB used, 20 TiB / 20 TiB avail
pgs: 321 active+clean

1.3.3 验证radosgw服务进程

cephadmin@ceph-mgr1:~$ ps -ef|grep radosgw
ceph 49861 1 0 05:42 ? 00:00:00 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr1 --setuser ceph --setgroup ceph
cephadm+ 50649 50608 0 05:44 pts/0 00:00:00 grep --color=auto radosgw
cephadmin@ceph-mgr2:~$ ps -ef|grep radosgw
ceph 50222 1 0 05:42 ? 00:00:04 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr2 --setuser ceph --setgroup ceph
cephadm+ 51421 51408 0 06:00 pts/0 00:00:00 grep --color=auto radosgw

1.3.4 radosgw存储池类型

cephadmin@ceph-deploy:/data/ceph-cluster$ ceph osd pool ls
device_health_metrics
mypool
myrbd1
rbd-data1
cephfs-metadata
cephfs-data
.rgw.root
default.rgw.log
default.rgw.control
default.rgw.meta
# 查看默认radosgw存储池信息
cephadmin@ceph-deploy:/data/ceph-cluster$ radosgw-admin zone get --rgw-zone=default --rgw-zonegroup=default
{
"id": "a202e4c0-376b-4848-956f-5b072662c3a3", # 区域的唯一标识符
"name": "default", # 默认区域的名称
"domain_root": "default.rgw.meta:root", # 区域的根域名
"control_pool": "default.rgw.control", # 系统控制池,在有数据更新是,通知其他RGW更新缓存
"gc_pool": "default.rgw.log:gc", # 用于垃圾回收的存储池
"lc_pool": "default.rgw.log:lc", # 用于存储日志的存储池
"log_pool": "default.rgw.log", # 存储日志信息,用于记录各种log信息
"intent_log_pool": "default.rgw.log:intent",
"usage_log_pool": "default.rgw.log:usage",
"roles_pool": "default.rgw.meta:roles", # default.rgw.meta:元数据存储池,通过不同的名称空间分别存储不同的rados对象
"reshard_pool": "default.rgw.log:reshard",
"user_keys_pool": "default.rgw.meta:users.keys", # 用户的密钥名称空间users.keys
"user_email_pool": "default.rgw.meta:users.email", # 用户的email名称空间users.email
"user_swift_pool": "default.rgw.meta:users.swift", # 用户的subuser的名称空间users.swift
"user_uid_pool": "default.rgw.meta:users.uid", # 用户UID
"otp_pool": "default.rgw.otp",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "default.rgw.buckets.index", # 存放bucket到object的索引信息
"storage_classes": {
"STANDARD": {
"data_pool": "default.rgw.buckets.data" # 存放对象的数据
}
},
"data_extra_pool": "default.rgw.buckets.non-ec", # 数据的额外信息存储池
"index_type": 0
}
}
],
"realm_id": "",
"notif_pool": "default.rgw.log:notif"
}
# 默认crush规则是副本池即一主两备的三副本机制
cephadmin@ceph-deploy:/data/ceph-cluster$ ceph osd pool get default.rgw.meta crush_rule
crush_rule: replicated_rule
# 默认副本数为3
cephadmin@ceph-deploy:/data/ceph-cluster$ ceph osd pool get default.rgw.meta size
size: 3
# 默认pgp数量为32
cephadmin@ceph-deploy:/data/ceph-cluster$ ceph osd pool get default.rgw.meta pgp_num
pgp_num: 32
# 默认pg数量为32
cephadmin@ceph-deploy:/data/ceph-cluster$ ceph osd pool get default.rgw.meta pg_num
pg_num: 32

1.3.5 radosgw存储池功能

cephadmin@ceph-deploy:/data/ceph-cluster$ ceph osd lspools
1 device_health_metrics
2 mypool
3 myrbd1
4 rbd-data1
5 cephfs-metadata
6 cephfs-data
7 .rgw.root
8 default.rgw.log
9 default.rgw.control
10 default.rgw.meta

1.3.6 验证radosgw zone信息

cephadmin@ceph-deploy:/data/ceph-cluster$ radosgw-admin zone get --rgw-zone=default
{
"id": "a202e4c0-376b-4848-956f-5b072662c3a3",
"name": "default",
"domain_root": "default.rgw.meta:root",
"control_pool": "default.rgw.control",
"gc_pool": "default.rgw.log:gc",
"lc_pool": "default.rgw.log:lc",
"log_pool": "default.rgw.log",
"intent_log_pool": "default.rgw.log:intent",
"usage_log_pool": "default.rgw.log:usage",
"roles_pool": "default.rgw.meta:roles",
"reshard_pool": "default.rgw.log:reshard",
"user_keys_pool": "default.rgw.meta:users.keys",
"user_email_pool": "default.rgw.meta:users.email",
"user_swift_pool": "default.rgw.meta:users.swift",
"user_uid_pool": "default.rgw.meta:users.uid",
"otp_pool": "default.rgw.otp",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"storage_classes": {
"STANDARD": {
"data_pool": "default.rgw.buckets.data"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 0
}
}
],
"realm_id": "",
"notif_pool": "default.rgw.log:notif"
}

1.3.7 访问radosgw服务

1.4 radosgw高可用

1.4.1 radosgw http高可用

1.4.1.1 自定义http端口

配置文件可以在ceph deploy服务器修改然后统一推送,或者单独修改每个radosgw服务器的配置,然后重启RGW服务

https://docs.ceph.com/en/latest/radosgw/frontends/

cat /etc/ceph/ceph.conf
...
# 添加如下配置
[client.rgw.ceph-mgr1]
rgw_host = ceph-mgr1
rgw_frontends = civetweb port=9900
[client.rgw.ceph-mgr2]
rgw_host = ceph-mgr2
rgw_frontends = civetweb port=9900
# 统一推送配置文件至radosgw服务器
cephadmin@ceph-deploy:/data/ceph-cluster$ sudo scp ceph.conf 10.0.0.54:/etc/ceph/ 100% 763 743.7KB/s 00:00
cephadmin@ceph-deploy:/data/ceph-cluster$ sudo scp ceph.conf 10.0.0.55:/etc/ceph/
# 重启服务
[root@ceph-mgr1 ~]#systemctl restart ceph-radosgw@rgw.ceph-mgr1.service
[root@ceph-mgr2 ~]#systemctl restart ceph-radosgw@rgw.ceph-mgr2.service

验证

1.4.1.2 高可用配置

使用haproxy进行反向代理

[root@lb1 ~]#cat /etc/haproxy/haproxy.cfg
# 添加如下配置
listen ceph-rgw
bind 10.0.0.63:80
mode tcp
server 10.0.0.54 10.0.0.54:9900 check inter 3s fall 3 rise 5
server 10.0.0.55 10.0.0.55:9900 check inter 3s fall 3 rise 5

1.4.1.3 测试http反向代理

1.4.2 radosgw https

在rgw节点生成签名证书并配置radosgw启用SSL

1.4.2.1 自签名证书

[root@ceph-mgr1 ~]#mkdir -p /etc/ceph/certs
[root@ceph-mgr1 ~]#cd /etc/ceph/certs/
[root@ceph-mgr1 certs]#openssl genrsa -out civetweb.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
............................................................................+++++
.......+++++
e is 65537 (0x010001)
[root@ceph-mgr1 certs]#openssl req -new -x509 -key civetweb.key -out civetweb.crt -subj "/CN=rgw.chu.net"
[root@ceph-mgr1 certs]#cat civetweb.key civetweb.crt > civetweb.pem
[root@ceph-mgr1 certs]#tree
.
├── civetweb.crt
├── civetweb.key
└── civetweb.pem
0 directories, 3 files
# 可直接将证书复制到ceph-mgr2节点上
[root@ceph-mgr1 ceph]#scp -r /etc/ceph/certs 10.0.0.55:/etc/ceph/

1.4.2.2 SSL配置

[root@ceph-mgr1 certs]#vim /etc/ceph/ceph.conf
....
# ceph-mgr1节点配置
[client.rgw.ceph-mgr1]
rgw_host = ceph-mgr1
rgw_frontends = "civetweb port=9900+9443s ssl_certificate=/etc/ceph/certs/civetweb.pem"
# ceph-mgr2节点配置
[client.rgw.ceph-mgr2]
rgw_host = ceph-mgr2
rgw_frontends = "civetweb port=9900+9443s ssl_certificate=/etc/ceph/certs/civetweb.pem"
# 重启服务
[root@ceph-mgr1 certs]#systemctl restart ceph-radosgw@rgw.ceph-mgr1.service
[root@ceph-mgr2 certs]#systemctl restart ceph-radosgw@rgw.ceph-mgr2.service

1.4.2.3 验证https端口

[root@ceph-mgr1 certs]#netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9900 0.0.0.0:* LISTEN 80864/radosgw # 9900端口
tcp 0 0 127.0.0.1:38351 0.0.0.0:* LISTEN 991/containerd
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/init
tcp 0 0 10.0.0.54:6800 0.0.0.0:* LISTEN 46495/ceph-mds
tcp 0 0 10.0.0.54:6801 0.0.0.0:* LISTEN 46495/ceph-mds
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 919/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1017/sshd: /usr/sbi
tcp 0 0 0.0.0.0:37015 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:35481 0.0.0.0:* LISTEN 942/rpc.mountd
tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 30872/sshd: root@pt
tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:9443 0.0.0.0:* LISTEN 80864/radosgw # 9443端口
tcp 0 0 0.0.0.0:35875 0.0.0.0:* LISTEN 942/rpc.mountd
tcp 0 0 0.0.0.0:37639 0.0.0.0:* LISTEN 942/rpc.mountd
[root@ceph-mgr1 certs]#lsof -i:9443
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
radosgw 80864 ceph 80u IPv4 343956 0t0 TCP *:9443 (LISTEN)
[root@ceph-mgr2 ~]#lsof -i:9443
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
radosgw 81763 ceph 80u IPv4 371937 0t0 TCP *:9443 (LISTEN)

1.4.2.4 验证访问

在本地host添加域名解析

echo "10.0.0.54 rgw.chu.net" >> /etc/hosts
# 或者 echo "10.0.0.55 rgw.chu.net" >> /etc/hosts
  • http访问

  • https访问

证书信息

1.4.3 radosgw高可用

通过负载均衡haproxy对radosgw进行反向代理,实现高可用

1.4.3.1 域名解析至负载均衡

将域名解析至负载均衡(VIP)

10.0.0.63 rgw.chu.net

1.4.3.2 负载均衡配置

如下单haproxy配置,也可结合keepalived实现负载均衡集群

# 配置监听
listen ceph-rgw-http
bind 10.0.0.63:80
mode tcp
server 10.0.0.54 10.0.0.54:9900 check inter 3s fall 3 rise 5
server 10.0.0.55 10.0.0.55:9900 check inter 3s fall 3 rise 5
listen ceph-rgw-https
bind 10.0.0.63:443
mode tcp
server 10.0.0.54 10.0.0.54:9443 check inter 3s fall 3 rise 5
server 10.0.0.55 10.0.0.55:9443 check inter 3s fall 3 rise 5

重启服务

[root@lb1 ~]#systemctl restart haproxy.service

1.4.3.3 测试访问


1.4.4 添加日志记录功能

增加日志及其他优化配置

  1. 新建日志目录并授权

    [root@ceph-mgr1 ceph]#mkdir -p /var/log/radosgw
    [root@ceph-mgr1 ceph]#chown -R ceph.ceph /var/log/radosgw
  2. 修改配置文件

    [root@ceph-mgr1 ceph]#cat /etc/ceph/ceph.conf
    ...
    # 修改rgw本节点配置
    [client.rgw.ceph-mgr1]
    rgw_host = ceph-mgr1
    rgw_frontends = "civetweb port=9900+9443s ssl_certificate=/etc/ceph/certs/civetweb.pem error_log_file=/var/log/radosgw/civetweb.error.log access_log_file=/var/log/radosgw/civetweb.access.log request_timeout_ms=30000 num_threads=200"

    num_threads默认值等于rgw_thread_pool_size=100

  3. 重启服务

    # systemctl daemon-reload
    [root@ceph-mgr1 ceph]#systemctl restart ceph-radosgw@rgw.ceph-mgr1.service
  4. 服务测试

    本地curl命令和客户端浏览器访问

    [root@ceph-mgr1 ceph]#curl -k https://10.0.0.54:9443
    <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner>
    <ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
  5. 验证日志

    [root@ceph-mgr1 ceph]#tail /var/log/radosgw/civetweb.access.log
    10.0.0.54 - - [26/Sep/2023:01:29:11 +0800] "GET / HTTP/1.1" 200 413 - curl/7.68.0
    10.0.0.63 - - [26/Sep/2023:01:29:16 +0800] "GET / HTTP/1.1" 200 437 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36
    10.0.0.63 - - [26/Sep/2023:01:29:16 +0800] "GET / HTTP/1.1" 200 437 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36
    10.0.0.63 - - [26/Sep/2023:01:29:16 +0800] "GET / HTTP/1.1" 200 437 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36
    10.0.0.63 - - [26/Sep/2023:01:29:16 +0800] "GET / HTTP/1.1" 200 437 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36

二、s3cmd客户端使用、实现基于nginx实现短视频的业务案例

https://docs.ceph.com/en/latest/radosgw/s3/

2.1 RGW Server配置

通常情况下,RGW1(10.0.0.54)和RGW2(10.0.0.55)参数配置是完全一样的

[root@ceph-mgr1 ceph]#cat /etc/ceph/ceph.conf
[global]
fsid = 28820ae5-8747-4c53-827b-219361781ada
public_network = 10.0.0.0/24
cluster_network = 192.168.10.0/24
mon_initial_members = ceph-mon1,ceph-mon2,ceph-mon3
mon_host = 10.0.0.51,10.0.0.52,10.0.0.53
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
...
[client.rgw.ceph-mgr1]
rgw_host = ceph-mgr1
rgw_frontends = "civetweb port=9900"
[client.rgw.ceph-mgr2]
rgw_host = ceph-mgr2
rgw_frontends = "civetweb port=9900"

2.2 创建RGW用户

用于s3cmd客户端连接RGW

cephadmin@ceph-deploy:/data/ceph-cluster$ radosgw-admin user create --uid="user1" --display-name="user1"
{
"user_id": "user1",
"display_name": "user1",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "user1",
"access_key": "26PBQL1JUTRAF3JAGZRA",
"secret_key": "yJkLphACBrlWNu2rCaQbC8MO3uAFnr0NPYmPEUew"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}

保存好user1用户的access_key、secret_key信息。

2.3 安装s3cmd客户端

s3cmd是一个通过命令行访问ceph RGW实现创建存储桶、上传、下载以及管理数据到对象存储的命令行工具。

cephadmin@ceph-deploy:/data/ceph-cluster$ sudo apt-cache madison s3cmd
cephadmin@ceph-deploy:/data/ceph-cluster$ sudo apt install -y s3cmd

2.4 配置s3cmd客户端执行环境

  1. s3cmd客户端添加域名解析
cephadmin@ceph-deploy:/data/ceph-cluster$ cat /etc/hosts
....
10.0.0.63 rgw.chu.net # 负载均衡地址或RGW网关地址
  1. 配置命令执行环境
[root@ceph-deploy ~]#s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: 26PBQL1JUTRAF3JAGZRA # 输入之前创建用于客户端连接RGW网关的user1用户access key
Secret Key: yJkLphACBrlWNu2rCaQbC8MO3uAFnr0NPYmPEUew # 输入用户secret key
Default Region [US]: # 地域,直接回车
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: rgw.chu.net # RGW域名,若直连RGW设置为rgw.chu.net:9900
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: rgw.chu.net/%(bucket) # bucket域名格式
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: 123456 # 密码
Path to GPG program [/usr/bin/gpg]: # gpg命令路径,用于认证管理,直接回车
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: No # 是否使用HTTPS
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: # 代理
New settings: # 最终配置
Access Key: 26PBQL1JUTRAF3JAGZRA
Secret Key: yJkLphACBrlWNu2rCaQbC8MO3uAFnr0NPYmPEUew
Default Region: US
S3 Endpoint: rgw.chu.net
DNS-style bucket+hostname:port template for accessing a bucket: rgw.chu.net/%(bucket)
Encryption password: 123456
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] y # 是否测试
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)
Save settings? [y/N] # 保存配置
Configuration saved to '/root/.s3cfg' # 配置文件保存路径
  1. 验证认证文件
[root@ceph-deploy ~]#cat /root/.s3cfg
[default]
access_key = 26PBQL1JUTRAF3JAGZRA
...
host_base = rgw.chu.net
host_bucket = rgw.chu.net/%(bucket)
...
secret_key = yJkLphACBrlWNu2rCaQbC8MO3uAFnr0NPYmPEUew
send_chunk = 65536
server_side_encryption = False
...

2.5 s3cmd常见使用命令

使用s3cmd --help​查看帮助信息,详细使用方法见:http://s3tools.org

Make bucket # 创建bucket
s3cmd mb s3://BUCKET
Remove bucket # 删除bucket,只能删除空的bucket,如果bucket中有内容,需要先删除内容,才能删除bucket。
s3cmd rb s3://BUCKET
List objects or buckets # 列出bucket中的文件
s3cmd ls [s3://BUCKET[/PREFIX]]
List all object in all buckets # 列出所有bucket中的所有文件
s3cmd la
Put file into bucket # 上传文件到bucket中
s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX]
Get file from bucket # 从bucket中下载文件到本地
s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
Delete file from bucket # 删除文件
s3cmd del s3://BUCKET/OBJECT
Delete file from bucket (alias for del) # 删除文件,del命令的别名
s3cmd rm s3://BUCKET/OBJECT
Restore file from Glacier storage # 恢复文件
s3cmd restore s3://BUCKET/OBJECT
Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overridden by options, see below)
s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR # 同步目录树
Disk usage by buckets # 空间使用
s3cmd du [s3://BUCKET[/PREFIX]]
Get various information about Buckets or Files # 获取bucket或文件的详细信息
s3cmd info s3://BUCKET[/OBJECT]
Copy object # 复制文件
s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
Modify object metadata # 修改文件属性
s3cmd modify s3://BUCKET1/OBJECT
Move object # 移动文件
s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
Modify Access control list for Bucket or Files
s3cmd setacl s3://BUCKET[/OBJECT]
Modify Bucket Policy
s3cmd setpolicy FILE s3://BUCKET
Delete Bucket Policy
s3cmd delpolicy s3://BUCKET
Modify Bucket CORS
s3cmd setcors FILE s3://BUCKET
Delete Bucket CORS
s3cmd delcors s3://BUCKET
Modify Bucket Requester Pays policy
s3cmd payer s3://BUCKET
Show multipart uploads
s3cmd multipart s3://BUCKET [Id]
Abort a multipart upload
s3cmd abortmp s3://BUCKET/OBJECT Id
List parts of a multipart upload
s3cmd listmp s3://BUCKET/OBJECT Id
Enable/disable bucket access logging
s3cmd accesslog s3://BUCKET
Sign arbitrary string using the secret key
s3cmd sign STRING-TO-SIGN
Sign an S3 URL to provide limited public access with expiry
s3cmd signurl s3://BUCKET/OBJECT <expiry_epoch|+expiry_offset>
Fix invalid file names in a bucket
s3cmd fixbucket s3://BUCKET[/PREFIX]
Create Website from bucket
s3cmd ws-create s3://BUCKET
Delete Website
s3cmd ws-delete s3://BUCKET
Info about Website
s3cmd ws-info s3://BUCKET
Set or delete expiration rule for the bucket
s3cmd expire s3://BUCKET
Upload a lifecycle policy for the bucket
s3cmd setlifecycle FILE s3://BUCKET
Get a lifecycle policy for the bucket
s3cmd getlifecycle s3://BUCKET
Remove a lifecycle policy for the bucket
s3cmd dellifecycle s3://BUCKET
List CloudFront distribution points
s3cmd cflist
Display CloudFront distribution point parameters
s3cmd cfinfo [cf://DIST_ID]
Create CloudFront distribution point
s3cmd cfcreate s3://BUCKET
Delete CloudFront distribution point
s3cmd cfdelete cf://DIST_ID
Change CloudFront distribution point parameters
s3cmd cfmodify cf://DIST_ID
Display CloudFront invalidation request(s) status
s3cmd cfinvalinfo cf://DIST_ID[/INVAL_ID]

2.6 s3cmd测试数据上传、下载

2.6.1 创建bucket

存储空间Bucket是用于存储对象Object的容器,在上传任意类型的Object前,需要先创建Bucket

bucket/object授权参考:https://docs.amazonaws.cn/AmazonS3/latest/userguide/access-policy-language-overview.html

[root@ceph-deploy ~]#s3cmd mb s3://mybucket
Bucket 's3://mybucket/' created
[root@ceph-deploy ~]#s3cmd mb s3://css
Bucket 's3://css/' created
[root@ceph-deploy ~]#s3cmd mb s3://images
Bucket 's3://images/' created

2.6.2 上传文件

# 上传文件至mybucket
[root@ceph-deploy ~]#s3cmd put /etc/passwd s3://mybucket
upload: '/etc/passwd' -> 's3://mybucket/passwd' [1 of 1]
2104 of 2104 100% in 1s 1182.09 B/s done
# /不表示文件目录层级关系,只表示地址信息
[root@ceph-deploy ~]#s3cmd put /var/log/syslog s3://images/log/
upload: '/var/log/syslog' -> 's3://images/log/syslog' [1 of 1]
614406 of 614406 100% in 0s 14.23 MB/s done
# 验证bucket中的文件
[root@ceph-deploy ~]#s3cmd ls s3://mybucket
2023-09-25 18:23 2104 s3://mybucket/passwd
# 可先查看逻辑上bucket根目录s3://images
[root@ceph-deploy ~]#s3cmd ls s3://images
DIR s3://images/log/
[root@ceph-deploy ~]#s3cmd ls s3://images/log/
2023-09-25 18:26 614406 s3://images/log/syslog

2.6.3 下载文件

# 下载文件
[root@ceph-deploy ~]#s3cmd get s3://images/log/syslog /opt/
download: 's3://images/log/syslog' -> '/opt/syslog' [1 of 1]
614406 of 614406 100% in 0s 38.28 MB/s done
# 验证文件
[root@ceph-deploy ~]#ls /opt/syslog
/opt/syslog

2.6.4 删除文件

# 查看bucket中文件
[root@ceph-deploy ~]#s3cmd ls s3://mybucket/
2023-09-25 18:23 2104 s3://mybucket/passwd
# 删除
[root@ceph-deploy ~]#s3cmd rm s3://mybucket/passwd
delete: 's3://mybucket/passwd'
# 查看bucket文件删除
[root@ceph-deploy ~]#s3cmd ls s3://mybucket/

2.7 结合nginx实现短视频业务

2.7.1 上传视频文件至bucket

  1. 创建bucket

    [root@ceph-deploy opt]#s3cmd mb s3://video
    Bucket 's3://video/' created
  2. 上传文件

    [root@ceph-deploy opt]#s3cmd put 1656753768362045.mp4 s3://video
    upload: '1656753768362045.mp4' -> 's3://video/1656753768362045.mp4' [1 of 1]
    15061866 of 15061866 100% in 0s 40.93 MB/s done
  3. 授权bucket匿名用户只读权限,用于浏览器访问

    https://docs.amazonaws.cn/AmazonS3/latest/userguide/example-bucket-policies.html

    # 编辑授权文件,对video bucket用户授予读权限
    cat video-bucket-single-policy.json
    {
    "Version": "2012-10-17",
    "Statement": [{
    "Effect": "Allow" ,
    "Principal": "*",
    "Action": "s3:GetObject",
    "Resource": [
    "arn:aws:s3:::video/*"
    ]
    }]
    }
    # 执行授权
    s3cmd setpolicy video-bucket-single-policy.json s3://video

  4. 配置haproxy

    [root@lb1 ~]#cat /etc/haproxy/haproxy.cfg
    ....
    listen ceph-rgw-http
    bind 10.0.0.63:9900
    mode tcp
    server 10.0.0.54 10.0.0.54:9900 check inter 3s fall 3 rise 5
    server 10.0.0.55 10.0.0.55:9900 check inter 3s fall 3 rise 5

    重启服务

    [root@lb1 ~]#systemctl restart haproxy.service
  5. 浏览器直接访问测试

    访问负载均衡测试

    ​​

​​​

2.7.2 配置nginx反向代理

nginx服务器为10.0.0.60,实现访问www.chu.net的视频内容将请求转向访问ceph文件存储中的video视频

[root@nginx ~]#cat /etc/nginx/nginx.conf
...
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# 配置server块内容
server {
listen 80;
server_name chu.net www.chu.net;
location / {
root html;
index index.html index.htm;
}
location ~* \.(mp4|avi)$ { # 以mp4或avi结尾的请求转向10.0.0.63(VIP)
proxy_pass http://10.0.0.63:9900;
}
}
}

重启nginx

nginx -s reload

2.7.3 访问测试

配置本地hosts域名解析

10.0.0.60 chu.net www.chu.net

浏览器访问视频

posted @   areke  阅读(1917)  评论(0编辑  收藏  举报
相关博文:
阅读排行:
· DeepSeek “源神”启动!「GitHub 热点速览」
· 微软正式发布.NET 10 Preview 1:开启下一代开发框架新篇章
· 我与微信审核的“相爱相杀”看个人小程序副业
· C# 集成 DeepSeek 模型实现 AI 私有化(本地部署与 API 调用教程)
· DeepSeek R1 简明指南:架构、训练、本地部署及硬件要求
点击右上角即可分享
微信分享提示