Ceph Reef(18.2.X)之python操作对象存储网关

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

目录

一.环境准备

1.Ceph Reef(18.2.X)的对象存储网关(rgw)组件搭建

推荐阅读:
    https://www.cnblogs.com/yinzhengjie/p/18377291

2.创建账号

[root@ceph141 ~]# radosgw-admin user create --uid "yinzhengjie" --display-name "尹正杰"
{
    "user_id": "yinzhengjie",
    "display_name": "尹正杰",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "yinzhengjie",
            "access_key": "M25RJ5F8XLNVUY4ORF6Z",
            "secret_key": "lk7c4eNCAkTOfaI3BuOcct70peebF2CCPUKpR6s5"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

[root@ceph141 ~]#

3.s3cmd工具

    1.安装s3cmd
apt -y install s3cmd

    2.查看rgw所在节点
[root@ceph141 ~]# ceph orch ls  | grep rgw
rgw.yinzhengjie  ?:80             1/1  10m ago    31m  ceph142      
[root@ceph141 ~]# 
[root@ceph141 ~]# echo 172.30.100.142 www.yinzhengjie.com >> /etc/hosts
[root@ceph141 ~]# 

    3.运行s3cmd的运行环境，生成"/root/.s3cfg"配置文件
[root@ceph141 ~]# s3cmd --configure 

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: M25RJ5F8XLNVUY4ORF6Z
Secret Key: lk7c4eNCAkTOfaI3BuOcct70peebF2CCPUKpR6s5
Default Region [US]: 

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: www.yinzhengjie.com

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: www.yinzhengjie.com/%(bucket)

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: 
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: No

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: 

New settings:
  Access Key: M25RJ5F8XLNVUY4ORF6Z
  Secret Key: lk7c4eNCAkTOfaI3BuOcct70peebF2CCPUKpR6s5
  Default Region: US
  S3 Endpoint: www.yinzhengjie.com
  DNS-style bucket+hostname:port template for accessing a bucket: www.yinzhengjie.com/%(bucket)
  Encryption password: 
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: 
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'
[root@ceph141 ~]#

二.Python操作对象存储

1.安装python环境

    1.安装pip工具包
[root@ceph141 ~]# apt -y install python3-pip

    2.配置pip3软件源
[root@ceph141 ~]# mkdir ~/.pip
[root@ceph141 ~]# vim ~/.pip/pip.conf
[root@ceph141 ~]# cat  ~/.pip/pip.conf
[global]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple
[root@ceph141 ~]# 

    3.安装boto包
[root@ceph141 ~]# pip install boto

2.编写python程序

[root@ceph141 ~]# cat rgw-yinzhengjie.py 
import boto
import boto.s3.connection

access_key = 'M25RJ5F8XLNVUY4ORF6Z'
secret_key = 'lk7c4eNCAkTOfaI3BuOcct70peebF2CCPUKpR6s5'

# 连接rgw
conn = boto.connect_s3(
        aws_access_key_id = access_key,
        aws_secret_access_key = secret_key,
        host = 'www.yinzhengjie.com',
        is_secure=False,
        calling_format = boto.s3.connection.OrdinaryCallingFormat(),
        )

# 创建bucket
bucket = conn.create_bucket('yinzhengjie-rgw')

# 查看bucket列表
for bucket in conn.get_all_buckets():
        print("{name}\t{created}".format(
                name = bucket.name,
                created = bucket.creation_date,
        ))


# 查看bucket内容
for key in bucket.list():
        print("{name}\t{size}\t{modified}".format(
                name = key.name,
                size = key.size,
                modified = key.last_modified,
        ))

# 创建一个对象
key = bucket.new_key('blog.txt')
key.set_contents_from_string('https://www.cnblogs.com/yinzhengjie')

# 生成对象下载的URL
hello_key = bucket.get_key('blog.txt')
hello_url = hello_key.generate_url(0, query_auth=False, force_http=True)
print(hello_url)
[root@ceph141 ~]# 

参考链接：
      https://docs.ceph.com/en/latest/radosgw/s3/python/

3.测试python代码测试

[root@ceph141 ~]# python3 rgw-yinzhengjie.py 
yinzhengjie-rgw 2024-08-29T23:40:36.356Z
blog.txt        35      2024-08-29T23:44:19.424Z
http://www.yinzhengjie.com/yinzhengjie-rgw/blog.txt
[root@ceph141 ~]#

4.使用s3cmd命令访问测试

[root@ceph141 ~]# s3cmd get s3://yinzhengjie-rgw/blog.txt
download: 's3://yinzhengjie-rgw/blog.txt' -> './blog.txt'  [1 of 1]
 35 of 35   100% in    0s   712.50 B/s  done
[root@ceph141 ~]# 
[root@ceph141 ~]# more ./blog.txt
https://www.cnblogs.com/yinzhengjie
[root@ceph141 ~]#

5.创建访问策略

    1.编写策略配置文件
[root@ceph141 ~]# cat yinzhengjie-anonymous-access-policy.json 
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": ["*"]},
    "Action": "s3:GetObject",
    "Resource": [
      "arn:aws:s3:::yinzhengjie-rgw/*"
    ]
  }]
}
[root@ceph141 ~]# 

    2.应用策略
[root@ceph141 ~]# s3cmd info s3://yinzhengjie-rgw
s3://yinzhengjie-rgw/ (bucket):
   Location:  default
   Payer:     BucketOwner
   Expiration Rule: none
   Policy:    none
   CORS:      none
   ACL:       尹正杰: FULL_CONTROL
[root@ceph141 ~]# 
[root@ceph141 ~]# s3cmd setpolicy yinzhengjie-anonymous-access-policy.json s3://yinzhengjie-rgw
s3://yinzhengjie-rgw/: Policy updated
[root@ceph141 ~]# 
[root@ceph141 ~]# s3cmd info s3://yinzhengjie-rgw
s3://yinzhengjie-rgw/ (bucket):
   Location:  default
   Payer:     BucketOwner
   Expiration Rule: none
   Policy:    {
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": ["*"]},
    "Action": "s3:GetObject",
    "Resource": [
      "arn:aws:s3:::yinzhengjie-rgw/*"
    ]
  }]
}

   CORS:      none
   ACL:       尹正杰: FULL_CONTROL
[root@ceph141 ~]#

6.发起http请求测试

[root@ceph141 ~]# curl -s http://www.yinzhengjie.com/yinzhengjie-rgw/blog.txt | more 
https://www.cnblogs.com/yinzhengjie
[root@ceph141 ~]# 


温馨提示:
   由于咱们访问的并不是html文件，不建议使用浏览器访问，而是用curl来模拟http请求即可。
posted @ 2024-08-30 05:01  尹正杰  阅读(50)  评论(0)  编辑  收藏  举报