当前位置：首页 > news >正文

ceph的RGW对象存储网关部署和使用

news 来源：原创 2025/6/17 1:12:50

RGW对象存储网关部署和使用

Rados GateWay简称rgw，底层采用亚马逊的S3作为对象存储实现。

参考链接：

https://docs.ceph.com/en/reef/radosgw/
https://docs.ceph.com/en/reef/radosgw/bucketpolicy/
https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/bucketnamingrules.html
https://www.s3express.com/help/help.html

01 部署rgw组件

1 部署之前查看集群状态
[root@ceph141 ~]# ceph -scluster:id:     3cb12fba-5f6e-11ef-b412-9d303a22b70fhealth: HEALTH_OKservices:mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 11m)mgr: ceph141.cwgrgj(active, since 10m), standbys: ceph142.ymuzfemds: 1/1 daemons up, 1 standbyosd: 7 osds: 7 up (since 11m), 7 in (since 16h)data:volumes: 1/1 healthypools:   3 pools, 65 pgsobjects: 48 objects, 492 KiBusage:   329 MiB used, 3.3 TiB / 3.3 TiB availpgs:     65 active+clean2 创建一个服务
[root@ceph141 ~]# ceph orch apply rgw wenzhiyong
Scheduled rgw.wenzhiyong update...
[root@ceph141 ~]# 3 部署rgw组件 
[root@ceph141 ~]# ceph orch daemon add rgw wenzhiyong ceph142
Deployed rgw.wenzhiyong.ceph141.csxaif on host 'ceph142'
[root@ceph141 ~]# 4 检查rgw组件是否部署成功
[root@ceph141 ~]# ceph -scluster:id:     3cb12fba-5f6e-11ef-b412-9d303a22b70fhealth: HEALTH_OKservices:mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 23m)mgr: ceph141.cwgrgj(active, since 23m), standbys: ceph142.ymuzfemds: 1/1 daemons up, 1 standbyosd: 7 osds: 7 up (since 23m), 7 in (since 16h)rgw: 1 daemon active (1 hosts, 1 zones)  # Duang~不难发现，多了一个rgw组件！data:volumes: 1/1 healthypools:   7 pools, 193 pgsobjects: 274 objects, 499 KiBusage:   430 MiB used, 3.3 TiB / 3.3 TiB availpgs:     193 active+clean5 查看rgw默认创建的存储池信息
[root@ceph141 ~]# ceph osd pool ls
...
.rgw.root
default.rgw.log
default.rgw.control
default.rgw.meta
[root@ceph141 ~]# 
[root@ceph141 ~]# radosgw-admin zone get --rgw-zone=default --rgw-zonegroup=default
{"id": "10c61974-a41b-438d-ac2e-942b00e11d53","name": "default","domain_root": "default.rgw.meta:root","control_pool": "default.rgw.control","gc_pool": "default.rgw.log:gc","lc_pool": "default.rgw.log:lc","log_pool": "default.rgw.log","intent_log_pool": "default.rgw.log:intent","usage_log_pool": "default.rgw.log:usage","roles_pool": "default.rgw.meta:roles","reshard_pool": "default.rgw.log:reshard","user_keys_pool": "default.rgw.meta:users.keys","user_email_pool": "default.rgw.meta:users.email","user_swift_pool": "default.rgw.meta:users.swift","user_uid_pool": "default.rgw.meta:users.uid","otp_pool": "default.rgw.otp","system_key": {"access_key": "","secret_key": ""},"placement_pools": [{"key": "default-placement","val": {"index_pool": "default.rgw.buckets.index","storage_classes": {"STANDARD": {"data_pool": "default.rgw.buckets.data"}},"data_extra_pool": "default.rgw.buckets.non-ec","index_type": 0,"inline_data": true}}],"realm_id": "","notif_pool": "default.rgw.log:notif"
}
[root@ceph141 ~]# 6.查看ceph集群各组件部署信息
[root@ceph141~]# ceph orch ls
NAME                  PORTS        RUNNING  REFRESHED  AGE  PLACEMENT    
alertmanager          ?:9093,9094      1/1  2m ago     2d   count:1      
ceph-exporter                          3/3  2m ago     2d   *            
crash                                  3/3  2m ago     2d   *            
grafana               ?:3000           1/1  2m ago     2d   count:1      
mds.zhiyong18-cephfs                   2/2  2m ago     21h  count:2      
mgr                                    2/2  2m ago     2d   count:2      
mon                                    3/5  2m ago     2d   count:5      
node-exporter         ?:9100           3/3  2m ago     2d   *            
osd                                      7  2m ago     -    <unmanaged>  
prometheus            ?:9095           1/1  2m ago     2d   count:1      
rgw.wenzhiyong        ?:80             1/1  4s ago     10m  ceph142  7 访问对象存储的WebUI
http://10.0.0.142/

在这里插入图片描述

可能遇到的故障：

重复添加142节点，导致提示80端口被占用

02 s3cmd工具安装

S3cmd 是一个用于与兼容 Amazon S3 的云存储服务交互的命令行工具。支持多种 S3 兼容的存储平台（如 MinIO、阿里云 OSS 等），可以用来执行常见的存储操作，例如上传、下载、同步文件和管理存储桶等

1 安装s3cmd工具
[root@ceph141 ~]# echo 10.0.0.142 www.wenzhiyong.com >> /etc/hosts
[root@ceph141 ~]# 
[root@ceph141 ~]# apt -y install s3cmd2 创建rgw账号
[root@ceph141 ~]# radosgw-admin user create --uid "wzy666" --display-name "文治勇"
{"user_id": "wzy666","display_name": "文治勇","email": "","suspended": 0,"max_buckets": 1000,"subusers": [],"keys": [{"user": "wzy666","access_key": "ZHOE7MVPLJFE5EIU738W",                      # 注意，别丢了，下面要用！"secret_key": "VUNbdDwAGIq9AZv5f55e2gzptK1PUOnWg9nc44pE"   # 注意，别丢了，下面要用！}],"swift_keys": [],"caps": [],"op_mask": "read, write, delete","default_placement": "","default_storage_class": "","placement_tags": [],"bucket_quota": {"enabled": false,"check_on_raw": false,"max_size": -1,"max_size_kb": 0,"max_objects": -1},"user_quota": {"enabled": false,"check_on_raw": false,"max_size": -1,"max_size_kb": 0,"max_objects": -1},"temp_url_keys": [],"type": "rgw","mfa_ids": []
}3 运行s3cmd的运行环境，生成"/root/.s3cfg"配置文件
[root@ceph141 ~]# ll /root/.s3cfg
ls: cannot access '/root/.s3cfg': No such file or directory
[root@ceph141 ~]# 
[root@ceph141 ~]# s3cmd --configure Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: ZHOE7MVPLJFE5EIU738W  # rgw账号的access_key
Secret Key: VUNbdDwAGIq9AZv5f55e2gzptK1PUOnWg9nc44pE   # rgw账号的secret_key
Default Region [US]:  # 直接回车即可Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: www.wenzhiyong.com  # 用于访问rgw的地址Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: www.wenzhiyong.com/%(bucket)  # 设置DNS解析风格Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:  # 文件不加密，直接回车即可 
Path to GPG program [/usr/bin/gpg]:  # 指定自定义的gpg程序路径，直接回车即可When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: No  # 你的rgw是否是https，如果不是设置为NoOn some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:   # 代理服务器的地址，我并没有配置代理服务器，因此直接回车即可New settings:  # 注意，下面的信息是上面咱们填写时一个总的预览信息Access Key: ZHOE7MVPLJFE5EIU738WSecret Key: VUNbdDwAGIq9AZv5f55e2gzptK1PUOnWg9nc44pEDefault Region: USS3 Endpoint: www.wenzhiyong.comDNS-style bucket+hostname:port template for accessing a bucket: www.wenzhiyong.com/%(bucket)Encryption password: Path to GPG program: /usr/bin/gpgUse HTTPS protocol: FalseHTTP Proxy server name: HTTP Proxy server port: 0Test access with supplied credentials? [Y/n] Y  # 如果确认上述信息没问题的话，则输入字母Y即可。
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)Now verifying that encryption works...
Not configured. Never mind.Save settings? [y/N] y  # 是否保存配置，我们输入y，默认是不保存配置的。
Configuration saved to '/root/.s3cfg'
[root@ceph141 ~]# 
[root@ceph141 ~]# 
[root@ceph141 ~]# ll /root/.s3cfg
-rw------- 1 root root 2269 Aug 23 09:59 /root/.s3cfg
[root@ceph141 ~]#

03 创建存储桶bucket

4 创建buckets
[root@ceph141 ~]# s3cmd mb s3://wenzhiyong-bucket
Bucket 's3://wenzhiyong-bucket/' created
[root@ceph141 ~]#

以下命名规则适用于通用存储桶

1存储桶名称必须介于 3（最少）到 63（最多）个字符之间。
2.存储桶名称只能由小写字母、数字、句点（.）和连字符（-）组成。
3.存储桶名称必须以字母或数字开头和结尾。
4.存储桶名称不得包含两个相邻的句点。
5.存储桶名称不得采用 IP 地址格式（例如，192.168.5.4）。
6.存储桶名称不得以前缀 xn-- 开头。
7.存储桶名称不得以前缀 sthree- 开头。
8.存储桶名称不得以前缀 sthree-configurator 开头。
9.存储桶名称不得以前缀 amzn-s3-demo- 开头。
10.存储桶名称不得以后缀 -s3alias 结尾。此后缀是为接入点别名预留的。有关更多信息，请参阅为您的 S3 存储桶接入点使用存储桶式别名。
11.存储桶名称不得以后缀 --ol-s3 结尾。此后缀是为对象 Lambda 接入点别名预留的。有关更多信息，请参阅如何为您的 S3 存储桶对象 Lambda 接入点使用存储桶式别名。
12.存储桶名称不得以后缀 .mrap 结尾。此后缀预留用于多区域接入点名称。有关更多信息，请参阅命名 Amazon S3 多区域接入点的规则。
13.存储桶名称不得以后缀 --x-s3 结尾。此后缀预留用于目录存储桶。有关更多信息，请参阅目录存储桶命名规则。
14.存储桶名称在分区内所有 AWS 区域中的所有 AWS 账户间必须是唯一的。分区是一组区域。AWS 目前有三个分区：aws（标准区域）、aws-cn（中国区域）和 aws-us-gov（AWS GovCloud (US)）。
15.存储桶名称不能被同一分区中的另一个 AWS 账户使用，直到存储桶被删除。
16.与 Amazon S3 Transfer Acceleration 一起使用的存储桶名称中不能有句点（.）。

04 上传视频到bucket

1.查看buckets

	
[root@ceph141 ~]# s3cmd ls
2024-08-23 02:03  s3://wenzhiyong-bucket
[root@ceph141 ~]# 
[root@ceph141 ~]# 
[root@ceph141 ~]# radosgw-admin buckets list

2.上传1个53MB的视频，如果文件体积足够大的话，才会有分片显示。下面的数字加起来正好是53MB

[root@ceph141~]# ll -h 01-MD5+简述.mp4 
-rw-r--r-- 1 root root 53M Apr 13  2024 01-MD5+简述.mp4
[root@ceph141~]# s3cmd put 01-MD5+简述.mp4 s3://wenzhiyong-bucket
upload: '01-MD5+简述.mp4' -> 's3://wenzhiyong-bucket/01-MD5+简述.mp4'  [part 1 of 4, 15MB] [1 of 1]15728640 of 15728640   100% in    0s    40.55 MB/s  done
upload: '01-MD5+简述.mp4' -> 's3://wenzhiyong-bucket/01-MD5+简述.mp4'  [part 2 of 4, 15MB] [1 of 1]15728640 of 15728640   100% in    0s    58.86 MB/s  done
upload: '01-MD5+简述.mp4' -> 's3://wenzhiyong-bucket/01-MD5+简述.mp4'  [part 3 of 4, 15MB] [1 of 1]15728640 of 15728640   100% in    0s    51.43 MB/s  done
upload: '01-MD5+简述.mp4' -> 's3://wenzhiyong-bucket/01-MD5+简述.mp4'  [part 4 of 4, 7MB] [1 of 1]8383328 of 8383328   100% in    0s    34.73 MB/s  done[root@ceph141~]# echo 15728640 + 15728640 + 15728640 + 8383328 | bc
55569248
[root@ceph141~]# echo 55569248 / 1024 /1024 | bc
52

3.使用s3cmd下载数据

[root@ceph141~]# s3cmd get s3://wenzhiyong-bucket/01-MD5+简述.mp4 /tmp/
download: 's3://wenzhiyong-bucket/01-MD5+简述.mp4' -> '/tmp/01-MD5+简述.mp4'  [1 of 1]55569248 of 55569248   100% in    0s   130.01 MB/s  done
[root@ceph141~]# ll /tmp/01-MD5+简述.mp4 
-rw-r--r-- 1 root root 55569248 Nov  7 11:15 /tmp/01-MD5+简述.mp4

05 创建授权策略以访问web页面

要通过web页面访问数据需要授权，否则就是下图这样显示权限拒绝：

在这里插入图片描述

1.创建策略

cat > wenzhiyong-anonymous-access-policy.json <<EOF
{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Principal": {"AWS": ["*"]},"Action": "s3:GetObject","Resource": ["arn:aws:s3:::wenzhiyong-bucket/*"]}]
}
EOF

2.应用策略

[root@ceph141~]# s3cmd setpolicy wenzhiyong-anonymous-access-policy.json s3://wenzhiyong-bucket
s3://wenzhiyong-bucket/: Policy updated

3.再次访问，http://10.0.0.142/wenzhiyong-bucket/01-MD5+简述.mp4。如果只有声音没有画面可能是浏览器问题，建议换成Chrome。

在这里插入图片描述

高可用提示：

于对象存储网关而言，需要将www.wenzhiyong.com解析到ceph141，ceph142,ceph143的任意一个节点上。
生产环境中，建议在rgw设备前加一个负载均衡器，以防止后端rgw宕机的情况，以减少单点故障的问题。

06 策略的删除

策略删除前后对比

[root@ceph141~]# s3cmd info s3://wenzhiyong-bucket
s3://wenzhiyong-bucket/ (bucket):Location:  defaultPayer:     BucketOwnerExpiration Rule: nonePolicy:    {"Version": "2012-10-17","Statement": [{"Effect": "Allow","Principal": {"AWS": ["*"]},"Action": "s3:GetObject","Resource": ["arn:aws:s3:::wenzhiyong-bucket/*"]}]
}CORS:      noneACL:       文治勇: FULL_CONTROL[root@ceph141~]# s3cmd delpolicy s3://wenzhiyong-bucket
s3://wenzhiyong-bucket/: Policy deleted[root@ceph141~]# s3cmd info s3://wenzhiyong-bucket
s3://wenzhiyong-bucket/ (bucket):Location:  defaultPayer:     BucketOwnerExpiration Rule: nonePolicy:    noneCORS:      noneACL:       文治勇: FULL_CONTROL

再次访问web页面就会和之前一样提示权限拒绝

GAoBrduprZSwWqhuUAp4uxJekCEizwZCuG2QnxDB
7845VJXFP14XFBAL648B
wenzhiyong-bucket
nzhiyong-bucket
s3://wenzhiyong-bucket/: Policy deleted[root@ceph141~]# s3cmd info s3://wenzhiyong-bucket
s3://wenzhiyong-bucket/ (bucket):Location:  defaultPayer:     BucketOwnerExpiration Rule: nonePolicy:    noneCORS:      noneACL:       文治勇: FULL_CONTROL

再次访问web页面就会和之前一样提示权限拒绝

GAoBrduprZSwWqhuUAp4uxJekCEizwZCuG2QnxDB
7845VJXFP14XFBAL648B
wenzhiyong-bucket

RGW对象存储网关部署和使用

01 部署rgw组件

02 s3cmd工具安装

03 创建存储桶bucket

04 上传视频到bucket

05 创建授权策略以访问web页面

06 策略的删除

相关文章：