ceph的用户管理和cephx认证

用户权限概述

用户格式

参考链接：

权限：https://docs.ceph.com/en/latest/rados/operations/user-management/#authorization-capabilities
用户：https://docs.ceph.com/en/reef/rados/operations/user-management/

ceph的用户格式TYPEID.USERID

• TYPEID也叫用户类型，有2用户类型；内置组件用户(mon,mds,rgw,osd,mgr)和普通用户(client)

• USERID，就是用户名，可以是数字。

  • 比如表示ods的第0块磁盘，对应的是ods.0
  • 也可以是字符串，比如管理员用户，对应的是client.admin
  • 用户可以自定义USERID，比如client.wzy，client.wenzhiyong

用户权限

每个用户都可以授权，使用caps字段关联。授权的格式allow 权限

• r：读权限

• w: 写权限

• x：执行权限，可以调用方法(这些方法可能存在读写等操作)，还可以执行mon的auth等相关命令

• *：拥有rwx等权限

• profile osd：可以获取OSD的状态信息

• profile mds：可以获取mds的状态信息

举例ceph系统组件的权限就在授权文件中体现：

[root@ceph141~]# cat /etc/ceph/ceph.client.admin.keyring
[client.admin]key = AQAlsChnHubLJRAAH2s3vhyGrxgba8anloPDtg==caps mds = "allow *"caps mgr = "allow *"caps mon = "allow *"caps osd = "allow *"

查看管理员权限

[root@ceph141~]# ceph auth get client.admin
[client.admin]key = AQAlsChnHubLJRAAH2s3vhyGrxgba8anloPDtg==caps mds = "allow *"caps mgr = "allow *"caps mon = "allow *"caps osd = "allow *"

查看其他用户权限，可以发现osd也算用户

[root@ceph141~]# ceph auth list
osd.0key: AQAJ1Chn4kJoMxAAO/sYaCTyTyJE6TSclIxKsA==caps: [mgr] allow profile osdcaps: [mon] allow profile osdcaps: [osd] allow *
osd.1key: AQA21ChniKrACRAANYkBLMXK5BThtHgTrNVqNw==caps: [mgr] allow profile osdcaps: [mon] allow profile osdcaps: [osd] allow *
...
client.adminkey: AQAlsChnHubLJRAAH2s3vhyGrxgba8anloPDtg==caps: [mds] allow *caps: [mgr] allow *caps: [mon] allow *caps: [osd] allow *
client.bootstrap-mdskey: AQAnsChncF9lOxAAGmqKpDlaOTzxCAX20uo6EA==caps: [mon] allow profile bootstrap-mds
client.bootstrap-mgrkey: AQAnsChnx2VlOxAABgp0KiClbDnraMQ6ZGEpBQ==caps: [mon] allow profile bootstrap-mgr
client.bootstrap-osdkey: AQAnsChnxGtlOxAAkCnj4ZlBhzIpr4vk6pcUdA==caps: [mon] allow profile bootstrap-osd
client.bootstrap-rbdkey: AQAnsChnjnFlOxAAQUXJdflbTiKjW/ZbKGgE1w==caps: [mon] allow profile bootstrap-rbd
client.bootstrap-rbd-mirrorkey: AQAnsChni3dlOxAAb6TImPKkGrR1baZO8AdYGg==caps: [mon] allow profile bootstrap-rbd-mirror
client.bootstrap-rgwkey: AQAnsChnm39lOxAAy6Qs5u3d5YidcT6cWaOH6A==caps: [mon] allow profile bootstrap-rgw
client.ceph-exporter.ceph141key: AQBgsChn0hbwGxAA6y6Op/+2zPirhwH4UqV5UQ==caps: [mgr] allow rcaps: [mon] allow rcaps: [osd] allow r
client.ceph-exporter.ceph142key: AQBMzyhnBYIxOxAAF4seBajmPKYWmzuM6XKqqQ==caps: [mgr] allow rcaps: [mon] allow rcaps: [osd] allow r
client.ceph-exporter.ceph143key: AQBjzyhnUbSSGRAAtt4r+evuoNE+ciwx/ymv1A==caps: [mgr] allow rcaps: [mon] allow rcaps: [osd] allow r
client.crash.ceph141key: AQBhsChngfrUIRAA2TjOYgDQQ4NENaU7p3EwHw==caps: [mgr] profile crashcaps: [mon] profile crash
client.crash.ceph142key: AQBPzyhnKwm4ExAAZ/0a6FVAWJFjSbRozum/PA==caps: [mgr] profile crashcaps: [mon] profile crash
client.crash.ceph143key: AQBlzyhn9+GPNBAA3NZddZGiXoyLrf9J9M7wQw==caps: [mgr] profile crashcaps: [mon] profile crash
mgr.ceph141.yvswvfkey: AQAlsChnJpeKMhAAsiyirSCpqTIgh3mB7o4V7g==caps: [mds] allow *caps: [mon] profile mgrcaps: [osd] allow *
mgr.ceph142.gtcikxkey: AQBRzyhnal2kLhAA4DvZbY7TiWIxWSg1Tw3ZQw==caps: [mds] allow *caps: [mon] profile mgrcaps: [osd] allow *

三种方式自定义普通用户

创建用户方式参考链接:：https://docs.ceph.com/en/nautilus/rados/operations/user-management/#add-a-user

1 直接创建

[root@ceph141~]# ceph auth add client.wzy666 mon 'allow r' osd 'allow * pool=zhiyong18-rbd'
added key for client.wzy666

client.wzy666：这是客户端名称，表示要为此客户端添加权限。

mon 'allow r'：为该客户端授予对 monitor（监视器）的读取权限 (r)，意味着该客户端可以查看集群状态、查询信息等。

osd 'allow * pool=zhiyong18-rbd'：为该客户端授予对 OSD（对象存储设备）上名为 zhiyong18-rbd 的池的所有权限。allow * 表示允许所有操作（如读写），但限制在 zhiyong18-rbd 这个特定的池上

验证用户wzy666的权限

[root@ceph141~]# ceph auth get client.wzy666
[client.wzy666]key = AQB+KypnuKsqDhAA1VYxg0qKjp4G3Lr+CUebHA==caps mon = "allow r"caps osd = "allow * pool=zhiyong18-rbd"

2 查看若不存在则创建

1.查看用户是否存在

[root@ceph141~]# ceph auth get client.wenzhiyong
Error ENOENT: failed to find client.wenzhiyong in keyring

2.若用户不存在则创建

[root@ceph141~]# ceph auth get-or-create client.wenzhiyong mon 'allow r' osd 'allow rwx'
[client.wenzhiyong]key = AQBgLypnfvLQBxAApSe9WoyC5ys1mySFPzjTfw==

再次查看用户信息

[root@ceph141~]# ceph auth get client.wenzhiyong
[client.wenzhiyong]key = AQBgLypnfvLQBxAApSe9WoyC5ys1mySFPzjTfw==caps mon = "allow r"caps osd = "allow rwx"

4.如果用户存在，再去创建是会报错的

[root@ceph141~]# ceph auth get-or-create client.wenzhiyong mon 'allow r' osd 'allow *'
Error EINVAL: key for client.wenzhiyong exists but cap osd does not match

5.若用户存在且权限匹配则打印KEY

[root@ceph141~]# ceph auth get-or-create client.wenzhiyong mon 'allow r' osd 'allow rwx'
[client.wenzhiyong]key = AQBgLypnfvLQBxAApSe9WoyC5ys1mySFPzjTfw==

6.查看最终的权限

[root@ceph141~]# ceph auth get client.wenzhiyong
[client.wenzhiyong]key = AQBgLypnfvLQBxAApSe9WoyC5ys1mySFPzjTfw==caps mon = "allow r"caps osd = "allow rwx"

3 查看权限若没有就创建

1.查看用户k8s不存在

[root@ceph141~]# ceph auth get client.k8s
Error ENOENT: failed to find client.k8s in keyring

2.创建用户并返回KEY

ceph auth get-or-create-key client.k8s mon 'allow r' osd 'allow rwx'

再次查看用户信息

[root@ceph141~]# ceph auth get client.k8s
[client.k8s]key = AQCfMCpnrHrtJBAAoLnVptDFXrhIzZKWIp16nw==caps mon = "allow r"caps osd = "allow rwx"

3.若用户存在则且权限不匹配则报错

[root@ceph141~]# ceph auth get-or-create-key client.k8s mon 'allow r' osd 'allow *'
Error EINVAL: key for client.k8s exists but cap osd does not match

若用户存在且权限匹配则打印KEY

[root@ceph141~]# ceph auth get-or-create-key client.k8s mon 'allow r' osd 'allow rwx'
AQCfMCpnrHrtJBAAoLnVptDFXrhIzZKWIp16nw==

ceph auth print-key打印已经存在用户的KEY，如果用户不存在则报错，如果用户存在则打印该用户对应的KEY信息

[root@ceph141~]# ceph auth print-key client.wzy666 | more
AQB+KypnuKsqDhAA1VYxg0qKjp4G3Lr+CUebHA==
[root@ceph141~]# ceph auth get client.wzy666
[client.wzy666]key = AQB+KypnuKsqDhAA1VYxg0qKjp4G3Lr+CUebHA==caps mon = "allow r"caps osd = "allow * pool=zhiyong18-rbd"

用户权限修改

修改权限参考链接：https://docs.ceph.com/en/nautilus/rados/operations/user-management/#modify-user-capabilities

1.查看权限后，进行修改

[root@ceph141~]# ceph auth get client.wzy666
[client.wzy666]key = AQB+KypnuKsqDhAA1VYxg0qKjp4G3Lr+CUebHA==caps mon = "allow r"caps osd = "allow * pool=zhiyong18-rbd"
[root@ceph141~]# ceph auth caps client.wzy666 mon 'allow rx' osd 'allow r pool=wenzhiyong18-rbd'
updated caps for client.wzy666

2.查看修改权后的auth

[root@ceph141~]# ceph auth get client.wzy666
[client.wzy666]key = AQB+KypnuKsqDhAA1VYxg0qKjp4G3Lr+CUebHA==caps mon = "allow rx"caps osd = "allow r pool=wenzhiyong18-rbd"

用户的删除

用户删除参考链接：https://docs.ceph.com/en/nautilus/rados/operations/user-management/#delete-a-user

1.直接删除用户wzy666

[root@ceph141~]# ceph auth get client.wzy666
[client.wzy666]key = AQB+KypnuKsqDhAA1VYxg0qKjp4G3Lr+CUebHA==caps mon = "allow rx"caps osd = "allow r pool=wenzhiyong18-rbd"[root@ceph141~]# ceph auth del client.wzy666

ceph用户的备份和恢复

用户数据备份

参考链接：

https://docs.ceph.com/en/nautilus/rados/operations/user-management/#get-a-user
https://docs.ceph.com/en/nautilus/rados/operations/user-management/#import-a-user-s

1.创建测试用户

[root@ceph141~]# ceph auth add client.wzy666 mon 'allow r' osd 'allow * pool=zhiyong18-rbd'
added key for client.wzy666
[root@ceph141~]# ceph auth get client.wzy666
[client.wzy666]key = AQB2NipneGZcCBAAqL6zGHCpU2uwM15R05uHzQ==caps mon = "allow r"caps osd = "allow * pool=zhiyong18-rbd"

2.导出用户到文件，用于模拟备份。这一步只是创建文件并不会写入

[root@ceph141~]# ceph-authtool --create-keyring ceph.client.wzy666.keyring
creating ceph.client.wzy666.keyring
[root@ceph141~]# ls
ceph.client.wzy666.keyring
[root@ceph141~]# cat ceph.client.wzy666.keyring 
[root@ceph141~]# 

3.将内容导出到指定文件

[root@ceph141~]# ceph auth get client.wzy666 -o ceph.client.wzy666.keyring

4.查看文件内容

[root@ceph141~]# cat ceph.client.wzy666.keyring 
[client.wzy666]key = AQB2NipneGZcCBAAqL6zGHCpU2uwM15R05uHzQ==caps mon = "allow r"caps osd = "allow * pool=zhiyong18-rbd"

总结：不如ceph auth get client.wzy666 > ceph.client.wzy666.keyring

用户数据导入

1.删除用户

ceph auth del client.wzy666

2.导入用户文件信息

[root@ceph141~]# ceph auth import -i ceph.client.wzy666.keyring 

3.验证用户信息完整性

[root@ceph141~]# ceph auth get client.wzy666
[client.wzy666]key = AQB2NipneGZcCBAAqL6zGHCpU2uwM15R05uHzQ==caps mon = "allow r"caps osd = "allow * pool=zhiyong18-rbd"

导出授权文件并验证用户权限

1.ceph141节点创建1个普通用户并保存到一个文件中

[root@ceph141~]# ceph auth get-or-create client.k3s mon 'allow r' osd 'allow * pool=zhiyong18-rdb'
[client.k3s]key = AQCzRSpn1SShChAAPmJUYIvCKsuAH47HDNWD0A==[root@ceph141~]# ceph auth export client.k3s -o ceph.client.k3s.keyring
[root@ceph141~]# cat ceph.client.k3s.keyring
[client.k3s]key = AQCzRSpn1SShChAAPmJUYIvCKsuAH47HDNWD0A==caps mon = "allow r"caps osd = "allow * pool=zhiyong18-rdb"

2.ceph142节点删除原来的管理员授权文件，再次访问权限报错

[root@ceph142~]# rm -f  /etc/ceph/ceph.client.admin.keyring
[root@ceph142~]# ceph -s
2024-11-05T23:38:38.932+0800 7f7fe4d69640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2024-11-05T23:38:38.932+0800 7f7fe4d69640 -1 AuthRegistry(0x7f7fe00672a0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx
2024-11-05T23:38:38.936+0800 7f7fe4d69640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2024-11-05T23:38:38.936+0800 7f7fe4d69640 -1 AuthRegistry(0x7f7fe4d67f60) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx
2024-11-05T23:38:38.936+0800 7f7fde59c640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2024-11-05T23:38:38.936+0800 7f7fded9d640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2024-11-05T23:38:38.936+0800 7f7fddd9b640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
2024-11-05T23:38:38.936+0800 7f7fe4d69640 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication
[errno 13] RADOS permission denied (error connecting to the cluster)

3.服务端将认证文件拷贝到客户端

[root@ceph141~]# scp ceph.client.k3s.keyring ceph142:/etc/ceph/

4.客户端验证权限

[root@ceph142~]# ceph -s --user k3scluster:id:     12fad866-9aa0-11ef-8656-6516a17ad6ddhealth: HEALTH_WARN
...[root@ceph142~]# cat /etc/ceph/ceph.client.k3s.keyring
[client.k3s]key = AQCzRSpn1SShChAAPmJUYIvCKsuAH47HDNWD0A==caps mon = "allow r"caps osd = "allow * pool=zhiyong18-rdb"

[root@ceph142~]# ceph --user k3s auth get client.k3s
Error EACCES: access denied

这是因为对用户相关的操作还没有执行权限，不能调用相关函数。后期添加上去就可以了

5.服务端尝试修改k3s用户权限

[root@ceph141~]# ceph auth caps client.k3s mon 'allow rx' 
updated caps for client.k3

6.客户端再次验证权限。虽然客户端可以查看用户信息了，但是此时/etc/ceph/ceph.client.k3s.keyring是没有任何变化的；也就是说：本地的keyring文件的caps字段并没有作用，而是基于KEY访问集群进行验证的！

[root@ceph142~]# ceph --user k3s auth get client.k3s
[client.k3s]key = AQCzRSpn1SShChAAPmJUYIvCKsuAH47HDNWD0A==caps mon = "allow rx"[root@ceph142~]# cat /etc/ceph/ceph.client.k3s.keyring 
[client.k3s]key = AQCzRSpn1SShChAAPmJUYIvCKsuAH47HDNWD0A==caps mon = "allow r"caps osd = "allow * pool=zhiyong18-rdb"

7.进一步验证k3s用户的权限，可以查看池列表

[root@ceph142~]# ceph --user k3s osd pool ls
.mgr
zhiyong-rbd
zhiyong18-rbd
zhiyong

但是没有权限访问存储池下的镜像文件

[root@ceph142~]# rbd --id k3s -p zhiyong ls -l
2024-11-05T23:47:24.820+0800 7f8de091de00 -1 librbd::api::Image: list_images: error listing v1 images: (1) Operation not permitted
rbd: listing images failed: (1) Operation not permitted
[root@ceph142~]# rbd --id k3s -p zhiyong18-rbd ls -l
2024-11-05T23:48:00.588+0800 7f38f923ce00 -1 librbd::api::Image: list_images: error listing v1 images: (1) Operation not permitted
rbd: listing images failed: (1) Operation not permitted

8.服务端再次修改权限

[root@ceph141~]# ceph auth get client.k3s
[client.k3s]key = AQCzRSpn1SShChAAPmJUYIvCKsuAH47HDNWD0A==caps mon = "allow rx"
[root@ceph141~]# ceph auth caps client.k3s  mon 'allow *'  osd 'allow *'
updated caps for client.k3s

10.客户端再次验证权限

[root@ceph142~]# rbd --id k3s -p zhiyong18-rbd ls -l
NAME        SIZE   PARENT  FMT  PROT  LOCK
mysqld      5 GiB            2            
rbd-snap    2 GiB            2            
wordpress   2 GiB            2            
zhiyong     5 GiB            2            
zhiyong@v1  5 GiB            2            
zhiyong@v2  5 GiB            2            
zhiyong@v3  5 GiB            2            
zhiyong@v4  5 GiB            2            
zhiyong@v5  5 GiB            2            
zhiyong@v6  5 GiB            2

用户授权总结

1.如果使用"–user k3s"指定用户，则默认去找以下文件，找不到就报错:

• /etc/ceph/ceph.client.k3s.keyring
• /etc/ceph/ceph.keyring
• /etc/ceph/keyring
• /etc/ceph/keyring.bin

2.如果不使用"–user"选项，咱们可以立即为默认为"–user amdin"

• /etc/ceph/ceph.client.admin.keyring
• /etc/ceph/ceph.keyring
• /etc/ceph/keyring
• /etc/ceph/keyring.bin

3.对于认证文件不能随便起名字，而是需要遵循上述2条的规范文件命名，否则ceph不识别用户的配置文件

4 客户端在连接ceph集群时，仅需要读取keyring文件中的KEY值；其他caps字段会被忽视。也就是说，对于文件中只要保留key值依旧是有效的

cephx认证

01 cephx认证概述

参考链接：

https://docs.ceph.com/en/nautilus/rados/configuration/auth-config-ref/
https://docs.ceph.com/en/nautilus/rados/operations/operating/
https://docs.ceph.com/en/nautilus/architecture/#high-availability-authentication

• 为了识别用户并防止中间人攻击，Ceph提供了cephx身份验证系统来验证用户和守护进程。但是注意cephx协议不解决传输中的数据加密（例如SSL/TLS）或静止时的加密问题

• 不建议关闭cephx认证，因为没有认证则集群任意节点都可以直接操作，除非内环环境相对安全

02 cephx相关参数说明

• auth_cluster_required
  • 如果启用，Ceph存储群集守护进程（即Ceph-mon、Ceph-osd、Ceph-mds和Ceph-mgr）必须相互进行身份验证
  • 有效设置为cephx或none，默认值为cephx
• auth_service_required
  • 如果启用，则Ceph存储群集守护进程要求Ceph客户端向Ceph存储集群进行身份验证，以便访问Ceph服务
  • 有效设置为cephx或none，默认值为cephx
• 有效设置为cephx或none，默认值为cephx
  • 如果启用，Ceph客户端需要Ceph存储群集向Ceph客户端进行身份验证
  • 有效设置为cephx或none，默认值为cephx

03 cephx启动和关闭

1.找到mon组件的容器

[root@ceph141~]# docker ps -a | grep mon
aa345967806c   quay.io/ceph/ceph:v18                     "/usr/bin/ceph-mon -…"

2.进入容器，再关闭认证：在/etc/ceph/ceph.conf增加以下参数，修改后需重启集群

auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

关闭认证：在vim /etc/ceph/ceph.conf改为以下参数

auth_cluster_required = none
auth_service_required = none
auth_client_required = none- 有效设置为cephx或none，默认值为cephx
- 有效设置为cephx或none，默认值为cephx- 如果启用，Ceph客户端需要Ceph存储群集向Ceph客户端进行身份验证- 有效设置为cephx或none，默认值为cephx## 03 cephx启动和关闭1.找到mon组件的容器```bash
[root@ceph141~]# docker ps -a | grep mon
aa345967806c   quay.io/ceph/ceph:v18                     "/usr/bin/ceph-mon -…"

2.进入容器，再关闭认证：在/etc/ceph/ceph.conf增加以下参数，修改后需重启集群

auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

关闭认证：在vim /etc/ceph/ceph.conf改为以下参数

auth_cluster_required = none
auth_service_required = none
auth_client_required = none