Ceph使用---Crush Map进阶
一、Ceph Crush Map介绍
ceph 集群中由 mon 服务器维护的的五种运行图：
Monitor map #监视器运行图
OSD map #OSD 运行图
PG map #PG 运行图
Crush map #(Controllers replication under scalable hashing #可控的、 可复制的、 可伸缩的一致性 hash 算法。
MDS map #cephfs metadata 运行图
crush 运行图,当新建存储池时会基于 OSD map 创建出新的 PG 组合列表用于存储数据
crush 算法针对目的节点的选择：
目前有 5 种算法来实现节点的选择， 包括 Uniform、 List、 Tree、 Straw、 Straw2， 早期版本使用的是 ceph 项目的发起者发明的算法 straw， 目前已经发展社区优化的 straw2 版本。
straw(抽签算法)：
抽签是指挑取一个最长的签， 而这个签值就是 OSD 的权重， 当创建存储池的时候会向 PG分配 OSD， straw 算法会遍历当前可用的 OSD 并优先使用中签的 OSD， 以让权重高的 OSD被分配较多的 PG 以存储更多
的数据。
二、PG 与 OSD 映射调整
默认情况下,crush 算法自行对创建的 pool 中的 PG 分配 OSD， 但是可以手动基于权重设置crush 算法分配数据的倾向性， 比如 1T 的磁盘权重是 1， 2T 的就是 2， 推荐使用相同大小的设备。
查看当前状态：
weight 表示设备(device)的容量相对值， 比如 1TB 对应 1.00， 那么 500G 的 OSD 的 weight就应该是 0.5， weight 是基于磁盘空间分配 PG 的数量， 让 crush 算法尽可能往磁盘空间大的 OSD 多分
配 PG.往磁盘空间小的 OSD 分配较少的 PG。 Reweight 参数的目的是重新平衡 ceph 的 CRUSH 算法随机分配的 PG， 默认的分配是概率上的均衡， 即使 OSD 都是一样的磁盘空间也会产生一些 PG 分布不均匀的情况， 此时可以通过调整 reweight 参数，
让 ceph 集群立即重新平衡当前磁盘的 PG， 以达到数据均衡分布的目的， REWEIGHT 是 PG 已经分配完成， 要在 ceph 集群重新平衡 PG 的分布。
修改 WEIGHT 并验证:
[root@ceph-deploy ~]# ceph osd crush reweight osd.1 1.5 #此操作属于高危操作，会立即触发ceph集群数据重平衡，谨慎操作！！！！ reweighted item id 1 name 'osd.1' to 1.5 in crush map [root@ceph-deploy ~]# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0.09769 1.00000 100 GiB 770 MiB 719 MiB 20 KiB 51 MiB 99 GiB 0.75 1.14 133 up 1 hdd 1.50000 1.00000 100 GiB 695 MiB 627 MiB 7 KiB 68 MiB 99 GiB 0.68 1.03 147 up 2 hdd 0.09769 1.00000 100 GiB 555 MiB 511 MiB 6 KiB 44 MiB 99 GiB 0.54 0.82 118 up 3 hdd 0.09769 1.00000 100 GiB 679 MiB 625 MiB 12 KiB 53 MiB 99 GiB 0.66 1.00 121 up 4 hdd 0.09769 1.00000 100 GiB 652 MiB 595 MiB 13 KiB 58 MiB 99 GiB 0.64 0.97 132 up 5 hdd 0.09769 1.00000 100 GiB 688 MiB 636 MiB 8 KiB 52 MiB 99 GiB 0.67 1.02 147 up 6 hdd 0.09769 1.00000 100 GiB 638 MiB 571 MiB 7 KiB 67 MiB 99 GiB 0.62 0.94 125 up 7 hdd 0.09769 1.00000 100 GiB 669 MiB 611 MiB 13 KiB 58 MiB 99 GiB 0.65 0.99 141 up 8 hdd 0.09769 1.00000 100 GiB 734 MiB 674 MiB 13 KiB 60 MiB 99 GiB 0.72 1.09 135 up TOTAL 900 GiB 5.9 GiB 5.4 GiB 104 KiB 510 MiB 894 GiB 0.66 MIN/MAX VAR: 0.82/1.14 STDDEV: 0.06 [root@ceph-deploy ~]# ceph -s cluster: id: 8dc32c41-121c-49df-9554-dfb7deb8c975 health: HEALTH_WARN Degraded data redundancy: 1484/2529 objects degraded (58.679%), 151 pgs degraded #此操作属于高危操作，会立即触发ceph集群数据重平衡，谨慎操作！！！！ services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 21h) mgr: ceph-mgr1(active, since 21h), standbys: ceph-mgr2 mds: 2/2 daemons up, 2 standby osd: 9 osds: 9 up (since 21h), 9 in (since 6d); 58 remapped pgs rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 12 pools, 401 pgs objects: 843 objects, 1.8 GiB usage: 6.0 GiB used, 894 GiB / 900 GiB avail pgs: 1484/2529 objects degraded (58.679%) 571/2529 objects misplaced (22.578%) 211 active+clean 127 active+recovery_wait+degraded 31 active+remapped+backfill_wait 24 active+recovery_wait+undersized+degraded+remapped 4 active+recovery_wait 3 active+recovery_wait+remapped 1 active+recovering io: client: 511 KiB/s wr, 0 op/s rd, 0 op/s wr recovery: 4.5 MiB/s, 1 keys/s, 3 objects/s #数据量过大，此处值阈值不放开，会造成osd拥塞，最终集群无法正常提供服务 [root@ceph-deploy ~]#
[root@ceph-deploy ~]# ceph health detail
HEALTH_WARN Degraded data redundancy: 1143/2529 objects degraded (45.196%), 111 pgs degraded [WRN] PG_DEGRADED: Degraded data redundancy: 1143/2529 objects degraded (45.196%), 111 pgs degraded pg 2.0 is active+recovery_wait+degraded, acting [3,1,8] pg 2.1 is active+recovery_wait+degraded, acting [1,3,8] pg 2.2 is active+recovery_wait+degraded, acting [1,5,6] pg 2.11 is active+recovery_wait+degraded, acting [1,3,8] pg 2.13 is active+recovery_wait+degraded, acting [1,4,6] pg 2.15 is active+recovery_wait+degraded, acting [1,8,5] pg 2.16 is active+recovery_wait+degraded, acting [1,7,4] pg 2.18 is active+recovery_wait+degraded, acting [1,6,3] pg 2.19 is active+recovery_wait+degraded, acting [1,7,3] pg 2.1a is active+recovery_wait+degraded, acting [1,4,7] pg 2.1b is active+recovery_wait+degraded, acting [6,1,5] pg 2.39 is active+recovery_wait+degraded, acting [7,5,1] pg 2.3d is active+recovery_wait+degraded, acting [1,7,4] pg 2.3e is active+recovery_wait+degraded, acting [7,1,4] pg 3.f is active+recovery_wait+degraded, acting [1,8,4] pg 6.5 is active+recovery_wait+degraded, acting [0,4,8] pg 6.6 is active+recovery_wait+degraded, acting [1,3,8] pg 6.7 is active+recovery_wait+degraded, acting [1,6,3] pg 8.4 is active+recovery_wait+degraded, acting [1,5,7] pg 8.5 is active+recovery_wait+degraded, acting [6,2,5] pg 8.6 is active+recovery_wait+degraded, acting [6,0,3] pg 8.7 is active+recovery_wait+degraded, acting [1,6,3] pg 8.8 is active+recovery_wait+degraded, acting [1,7,5] pg 8.9 is active+recovery_wait+degraded, acting [1,4,7] pg 8.a is active+recovery_wait+degraded, acting [1,7,3] pg 8.10 is active+recovery_wait+degraded, acting [3,7,1] pg 8.13 is active+recovery_wait+degraded, acting [1,7,5] pg 8.18 is active+recovery_wait+degraded, acting [1,5,6] pg 8.1a is active+recovery_wait+degraded, acting [1,3,7] pg 8.1b is active+recovery_wait+degraded, acting [1,5,8] pg 8.1c is active+recovery_wait+degraded, acting [1,3,7] pg 8.1d is active+recovery_wait+degraded, acting [1,4,8] pg 8.1f is active+recovery_wait+degraded, acting [7,1,3] pg 8.37 is active+recovery_wait+degraded, acting [1,3,6] pg 13.0 is active+recovery_wait+undersized+degraded+remapped, acting [7,4] pg 13.2 is active+recovery_wait+undersized+degraded+remapped, acting [5,6] pg 13.3 is active+recovery_wait+undersized+degraded+remapped, acting [0,4] pg 13.16 is active+recovery_wait+undersized+degraded+remapped, acting [8,3] pg 13.18 is active+recovery_wait+undersized+degraded+remapped, acting [5,8] pg 13.1c is active+recovery_wait+undersized+degraded+remapped, acting [5,1] pg 13.1d is active+recovery_wait+undersized+degraded+remapped, acting [3,7] pg 13.1e is active+recovery_wait+undersized+degraded+remapped, acting [1,5] pg 13.1f is active+recovery_wait+undersized+degraded+remapped, acting [0,5] pg 15.1 is active+recovery_wait+degraded, acting [1,3,7] pg 15.3 is active+recovery_wait+degraded, acting [1,6,4] pg 15.c is active+recovery_wait+degraded, acting [7,1,4] pg 15.e is active+recovery_wait+degraded, acting [1,6,4] pg 15.15 is active+recovery_wait+degraded, acting [5,7,1] pg 15.16 is active+recovery_wait+degraded, acting [1,4,8] pg 15.1d is active+recovery_wait+degraded, acting [1,7,3] pg 15.1f is active+recovery_wait+degraded, acting [1,4,8]
修改 REWEIGHT 并验证：
OSD 的 REWEIGHT 的值默认为 1， 值可以调整， 范围在 0~1 之间， 值越低 PG 越小， 如果调整了任何一个 OSD 的 REWEIGHT 值， 那么 OSD 的 PG 会立即和其它 OSD 进行重新平衡， 即数据的重新分配， 用于当某个 OSD 的 PG 相对较多需要降低其 PG 数量的场景。
[root@ceph-deploy ~]# ceph osd reweight 2 0.6 reweighted osd.2 to 0.6 (9999) [root@ceph-deploy ~]# [root@ceph-deploy ~]# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0.09769 1.00000 100 GiB 828 MiB 771 MiB 20 KiB 57 MiB 99 GiB 0.81 1.22 163 up 1 hdd 0.09769 1.00000 100 GiB 772 MiB 677 MiB 7 KiB 96 MiB 99 GiB 0.75 1.13 153 up 2 hdd 0.09769 0.59999 100 GiB 454 MiB 400 MiB 6 KiB 53 MiB 100 GiB 0.44 0.67 85 up 3 hdd 0.09769 1.00000 100 GiB 684 MiB 623 MiB 12 KiB 61 MiB 99 GiB 0.67 1.00 127 up 4 hdd 0.09769 1.00000 100 GiB 661 MiB 592 MiB 13 KiB 69 MiB 99 GiB 0.65 0.97 135 up 5 hdd 0.09769 1.00000 100 GiB 696 MiB 633 MiB 8 KiB 62 MiB 99 GiB 0.68 1.02 139 up 6 hdd 0.09769 1.00000 100 GiB 662 MiB 580 MiB 7 KiB 82 MiB 99 GiB 0.65 0.97 134 up 7 hdd 0.09769 1.00000 100 GiB 662 MiB 608 MiB 13 KiB 54 MiB 99 GiB 0.65 0.97 134 up 8 hdd 0.09769 1.00000 100 GiB 713 MiB 659 MiB 13 KiB 54 MiB 99 GiB 0.70 1.05 133 up TOTAL 900 GiB 6.0 GiB 5.4 GiB 104 KiB 588 MiB 894 GiB 0.67 MIN/MAX VAR: 0.67/1.22 STDDEV: 0.08 [root@ceph-deploy ~]#
三、crush 运行图管理
通过工具将 ceph 的 crush 运行图导出并进行编辑， 然后导入
导出 crush 运行图：
注： 导出的 crush 运行图为二进制格式， 无法通过文本编辑器直接打开， 需要使用 crushtool工具转换为文本格式后才能通过 vim 等文本编辑宫工具打开和编辑。
[root@ceph-deploy ~]# mkdir /data/ceph -p [root@ceph-deploy ~]# ceph osd getcrushmap -o /data/ceph/crushmap-v1 32 [root@ceph-deploy ~]# crushtool -d /data/ceph/crushmap-v1 > /data/ceph/crushmap-v1.txt #二进制转文本文件 [root@ceph-deploy ~]# file /data/ceph/crushmap-v1.txt /data/ceph/crushmap-v1.txt: ASCII text [root@ceph-deploy ~]#
[root@ceph-deploy ~]# cat /data/ceph/crushmap-v1.txt
# begin crush map #可调整的 crush map 参数 tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices #当前的设备列表 device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd # types #当前支持的 bucket 类型 type 0 osd #osd 守护进程， 对应到一个磁盘设备 type 1 host #一个主机 type 2 chassis #刀片服务器的机箱 type 3 rack #包含若干个服务器的机柜/机架 type 4 row #包含若干个机柜的一排机柜(一行机柜) type 5 pdu #机柜的接入电源插座 type 6 pod #一个机房中的若干个小房间 type 7 room #包含若干机柜的房间， 一个数据中心有好多这样的房间组成 type 8 datacenter #一个数据中心或 IDC type 9 zone #区域， 地区 type 10 region #可用域， 比如 AWS 宁夏中卫数据中心 type 11 root #bucket 分层的最顶部， 根 # buckets host ceph-node1 { #类型 Host 名称为 ceph-node1 id -3 # do not change unnecessarily #ceph 生成的 OSD ID， 非必要不要改 id -4 class hdd # do not change unnecessarily # weight 0.293 #crush 算法， 管理 OSD 角色 alg straw2 hash 0 # rjenkins1 # rjenkins1 #使用是哪个 hash 算法， 0 表示选择 rjenkins1 这种 hash算法 item osd.0 weight 0.098 #osd0 权重比例， crush 会自动根据磁盘空间计算， 不同的磁盘空间的权重不一样 item osd.1 weight 0.098 item osd.2 weight 0.098 } host ceph-node2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily # weight 0.293 alg straw2 hash 0 # rjenkins1 item osd.3 weight 0.098 item osd.4 weight 0.098 item osd.5 weight 0.098 } host ceph-node3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily # weight 0.293 alg straw2 hash 0 # rjenkins1 item osd.6 weight 0.098 item osd.7 weight 0.098 item osd.8 weight 0.098 } root default { #根的配置 id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily # weight 0.879 alg straw2 hash 0 # rjenkins1 item ceph-node1 weight 0.293 item ceph-node2 weight 0.293 item ceph-node3 weight 0.293 } # rules rule replicated_rule { #副本池的默认配置 id 0 type replicated min_size 1 max_size 10 #默认最大副本为 10 step take default #基于 default 定义的主机分配 OSD step chooseleaf firstn 0 type host #选择主机， 故障域类型为主机 step emit #弹出配置即返回给客户端 } rule erasure-code { #纠删码池的默认配置 id 1 type erasure min_size 3 max_size 4 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } # end crush map
将文本转换为 crush 格式
root@ceph-deploy:~# crushtool -c /data/ceph/crushmap-v1.txt -o /data/ceph/crushmap-v2
导入新的 crush：
导入的运行图会立即覆盖原有的运行图并立即生效。
root@ceph-deploy:~# ceph osd setcrushmap -i /data/ceph/crushmap-v2
验证 crush 运行图是否生效：
[root@ceph-deploy ~]# ceph osd crush rule dump
[ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 8, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "erasure-code", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 4, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ]
四、crush 数据分类管理
Ceph crush 算法分配的 PG 的时候可以将 PG 分配到不同主机的 OSD 上， 以实现以主机为单位的高可用， 这也是默认机制， 但是无法保证不同 PG 位于不同机柜或者机房的主机， 如果要实现基于机柜或者是更高级的 IDC 等方式的数据高可用， 而且也不能实现 A 项目的数据在 SSD， B 项目的数据在机械盘,如果想要实现此功能则需要导出 crush 运行图并手动编辑， 之后再导入并覆盖原有的 crush 运行图。
五、集群中ssh、hdd分类管理
模拟集群内，物理机有ssd、hdd两种不同类型磁盘，然后进行分类管理
[root@ceph-deploy ~]# ceph osd getcrushmap -o /data/ceph/crushmap-v3
[root@ceph-deploy ~]# crushtool -d /data/ceph/crushmap-v3 > /data/ceph/crushmap-v3.txt
[root@ceph-deploy ~]# vi /data/ceph/crushmap-v3.txt
# begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 zone type 10 region type 11 root # buckets host ceph-node1 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 0.293 alg straw2 hash 0 # rjenkins1 item osd.0 weight 0.098 item osd.1 weight 0.098 } host ceph-node2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily # weight 0.293 alg straw2 hash 0 # rjenkins1 item osd.3 weight 0.098 item osd.4 weight 0.098 } host ceph-node3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily # weight 0.293 alg straw2 hash 0 # rjenkins1 item osd.6 weight 0.098 item osd.7 weight 0.098 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily # weight 0.879 alg straw2 hash 0 # rjenkins1 item ceph-node1 weight 0.293 item ceph-node2 weight 0.293 item ceph-node3 weight 0.293 } #ssd node host ceph-ssd-node1 { id -13 # do not change unnecessarily id -14 class ssd # do not change unnecessarily # weight 0.293 alg straw2 hash 0 # rjenkins1 item osd.2 weight 0.098 } host ceph-ssd-node2 { id -15 # do not change unnecessarily id -16 class ssd # do not change unnecessarily # weight 0.293 alg straw2 hash 0 # rjenkins1 item osd.5 weight 0.098 } host ceph-ssd-node3 { id -17 # do not change unnecessarily id -18 class ssd # do not change unnecessarily # weight 0.293 alg straw2 hash 0 # rjenkins1 item osd.8 weight 0.098 } root ssd { id -11 # do not change unnecessarily id -12 class hdd # do not change unnecessarily #真实环境这里需要写真是ssd类型 # weight 0.879 alg straw2 hash 0 # rjenkins1 item ceph-ssd-node1 weight 0.293 item ceph-ssd-node2 weight 0.293 item ceph-ssd-node3 weight 0.293 } # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } rule ssd_rule { id 10 type replicated min_size 1 max_size 5 step take ssd step chooseleaf firstn 0 type host step emit } rule erasure-code { id 1 type erasure min_size 3 max_size 4 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } # end crush map [root@ceph-deploy ~]#
将v3文本文件编译成v4二进制文件导入
[root@ceph-deploy ~]# crushtool -c /data/ceph/crushmap-v3.txt -o /data/ceph/crushmap-v4 [root@ceph-deploy ~]# ceph osd setcrushmap -i /data/ceph/crushmap-v4 33 [root@ceph-deploy ~]#
[root@ceph-deploy ~]# ceph osd crush rule dump
[ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "erasure-code", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 4, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 10, "rule_name": "ssd_rule", "ruleset": 10, "type": 1, "min_size": 1, "max_size": 5, "steps": [ { "op": "take", "item": -11, "item_name": "ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ]
查看集群状态
[root@ceph-deploy ~]# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2 hdd 0.09799 1.00000 100 GiB 63 MiB 10 MiB 6 KiB 53 MiB 100 GiB 0.06 0.09 0 up #分类后的osd pgs会自动移除 5 hdd 0.09799 1.00000 100 GiB 87 MiB 10 MiB 8 KiB 77 MiB 100 GiB 0.09 0.13 0 up 8 hdd 0.09799 1.00000 100 GiB 78 MiB 10 MiB 13 KiB 68 MiB 100 GiB 0.08 0.11 0 up 0 hdd 0.09799 1.00000 100 GiB 989 MiB 911 MiB 20 KiB 78 MiB 99 GiB 0.97 1.42 197 up 1 hdd 0.09799 1.00000 100 GiB 1.0 GiB 932 MiB 7 KiB 111 MiB 99 GiB 1.02 1.50 204 up 3 hdd 0.09799 1.00000 100 GiB 881 MiB 816 MiB 12 KiB 65 MiB 99 GiB 0.86 1.27 200 up 4 hdd 0.09799 1.00000 100 GiB 1.1 GiB 1.0 GiB 13 KiB 78 MiB 99 GiB 1.08 1.59 201 up 6 hdd 0.09799 1.00000 100 GiB 1.0 GiB 967 MiB 7 KiB 91 MiB 99 GiB 1.03 1.52 197 up 7 hdd 0.09799 1.00000 100 GiB 949 MiB 876 MiB 13 KiB 73 MiB 99 GiB 0.93 1.37 204 up TOTAL 900 GiB 6.1 GiB 5.4 GiB 104 KiB 694 MiB 894 GiB 0.68 MIN/MAX VAR: 0.09/1.59 STDDEV: 0.43 [root@ceph-deploy ~]# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -11 0.87900 - 300 GiB 228 MiB 30 MiB 27 KiB 198 MiB 300 GiB 0.07 0.11 - root ssd -13 0.29300 - 100 GiB 63 MiB 10 MiB 6 KiB 53 MiB 100 GiB 0.06 0.09 - host ceph-ssd-node1 2 hdd 0.09799 1.00000 100 GiB 63 MiB 10 MiB 6 KiB 53 MiB 100 GiB 0.06 0.09 0 up osd.2 -15 0.29300 - 100 GiB 87 MiB 10 MiB 8 KiB 77 MiB 100 GiB 0.09 0.13 - host ceph-ssd-node2 5 hdd 0.09799 1.00000 100 GiB 87 MiB 10 MiB 8 KiB 77 MiB 100 GiB 0.09 0.13 0 up osd.5 -17 0.29300 - 100 GiB 78 MiB 10 MiB 13 KiB 68 MiB 100 GiB 0.08 0.11 - host ceph-ssd-node3 8 hdd 0.09799 1.00000 100 GiB 78 MiB 10 MiB 13 KiB 68 MiB 100 GiB 0.08 0.11 0 up osd.8 -1 0.87900 - 600 GiB 5.9 GiB 5.4 GiB 72 KiB 500 MiB 594 GiB 0.98 1.45 - root default -3 0.29300 - 200 GiB 2.0 GiB 1.8 GiB 27 KiB 189 MiB 198 GiB 0.99 1.46 - host ceph-node1 0 hdd 0.09799 1.00000 100 GiB 990 MiB 911 MiB 20 KiB 78 MiB 99 GiB 0.97 1.42 197 up osd.0 1 hdd 0.09799 1.00000 100 GiB 1.0 GiB 932 MiB 7 KiB 111 MiB 99 GiB 1.02 1.50 204 up osd.1 -5 0.29300 - 200 GiB 1.9 GiB 1.8 GiB 25 KiB 147 MiB 198 GiB 0.97 1.43 - host ceph-node2 3 hdd 0.09799 1.00000 100 GiB 885 MiB 816 MiB 12 KiB 69 MiB 99 GiB 0.86 1.27 200 up osd.3 4 hdd 0.09799 1.00000 100 GiB 1.1 GiB 1.0 GiB 13 KiB 78 MiB 99 GiB 1.08 1.59 201 up osd.4 -7 0.29300 - 200 GiB 2.0 GiB 1.8 GiB 20 KiB 163 MiB 198 GiB 0.98 1.44 - host ceph-node3 6 hdd 0.09799 1.00000 100 GiB 1.0 GiB 967 MiB 7 KiB 91 MiB 99 GiB 1.03 1.52 197 up osd.6 7 hdd 0.09799 1.00000 100 GiB 949 MiB 876 MiB 13 KiB 73 MiB 99 GiB 0.93 1.36 204 up osd.7 TOTAL 900 GiB 6.1 GiB 5.4 GiB 104 KiB 698 MiB 894 GiB 0.68 MIN/MAX VAR: 0.09/1.59 STDDEV: 0.43 [root@ceph-deploy ~]# ceph -s cluster: id: 8dc32c41-121c-49df-9554-dfb7deb8c975 health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 23h) mgr: ceph-mgr1(active, since 23h), standbys: ceph-mgr2 mds: 2/2 daemons up, 2 standby osd: 9 osds: 9 up (since 23h), 9 in (since 7d) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 12 pools, 401 pgs objects: 843 objects, 1.8 GiB usage: 6.1 GiB used, 894 GiB / 900 GiB avail pgs: 401 active+clean [root@ceph-deploy ~]#
再次导出验证crush map文件（一般主体文件不写错，次要配置ceph会自动帮忙修正）
[root@ceph-deploy ~]# ceph osd getcrushmap -o /data/ceph/crushmap-v5 33 [root@ceph-deploy ~]# crushtool -d /data/ceph/crushmap-v5 > /data/ceph/crushmap-v5.txt [root@ceph-deploy ~]#
[root@ceph-deploy ceph]# cat crushmap-v5.txt
# begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 zone type 10 region type 11 root # buckets host ceph-node1 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily id -21 class ssd # do not change unnecessarily # weight 0.196 alg straw2 hash 0 # rjenkins1 item osd.0 weight 0.098 item osd.1 weight 0.098 } host ceph-node2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily id -22 class ssd # do not change unnecessarily # weight 0.196 alg straw2 hash 0 # rjenkins1 item osd.3 weight 0.098 item osd.4 weight 0.098 } host ceph-node3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily id -23 class ssd # do not change unnecessarily # weight 0.196 alg straw2 hash 0 # rjenkins1 item osd.6 weight 0.098 item osd.7 weight 0.098 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily id -24 class ssd # do not change unnecessarily # weight 0.879 alg straw2 hash 0 # rjenkins1 item ceph-node1 weight 0.293 item ceph-node2 weight 0.293 item ceph-node3 weight 0.293 } host ceph-ssd-node1 { id -13 # do not change unnecessarily id -9 class hdd # do not change unnecessarily id -14 class ssd # do not change unnecessarily # weight 0.098 alg straw2 hash 0 # rjenkins1 item osd.2 weight 0.098 } host ceph-ssd-node2 { id -15 # do not change unnecessarily id -10 class hdd # do not change unnecessarily id -16 class ssd # do not change unnecessarily # weight 0.098 alg straw2 hash 0 # rjenkins1 item osd.5 weight 0.098 } host ceph-ssd-node3 { id -17 # do not change unnecessarily id -19 class hdd # do not change unnecessarily id -18 class ssd # do not change unnecessarily # weight 0.098 alg straw2 hash 0 # rjenkins1 item osd.8 weight 0.098 } root ssd { id -11 # do not change unnecessarily id -12 class hdd # do not change unnecessarily id -20 class ssd # do not change unnecessarily # weight 0.879 alg straw2 hash 0 # rjenkins1 item ceph-ssd-node1 weight 0.293 item ceph-ssd-node2 weight 0.293 item ceph-ssd-node3 weight 0.293 } # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } rule erasure-code { id 1 type erasure min_size 3 max_size 4 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } rule ssd_rule { id 10 type replicated min_size 1 max_size 5 step take ssd step chooseleaf firstn 0 type host step emit } # end crush map
测试创建存储池：
[root@ceph-deploy ~]# ceph osd pool create ssd-pool 32 32 ssd_rule pool 'ssd-pool' created [root@ceph-deploy ~]# [root@ceph-deploy ~]# ceph osd pool ls device_health_metrics myrbd1 .rgw.root default.rgw.log default.rgw.control default.rgw.meta cephfs-metadata cephfs-data mypool rbd-data1 default.rgw.buckets.index default.rgw.buckets.data ssd-pool [root@ceph-deploy ~]#
验证 pgp 状态:
[root@ceph-deploy ~]# ceph pg ls-by-pool ssd-pool | awk '{print $1,$2,$15}' PG OBJECTS ACTING 16.0 0 [8,5,2]p8 16.1 0 [2,5,8]p2 16.2 0 [2,8,5]p2 16.3 0 [2,5,8]p2 16.4 0 [5,8,2]p5 16.5 0 [8,2,5]p8 16.6 0 [2,8,5]p2 16.7 0 [8,2,5]p8 16.8 0 [5,2,8]p5 16.9 0 [8,2,5]p8 16.a 0 [2,8,5]p2 16.b 0 [8,2,5]p8 16.c 0 [8,5,2]p8 16.d 0 [5,2,8]p5 16.e 0 [8,2,5]p8 16.f 0 [8,2,5]p8 16.10 0 [8,5,2]p8 16.11 0 [8,5,2]p8 16.12 0 [8,5,2]p8 16.13 0 [8,5,2]p8 16.14 0 [2,5,8]p2 16.15 0 [8,5,2]p8 16.16 0 [5,2,8]p5 16.17 0 [2,8,5]p2 16.18 0 [2,5,8]p2 16.19 0 [5,2,8]p5 16.1a 0 [2,8,5]p2 16.1b 0 [5,8,2]p5 16.1c 0 [2,8,5]p2 16.1d 0 [5,8,2]p5 16.1e 0 [2,5,8]p2 16.1f 0 [8,5,2]p8 * NOTE: afterwards [root@ceph-deploy ~]#
真是生产环境中ssd、hdd节点分类管理情形
# begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd device 9 osd.9 class hdd device 10 osd.10 class hdd device 11 osd.11 class hdd device 12 osd.12 class hdd device 13 osd.13 class hdd device 14 osd.14 class hdd device 20 osd.20 class hdd device 21 osd.21 class hdd device 22 osd.22 class hdd device 23 osd.23 class hdd device 24 osd.24 class hdd device 25 osd.25 class hdd device 26 osd.26 class hdd device 27 osd.27 class hdd device 28 osd.28 class hdd device 29 osd.29 class hdd device 30 osd.30 class hdd device 31 osd.31 class hdd device 32 osd.32 class hdd device 33 osd.33 class hdd device 34 osd.34 class hdd device 40 osd.40 class hdd device 41 osd.41 class hdd device 42 osd.42 class hdd device 43 osd.43 class hdd device 44 osd.44 class hdd device 45 osd.45 class hdd device 46 osd.46 class hdd device 47 osd.47 class hdd device 48 osd.48 class hdd device 49 osd.49 class hdd device 50 osd.50 class hdd device 51 osd.51 class hdd device 52 osd.52 class hdd device 53 osd.53 class hdd device 54 osd.54 class hdd device 60 osd.60 class hdd device 61 osd.61 class hdd device 62 osd.62 class hdd device 63 osd.63 class hdd device 64 osd.64 class hdd device 65 osd.65 class hdd device 66 osd.66 class hdd device 67 osd.67 class hdd device 68 osd.68 class hdd device 69 osd.69 class hdd device 70 osd.70 class hdd device 71 osd.71 class hdd device 72 osd.72 class hdd device 73 osd.73 class hdd device 74 osd.74 class hdd device 80 osd.80 class hdd device 81 osd.81 class hdd device 82 osd.82 class hdd device 83 osd.83 class hdd device 84 osd.84 class hdd device 85 osd.85 class hdd device 86 osd.86 class hdd device 87 osd.87 class hdd device 88 osd.88 class hdd device 89 osd.89 class hdd device 90 osd.90 class hdd device 91 osd.91 class hdd device 92 osd.92 class hdd device 93 osd.93 class hdd device 94 osd.94 class hdd device 100 osd.100 class hdd device 101 osd.101 class hdd device 102 osd.102 class hdd device 103 osd.103 class hdd device 104 osd.104 class hdd device 105 osd.105 class hdd device 106 osd.106 class hdd device 107 osd.107 class hdd device 108 osd.108 class hdd device 109 osd.109 class hdd device 110 osd.110 class hdd device 111 osd.111 class hdd device 112 osd.112 class hdd device 113 osd.113 class hdd device 114 osd.114 class hdd device 120 osd.120 class hdd device 121 osd.121 class hdd device 122 osd.122 class hdd device 123 osd.123 class hdd device 124 osd.124 class hdd device 125 osd.125 class hdd device 126 osd.126 class hdd device 127 osd.127 class hdd device 128 osd.128 class hdd device 129 osd.129 class hdd device 130 osd.130 class hdd device 131 osd.131 class hdd device 132 osd.132 class hdd device 133 osd.133 class hdd device 134 osd.134 class hdd device 140 osd.140 class hdd device 141 osd.141 class hdd device 142 osd.142 class hdd device 143 osd.143 class hdd device 144 osd.144 class hdd device 145 osd.145 class hdd device 146 osd.146 class hdd device 147 osd.147 class hdd device 148 osd.148 class hdd device 149 osd.149 class hdd device 150 osd.150 class hdd device 151 osd.151 class hdd device 152 osd.152 class hdd device 153 osd.153 class hdd device 154 osd.154 class hdd device 160 osd.160 class hdd device 161 osd.161 class hdd device 162 osd.162 class hdd device 163 osd.163 class hdd device 164 osd.164 class hdd device 165 osd.165 class hdd device 166 osd.166 class hdd device 167 osd.167 class hdd device 168 osd.168 class hdd device 169 osd.169 class hdd device 170 osd.170 class hdd device 171 osd.171 class hdd device 172 osd.172 class hdd device 173 osd.173 class hdd device 174 osd.174 class hdd device 180 osd.180 class hdd device 181 osd.181 class hdd device 182 osd.182 class hdd device 183 osd.183 class hdd device 184 osd.184 class hdd device 185 osd.185 class hdd device 186 osd.186 class hdd device 187 osd.187 class hdd device 188 osd.188 class hdd device 189 osd.189 class hdd device 190 osd.190 class hdd device 191 osd.191 class hdd device 192 osd.192 class hdd device 193 osd.193 class hdd device 194 osd.194 class hdd device 200 osd.200 class hdd device 201 osd.201 class hdd device 202 osd.202 class hdd device 203 osd.203 class hdd device 204 osd.204 class hdd device 205 osd.205 class hdd device 206 osd.206 class hdd device 207 osd.207 class hdd device 208 osd.208 class hdd device 209 osd.209 class hdd device 210 osd.210 class hdd device 211 osd.211 class hdd device 212 osd.212 class hdd device 213 osd.213 class hdd device 214 osd.214 class hdd device 220 osd.220 class hdd device 221 osd.221 class hdd device 222 osd.222 class hdd device 223 osd.223 class hdd device 224 osd.224 class hdd device 225 osd.225 class hdd device 226 osd.226 class hdd device 227 osd.227 class hdd device 228 osd.228 class hdd device 229 osd.229 class hdd device 230 osd.230 class hdd device 231 osd.231 class hdd device 232 osd.232 class hdd device 233 osd.233 class hdd device 234 osd.234 class hdd device 240 osd.240 class hdd device 241 osd.241 class hdd device 242 osd.242 class hdd device 243 osd.243 class hdd device 244 osd.244 class hdd device 245 osd.245 class hdd device 246 osd.246 class hdd device 247 osd.247 class hdd device 248 osd.248 class hdd device 249 osd.249 class hdd device 250 osd.250 class hdd device 251 osd.251 class hdd device 252 osd.252 class hdd device 253 osd.253 class hdd device 254 osd.254 class hdd device 260 osd.260 class hdd device 261 osd.261 class hdd device 262 osd.262 class hdd device 263 osd.263 class hdd device 264 osd.264 class hdd device 265 osd.265 class hdd device 266 osd.266 class hdd device 267 osd.267 class hdd device 268 osd.268 class hdd device 269 osd.269 class hdd device 270 osd.270 class hdd device 271 osd.271 class hdd device 272 osd.272 class hdd device 273 osd.273 class hdd device 274 osd.274 class hdd device 280 osd.280 class hdd device 281 osd.281 class hdd device 282 osd.282 class hdd device 283 osd.283 class hdd device 284 osd.284 class hdd device 285 osd.285 class hdd device 286 osd.286 class hdd device 287 osd.287 class hdd device 288 osd.288 class hdd device 289 osd.289 class hdd device 290 osd.290 class hdd device 291 osd.291 class hdd device 292 osd.292 class hdd device 293 osd.293 class hdd device 294 osd.294 class hdd device 300 osd.300 class hdd device 301 osd.301 class hdd device 302 osd.302 class hdd device 303 osd.303 class hdd device 304 osd.304 class hdd device 305 osd.305 class hdd device 306 osd.306 class hdd device 307 osd.307 class hdd device 308 osd.308 class hdd device 309 osd.309 class hdd device 310 osd.310 class hdd device 311 osd.311 class hdd device 312 osd.312 class hdd device 313 osd.313 class hdd device 314 osd.314 class hdd device 320 osd.320 class ssd device 321 osd.321 class ssd device 322 osd.322 class ssd device 323 osd.323 class ssd device 324 osd.324 class ssd device 325 osd.325 class ssd device 326 osd.326 class ssd device 327 osd.327 class ssd device 328 osd.328 class ssd device 329 osd.329 class ssd device 330 osd.330 class ssd device 331 osd.331 class ssd device 332 osd.332 class ssd device 333 osd.333 class ssd device 334 osd.334 class ssd device 335 osd.335 class ssd device 336 osd.336 class ssd device 337 osd.337 class ssd device 338 osd.338 class ssd device 339 osd.339 class ssd device 340 osd.340 class ssd device 341 osd.341 class ssd device 342 osd.342 class ssd device 343 osd.343 class ssd device 344 osd.344 class ssd device 345 osd.345 class ssd device 346 osd.346 class ssd device 347 osd.347 class ssd device 348 osd.348 class ssd device 349 osd.349 class ssd device 350 osd.350 class ssd device 351 osd.351 class ssd device 352 osd.352 class ssd device 353 osd.353 class ssd device 354 osd.354 class ssd device 355 osd.355 class ssd device 356 osd.356 class ssd device 357 osd.357 class ssd device 358 osd.358 class ssd device 359 osd.359 class ssd device 360 osd.360 class ssd device 361 osd.361 class ssd device 362 osd.362 class ssd device 363 osd.363 class ssd device 364 osd.364 class ssd device 365 osd.365 class ssd device 366 osd.366 class ssd device 367 osd.367 class ssd device 368 osd.368 class ssd device 369 osd.369 class ssd device 370 osd.370 class ssd device 371 osd.371 class ssd device 372 osd.372 class ssd device 373 osd.373 class ssd device 374 osd.374 class ssd device 375 osd.375 class ssd device 376 osd.376 class ssd device 377 osd.377 class ssd device 378 osd.378 class ssd device 379 osd.379 class ssd device 380 osd.380 class ssd device 381 osd.381 class ssd device 382 osd.382 class ssd device 383 osd.383 class ssd device 384 osd.384 class ssd device 385 osd.385 class ssd device 386 osd.386 class ssd device 387 osd.387 class ssd device 388 osd.388 class ssd device 389 osd.389 class ssd device 390 osd.390 class ssd device 391 osd.391 class ssd device 392 osd.392 class ssd device 393 osd.393 class ssd device 394 osd.394 class ssd device 395 osd.395 class ssd device 396 osd.396 class ssd device 397 osd.397 class ssd device 398 osd.398 class ssd device 399 osd.399 class ssd device 400 osd.400 class ssd device 401 osd.401 class ssd device 402 osd.402 class ssd device 403 osd.403 class ssd device 404 osd.404 class ssd device 405 osd.405 class ssd device 406 osd.406 class ssd device 407 osd.407 class ssd device 408 osd.408 class ssd device 409 osd.409 class ssd device 410 osd.410 class ssd device 411 osd.411 class ssd device 412 osd.412 class ssd device 413 osd.413 class ssd device 414 osd.414 class ssd device 415 osd.415 class ssd device 416 osd.416 class ssd device 417 osd.417 class ssd device 418 osd.418 class ssd device 419 osd.419 class ssd device 420 osd.420 class ssd device 421 osd.421 class ssd device 422 osd.422 class ssd device 423 osd.423 class ssd device 424 osd.424 class ssd device 425 osd.425 class ssd device 426 osd.426 class ssd device 427 osd.427 class ssd device 428 osd.428 class ssd device 429 osd.429 class ssd device 430 osd.430 class ssd device 431 osd.431 class ssd device 432 osd.432 class ssd device 433 osd.433 class ssd device 434 osd.434 class ssd device 435 osd.435 class ssd device 436 osd.436 class ssd device 437 osd.437 class ssd device 438 osd.438 class ssd device 439 osd.439 class ssd # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 region type 10 root # buckets host dt-1ap213-proxmox-01 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily id -35 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.0 weight 2.183 item osd.1 weight 2.183 item osd.2 weight 2.183 item osd.3 weight 2.183 item osd.4 weight 2.183 item osd.5 weight 2.183 item osd.6 weight 2.183 item osd.7 weight 2.183 item osd.8 weight 2.183 item osd.9 weight 2.183 item osd.10 weight 2.183 item osd.11 weight 2.183 item osd.12 weight 2.183 item osd.13 weight 2.183 item osd.14 weight 2.183 } host dt-1ap213-proxmox-02 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily id -36 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.20 weight 2.183 item osd.21 weight 2.183 item osd.22 weight 2.183 item osd.23 weight 2.183 item osd.24 weight 2.183 item osd.25 weight 2.183 item osd.26 weight 2.183 item osd.27 weight 2.183 item osd.28 weight 2.183 item osd.29 weight 2.183 item osd.30 weight 2.183 item osd.31 weight 2.183 item osd.32 weight 2.183 item osd.33 weight 2.183 item osd.34 weight 2.183 } host dt-1ap213-proxmox-03 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily id -37 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.40 weight 2.183 item osd.41 weight 2.183 item osd.42 weight 2.183 item osd.43 weight 2.183 item osd.44 weight 2.183 item osd.45 weight 2.183 item osd.46 weight 2.183 item osd.47 weight 2.183 item osd.48 weight 2.183 item osd.49 weight 2.183 item osd.50 weight 2.183 item osd.51 weight 2.183 item osd.52 weight 2.183 item osd.53 weight 2.183 item osd.54 weight 2.183 } host dt-1ap213-proxmox-04 { id -9 # do not change unnecessarily id -10 class hdd # do not change unnecessarily id -38 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.60 weight 2.183 item osd.61 weight 2.183 item osd.62 weight 2.183 item osd.63 weight 2.183 item osd.64 weight 2.183 item osd.65 weight 2.183 item osd.66 weight 2.183 item osd.67 weight 2.183 item osd.68 weight 2.183 item osd.69 weight 2.183 item osd.70 weight 2.183 item osd.71 weight 2.183 item osd.72 weight 2.183 item osd.73 weight 2.183 item osd.74 weight 2.183 } host dt-1ap213-proxmox-05 { id -11 # do not change unnecessarily id -12 class hdd # do not change unnecessarily id -39 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.80 weight 2.183 item osd.81 weight 2.183 item osd.82 weight 2.183 item osd.83 weight 2.183 item osd.84 weight 2.183 item osd.85 weight 2.183 item osd.86 weight 2.183 item osd.87 weight 2.183 item osd.88 weight 2.183 item osd.89 weight 2.183 item osd.90 weight 2.183 item osd.91 weight 2.183 item osd.92 weight 2.183 item osd.93 weight 2.183 item osd.94 weight 2.183 } host dt-1ap213-proxmox-06 { id -13 # do not change unnecessarily id -14 class hdd # do not change unnecessarily id -40 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.100 weight 2.183 item osd.101 weight 2.183 item osd.102 weight 2.183 item osd.103 weight 2.183 item osd.104 weight 2.183 item osd.105 weight 2.183 item osd.106 weight 2.183 item osd.107 weight 2.183 item osd.108 weight 2.183 item osd.109 weight 2.183 item osd.110 weight 2.183 item osd.111 weight 2.183 item osd.112 weight 2.183 item osd.113 weight 2.183 item osd.114 weight 2.183 } host dt-1ap213-proxmox-07 { id -15 # do not change unnecessarily id -16 class hdd # do not change unnecessarily id -41 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.120 weight 2.183 item osd.121 weight 2.183 item osd.122 weight 2.183 item osd.123 weight 2.183 item osd.124 weight 2.183 item osd.125 weight 2.183 item osd.126 weight 2.183 item osd.127 weight 2.183 item osd.128 weight 2.183 item osd.129 weight 2.183 item osd.130 weight 2.183 item osd.131 weight 2.183 item osd.132 weight 2.183 item osd.133 weight 2.183 item osd.134 weight 2.183 } host dt-1ap213-proxmox-08 { id -17 # do not change unnecessarily id -18 class hdd # do not change unnecessarily id -42 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.140 weight 2.183 item osd.141 weight 2.183 item osd.142 weight 2.183 item osd.143 weight 2.183 item osd.144 weight 2.183 item osd.145 weight 2.183 item osd.146 weight 2.183 item osd.147 weight 2.183 item osd.148 weight 2.183 item osd.149 weight 2.183 item osd.150 weight 2.183 item osd.151 weight 2.183 item osd.152 weight 2.183 item osd.153 weight 2.183 item osd.154 weight 2.183 } host dt-1ap214-proxmox-01 { id -19 # do not change unnecessarily id -20 class hdd # do not change unnecessarily id -43 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.160 weight 2.183 item osd.161 weight 2.183 item osd.162 weight 2.183 item osd.163 weight 2.183 item osd.164 weight 2.183 item osd.165 weight 2.183 item osd.166 weight 2.183 item osd.167 weight 2.183 item osd.168 weight 2.183 item osd.169 weight 2.183 item osd.170 weight 2.183 item osd.171 weight 2.183 item osd.172 weight 2.183 item osd.173 weight 2.183 item osd.174 weight 2.183 } host dt-1ap214-proxmox-02 { id -21 # do not change unnecessarily id -22 class hdd # do not change unnecessarily id -44 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.180 weight 2.183 item osd.181 weight 2.183 item osd.182 weight 2.183 item osd.183 weight 2.183 item osd.184 weight 2.183 item osd.185 weight 2.183 item osd.186 weight 2.183 item osd.187 weight 2.183 item osd.188 weight 2.183 item osd.189 weight 2.183 item osd.190 weight 2.183 item osd.191 weight 2.183 item osd.192 weight 2.183 item osd.193 weight 2.183 item osd.194 weight 2.183 } host dt-1ap214-proxmox-03 { id -23 # do not change unnecessarily id -24 class hdd # do not change unnecessarily id -45 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.200 weight 2.183 item osd.201 weight 2.183 item osd.202 weight 2.183 item osd.203 weight 2.183 item osd.204 weight 2.183 item osd.205 weight 2.183 item osd.206 weight 2.183 item osd.207 weight 2.183 item osd.208 weight 2.183 item osd.209 weight 2.183 item osd.210 weight 2.183 item osd.211 weight 2.183 item osd.212 weight 2.183 item osd.213 weight 2.183 item osd.214 weight 2.183 } host dt-1ap214-proxmox-04 { id -25 # do not change unnecessarily id -26 class hdd # do not change unnecessarily id -46 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.220 weight 2.183 item osd.221 weight 2.183 item osd.222 weight 2.183 item osd.223 weight 2.183 item osd.224 weight 2.183 item osd.225 weight 2.183 item osd.226 weight 2.183 item osd.227 weight 2.183 item osd.228 weight 2.183 item osd.229 weight 2.183 item osd.230 weight 2.183 item osd.231 weight 2.183 item osd.232 weight 2.183 item osd.233 weight 2.183 item osd.234 weight 2.183 } host dt-1ap214-proxmox-05 { id -27 # do not change unnecessarily id -28 class hdd # do not change unnecessarily id -47 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.240 weight 2.183 item osd.241 weight 2.183 item osd.242 weight 2.183 item osd.243 weight 2.183 item osd.244 weight 2.183 item osd.245 weight 2.183 item osd.246 weight 2.183 item osd.247 weight 2.183 item osd.248 weight 2.183 item osd.249 weight 2.183 item osd.250 weight 2.183 item osd.251 weight 2.183 item osd.252 weight 2.183 item osd.253 weight 2.183 item osd.254 weight 2.183 } host dt-1ap214-proxmox-06 { id -29 # do not change unnecessarily id -30 class hdd # do not change unnecessarily id -48 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.260 weight 2.183 item osd.261 weight 2.183 item osd.262 weight 2.183 item osd.263 weight 2.183 item osd.264 weight 2.183 item osd.265 weight 2.183 item osd.266 weight 2.183 item osd.267 weight 2.183 item osd.268 weight 2.183 item osd.269 weight 2.183 item osd.270 weight 2.183 item osd.271 weight 2.183 item osd.272 weight 2.183 item osd.273 weight 2.183 item osd.274 weight 2.183 } host dt-1ap214-proxmox-07 { id -31 # do not change unnecessarily id -32 class hdd # do not change unnecessarily id -49 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.280 weight 2.183 item osd.281 weight 2.183 item osd.282 weight 2.183 item osd.283 weight 2.183 item osd.284 weight 2.183 item osd.285 weight 2.183 item osd.286 weight 2.183 item osd.287 weight 2.183 item osd.288 weight 2.183 item osd.289 weight 2.183 item osd.290 weight 2.183 item osd.291 weight 2.183 item osd.292 weight 2.183 item osd.293 weight 2.183 item osd.294 weight 2.183 } host dt-1ap214-proxmox-08 { id -33 # do not change unnecessarily id -34 class hdd # do not change unnecessarily id -50 class ssd # do not change unnecessarily # weight 32.745 alg straw2 hash 0 # rjenkins1 item osd.300 weight 2.183 item osd.301 weight 2.183 item osd.302 weight 2.183 item osd.303 weight 2.183 item osd.304 weight 2.183 item osd.305 weight 2.183 item osd.306 weight 2.183 item osd.307 weight 2.183 item osd.308 weight 2.183 item osd.309 weight 2.183 item osd.310 weight 2.183 item osd.311 weight 2.183 item osd.312 weight 2.183 item osd.313 weight 2.183 item osd.314 weight 2.183 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily id -51 class ssd # do not change unnecessarily # weight 523.920 alg straw2 hash 0 # rjenkins1 item dt-1ap213-proxmox-01 weight 32.745 item dt-1ap213-proxmox-02 weight 32.745 item dt-1ap213-proxmox-03 weight 32.745 item dt-1ap213-proxmox-04 weight 32.745 item dt-1ap213-proxmox-05 weight 32.745 item dt-1ap213-proxmox-06 weight 32.745 item dt-1ap213-proxmox-07 weight 32.745 item dt-1ap213-proxmox-08 weight 32.745 item dt-1ap214-proxmox-01 weight 32.745 item dt-1ap214-proxmox-02 weight 32.745 item dt-1ap214-proxmox-03 weight 32.745 item dt-1ap214-proxmox-04 weight 32.745 item dt-1ap214-proxmox-05 weight 32.745 item dt-1ap214-proxmox-06 weight 32.745 item dt-1ap214-proxmox-07 weight 32.745 item dt-1ap214-proxmox-08 weight 32.745 } host dt-1ap215-proxmox-01 { id -52 # do not change unnecessarily id -53 class hdd # do not change unnecessarily id -54 class ssd # do not change unnecessarily # weight 10.476 alg straw2 hash 0 # rjenkins1 item osd.320 weight 0.873 item osd.321 weight 0.873 item osd.322 weight 0.873 item osd.323 weight 0.873 item osd.324 weight 0.873 item osd.325 weight 0.873 item osd.326 weight 0.873 item osd.327 weight 0.873 item osd.328 weight 0.873 item osd.329 weight 0.873 item osd.330 weight 0.873 item osd.331 weight 0.873 } host dt-1ap215-proxmox-02 { id -55 # do not change unnecessarily id -56 class hdd # do not change unnecessarily id -57 class ssd # do not change unnecessarily # weight 10.476 alg straw2 hash 0 # rjenkins1 item osd.332 weight 0.873 item osd.333 weight 0.873 item osd.334 weight 0.873 item osd.335 weight 0.873 item osd.336 weight 0.873 item osd.337 weight 0.873 item osd.338 weight 0.873 item osd.339 weight 0.873 item osd.340 weight 0.873 item osd.341 weight 0.873 item osd.342 weight 0.873 item osd.343 weight 0.873 } host dt-1ap215-proxmox-03 { id -58 # do not change unnecessarily id -59 class hdd # do not change unnecessarily id -60 class ssd # do not change unnecessarily # weight 10.476 alg straw2 hash 0 # rjenkins1 item osd.344 weight 0.873 item osd.345 weight 0.873 item osd.346 weight 0.873 item osd.347 weight 0.873 item osd.348 weight 0.873 item osd.349 weight 0.873 item osd.350 weight 0.873 item osd.351 weight 0.873 item osd.352 weight 0.873 item osd.353 weight 0.873 item osd.354 weight 0.873 item osd.355 weight 0.873 } host dt-1ap215-proxmox-04 { id -61 # do not change unnecessarily id -62 class hdd # do not change unnecessarily id -63 class ssd # do not change unnecessarily # weight 10.476 alg straw2 hash 0 # rjenkins1 item osd.356 weight 0.873 item osd.357 weight 0.873 item osd.358 weight 0.873 item osd.359 weight 0.873 item osd.360 weight 0.873 item osd.361 weight 0.873 item osd.362 weight 0.873 item osd.363 weight 0.873 item osd.364 weight 0.873 item osd.365 weight 0.873 item osd.366 weight 0.873 item osd.367 weight 0.873 } host dt-1ap215-proxmox-05 { id -64 # do not change unnecessarily id -65 class hdd # do not change unnecessarily id -66 class ssd # do not change unnecessarily # weight 10.476 alg straw2 hash 0 # rjenkins1 item osd.368 weight 0.873 item osd.369 weight 0.873 item osd.370 weight 0.873 item osd.371 weight 0.873 item osd.373 weight 0.873 item osd.372 weight 0.873 item osd.374 weight 0.873 item osd.375 weight 0.873 item osd.376 weight 0.873 item osd.377 weight 0.873 item osd.378 weight 0.873 item osd.379 weight 0.873 } host dt-1ap215-proxmox-06 { id -67 # do not change unnecessarily id -68 class hdd # do not change unnecessarily id -69 class ssd # do not change unnecessarily # weight 10.476 alg straw2 hash 0 # rjenkins1 item osd.380 weight 0.873 item osd.381 weight 0.873 item osd.382 weight 0.873 item osd.383 weight 0.873 item osd.384 weight 0.873 item osd.385 weight 0.873 item osd.386 weight 0.873 item osd.387 weight 0.873 item osd.388 weight 0.873 item osd.389 weight 0.873 item osd.390 weight 0.873 item osd.391 weight 0.873 } host dt-1ap215-proxmox-07 { id -70 # do not change unnecessarily id -71 class hdd # do not change unnecessarily id -72 class ssd # do not change unnecessarily # weight 10.476 alg straw2 hash 0 # rjenkins1 item osd.392 weight 0.873 item osd.393 weight 0.873 item osd.394 weight 0.873 item osd.395 weight 0.873 item osd.396 weight 0.873 item osd.397 weight 0.873 item osd.398 weight 0.873 item osd.399 weight 0.873 item osd.400 weight 0.873 item osd.401 weight 0.873 item osd.402 weight 0.873 item osd.403 weight 0.873 } host dt-1ap215-proxmox-08 { id -73 # do not change unnecessarily id -74 class hdd # do not change unnecessarily id -75 class ssd # do not change unnecessarily # weight 10.476 alg straw2 hash 0 # rjenkins1 item osd.404 weight 0.873 item osd.405 weight 0.873 item osd.406 weight 0.873 item osd.407 weight 0.873 item osd.408 weight 0.873 item osd.409 weight 0.873 item osd.410 weight 0.873 item osd.411 weight 0.873 item osd.412 weight 0.873 item osd.413 weight 0.873 item osd.414 weight 0.873 item osd.415 weight 0.873 } host dt-1ap216-proxmox-05 { id -76 # do not change unnecessarily id -77 class hdd # do not change unnecessarily id -78 class ssd # do not change unnecessarily # weight 10.476 alg straw2 hash 0 # rjenkins1 item osd.416 weight 0.873 item osd.417 weight 0.873 item osd.418 weight 0.873 item osd.419 weight 0.873 item osd.420 weight 0.873 item osd.421 weight 0.873 item osd.422 weight 0.873 item osd.423 weight 0.873 item osd.424 weight 0.873 item osd.425 weight 0.873 item osd.426 weight 0.873 item osd.427 weight 0.873 } host dt-1ap216-proxmox-06 { id -79 # do not change unnecessarily id -80 class hdd # do not change unnecessarily id -81 class ssd # do not change unnecessarily # weight 10.476 alg straw2 hash 0 # rjenkins1 item osd.428 weight 0.873 item osd.429 weight 0.873 item osd.430 weight 0.873 item osd.431 weight 0.873 item osd.432 weight 0.873 item osd.433 weight 0.873 item osd.434 weight 0.873 item osd.435 weight 0.873 item osd.436 weight 0.873 item osd.437 weight 0.873 item osd.438 weight 0.873 item osd.439 weight 0.873 } root ssd { id -82 # do not change unnecessarily id -83 class hdd # do not change unnecessarily id -84 class ssd # do not change unnecessarily # weight 104.780 alg straw2 hash 0 # rjenkins1 item dt-1ap215-proxmox-01 weight 10.478 item dt-1ap215-proxmox-02 weight 10.478 item dt-1ap215-proxmox-03 weight 10.478 item dt-1ap215-proxmox-04 weight 10.478 item dt-1ap215-proxmox-05 weight 10.478 item dt-1ap215-proxmox-06 weight 10.478 item dt-1ap215-proxmox-07 weight 10.478 item dt-1ap215-proxmox-08 weight 10.478 item dt-1ap216-proxmox-05 weight 10.478 item dt-1ap216-proxmox-06 weight 10.478 } # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } rule ssd_replicated_rule { id 1 type replicated min_size 1 max_size 10 step take ssd step chooseleaf firstn 0 type host step emit }
