八、Ceph Crush map
2、CRUSH算法的流程
CRUSH算法的全程为：Controlled、Scalable、Decentralized Placement of Replicated Data,可控的、可扩展的、分布式的副本数据放置算法。
对 Ceph 集群的一个读写操作，客户端首先访问 Ceph monitor 来获取 cluster map 的一份副本，它包含五个map，分别是 monitor map、OSD map、MDS map、CRUSH map 和 PG map。这些 cluster map 将有助于客户端知晓 Ceph集群的状态和配置。然后，数据被转化为一个或者多个对象，每个对象都具有对象名称（name）和存储池名称/ID（pool name/ID）。接着，该对象会被以配置组（PG）的数目为基数做哈希，在指定的 Ceph 存储池中生成最终的 PG。这个计算出的 PG 再通过 CRUSH 查询来确定存放或者获取数据的主（primary）、次（secondary）和再次（tertiary）OSD 的位置。
一旦客户端获得了精确的 OSD ID，将会直接和这些 OSD 通信并存放数据。所有这些计算操作都是在客户端完成的，因此它们不会影响 Ceph 集群服务器端的性能。
1. 上传一个文件时，使用如下命令： # rados -p ssd-pool put dummy_object1 /etc/ hosts 2. 这里上传到那个 pool(ssd-pool)，以及对象的名称(dummy_object1)已经确定； 3. 一个 pool 在创建的时候，其 PG 的数量已确定； 4. 通过对 object 进行哈希，公式为：hash(dummy_object1)%PG_num；这样可以得到对象所在的 PG(这里是 4.32) 5. PG 根据根据 CRUSH map 的 rule 最终确定落在哪些 OSD 上； 6. 所以客户端在访问 ceph 时，可以通过计算来得出 object 所在的 OSD，消除了中心节点的 metadata。
整个流程如下:
1）取得Object的name进行Hash运算。 2）取得的Hash值与PG数取余，得到的结果与Pool ID结成PG的编号（如图10-5中的4.32）。 3）通过CRUSH算法，把PG映射到具体的OSD。 在Ceph里，PG是数据存储的管理单元，如果把PG当作一致性Hash里的存储节点，那么它就是最简单的数据分布（即取余算法）方式。不同的是，PG是抽象的存储节点，它不会随着物理节点的加入或者离开而增加或减少，因此数据到PG的映射是稳定的。 当在Ceph里创建存储池（指定PG的数量），整个Hash环就固定了，对象到PG的映射就唯一确认，即数据具体存储到哪个PG是确定的，不会随着下层OSD的增删而改变，这充分体现了Ceph在数据可靠性的特征之一。
实验案例：
#1、创建pool2地址池 cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool create pool2 64 64 pool 'pool2' created #2、上传一个名为object1到pool2存储池 cephadmin@ceph-deploy:~/ceph-cluster$ rados -p pool2 put object1 /etc/hosts #3、查看pool2中的存储池对象 cephadmin@ceph-deploy:~/ceph-cluster$ rados -p pool2 ls object1 #4、查看pool2存储池中的object1在OSD上的分布及PGid cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd map pool2 object1 osdmap e439 pool 'pool2' (11) object 'object1' -> pg 11.bac5debc (11.3c) -> up ([1,6,10], p1) acting ([1,6,10], p1) #这里的PGid为 pg 11.bac5debc (11.3c)，数据分布存数在osd.1,osd.6和osd.10上，其中osd1为主OSD
2）、删除pool2中的object2,让后上传内容不一样，单对象名认为object1。
#1、删除object1对象 cephadmin@ceph-deploy:~/ceph-cluster$ rados -p pool2 rm object1 cephadmin@ceph-deploy:~/ceph-cluster$ rados -p pool2 ls cephadmin@ceph-deploy:~/ceph-cluster$ #2、上传/etc/passwd,对象名称为objetct1 cephadmin@ceph-deploy:~/ceph-cluster$ rados -p pool2 put object1 /etc/passwd #3、查看pool2存储池中的object1在OSD上的分布及PGid cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd map pool2 object1 osdmap e439 pool 'pool2' (11) object 'object1' -> pg 11.bac5debc (11.3c) -> up ([1,6,10], p1) acting ([1,6,10], p1) #这里的PGid为 pg 11.bac5debc (11.3c)，数据分布存数在osd.1,osd.6和osd.10上，其中osd1为master
上面2个实验说明了在同一个pool存储池中，只要对象名称不变，那么数据的分布就固定不变
Ceph 的后端是一个对象存储（RADOS），所以所有的数据都会按照一个特定的 size（ceph 系统默认是 4M）被切分成若干个对象，也就是上面的 Objects。每一个 Object 都有一个 Objectid（oid），Objectid 的命名规则是数据所在image 的 block_name_prefix 再跟上一个编号，这个编号是顺序递增的。通过(pool id, hash(oid) & mask)，每个object 都可以得到它对应的 pgid。有了这个 pgid 之后，Client 就会执行 CRUSH 算法得到一个 OSD 列表（OSD1，OSD2，OSD3）。然后对它们进行筛选，根据副本数找出符合要求的 OSD，比如 OSD 不能是 failed、overloaded的。知道数据要存在哪些 OSD 上之后，Client 会向列表中的第一个 OSD（primary osd）发起 IO 请求。然后这个OSD 按照读写请求分别做相应的处理。
4、ceph cluster map介绍
Ceph monitor负责监控整个集群的监控状态，以及维护集群成员关系状态、对等节点的状态，和集群的配置信息等。Ceph monitor通过维护cluster map的主复制来实现这些功能。
1）monitor map: 它包含监视节点端到端的信息，包括Ceph集群ID、monitor节点名称（hostname）、IP地址和端口号等。它还保存自monitor map被创建以来的最新版本号（epoch:每种map都维护着其历史版本，每个版本被称为一个epoch，epoch是一个单调递增的序列）。以及最后修改时间等。查看monitor map的命令
cephadmin@ceph-deploy:~/ceph-cluster$ ceph mon dump epoch 3 fsid d1f2683e-ac8d-4a02-8b94-9f7517d32c50 last_changed 2022-04-03T16:49:08.531006+0800 created 2022-04-03T16:28:03.331168+0800 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:192.168.1.203:3300/0,v1:192.168.1.203:6789/0] mon.ceph-mon1 1: [v2:192.168.1.204:3300/0,v1:192.168.1.204:6789/0] mon.ceph-mon2 2: [v2:192.168.1.205:3300/0,v1:192.168.1.205:6789/0] mon.ceph-mon3 dumped monmap epoch 3
2) OSD map: 它保存一些常用的信息，包括集群ID、OSD map自创建以来的最新版本号（epoch）及其最后修改时间，以及存储池相关的信息，包括存储池名称、ID、类型、副本级别（replication level)和PG。它还保存着OSD的信息，比如数量、状态、权重、最后清理间隔（last clean interval）以及OSD节点的信息。查看OSD map的命令
cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd dump epoch 378 fsid d1f2683e-ac8d-4a02-8b94-9f7517d32c50 created 2022-04-03T16:28:04.210492+0800 modified 2022-04-04T17:02:50.538648+0800 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 35 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client luminous min_compat_client luminous require_osd_release pacific stretch_mode_enabled false pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 226 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 73 flags hashpspool stripe_width 0 application rgw pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 75 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 77 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 202 lfor 0/202/200 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 6 'cephfs-metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 156 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs pool 7 'cephfs-data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 156 flags hashpspool stripe_width 0 application cephfs pool 8 'myrbd1' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 212 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 9 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 377 lfor 0/377/375 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 10 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 258 flags hashpspool stripe_width 0 application rgw max_osd 12 osd.0 up in weight 1 up_from 229 up_thru 256 down_at 219 last_clean_interval [65,218) [v2:192.168.1.206:6800/1363,v1:192.168.1.206:6805/1363] [v2:192.168.88.206:6804/1363,v1:192.168.88.206:6805/1363] exists,up 2ed474c5-6975-4400-b1cf-e7a528c19842 osd.1 up in weight 1 up_from 228 up_thru 377 down_at 219 last_clean_interval [65,218) [v2:192.168.1.206:6808/1367,v1:192.168.1.206:6809/1367] [v2:192.168.88.206:6808/1367,v1:192.168.88.206:6809/1367] exists,up 9f0d23e1-2c70-4f8f-b19d-6b416469debb osd.2 up in weight 1 up_from 230 up_thru 339 down_at 221 last_clean_interval [17,220) [v2:192.168.1.207:6800/1496,v1:192.168.1.207:6805/1496] [v2:192.168.88.207:6804/1496,v1:192.168.88.207:6805/1496] exists,up 1a26c5b1-18b1-429a-8334-e9cf154c1e6c osd.3 up in weight 1 up_from 231 up_thru 344 down_at 221 last_clean_interval [23,220) [v2:192.168.1.207:6808/1498,v1:192.168.1.207:6809/1498] [v2:192.168.88.207:6808/1498,v1:192.168.88.207:6809/1498] exists,up 1a183a93-5ccd-45b2-89fc-2dc8ad2914c5 osd.4 up in weight 1 up_from 230 up_thru 256 down_at 221 last_clean_interval [28,220) [v2:192.168.1.207:6801/1497,v1:192.168.1.207:6802/1497] [v2:192.168.88.207:6800/1497,v1:192.168.88.207:6801/1497] exists,up 76526a2b-a3d3-4e66-82e8-e26754115cfc osd.5 up in weight 1 up_from 235 up_thru 354 down_at 223 last_clean_interval [33,222) [v2:192.168.1.208:6800/1532,v1:192.168.1.208:6801/1532] [v2:192.168.88.208:6800/1532,v1:192.168.88.208:6801/1532] exists,up 7612de2d-94a0-4072-986e-e51d37d407d3 osd.6 up in weight 1 up_from 234 up_thru 256 down_at 223 last_clean_interval [38,222) [v2:192.168.1.208:6808/1534,v1:192.168.1.208:6809/1534] [v2:192.168.88.208:6808/1534,v1:192.168.88.208:6809/1534] exists,up a88f02a6-1c08-4dca-bfb3-3d5981595fb8 osd.7 up in weight 1 up_from 235 up_thru 365 down_at 223 last_clean_interval [43,222) [v2:192.168.1.208:6804/1533,v1:192.168.1.208:6805/1533] [v2:192.168.88.208:6804/1533,v1:192.168.88.208:6805/1533] exists,up 7dd759be-e341-4f62-9378-1be6171519f7 osd.8 up in weight 1 up_from 238 up_thru 369 down_at 230 last_clean_interval [48,224) [v2:192.168.1.209:6808/1526,v1:192.168.1.209:6809/1526] [v2:192.168.88.209:6808/1526,v1:192.168.88.209:6809/1526] exists,up 99260fce-64ce-4426-a0a7-d8819b54f0a2 osd.9 up in weight 1 up_from 238 up_thru 373 down_at 230 last_clean_interval [54,224) [v2:192.168.1.209:6804/1525,v1:192.168.1.209:6805/1525] [v2:192.168.88.209:6804/1525,v1:192.168.88.209:6805/1525] exists,up ab8ab101-60b5-4943-8b1a-a32d76084b6d osd.10 up in weight 1 up_from 238 up_thru 259 down_at 229 last_clean_interval [59,224) [v2:192.168.1.209:6800/1523,v1:192.168.1.209:6801/1523] [v2:192.168.88.209:6800/1523,v1:192.168.88.209:6801/1523] exists,up b4998715-69bb-4140-a18f-dfb112783e8c osd.11 up in weight 1 up_from 229 up_thru 256 down_at 219 last_clean_interval [69,218) [v2:192.168.1.206:6801/1376,v1:192.168.1.206:6802/1376] [v2:192.168.88.206:6800/1376,v1:192.168.88.206:6801/1376] exists,up 94462cbe-6ddc-4edb-86e9-f78b1f43bea8 pg_upmap_items 6.0 [10,9] pg_upmap_items 6.2 [11,0] pg_upmap_items 6.c [11,0] pg_upmap_items 6.e [10,9] pg_upmap_items 6.1a [1,0] pg_upmap_items 7.a [0,1] pg_upmap_items 7.17 [2,4] pg_upmap_items 7.2a [0,4] pg_upmap_items 7.3e [2,4] pg_upmap_items 8.2 [10,3] pg_upmap_items 8.3 [1,11] pg_upmap_items 8.a [5,3] pg_upmap_items 8.20 [2,3] pg_upmap_items 8.2e [8,9] pg_upmap_items 10.0 [0,11] blocklist 192.168.1.201:6801/9150 expires 2022-04-05T15:21:56.596516+0800 blocklist 192.168.1.201:6800/9150 expires 2022-04-05T15:21:56.596516+0800 blocklist 192.168.1.201:0/3611889504 expires 2022-04-05T15:21:56.596516+0800 blocklist 192.168.1.201:0/1264985371 expires 2022-04-05T15:21:56.596516+0800 blocklist 192.168.1.205:6800/2648602275 expires 2022-04-05T00:02:38.304109+0800 blocklist 192.168.1.205:6801/2648602275 expires 2022-04-05T00:02:38.304109+0800 blocklist 192.168.1.203:6801/3665534120 expires 2022-04-05T00:02:25.602324+0800 blocklist 192.168.1.204:6800/1132501689 expires 2022-04-05T00:02:33.275718+0800 blocklist 192.168.1.203:6800/3665534120 expires 2022-04-05T00:02:25.602324+0800 blocklist 192.168.1.204:6801/1132501689 expires 2022-04-05T00:02:33.275718+0800
3) PG map:它保存的信息包括PG的版本、时间戳、OSD map的最新版本号（epoch）、容量已满百分比（full ratio,是指集群被认为容量已满时的最大使用百分比，此时Ceph集群禁止客户端读写数据）、容量将满百分比（near full ration,是指集群被认为容量将满时的最大使用百分比，此时Ceph集群将发出告警）等。它还记录了每个PG的ID（其格式为{pool-num}.{pg-id}）、对象数量、状态、状态时间戳（state stamp）、up OSD sets(一个up OSD set指某些特定PG map版本的PG的所有副本所在的OSD列表，该I列表是有序的，第一个OSD为主OSD，其余为从OSD)、acting OSD sets（acting OSD set指该PG的所有副本所在的OSD的列表），以及清理（scrub）的信息。查看PG map的命令
cephadmin@ceph-deploy:~/ceph-cluster$ ceph pg dump
version 13086
stamp 2022-04-04T22:33:26.923756+0800
last_osdmap_epoch 0
last_pg_scan 0
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN
8.30 3 0 0 0 0 12582912 0 0 34 34 active+clean 2022-04-04T19:54:12.287880+0800 215'34 378:232 [1,6,8] 1 [1,6,8] 1 215'34 2022-04-04T19:54:12.287820+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.3f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:09:39.827406+0800 0'0 378:270 [3,7,1] 3 [3,7,1] 3 0'0 2022-04-04T17:09:39.826881+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.31 5 0 0 0 0 18939904 0 0 42 42 active+clean 2022-04-04T20:03:16.350937+0800 215'42 378:253 [7,2,10] 7 [7,2,10] 7 215'42 2022-04-04T20:03:16.350878+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.3e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:44:29.338657+0800 0'0 378:198 [4,6,0] 4 [4,6,0] 4 0'0 2022-04-04T19:44:29.338594+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.32 3 0 0 0 0 8421376 0 0 34 34 active+clean 2022-04-04T15:22:25.982726+0800 215'34 377:228 [0,7,9] 0 [0,7,9] 0 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.3d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:14:07.110388+0800 0'0 378:270 [4,10,11] 4 [4,10,11] 4 0'0 2022-04-04T21:14:07.110321+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.33 5 0 0 0 0 20971520 0 0 49 49 active+clean 2022-04-04T22:17:05.995239+0800 215'49 378:244 [11,4,9] 11 [11,4,9] 11 215'49 2022-04-04T22:17:05.995185+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.3c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:37.784982+0800 0'0 377:264 [8,6,11] 8 [8,6,11] 8 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.34 6 0 0 0 0 25165824 0 0 47 47 active+clean 2022-04-04T20:40:33.146876+0800 378'47 378:244 [0,6,8] 0 [0,6,8] 0 378'47 2022-04-04T20:40:33.145571+0800 378'47 2022-04-04T20:40:33.145571+0800 0
7.3b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:23:02.672284+0800 0'0 378:270 [3,0,5] 3 [3,0,5] 3 0'0 2022-04-04T18:23:02.672233+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.35 3 0 0 0 0 12582912 0 0 29 29 active+clean 2022-04-04T15:22:26.524343+0800 215'29 377:221 [11,9,7] 11 [11,9,7] 11 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.3a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:51:17.299928+0800 0'0 378:269 [9,7,2] 9 [9,7,2] 9 0'0 2022-04-04T18:51:17.299877+0800 0'0 2022-04-04T18:51:17.299877+0800 0
8.36 0 0 0 0 0 0 0 0 31 31 active+clean 2022-04-04T18:05:56.349707+0800 215'31 378:229 [1,10,3] 1 [1,10,3] 1 215'31 2022-04-04T18:05:56.349639+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.39 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:39.225998+0800 0'0 377:267 [8,1,3] 8 [8,1,3] 8 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.37 5 0 0 0 0 20971520 0 0 34 34 active+clean 2022-04-04T20:50:35.958059+0800 378'34 378:230 [5,0,9] 5 [5,0,9] 5 378'34 2022-04-04T20:50:35.957568+0800 378'34 2022-04-04T20:50:35.957568+0800 0
7.38 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:05:55.131216+0800 0'0 378:273 [9,2,0] 9 [9,2,0] 9 0'0 2022-04-04T17:05:55.131130+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.38 0 0 0 0 0 0 0 0 21 21 active+clean 2022-04-04T18:08:44.937516+0800 215'21 378:217 [1,9,6] 1 [1,9,6] 1 215'21 2022-04-04T18:08:44.937459+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.37 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.439177+0800 0'0 377:264 [9,7,3] 9 [9,7,3] 9 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.39 3 0 0 0 0 12582912 0 0 20 20 active+clean 2022-04-04T21:35:13.985246+0800 215'20 378:217 [5,8,3] 5 [5,8,3] 5 215'20 2022-04-04T21:35:13.985046+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.36 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:34:58.813081+0800 0'0 378:265 [11,4,6] 11 [11,4,6] 11 0'0 2022-04-04T19:34:58.813040+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.3a 4 0 0 0 0 4333587 0 0 52 52 active+clean 2022-04-04T15:22:27.152561+0800 215'52 377:293 [7,0,10] 7 [7,0,10] 7 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.35 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:37.463043+0800 0'0 377:264 [8,6,0] 8 [8,6,0] 8 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.3b 1 0 0 0 0 4194304 0 0 25 25 active+clean 2022-04-04T20:57:41.855311+0800 215'25 378:219 [4,7,10] 4 [4,7,10] 4 215'25 2022-04-04T20:57:41.855265+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.34 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:37:27.479292+0800 0'0 378:269 [5,11,2] 5 [5,11,2] 5 0'0 2022-04-04T19:37:27.479245+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.3c 4 0 0 0 0 16777216 0 0 51 51 active+clean 2022-04-04T15:22:22.271509+0800 215'51 377:247 [2,7,0] 2 [2,7,0] 2 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.33 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:05:47.749529+0800 0'0 378:269 [8,5,0] 8 [8,5,0] 8 0'0 2022-04-04T19:05:47.749462+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.3d 6 0 0 0 0 10579968 0 0 52 52 active+clean 2022-04-04T15:22:38.013594+0800 215'52 377:399 [9,6,1] 9 [9,6,1] 9 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.32 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:25:31.727211+0800 0'0 378:267 [5,8,4] 5 [5,8,4] 5 0'0 2022-04-04T17:25:31.727159+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.3e 1 0 0 0 0 4194304 0 0 25 25 active+clean 2022-04-04T18:28:38.971941+0800 215'25 378:225 [7,2,1] 7 [7,2,1] 7 215'25 2022-04-04T18:28:38.971893+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.31 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:03:40.787280+0800 0'0 378:267 [5,9,3] 5 [5,9,3] 5 0'0 2022-04-04T17:03:40.787213+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.3f 1 0 0 0 0 8192 0 0 30 30 active+clean 2022-04-04T15:22:22.300929+0800 215'30 377:221 [1,3,7] 1 [1,3,7] 1 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.30 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.796519+0800 0'0 377:265 [9,5,1] 9 [9,5,1] 9 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.20 2 0 0 0 0 5926912 0 0 30 30 active+clean 2022-04-04T15:22:26.047927+0800 215'30 377:208 [3,6,8] 3 [3,6,8] 3 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.2f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:14:49.996984+0800 0'0 378:271 [2,10,1] 2 [2,10,1] 2 0'0 2022-04-04T17:14:49.996917+0800 0'0 2022-04-04T17:14:49.996917+0800 0
8.21 2 0 0 0 0 8388608 0 0 28 28 active+clean 2022-04-04T21:02:10.719437+0800 215'28 378:225 [0,5,9] 0 [0,5,9] 0 215'28 2022-04-04T21:02:10.719373+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.2e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:47:40.444977+0800 0'0 378:269 [8,11,7] 8 [8,11,7] 8 0'0 2022-04-04T20:47:40.444928+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.22 2 0 0 0 0 8388608 0 0 26 26 active+clean 2022-04-04T18:29:59.656872+0800 215'26 378:228 [10,1,4] 10 [10,1,4] 10 215'26 2022-04-04T18:29:59.656288+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.2d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:37.519631+0800 0'0 377:264 [9,6,3] 9 [9,6,3] 9 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.23 1 0 0 0 0 0 0 0 19 19 active+clean 2022-04-04T20:16:22.127774+0800 215'19 378:223 [8,4,7] 8 [8,4,7] 8 215'19 2022-04-04T20:16:22.127319+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.2c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:21:55.216352+0800 0'0 378:266 [11,7,8] 11 [11,7,8] 11 0'0 2022-04-04T18:21:55.216306+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.24 1 0 0 0 0 4194304 0 0 24 24 active+clean 2022-04-04T15:22:27.304174+0800 215'24 377:219 [4,0,8] 4 [4,0,8] 4 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.2b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:21.543563+0800 0'0 377:262 [1,2,6] 1 [1,2,6] 1 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.25 2 0 0 0 0 8388608 0 0 25 25 active+clean 2022-04-04T20:08:39.735770+0800 215'25 378:220 [1,10,4] 1 [1,10,4] 1 215'25 2022-04-04T20:08:39.735712+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.2a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:48:56.237374+0800 0'0 378:182 [4,9,7] 4 [4,9,7] 4 0'0 2022-04-04T21:48:56.237314+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.26 5 0 0 0 0 10645504 0 0 40 40 active+clean 2022-04-04T15:22:31.127727+0800 215'40 377:236 [5,1,3] 5 [5,1,3] 5 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.29 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:26.928825+0800 0'0 377:265 [3,9,0] 3 [3,9,0] 3 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.27 2 0 0 0 0 8388608 0 0 21 21 active+clean 2022-04-04T15:22:27.217084+0800 215'21 377:215 [6,9,4] 6 [6,9,4] 6 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.28 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.254525+0800 0'0 377:264 [6,9,2] 6 [6,9,2] 6 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.28 2 0 0 0 0 8208 0 0 24 24 active+clean 2022-04-04T15:22:21.411243+0800 215'24 377:223 [0,2,6] 0 [0,2,6] 0 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.27 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T22:19:23.172047+0800 0'0 378:272 [10,2,1] 10 [10,2,1] 10 0'0 2022-04-04T22:19:23.172000+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.29 6 0 0 0 0 20992000 0 0 41 41 active+clean 2022-04-04T15:22:27.318296+0800 215'41 377:234 [5,9,11] 5 [5,9,11] 5 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.26 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:37.519648+0800 0'0 377:264 [9,6,3] 9 [9,6,3] 9 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.2a 0 0 0 0 0 0 0 0 21 21 active+clean 2022-04-04T15:22:26.860642+0800 215'21 377:212 [5,8,0] 5 [5,8,0] 5 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.25 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.516771+0800 0'0 377:263 [8,2,7] 8 [8,2,7] 8 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.2b 2 0 0 0 0 4194320 0 0 29 29 active+clean 2022-04-04T18:31:10.563124+0800 215'29 378:238 [10,3,7] 10 [10,3,7] 10 215'29 2022-04-04T18:31:10.563077+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.24 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.320218+0800 0'0 377:265 [3,7,11] 3 [3,7,11] 3 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.2c 4 0 0 0 0 16777216 0 0 31 31 active+clean 2022-04-04T21:10:07.437378+0800 215'31 378:230 [6,4,8] 6 [6,4,8] 6 215'31 2022-04-04T21:10:07.437333+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.23 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:26.046295+0800 0'0 377:265 [3,8,11] 3 [3,8,11] 3 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.2d 5 0 0 0 0 16797696 0 0 47 47 active+clean 2022-04-04T22:28:46.212208+0800 215'47 378:244 [0,4,5] 0 [0,4,5] 0 215'47 2022-04-04T22:28:46.212164+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.22 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:07:21.379545+0800 0'0 378:273 [9,3,11] 9 [9,3,11] 9 0'0 2022-04-04T21:07:21.379485+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.2e 3 0 0 0 0 12582912 0 0 55 55 active+clean 2022-04-04T22:11:04.175352+0800 215'55 378:266 [2,9,0] 2 [2,9,0] 2 215'55 2022-04-04T22:11:04.175286+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.21 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.136684+0800 0'0 377:264 [6,10,3] 6 [6,10,3] 6 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
8.2f 2 0 0 0 0 8388608 0 0 33 33 active+clean 2022-04-04T15:22:30.234232+0800 215'33 377:229 [6,0,2] 6 [6,0,2] 6 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.20 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:56:57.627763+0800 0'0 378:266 [5,0,10] 5 [5,0,10] 5 0'0 2022-04-04T18:56:57.627716+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.6 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.429725+0800 0'0 377:130 [3,7,10] 3 [3,7,10] 3 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
9.5 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:17.627348+0800 264'2 377:170 [5,10,4] 5 [5,10,4] 5 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 0
8.4 4 0 0 0 0 16777216 0 0 37 37 active+clean 2022-04-04T15:22:39.073619+0800 215'37 377:231 [9,5,2] 9 [9,5,2] 9 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
4.8 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:04:18.042232+0800 0'0 378:325 [5,3,11] 5 [5,3,11] 5 0'0 2022-04-04T19:04:18.042184+0800 0'0 2022-04-03T16:53:03.970520+0800 0
3.f 7 0 0 0 0 110 0 0 2223 2223 active+clean 2022-04-04T20:34:07.641044+0800 378'2223 378:3684 [2,5,9] 2 [2,5,9] 2 378'1927 2022-04-04T20:34:07.640996+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:05:16.673743+0800 0'0 378:309 [1,5,8] 1 [1,5,8] 1 0'0 2022-04-04T21:05:16.673691+0800 0'0 2022-04-04T21:05:16.673691+0800 0
6.a 1 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T15:22:27.032486+0800 239'2 377:260 [4,10,6] 4 [4,10,6] 4 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:21.540510+0800 0'0 377:262 [1,2,6] 1 [1,2,6] 1 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.5 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.431801+0800 0'0 377:130 [9,0,2] 9 [9,0,2] 9 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
9.6 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:13.965116+0800 0'0 377:170 [9,1,6] 9 [9,1,6] 9 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 0
8.7 3 0 0 0 0 8409088 0 0 37 37 active+clean 2022-04-04T15:22:26.471176+0800 215'37 377:232 [1,5,9] 1 [1,5,9] 1 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
4.b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:37.825922+0800 0'0 377:318 [8,4,6] 8 [8,4,6] 8 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
3.c 5 0 0 0 0 0 0 0 1780 1780 active+clean 2022-04-04T15:22:22.345821+0800 378'1780 378:2993 [3,0,7] 3 [3,0,7] 3 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:45:42.990376+0800 0'0 378:326 [5,10,1] 5 [5,10,1] 5 0'0 2022-04-04T18:45:42.990329+0800 0'0 2022-04-03T16:52:59.854233+0800 0
6.9 2 0 0 0 0 90 0 0 3 3 active+clean 2022-04-04T17:05:46.918006+0800 205'3 378:282 [10,7,11] 10 [10,7,11] 10 205'3 2022-04-04T17:05:46.917940+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.8 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:39.450652+0800 0'0 377:266 [10,4,0] 10 [10,4,0] 10 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.4 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.386133+0800 0'0 377:130 [6,0,8] 6 [6,0,8] 6 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
9.7 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:09.858927+0800 264'4 377:170 [5,10,11] 5 [5,10,11] 5 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 0
8.6 4 0 0 0 0 16777216 0 0 40 40 active+clean 2022-04-04T21:40:30.063658+0800 215'40 378:238 [6,9,1] 6 [6,9,1] 6 215'40 2022-04-04T21:40:30.063442+0800 0'0 2022-04-03T17:52:00.867759+0800 0
4.a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:25.735285+0800 0'0 377:321 [2,9,11] 2 [2,9,11] 2 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
3.d 8 0 0 0 0 110 0 0 2579 2579 active+clean 2022-04-04T18:32:14.728114+0800 378'2579 378:4224 [3,7,11] 3 [3,7,11] 3 378'1891 2022-04-04T18:32:14.728064+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:30.527163+0800 0'0 377:325 [6,0,4] 6 [6,0,4] 6 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 0
6.8 1 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T21:42:52.073880+0800 205'2 378:264 [3,6,9] 3 [3,6,9] 3 205'2 2022-04-04T21:42:52.073825+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.9 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.267626+0800 0'0 377:261 [0,7,4] 0 [0,7,4] 0 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.3 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.359622+0800 0'0 377:130 [7,10,4] 7 [7,10,4] 7 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:50.598378+0800 264'2 378:170 [1,8,5] 1 [1,8,5] 1 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 0
8.1 4 0 0 0 0 16777216 0 0 37 37 active+clean 2022-04-04T15:22:25.964053+0800 215'37 377:230 [1,6,10] 1 [1,6,10] 1 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.a 10 0 0 0 0 110 0 0 2947 2947 active+clean 2022-04-04T15:22:26.926949+0800 378'2947 378:4776 [3,9,5] 3 [3,9,5] 3 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.b 1 0 0 0 0 358 0 0 1 1 active+clean 2022-04-04T15:22:27.073937+0800 72'1 377:348 [7,3,10] 7 [7,3,10] 7 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.d 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T19:09:01.929819+0800 251'37 378:20973 [3,5,10] 3 [3,5,10] 3 251'37 2022-04-04T19:09:01.929609+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.f 2 0 0 0 0 22 0 0 2 2 active+clean 2022-04-04T15:22:26.442454+0800 205'2 377:260 [11,10,7] 11 [11,10,7] 11 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.419514+0800 0'0 377:262 [1,5,3] 1 [1,5,3] 1 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.2 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.408617+0800 0'0 377:130 [1,10,4] 1 [1,10,4] 1 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
9.1 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:41.929605+0800 264'6 377:170 [9,0,5] 9 [9,0,5] 9 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 0
8.0 4 0 0 0 0 16777216 0 0 48 48 active+clean 2022-04-04T21:05:31.145331+0800 215'48 378:342 [10,4,7] 10 [10,4,7] 10 215'48 2022-04-04T21:05:31.145094+0800 0'0 2022-04-03T17:52:00.867759+0800 0
4.c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:26.813073+0800 0'0 377:300 [4,9,5] 4 [4,9,5] 4 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
3.b 5 0 0 0 0 110 0 0 1166 1166 active+clean 2022-04-04T15:22:26.034474+0800 378'1166 378:2106 [11,8,6] 11 [11,8,6] 11 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.255909+0800 0'0 377:322 [6,1,9] 6 [6,1,9] 6 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 0
6.e 1 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T17:41:49.130913+0800 156'2 378:294 [4,1,9] 4 [4,1,9] 4 156'2 2022-04-04T17:41:49.130851+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:26.415463+0800 0'0 377:261 [11,5,9] 11 [11,5,9] 11 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.1 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.433283+0800 0'0 377:130 [9,2,0] 9 [9,2,0] 9 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
9.2 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:37.789651+0800 264'2 377:180 [8,1,6] 8 [8,1,6] 8 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 0
8.3 2 0 0 0 0 8388608 0 0 35 35 active+clean 2022-04-04T15:22:38.607483+0800 215'35 377:242 [8,11,2] 8 [8,11,2] 8 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.8 9 0 0 0 0 110 0 0 2247 2247 active+clean 2022-04-04T15:22:38.434298+0800 378'2247 378:3744 [10,1,7] 10 [10,1,7] 10 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.9 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:53:24.234989+0800 0'0 378:304 [11,4,9] 11 [11,4,9] 11 0'0 2022-04-04T21:53:24.234942+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:26.486645+0800 0'0 377:301 [11,9,7] 11 [11,9,7] 11 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.d 1 0 0 0 0 70 0 0 1 1 active+clean 2022-04-04T15:22:38.649490+0800 156'1 377:273 [8,1,5] 8 [8,1,5] 8 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:09:43.454919+0800 0'0 378:271 [2,1,10] 2 [2,1,10] 2 0'0 2022-04-04T17:09:43.454870+0800 0'0 2022-04-04T17:09:43.454870+0800 0
10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:00:00.023163+0800 0'0 377:138 [10,5,11] 10 [10,5,11] 10 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
9.3 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:33.692220+0800 0'0 377:170 [7,9,3] 7 [7,9,3] 7 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 0
8.2 0 0 0 0 0 0 0 0 30 30 active+clean 2022-04-04T15:22:22.637181+0800 215'30 377:213 [3,0,5] 3 [3,0,5] 3 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.9 7 0 0 0 0 220 0 0 1266 1266 active+clean 2022-04-04T15:22:27.979630+0800 378'1266 378:2297 [5,10,4] 5 [5,10,4] 5 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.8 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:06:31.891710+0800 0'0 378:328 [8,7,0] 8 [8,7,0] 8 0'0 2022-04-04T20:06:31.891403+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.e 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T15:22:27.031381+0800 251'37 378:20980 [4,10,1] 4 [4,10,1] 4 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.c 2 0 0 0 0 554 0 0 3 3 active+clean 2022-04-04T18:07:06.713164+0800 205'3 378:185 [0,2,10] 0 [0,2,10] 0 205'3 2022-04-04T18:07:06.713106+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:26.940530+0800 0'0 377:265 [4,9,0] 4 [4,9,0] 4 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.440535+0800 0'0 377:129 [10,6,3] 10 [10,6,3] 10 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.d 3 0 0 0 0 12582912 0 0 57 57 active+clean 2022-04-04T21:40:30.572238+0800 215'57 378:257 [5,2,11] 5 [5,2,11] 5 215'57 2022-04-04T21:40:30.572183+0800 0'0 2022-04-03T17:52:00.867759+0800 0
4.1 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:30:40.645642+0800 0'0 378:323 [8,2,5] 8 [8,2,5] 8 0'0 2022-04-04T17:30:40.645586+0800 0'0 2022-04-03T16:53:03.970520+0800 0
3.6 6 0 0 0 0 0 0 0 2135 2135 active+clean 2022-04-04T15:22:25.812750+0800 378'2135 378:3519 [0,8,2] 0 [0,8,2] 0 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.7 1 0 0 0 0 873 0 0 1 1 active+clean 2022-04-04T15:22:25.969214+0800 72'1 377:352 [3,10,1] 3 [3,10,1] 3 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 0
5.0 0 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T15:22:30.851441+0800 264'2 377:356 [5,1,4] 5 [5,1,4] 5 0'0 2022-04-03T16:53:06.037035+0800 0'0 2022-04-03T16:53:06.037035+0800 0
6.3 3 0 0 0 0 34 0 0 3 3 active+clean 2022-04-04T17:09:33.559693+0800 205'3 378:267 [4,9,7] 4 [4,9,7] 4 205'3 2022-04-04T17:09:33.559632+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.2 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:16:48.183999+0800 0'0 378:266 [0,6,10] 0 [0,6,10] 0 0'0 2022-04-04T20:16:48.183949+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.386047+0800 0'0 377:129 [4,0,8] 4 [4,0,8] 4 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.c 3 0 0 0 0 8396800 0 0 46 46 active+clean 2022-04-04T18:23:31.595717+0800 378'46 378:242 [5,11,8] 5 [5,11,8] 5 378'46 2022-04-04T18:23:31.595542+0800 378'46 2022-04-04T18:23:31.595542+0800 0
4.0 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T15:22:25.982758+0800 251'37 378:20973 [0,4,9] 0 [0,4,9] 0 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
3.7 6 0 0 0 0 330 0 0 985 985 active+clean 2022-04-04T15:22:26.471763+0800 378'985 378:1893 [1,6,9] 1 [1,6,9] 1 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.6 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:29:52.083452+0800 0'0 378:309 [1,6,10] 1 [1,6,10] 1 0'0 2022-04-04T19:29:52.083357+0800 0'0 2022-04-03T16:52:59.854233+0800 0
5.1 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:54:15.604320+0800 0'0 378:360 [3,11,7] 3 [3,11,7] 3 0'0 2022-04-04T20:54:15.604265+0800 0'0 2022-04-03T16:53:06.037035+0800 0
6.2 0 0 0 0 0 0 0 0 1 1 active+clean 2022-04-04T15:22:37.663652+0800 205'1 377:282 [9,0,6] 9 [9,0,6] 9 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.3 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:25.812616+0800 0'0 377:260 [0,4,8] 0 [0,4,8] 0 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.408682+0800 0'0 377:130 [2,0,8] 2 [2,0,8] 2 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.f 4 0 0 0 0 10616832 0 0 51 51 active+clean 2022-04-04T20:38:14.483470+0800 215'51 378:412 [7,11,10] 7 [7,11,10] 7 215'51 2022-04-04T20:38:14.483430+0800 0'0 2022-04-03T17:52:00.867759+0800 0
4.3 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T15:22:39.066818+0800 251'37 378:20972 [9,5,2] 9 [9,5,2] 9 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
3.4 11 0 0 0 0 358 0 0 2434 2434 active+clean 2022-04-04T20:34:55.875668+0800 378'2434 378:24713 [1,5,8] 1 [1,5,8] 1 378'2122 2022-04-04T20:34:55.875495+0800 378'2122 2022-04-04T20:34:55.875495+0800 0
2.5 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:32:42.186686+0800 0'0 378:329 [7,0,4] 7 [7,0,4] 7 0'0 2022-04-04T19:32:42.186631+0800 0'0 2022-04-03T16:52:59.854233+0800 0
5.2 1 0 0 0 0 16 0 0 3 3 active+clean 2022-04-04T20:59:33.010620+0800 264'3 378:363 [3,0,5] 3 [3,0,5] 3 264'3 2022-04-04T20:59:33.010572+0800 0'0 2022-04-03T16:53:06.037035+0800 0
6.1 2 0 0 0 0 90 0 0 8 8 active+clean 2022-04-04T15:22:27.605063+0800 236'8 377:280 [6,9,3] 6 [6,9,3] 6 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:01:10.984418+0800 0'0 377:271 [2,5,11] 2 [2,5,11] 2 0'0 2022-04-04T17:01:10.984350+0800 0'0 2022-04-04T17:01:10.984350+0800 0
10.8 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.465454+0800 0'0 377:130 [9,5,3] 9 [9,5,3] 9 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.a 1 0 0 0 0 4194304 0 0 26 26 active+clean 2022-04-04T15:22:25.970557+0800 215'26 377:209 [3,11,10] 3 [3,11,10] 3 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
4.6 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:30.851935+0800 0'0 377:320 [5,1,4] 5 [5,1,4] 5 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:24:50.397324+0800 0'0 378:330 [3,5,11] 3 [3,5,11] 3 0'0 2022-04-04T18:24:50.397271+0800 0'0 2022-04-03T16:52:59.854233+0800 0
3.1 8 0 0 0 0 0 0 0 2506 2506 active+clean 2022-04-04T18:09:55.879975+0800 378'2506 378:4093 [0,4,10] 0 [0,4,10] 0 378'1764 2022-04-04T18:09:55.879822+0800 378'1764 2022-04-04T18:09:55.879822+0800 0
5.7 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T22:10:29.553385+0800 0'0 378:358 [8,1,7] 8 [8,1,7] 8 0'0 2022-04-04T22:10:29.553338+0800 0'0 2022-04-03T16:53:06.037035+0800 0
6.4 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:25:06.358558+0800 0'0 378:280 [9,2,11] 9 [9,2,11] 9 0'0 2022-04-04T19:25:06.358503+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.5 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:30.830605+0800 0'0 377:264 [7,11,2] 7 [7,11,2] 7 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.384146+0800 0'0 377:130 [2,1,8] 2 [2,1,8] 2 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.9 7 0 0 0 0 25186304 0 0 48 48 active+clean 2022-04-04T15:22:30.434864+0800 215'48 377:276 [6,2,0] 6 [6,2,0] 6 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
4.5 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:57:31.289390+0800 0'0 378:324 [9,3,5] 9 [9,3,5] 9 0'0 2022-04-04T17:57:31.289325+0800 0'0 2022-04-03T16:53:03.970520+0800 0
1.0 1 0 0 0 0 0 0 0 1 1 active+clean 2022-04-04T15:22:38.985703+0800 239'1 377:400 [8,5,2] 8 [8,5,2] 8 0'0 2022-04-03T16:36:15.631796+0800 0'0 2022-04-03T16:36:15.631796+0800 0
2.3 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:39.118775+0800 0'0 377:323 [10,2,5] 10 [10,2,5] 10 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 0
3.2 6 0 0 0 0 0 0 0 1792 1792 active+clean 2022-04-04T15:22:39.064858+0800 378'1792 378:3021 [10,5,3] 10 [10,5,3] 10 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
5.4 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.458830+0800 0'0 377:351 [7,11,8] 7 [7,11,8] 7 0'0 2022-04-03T16:53:06.037035+0800 0'0 2022-04-03T16:53:06.037035+0800 0
6.7 2 0 0 0 0 0 0 0 4 4 active+clean 2022-04-04T15:22:38.756756+0800 205'4 377:282 [10,1,2] 10 [10,1,2] 10 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.6 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:51:15.297786+0800 0'0 378:265 [0,4,8] 0 [0,4,8] 0 0'0 2022-04-04T18:51:15.297215+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.9 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.432790+0800 0'0 377:130 [3,11,9] 3 [3,11,9] 3 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.b 4 0 0 0 0 16777216 0 0 36 36 active+clean 2022-04-04T20:48:28.233988+0800 215'36 378:239 [8,4,1] 8 [8,4,1] 8 215'36 2022-04-04T20:48:28.233939+0800 0'0 2022-04-03T17:52:00.867759+0800 0
4.7 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:15:14.120970+0800 0'0 378:309 [1,10,3] 1 [1,10,3] 1 0'0 2022-04-04T20:15:14.120924+0800 0'0 2022-04-03T16:53:03.970520+0800 0
2.1 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:03:44.457237+0800 0'0 378:329 [9,0,5] 9 [9,0,5] 9 0'0 2022-04-04T19:03:44.457189+0800 0'0 2022-04-03T16:52:59.854233+0800 0
3.0 10 0 0 0 0 220 0 0 2334 2334 active+clean 2022-04-04T15:22:26.860295+0800 378'2334 378:3899 [5,2,8] 5 [5,2,8] 5 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
5.6 1 0 0 0 0 0 0 0 4 4 active+clean 2022-04-04T17:13:19.513024+0800 264'4 378:362 [11,2,10] 11 [11,2,10] 11 264'4 2022-04-04T17:13:19.512947+0800 0'0 2022-04-03T16:53:06.037035+0800 0
6.5 1 0 0 0 0 22 0 0 1 1 active+clean 2022-04-04T15:22:22.843630+0800 205'1 377:277 [2,5,11] 2 [2,5,11] 2 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.4 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:32:27.335291+0800 0'0 378:265 [0,9,2] 0 [0,9,2] 0 0'0 2022-04-04T21:32:27.335179+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.a 0 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T16:59:38.421333+0800 260'2 377:133 [10,7,2] 10 [10,7,2] 10 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.8 1 0 0 0 0 4194304 0 0 23 23 active+clean 2022-04-04T18:24:20.752914+0800 378'23 378:223 [4,0,7] 4 [4,0,7] 4 378'23 2022-04-04T18:24:20.752712+0800 378'23 2022-04-04T18:24:20.752712+0800 0
4.4 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:03:04.126035+0800 0'0 378:327 [8,0,3] 8 [8,0,3] 8 0'0 2022-04-04T18:03:04.125992+0800 0'0 2022-04-03T16:53:03.970520+0800 0
3.3 9 0 0 0 0 330 0 0 2051 2051 active+clean 2022-04-04T21:05:52.223272+0800 378'2051 378:3499 [10,4,11] 10 [10,4,11] 10 378'1853 2022-04-04T21:05:52.222814+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.2 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:32:07.667626+0800 0'0 378:330 [3,1,10] 3 [3,1,10] 3 0'0 2022-04-04T17:32:07.667466+0800 0'0 2022-04-03T16:52:59.854233+0800 0
5.5 1 0 0 0 0 373 0 0 1 1 active+clean 2022-04-04T15:22:26.414028+0800 251'1 377:315 [11,9,4] 11 [11,9,4] 11 0'0 2022-04-03T16:53:06.037035+0800 0'0 2022-04-03T16:53:06.037035+0800 0
6.6 2 0 0 0 0 0 0 0 6 6 active+clean 2022-04-04T17:44:13.862917+0800 239'6 378:281 [1,4,8] 1 [1,4,8] 1 239'6 2022-04-04T17:44:13.862840+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.7 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:43:39.845403+0800 0'0 378:270 [3,6,11] 3 [3,6,11] 3 0'0 2022-04-04T20:43:39.845354+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.383389+0800 0'0 377:129 [4,8,0] 4 [4,8,0] 4 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.e 2 0 0 0 0 8388608 0 0 20 20 active+clean 2022-04-04T15:22:26.531259+0800 215'20 377:211 [11,10,3] 11 [11,10,3] 11 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
4.2 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.520565+0800 0'0 377:319 [9,3,7] 9 [9,3,7] 9 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
3.5 7 0 0 0 0 154 0 0 2136 2136 active+clean 2022-04-04T15:22:21.396047+0800 378'2136 378:3527 [2,0,6] 2 [2,0,6] 2 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.4 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:16:00.165750+0800 0'0 378:306 [11,7,8] 11 [11,7,8] 11 0'0 2022-04-04T20:16:00.165699+0800 0'0 2022-04-03T16:52:59.854233+0800 0
5.3 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:46:48.189526+0800 0'0 378:330 [0,3,9] 0 [0,3,9] 0 0'0 2022-04-04T19:46:48.189479+0800 0'0 2022-04-03T16:53:06.037035+0800 0
6.0 1 0 0 0 0 0 0 0 5 5 active+clean 2022-04-04T20:08:24.618717+0800 236'5 378:205 [9,2,1] 9 [9,2,1] 9 236'5 2022-04-04T20:08:24.618669+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.1 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:26.299528+0800 0'0 377:266 [2,10,11] 2 [2,10,11] 2 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.7 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.405322+0800 0'0 377:129 [10,1,5] 10 [10,1,5] 10 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
9.4 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:21.774699+0800 264'2 377:180 [1,3,8] 1 [1,3,8] 1 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 0
8.5 4 0 0 0 0 16777216 0 0 36 36 active+clean 2022-04-04T15:22:27.779712+0800 215'36 377:229 [6,8,1] 6 [6,8,1] 6 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
4.9 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T15:22:30.851404+0800 251'37 378:20974 [5,1,2] 5 [5,1,2] 5 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
3.e 6 0 0 0 0 110 0 0 1522 1522 active+clean 2022-04-04T17:07:05.531122+0800 378'1522 378:2647 [7,10,4] 7 [7,10,4] 7 303'972 2022-04-04T17:07:05.531053+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.f 2 0 0 0 0 92 0 0 2 2 active+clean 2022-04-04T17:52:32.137249+0800 72'2 378:391 [6,9,4] 6 [6,9,4] 6 72'2 2022-04-04T17:52:32.137206+0800 0'0 2022-04-03T16:52:59.854233+0800 0
6.b 2 0 0 0 0 0 0 0 5 5 active+clean 2022-04-04T15:22:26.931126+0800 236'5 377:268 [1,3,8] 1 [1,3,8] 1 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.255720+0800 0'0 377:272 [6,1,9] 6 [6,1,9] 6 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.18 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.352338+0800 0'0 377:130 [11,6,4] 11 [11,6,4] 11 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.1a 0 0 0 0 0 0 0 0 17 17 active+clean 2022-04-04T15:22:26.529111+0800 215'17 377:208 [11,2,10] 11 [11,2,10] 11 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.11 6 0 0 0 0 110 0 0 1523 1523 active+clean 2022-04-04T19:08:46.182759+0800 378'1523 378:2649 [7,4,10] 7 [7,4,10] 7 378'1181 2022-04-04T19:08:46.182707+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.10 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:05:50.647261+0800 0'0 378:328 [8,7,0] 8 [8,7,0] 8 0'0 2022-04-04T21:05:50.647215+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.16 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:03:03.580096+0800 0'0 378:324 [10,0,5] 10 [10,0,5] 10 0'0 2022-04-04T18:03:03.580023+0800 0'0 2022-04-04T18:03:03.580023+0800 0
6.14 5 0 0 0 0 2631 0 0 18 18 active+clean 2022-04-04T15:22:38.803098+0800 239'18 377:291 [8,4,7] 8 [8,4,7] 8 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.15 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.432336+0800 0'0 377:264 [10,1,7] 10 [10,1,7] 10 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.19 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.422600+0800 0'0 377:129 [10,4,0] 10 [10,4,0] 10 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.1b 5 0 0 0 0 20971520 0 0 41 41 active+clean 2022-04-04T22:31:24.260509+0800 215'41 378:241 [4,11,10] 4 [4,11,10] 4 215'41 2022-04-04T22:31:24.260461+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.10 2 0 0 0 0 0 0 0 712 712 active+clean 2022-04-04T15:22:39.016737+0800 378'712 378:1389 [10,5,3] 10 [10,5,3] 10 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.11 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:35:04.352429+0800 0'0 378:331 [8,3,1] 8 [8,3,1] 8 0'0 2022-04-04T18:35:04.352376+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.17 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.540411+0800 0'0 377:300 [11,3,7] 11 [11,3,7] 11 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.15 3 0 0 0 0 644 0 0 4 4 active+clean 2022-04-04T15:22:25.640509+0800 205'4 377:265 [2,8,6] 2 [2,8,6] 2 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.14 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.851667+0800 0'0 377:268 [9,0,2] 9 [9,0,2] 9 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.1a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.429669+0800 0'0 377:130 [3,9,5] 3 [3,9,5] 3 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.18 1 0 0 0 0 4194304 0 0 23 23 active+clean 2022-04-04T18:18:24.550345+0800 215'23 378:214 [4,5,8] 4 [4,5,8] 4 215'23 2022-04-04T18:18:24.550285+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.13 5 0 0 0 0 110 0 0 1510 1510 active+clean 2022-04-04T15:22:26.930763+0800 378'1510 378:2609 [3,9,5] 3 [3,9,5] 3 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.12 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:30.851860+0800 0'0 377:324 [5,11,2] 5 [5,11,2] 5 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.14 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T15:22:30.280192+0800 251'37 378:20975 [6,1,2] 6 [6,1,2] 6 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.16 1 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T15:22:26.073214+0800 205'2 377:260 [0,4,9] 0 [0,4,9] 0 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.17 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:33:53.636731+0800 0'0 378:198 [4,11,10] 4 [4,11,10] 4 0'0 2022-04-04T18:33:53.636682+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.1b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.439698+0800 0'0 377:130 [0,4,10] 0 [0,4,10] 0 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.19 0 0 0 0 0 0 0 0 25 25 active+clean 2022-04-04T22:31:20.767031+0800 215'25 378:219 [2,5,8] 2 [2,5,8] 2 215'25 2022-04-04T22:31:20.766986+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.12 4 0 0 0 0 110 0 0 1154 1154 active+clean 2022-04-04T22:10:30.175172+0800 378'1154 378:2088 [9,3,0] 9 [9,3,0] 9 378'1122 2022-04-04T22:10:30.175126+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.13 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:39.709864+0800 0'0 377:327 [9,4,11] 9 [9,4,11] 9 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.15 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T15:22:38.353223+0800 251'37 378:20976 [8,7,0] 8 [8,7,0] 8 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.17 2 0 0 0 0 90 0 0 5 5 active+clean 2022-04-04T15:22:38.541873+0800 239'5 377:279 [10,11,5] 10 [10,11,5] 10 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.16 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.682357+0800 0'0 377:263 [6,8,0] 6 [6,8,0] 6 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.1c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.417948+0800 0'0 377:130 [2,8,7] 2 [2,8,7] 2 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.1e 4 0 0 0 0 16777216 0 0 55 55 active+clean 2022-04-04T15:22:21.530728+0800 215'55 377:331 [1,3,6] 1 [1,3,6] 1 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.15 5 0 0 0 0 110 0 0 1170 1170 active+clean 2022-04-04T15:22:37.773222+0800 378'1170 378:2112 [8,6,3] 8 [8,6,3] 8 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.14 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:58:25.270496+0800 0'0 378:305 [2,7,8] 2 [2,7,8] 2 0'0 2022-04-04T19:58:25.270445+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.12 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:14:03.606387+0800 0'0 378:309 [1,6,9] 1 [1,6,9] 1 0'0 2022-04-04T18:14:03.606340+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.10 8 0 0 0 0 112 0 0 15 15 active+clean 2022-04-04T15:22:39.183032+0800 205'15 377:292 [10,5,2] 10 [10,5,2] 10 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.11 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.518884+0800 0'0 377:264 [9,3,7] 9 [9,3,7] 9 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.1d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.414890+0800 0'0 377:129 [4,7,9] 4 [4,7,9] 4 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.1f 2 0 0 0 0 8388608 0 0 22 22 active+clean 2022-04-04T15:22:30.830221+0800 215'22 377:217 [7,4,11] 7 [7,4,11] 7 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.14 7 0 0 0 0 0 0 0 1804 1804 active+clean 2022-04-04T15:22:26.042066+0800 378'1804 378:3053 [3,8,0] 3 [3,8,0] 3 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.15 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:57:11.512647+0800 0'0 378:329 [9,1,5] 9 [9,1,5] 9 0'0 2022-04-04T20:57:11.512130+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.13 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:52:07.692891+0800 0'0 378:305 [4,8,6] 4 [4,8,6] 4 0'0 2022-04-04T20:52:07.692841+0800 0'0 2022-04-04T20:52:07.692841+0800 0
6.11 2 0 0 0 0 540 0 0 3 3 active+clean 2022-04-04T15:22:37.786957+0800 205'3 377:275 [8,6,11] 8 [8,6,11] 8 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.10 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T22:32:44.867665+0800 0'0 378:265 [11,10,5] 11 [11,10,5] 11 0'0 2022-04-04T22:32:44.867608+0800 0'0 2022-04-04T22:32:44.867608+0800 0
10.1e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.438344+0800 0'0 377:130 [5,9,0] 5 [5,9,0] 5 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.1c 3 0 0 0 0 4214784 0 0 41 41 active+clean 2022-04-04T15:22:38.468477+0800 215'41 377:257 [8,2,7] 8 [8,2,7] 8 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.17 4 0 0 0 0 0 0 0 1081 1081 active+clean 2022-04-04T15:22:38.843955+0800 378'1081 378:1954 [10,3,5] 10 [10,3,5] 10 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.16 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:40:03.197630+0800 0'0 378:305 [2,7,10] 2 [2,7,10] 2 0'0 2022-04-04T17:40:03.197577+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.10 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:18:30.917058+0800 0'0 378:309 [1,9,6] 1 [1,9,6] 1 0'0 2022-04-04T20:18:30.916510+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.12 1 0 0 0 0 0 0 0 5 5 active+clean 2022-04-04T15:22:27.436176+0800 236'5 377:278 [7,4,9] 7 [7,4,9] 7 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.13 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:59:13.229303+0800 0'0 378:268 [10,2,5] 10 [10,2,5] 10 0'0 2022-04-04T20:59:13.229253+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.1f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.374259+0800 0'0 377:130 [2,1,6] 2 [2,1,6] 2 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.1d 0 0 0 0 0 0 0 0 17 17 active+clean 2022-04-04T21:55:48.422035+0800 215'17 378:213 [0,10,5] 0 [0,10,5] 0 215'17 2022-04-04T21:55:48.421979+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.16 6 0 0 0 0 110 0 0 1179 1179 active+clean 2022-04-04T18:31:53.906728+0800 378'1179 378:2148 [2,1,9] 2 [2,1,9] 2 378'875 2022-04-04T18:31:53.906677+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.17 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.315533+0800 0'0 377:325 [4,5,11] 4 [4,5,11] 4 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.11 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:25.820179+0800 0'0 377:300 [0,4,8] 0 [0,4,8] 0 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.13 4 0 0 0 0 34 0 0 6 6 active+clean 2022-04-04T17:59:22.017997+0800 205'6 378:286 [8,1,7] 8 [8,1,7] 8 205'6 2022-04-04T17:59:22.017942+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.12 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.708233+0800 0'0 377:266 [10,11,2] 10 [10,11,2] 10 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.10 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.463131+0800 0'0 377:130 [9,4,0] 9 [9,4,0] 9 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.12 2 0 0 0 0 4202496 0 0 23 23 active+clean 2022-04-04T18:38:29.110382+0800 215'23 378:244 [3,9,6] 3 [3,9,6] 3 215'23 2022-04-04T18:38:29.110332+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.1d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.318007+0800 0'0 377:261 [5,9,0] 5 [5,9,0] 5 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
3.19 11 0 0 0 0 110 0 0 2271 2271 active+clean 2022-04-04T15:22:25.886617+0800 378'2271 378:3803 [1,4,10] 1 [1,4,10] 1 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.18 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:37.469975+0800 0'0 377:323 [9,2,6] 9 [9,2,6] 9 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.1e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:46:58.994342+0800 0'0 378:305 [0,4,8] 0 [0,4,8] 0 0'0 2022-04-04T21:46:58.994286+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.1c 1 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T19:21:58.590310+0800 156'2 378:280 [7,11,3] 7 [7,11,3] 7 156'2 2022-04-04T19:21:58.590251+0800 0'0 2022-04-03T16:54:54.909163+0800 0
10.11 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.410495+0800 0'0 377:130 [5,0,2] 5 [5,0,2] 5 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.13 3 0 0 0 0 12582912 0 0 34 34 active+clean 2022-04-04T21:08:38.667888+0800 215'34 378:231 [1,7,4] 1 [1,7,4] 1 215'34 2022-04-04T21:08:38.667292+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.18 5 0 0 0 0 110 0 0 1166 1166 active+clean 2022-04-04T15:22:37.764288+0800 378'1166 378:2109 [10,6,11] 10 [10,6,11] 10 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.19 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.540510+0800 0'0 377:300 [11,4,7] 11 [11,4,7] 11 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.1f 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T19:35:28.972562+0800 251'37 378:20986 [6,8,1] 6 [6,8,1] 6 251'37 2022-04-04T19:35:28.972508+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.1d 2 0 0 0 0 0 0 0 4 4 active+clean 2022-04-04T15:22:38.832675+0800 205'4 377:277 [8,5,1] 8 [8,5,1] 8 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.1c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:25.735065+0800 0'0 377:260 [2,5,9] 2 [2,5,9] 2 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.12 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.422445+0800 0'0 377:129 [10,5,2] 10 [10,5,2] 10 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.10 0 0 0 0 0 0 0 0 24 24 active+clean 2022-04-04T15:22:25.973210+0800 215'24 377:212 [3,5,10] 3 [3,5,10] 3 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.1f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.828665+0800 0'0 377:264 [8,5,1] 8 [8,5,1] 8 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
3.1b 3 0 0 0 0 110 0 0 455 455 active+clean 2022-04-04T20:08:00.827659+0800 378'455 378:1048 [6,4,8] 6 [6,4,8] 6 378'385 2022-04-04T20:08:00.827619+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.1a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:05:56.636457+0800 0'0 378:330 [3,5,1] 3 [3,5,1] 3 0'0 2022-04-04T19:05:56.636388+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.1c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:07:01.035323+0800 0'0 378:325 [5,1,4] 5 [5,1,4] 5 0'0 2022-04-04T17:07:01.035274+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.1e 0 0 0 0 0 0 0 0 3 3 active+clean 2022-04-04T22:07:55.342125+0800 239'3 378:280 [5,2,11] 5 [5,2,11] 5 239'3 2022-04-04T22:07:55.342047+0800 0'0 2022-04-03T16:54:54.909163+0800 0
10.13 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.421461+0800 0'0 377:130 [1,8,3] 1 [1,8,3] 1 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.11 6 0 0 0 0 25165824 0 0 30 30 active+clean 2022-04-04T15:22:26.860607+0800 215'30 377:222 [5,8,2] 5 [5,8,2] 5 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
7.1e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.312930+0800 0'0 377:260 [11,5,2] 11 [11,5,2] 11 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
3.1a 6 0 0 0 0 110 0 0 1179 1179 active+clean 2022-04-04T15:22:27.318326+0800 378'1179 378:2139 [5,11,9] 5 [5,11,9] 5 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.1b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.318197+0800 0'0 377:322 [5,2,9] 5 [5,2,9] 5 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.1d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:35:03.322133+0800 0'0 378:323 [7,9,3] 7 [7,9,3] 7 0'0 2022-04-04T21:35:03.322071+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.1f 4 0 0 0 0 554 0 0 10 10 active+clean 2022-04-04T17:59:11.539509+0800 236'10 378:271 [11,10,3] 11 [11,10,3] 11 236'10 2022-04-04T17:59:11.539462+0800 0'0 2022-04-03T16:54:54.909163+0800 0
10.14 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.393571+0800 0'0 377:130 [1,8,6] 1 [1,8,6] 1 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.16 1 0 0 0 0 8192 0 0 28 28 active+clean 2022-04-04T18:30:09.100578+0800 215'28 378:231 [9,0,2] 9 [9,0,2] 9 215'28 2022-04-04T18:30:09.100528+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.1d 6 0 0 0 0 220 0 0 1253 1253 active+clean 2022-04-04T19:48:25.408431+0800 378'1253 378:2270 [8,4,7] 8 [8,4,7] 8 378'1025 2022-04-04T19:48:25.408373+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.1c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:30.830653+0800 0'0 377:324 [7,4,1] 7 [7,4,1] 7 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.1a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:37.787869+0800 0'0 377:319 [8,11,6] 8 [8,11,6] 8 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.18 0 0 0 0 0 0 0 0 3 3 active+clean 2022-04-04T17:36:11.185304+0800 236'3 378:278 [11,8,7] 11 [11,8,7] 11 236'3 2022-04-04T17:36:11.185258+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.19 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:01:23.522037+0800 0'0 378:272 [8,11,4] 8 [8,11,4] 8 0'0 2022-04-04T19:01:23.521994+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.15 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.454770+0800 0'0 377:130 [11,9,2] 11 [11,9,2] 11 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.17 1 0 0 0 0 4194304 0 0 21 21 active+clean 2022-04-04T15:22:25.735027+0800 215'21 377:210 [2,9,6] 2 [2,9,6] 2 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.1c 7 0 0 0 0 110 0 0 2222 2222 active+clean 2022-04-04T17:12:49.656665+0800 378'2222 378:3688 [4,11,9] 4 [4,11,9] 4 378'1432 2022-04-04T17:12:49.656611+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.1d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:03:30.277360+0800 0'0 378:328 [10,6,3] 10 [10,6,3] 10 0'0 2022-04-04T21:03:30.277308+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.1b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.670932+0800 0'0 377:323 [9,1,4] 9 [9,1,4] 9 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.19 1 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T18:51:08.770487+0800 205'2 378:279 [10,1,7] 10 [10,1,7] 10 205'2 2022-04-04T18:51:08.770422+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.18 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:25.815539+0800 0'0 377:262 [0,8,3] 0 [0,8,3] 0 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.16 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.398330+0800 0'0 377:130 [2,1,9] 2 [2,1,9] 2 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.14 1 0 0 0 0 4194304 0 0 19 19 active+clean 2022-04-04T15:22:27.334764+0800 215'19 377:214 [4,8,11] 4 [4,8,11] 4 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.1f 6 0 0 0 0 110 0 0 1523 1523 active+clean 2022-04-04T16:56:19.427764+0800 378'1523 378:2653 [10,0,3] 10 [10,0,3] 10 251'957 2022-04-04T16:56:19.427716+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.1e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:16:01.298937+0800 0'0 378:306 [11,5,9] 11 [11,5,9] 11 0'0 2022-04-04T17:16:01.298887+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.18 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:46:56.364832+0800 0'0 378:326 [2,9,1] 2 [2,9,1] 2 0'0 2022-04-04T21:46:56.364784+0800 0'0 2022-04-04T21:46:56.364784+0800 0
6.1a 1 0 0 0 0 0 0 0 4 4 active+clean 2022-04-04T19:05:38.955643+0800 236'4 378:288 [10,0,5] 10 [10,0,5] 10 236'4 2022-04-04T19:05:38.955598+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.1b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:25.958790+0800 0'0 377:262 [0,5,8] 0 [0,5,8] 0 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10.17 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.460120+0800 0'0 377:130 [9,7,4] 9 [9,7,4] 9 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 0
8.15 6 0 0 0 0 22712320 0 0 78 78 active+clean 2022-04-04T15:22:30.830236+0800 215'78 377:679 [7,2,1] 7 [7,2,1] 7 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 0
3.1e 6 0 0 0 0 0 0 0 1793 1793 active+clean 2022-04-04T20:43:14.098177+0800 378'1793 378:3024 [11,6,8] 11 [11,6,8] 11 378'1573 2022-04-04T20:43:14.098134+0800 0'0 2022-04-03T16:53:01.903832+0800 0
2.1f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.313511+0800 0'0 377:300 [11,3,5] 11 [11,3,5] 11 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 0
4.19 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.197109+0800 0'0 377:317 [7,0,10] 7 [7,0,10] 7 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 0
6.1b 2 0 0 0 0 1638 0 0 14 14 active+clean 2022-04-04T15:22:27.569007+0800 239'14 377:292 [6,10,3] 6 [6,10,3] 6 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 0
7.1a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T22:02:20.057612+0800 0'0 378:264 [3,5,10] 3 [3,5,10] 3 0'0 2022-04-04T22:02:20.057536+0800 0'0 2022-04-03T16:55:03.148848+0800 0
10 0 0 0 0 0 0 0 0 2 2
9 0 0 0 0 0 0 0 0 0 0
8 176 0 0 0 0 624996403 0 0 2175 2175
7 0 0 0 0 0 0 0 0 0 0
2 4 0 0 0 0 1323 0 0 4 4
1 1 0 0 0 0 0 0 0 1 1
3 209 0 0 0 0 3702 0 0 54098 54098
4 8 0 0 0 0 0 0 0 296 296
5 3 0 0 0 0 389 0 0 10 10
6 60 0 0 0 0 7125 0 0 148 148
sum 461 0 0 0 0 625008942 0 0 56734 56734
OSD_STAT USED AVAIL USED_RAW TOTAL HB_PEERS PG_SUM PRIMARY_PG_SUM
10 144 MiB 9.9 GiB 144 MiB 10 GiB [0,1,2,3,4,5,6,7,9,11] 80 30
9 192 MiB 9.8 GiB 192 MiB 10 GiB [0,1,2,3,4,5,6,7,8,10,11] 89 31
8 193 MiB 9.8 GiB 193 MiB 10 GiB [0,1,2,3,4,5,6,7,9,11] 80 30
7 162 MiB 9.8 GiB 162 MiB 10 GiB [0,1,2,3,4,6,8,9,10,11] 69 19
5 191 MiB 9.8 GiB 191 MiB 10 GiB [0,1,2,3,4,6,8,9,10,11] 88 29
6 197 MiB 9.8 GiB 197 MiB 10 GiB [0,1,2,3,4,5,7,8,9,10,11] 64 19
0 177 MiB 9.8 GiB 177 MiB 10 GiB [1,2,3,4,5,6,7,8,9,10,11] 73 22
1 198 MiB 9.8 GiB 198 MiB 10 GiB [0,2,3,4,5,6,7,8,9,10] 77 27
11 160 MiB 9.8 GiB 160 MiB 10 GiB [0,2,3,4,5,6,7,8,9,10] 72 25
4 193 MiB 9.8 GiB 193 MiB 10 GiB [0,1,3,5,6,7,8,9,10,11] 76 22
2 198 MiB 9.8 GiB 198 MiB 10 GiB [0,1,3,5,6,7,8,9,10,11] 74 23
3 95 MiB 9.9 GiB 95 MiB 10 GiB [0,1,2,4,5,6,7,8,9,10,11] 73 28
sum 2.1 GiB 118 GiB 2.1 GiB 120 GiB
* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details.
dumped all
4) CRUSH map: 它保存的信息包括集群设备列表、bucket列表、故障域（failure domain）分层结构、保存数据时用到的为故障域定义的规则（rules）等。查看crush map命令
cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd crush dump
{
"devices": [
{
"id": 0,
"name": "osd.0",
"class": "hdd"
},
{
"id": 1,
"name": "osd.1",
"class": "hdd"
},
{
"id": 2,
"name": "osd.2",
"class": "hdd"
},
{
"id": 3,
"name": "osd.3",
"class": "hdd"
},
{
"id": 4,
"name": "osd.4",
"class": "hdd"
},
{
"id": 5,
"name": "osd.5",
"class": "hdd"
},
{
"id": 6,
"name": "osd.6",
"class": "hdd"
},
{
"id": 7,
"name": "osd.7",
"class": "hdd"
},
{
"id": 8,
"name": "osd.8",
"class": "hdd"
},
{
"id": 9,
"name": "osd.9",
"class": "hdd"
},
{
"id": 10,
"name": "osd.10",
"class": "hdd"
},
{
"id": 11,
"name": "osd.11",
"class": "hdd"
}
],
"types": [
{
"type_id": 0,
"name": "osd"
},
{
"type_id": 1,
"name": "host"
},
{
"type_id": 2,
"name": "chassis"
},
{
"type_id": 3,
"name": "rack"
},
{
"type_id": 4,
"name": "row"
},
{
"type_id": 5,
"name": "pdu"
},
{
"type_id": 6,
"name": "pod"
},
{
"type_id": 7,
"name": "room"
},
{
"type_id": 8,
"name": "datacenter"
},
{
"type_id": 9,
"name": "zone"
},
{
"type_id": 10,
"name": "region"
},
{
"type_id": 11,
"name": "root"
}
],
"buckets": [
{
"id": -1,
"name": "default",
"type_id": 11,
"type_name": "root",
"weight": 7704,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": -3,
"weight": 1926,
"pos": 0
},
{
"id": -5,
"weight": 1926,
"pos": 1
},
{
"id": -7,
"weight": 1926,
"pos": 2
},
{
"id": -9,
"weight": 1926,
"pos": 3
}
]
},
{
"id": -2,
"name": "default~hdd",
"type_id": 11,
"type_name": "root",
"weight": 7704,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": -4,
"weight": 1926,
"pos": 0
},
{
"id": -6,
"weight": 1926,
"pos": 1
},
{
"id": -8,
"weight": 1926,
"pos": 2
},
{
"id": -10,
"weight": 1926,
"pos": 3
}
]
},
{
"id": -3,
"name": "ceph-node1",
"type_id": 1,
"type_name": "host",
"weight": 1926,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": 0,
"weight": 642,
"pos": 0
},
{
"id": 1,
"weight": 642,
"pos": 1
},
{
"id": 11,
"weight": 642,
"pos": 2
}
]
},
{
"id": -4,
"name": "ceph-node1~hdd",
"type_id": 1,
"type_name": "host",
"weight": 1926,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": 0,
"weight": 642,
"pos": 0
},
{
"id": 1,
"weight": 642,
"pos": 1
},
{
"id": 11,
"weight": 642,
"pos": 2
}
]
},
{
"id": -5,
"name": "ceph-node2",
"type_id": 1,
"type_name": "host",
"weight": 1926,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": 2,
"weight": 642,
"pos": 0
},
{
"id": 3,
"weight": 642,
"pos": 1
},
{
"id": 4,
"weight": 642,
"pos": 2
}
]
},
{
"id": -6,
"name": "ceph-node2~hdd",
"type_id": 1,
"type_name": "host",
"weight": 1926,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": 2,
"weight": 642,
"pos": 0
},
{
"id": 3,
"weight": 642,
"pos": 1
},
{
"id": 4,
"weight": 642,
"pos": 2
}
]
},
{
"id": -7,
"name": "ceph-node3",
"type_id": 1,
"type_name": "host",
"weight": 1926,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": 5,
"weight": 642,
"pos": 0
},
{
"id": 6,
"weight": 642,
"pos": 1
},
{
"id": 7,
"weight": 642,
"pos": 2
}
]
},
{
"id": -8,
"name": "ceph-node3~hdd",
"type_id": 1,
"type_name": "host",
"weight": 1926,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": 5,
"weight": 642,
"pos": 0
},
{
"id": 6,
"weight": 642,
"pos": 1
},
{
"id": 7,
"weight": 642,
"pos": 2
}
]
},
{
"id": -9,
"name": "ceph-node4",
"type_id": 1,
"type_name": "host",
"weight": 1926,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": 8,
"weight": 642,
"pos": 0
},
{
"id": 9,
"weight": 642,
"pos": 1
},
{
"id": 10,
"weight": 642,
"pos": 2
}
]
},
{
"id": -10,
"name": "ceph-node4~hdd",
"type_id": 1,
"type_name": "host",
"weight": 1926,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": 8,
"weight": 642,
"pos": 0
},
{
"id": 9,
"weight": 642,
"pos": 1
},
{
"id": 10,
"weight": 642,
"pos": 2
}
]
}
],
"rules": [
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
],
"tunables": {
"choose_local_tries": 0,
"choose_local_fallback_tries": 0,
"choose_total_tries": 50,
"chooseleaf_descend_once": 1,
"chooseleaf_vary_r": 1,
"chooseleaf_stable": 1,
"straw_calc_version": 1,
"allowed_bucket_algs": 54,
"profile": "jewel",
"optimal_tunables": 1,
"legacy_tunables": 0,
"minimum_required_version": "jewel",
"require_feature_tunables": 1,
"require_feature_tunables2": 1,
"has_v2_rules": 0,
"require_feature_tunables3": 1,
"has_v3_rules": 0,
"has_v4_buckets": 1,
"require_feature_tunables5": 1,
"has_v5_rules": 0
},
"choose_args": {}
}
5) MDS map: 它保存的信息包括MDS map当前版本号（epoch）、MDS map的创建和修改时间、数据和元数据存储池的ID、集群MDS数量以及MDS状态。查看命令
cephadmin@ceph-deploy:~/ceph-cluster$ ceph mds metadata
[
{
"name": "ceph-mds2",
"addr": "[v2:192.168.1.204:6800/2803369705,v1:192.168.1.204:6801/2803369705]",
"arch": "x86_64",
"ceph_release": "pacific",
"ceph_version": "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)",
"ceph_version_short": "16.2.7",
"cpu": "11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz",
"distro": "ubuntu",
"distro_description": "Ubuntu 18.04.5 LTS",
"distro_version": "18.04",
"hostname": "ceph-mon2",
"kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020",
"kernel_version": "4.15.0-112-generic",
"mem_swap_kb": "969960",
"mem_total_kb": "1008784",
"os": "Linux"
},
{
"name": "ceph-mds1",
"addr": "[v2:192.168.1.203:6800/3979017759,v1:192.168.1.203:6801/3979017759]",
"arch": "x86_64",
"ceph_release": "pacific",
"ceph_version": "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)",
"ceph_version_short": "16.2.7",
"cpu": "11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz",
"distro": "ubuntu",
"distro_description": "Ubuntu 18.04.5 LTS",
"distro_version": "18.04",
"hostname": "ceph-mon1",
"kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020",
"kernel_version": "4.15.0-112-generic",
"mem_swap_kb": "969960",
"mem_total_kb": "1008784",
"os": "Linux"
},
{
"name": "ceph-mds3",
"addr": "[v2:192.168.1.205:6800/2412475465,v1:192.168.1.205:6801/2412475465]",
"arch": "x86_64",
"ceph_release": "pacific",
"ceph_version": "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)",
"ceph_version_short": "16.2.7",
"cpu": "11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz",
"distro": "ubuntu",
"distro_description": "Ubuntu 18.04.5 LTS",
"distro_version": "18.04",
"hostname": "ceph-mon3",
"kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020",
"kernel_version": "4.15.0-112-generic",
"mem_swap_kb": "969960",
"mem_total_kb": "1008784",
"os": "Linux"
}
]
# 查看版本
cephadmin@ceph-deploy:~/ceph-cluster$ ceph mds versions
{
"ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 3
}
# 查看状态
cephadmin@ceph-deploy:~/ceph-cluster$ ceph mds stat
mycephfs:2 {0=ceph-mds3=up:active,1=ceph-mds1=up:active} 1 up:standby
#1、从monitor节点上获取CRUSH map ceph osd getcrushmap -o crushmap_file #获取的crushmap_file文件为二进制文件 #2、获取该crushmap文件后，编译为可读文件 crushtool -d crushmap_file -o crushmap_file.txt #ubuntu无crushtool命令需要安装ceph-base软件包 #3、修改完成后，再次将crushmap_file.txt编译为二进制文件 crushtool -c crushmap_file.txt -o newcrushmap #4、将新的crushmap注入到ceph集群 ceph osd setcrushmap -i newcrushmap
cephadmin@ceph-deploy:~/ceph-cluster$ cat crushmap_file.txt
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54
# devices
#CRUSH map 文件的 devices 部分保存了 Ceph 集群中所有 OSD 设备。OSD 是和 ceph-osd 守护进程一一对应的物理磁盘。要将 PG 映射到 OSD，CRUSH 需要 OSD 设备列表。
#一般来说， 一个OSD映射到一个单独的硬盘或raid
device 0 osd.0 class hdd
device 1 osd.1 class hdd
device 2 osd.2 class hdd
device 3 osd.3 class hdd
device 4 osd.4 class hdd
device 5 osd.5 class hdd
device 6 osd.6 class hdd
device 7 osd.7 class hdd
device 8 osd.8 class hdd
device 9 osd.9 class hdd
device 10 osd.10 class hdd
device 11 osd.11 class hdd
# types
#CRUSHmap 文件的 types 部分定义了在 CRUSH 层次结构中用到的 bucket 类型。Bucket 由物理位置（例如， 行、机架、机箱、主机等）的分层聚合以及它们被分配的权重（weights）组成。它们使用节点（nodes）和叶子（leaves）两种层次，节点 bucket 表示物理位置，能够和层次结构中的其他节点和叶子 bucket 相聚合；叶子 bucket 表示cephosd守护进程和它们的底层物理设备。
type 0 osd #一个OSD守护进程（如：osd.1,osd.2等）
type 1 host #一个包含若干OSD的节点（如：主机）
type 2 chassis
type 3 rack #一个包含若干节点的机架
type 4 row #跨一组机架的一行
type 5 pdu
type 6 pod
type 7 room #一个包含若干机架和节点行的房间
type 8 datacenter #一个包含若干房间的物理数据中心
type 9 zone
type 10 region
type 11 root #这是bucket分层结构的根
# buckets
#Bucket instances(bucket 实例)逻辑的
#在定义了 bucket type 之后，需要为所有主机声明 bucket 的实例。声明 bucket 实例时，必须指定其 bucket 类型、一个唯一的名称（字符串）、一个用负整数表示的唯一的ID、和主机的总设备容量相关的权重（weight）、bucket 算法（默认为 straw），以及哈希算法（默认为0，表示使用 CRUSH哈希算法 rjenkins1）。一个 bucket 可以有一个或者多个项目（items），项目可能包含其他的 bucket 或者 OSD。每个项目都要有权重，它表示该项目的相对权重。
#bucket 实例的语法
[bucket-type] [bucket-name] {
id [a unique negative number ID]
weight [the relative capacity the item]
alg [the bucket type: uniform|list|tree|straw|straw2]
hash [the hash type: 0 by default]
item [item-name] weight [weight]
}
#bucket实例参数
bucket-type: bucket的类型，用来指定OSD在CRUSH分层结构中的位置。
bucket-name: 唯一的bucket名称
id:唯一的ID，用一个负整数表示
weight: Ceph将要在集群所有磁盘上均匀地写数据，这将有助于更好的分布数据和保证集群性能。这就要求所有的磁盘无论其容量大小都要被加入到集群中，被同等地利用。要达到该要求，需要Ceph使用加权机制（weighting mechanism）CRUSH给每个OSD分配一个权重。OSD的权重越高，说明它的物理存储容量越大。权重表示设备容量之间的相对差异。例如：1TB的存储设备设置相对权重1.00。相对的，权重0.5表示大约500G容量，权重3.00表示大约3TB容量。
alg: Ceph提供多个bucket类型的算法选择。每个算法都是在性能和重组效率之间的一种妥协。它包含uniform|list|tree|straw|straw2
uniform:当所有存储设备的权重都统一时可以使用这种类型的bucket,当权重不统一时不能使用。在这种类型的bucket中添加或者删除设备将会要求数据重新放置（reshuffling of data）,uniform类型缺乏效率。
list:链表类型的bucket将它们的内容聚合成链表（linked list）,可以包含任意权重的存储设备。当集群扩展时，新存储设备会加到链表头上，因此数据迁移将最少，然而，移除设备存储会产生相当多的数据移动。因此，这种类型只适合于很少或者从不添加新设备到集群的场景。另外，链表型bucket对小集合的项目时非常高效的，但不合适大集合项目。
tree:树形bucket将项目保存在一棵二叉树中。当一个bucket包含大集合项目时，会比链表式bucket更高效。树形bucket被组织为一棵加权二叉搜索数，项目都位于叶子节点上。每个根节点（interior node）都知道它的左子树和右子树的总权重，而且根据一个固定策略被打上了标签。树型bucket是全能的，它能提供优异的性能和相当不错的重组效率。
straw:在列表和树型bucket中选择一个项目时，需要计算一定数量的哈希值以及比较权重。它们使用分而治之的策略，给一些特定的项目赋予优先级（例如，哪些在列表开头的项目）。这会改善副本放置过程的性能，但在bucket项目被添加、删除、或调整权重时将会引入适量的重组。straw bucket类型允许所有的项目在副本放置时公平竞争。在有项目删除但是重组效率又是非常关键的场景中，straw类型的bucket将提供最佳的子树之间数据迁移行为。这种bucket类型允许在放置副本时所有项目公平竞争。
straw2:这是进过改进的straw bucket类型，它会在项目A和B的权重都没有改变时避免任何数据移动。换句话说，当增加一个设备给项目C从而改变它的权重后，或者删除项目C以后，数据只会移动到它上面或者从它上面移动到其它地址，而不会在bucket内的其他项目之间出现数据移动。因此，straw2 bucket算法减少了集群发生了改变后的数据移动。
hash: 每个bucket都具有哈希算法。目前，ceph支持rjenkins1算法。将hash值设为0，就将使用rjenkins1算法。
item:一个bucket可能有一个或者多个项目。这些项目可能包含若干节点bucket或者叶子bucket。项目的权重值（weight）放映了它的相对加权。
#bucket的真实案例
host ceph-node1 {
id -3 # do not change unnecessarily
id -4 class hdd # do not change unnecessarily
# weight 0.029
alg straw2
hash 0 # rjenkins1
item osd.0 weight 0.010
item osd.1 weight 0.010
item osd.11 weight 0.010
}
host ceph-node2 {
id -5 # do not change unnecessarily
id -6 class hdd # do not change unnecessarily
# weight 0.029
alg straw2
hash 0 # rjenkins1
item osd.2 weight 0.010
item osd.3 weight 0.010
item osd.4 weight 0.010
}
host ceph-node3 {
id -7 # do not change unnecessarily
id -8 class hdd # do not change unnecessarily
# weight 0.029
alg straw2
hash 0 # rjenkins1
item osd.5 weight 0.010
item osd.6 weight 0.010
item osd.7 weight 0.010
}
host ceph-node4 {
id -9 # do not change unnecessarily
id -10 class hdd # do not change unnecessarily
# weight 0.029
alg straw2
hash 0 # rjenkins1
item osd.8 weight 0.010
item osd.9 weight 0.010
item osd.10 weight 0.010
}
root default {
id -1 # do not change unnecessarily
id -2 class hdd # do not change unnecessarily
# weight 0.118
alg straw2
hash 0 # rjenkins1
item ceph-node1 weight 0.029
item ceph-node2 weight 0.029
item ceph-node3 weight 0.029
item ceph-node4 weight 0.029
}
# rules
#CRUSH map 包含了若干 CRUSH rules 来决定存储池内的数据存放方式。它们定义了存储池的属性，以及存储池中数据的存放方式。它们指定了复制（replication）和放置（placement）策略，该策略允许 CRUSH 将数据保存在 Ceph 集群中。中。默认的 CRUSH map 包含了适用于默认存储池 rbd 的一条规则。
#rules 的语法
rule <rulename>{
ruleset <ruleset>
type [replicated|erasure]
min_size <min_size>
max_size <mas_size>
step take <bucket-type>
step [choose|chooseleaf] [firstn] <num>
<bucket-type>
step emit
}
#rules 的参数
1. ruleset：一个整数值，它指定了这条规则所属的规则集。
2. type：一个字符串值，它指定存储池类型是复制型（replicated）还是纠删码类型（erasurecoded）。
3. min_size：一个整数值，如果存储池的副本份数小于该值，CRUSH 将不会为该存储池使用这条规则。
4. max_size：一个整数值，如果存储池的副本份数大于该值，CRUSH 将不会为该存储池使用这条规则。
5. steptake：获取一个 bucket 名称，开始遍历其树。
6. step choose firstn {num} type {busket-type}：选择某类型的若干（N）bucket，这里数字 N 通常是存储池的副本
份数（numberofreplicas）。
如果 num==0，选择 N 个 bucket。
如果 num>0 并且 num<N，选择 num 个 bucket。
如果 num<0，选择 N-num 个 bucket
例如，step choose firstn 1 type row。
本例中，num=1，假设存储池的副本份数也就是大小（poolsize）为 3，那么 CRUSH 会判断出 1>0 并且 1<3，因此，它会选择 1 个 row 类型 bucket。
7. Step chooseleaf firstn {num} type {bucket-type}：首先选择指定 bucket 类型的一组 bucket，然后从每个bucket 的子树中选择叶子节点。该组中 bucket 的数目 N 通常是该存储池的副本份数。
如果 num==0，选择 N 个 bucket。
如果 num>0 并且 num<N，选择 num 个 bucket。
如果 num<0，选择 N-num 个 bucket
例如， step chooseleaf firstn 0 type row。
本例中，num=0，假设存储池大小（poolsize）为 3，CRUSH 会判断条件 0==0，因此它会选择包含 3 个 bucket 的一个 row 类型 bucket 集合。然后它会从每个 bucket 的子树中选择叶子节点。此时，CRUSH 将会选择 3 个叶子节点。
8. stepemit：它首先弹出当前值，并清空栈。它会被典型地应用于 rule 结尾，也可用于组织同一条 rule 的不同树。
#rules的真实案例
rule replicated_rule {
id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
# end crush map
7、crush map 的使用示例
将osd.0 osd.4 osd.7和osd.10假设为机械硬盘SSD
编辑运文本格式运行图：
#ywx HDD node
host ceph-ssdnode1 {
id -103 # do not change unnecessarily
id -104 class hdd # do not change unnecessarily
# weight 0.098
alg straw2
hash 0 # rjenkins1
item osd.0 weight 0.098
}
host ceph-ssdhddnode2 {
id -105 # do not change unnecessarily
id -106 class hdd # do not change unnecessarily
# weight 0.098
alg straw2
hash 0 # rjenkins1
item osd.4 weight 0.098
}
host ceph-ssdnode3 {
id -107 # do not change unnecessarily
id -108 class hdd # do not change unnecessarily
# weight 0.098
alg straw2
hash 0 # rjenkins1
item osd.7 weight 0.098
}
host ceph-ssdnode4 {
id -109 # do not change unnecessarily
id -110 class hdd # do not change unnecessarily
# weight 0.098
alg straw2
hash 0 # rjenkins1
item osd.10 weight 0.098
}
#HDD bucket
root ssd {
id -127 # do not change unnecessarily
id -11 class hdd # do not change unnecessarily
# weight 1.952
alg straw
hash 0 # rjenkins1
item ceph-ssdnode1 weight 0.488
item ceph-ssdnode2 weight 0.488
item ceph-ssdnode3 weight 0.488
item ceph-ssdnode4 weight 0.488
}
#hdd rules
rule ssd_rule {
id 20
type replicated
min_size 1
max_size 5
step take ssd #选择root hdd作为输入的bucket
step chooseleaf firstn 0 type host #选择 1 个 host 类型，并递归选择叶子节点 osd
step emit #输出结果
}
1)、将上述配置编写进crushmap_file.txt 中
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54
# devices
device 0 osd.0 class hdd
device 1 osd.1 class hdd
device 2 osd.2 class hdd
device 3 osd.3 class hdd
device 4 osd.4 class hdd
device 5 osd.5 class hdd
device 6 osd.6 class hdd
device 7 osd.7 class hdd
device 8 osd.8 class hdd
device 9 osd.9 class hdd
device 10 osd.10 class hdd
device 11 osd.11 class hdd
# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 zone
type 10 region
type 11 root
# buckets
host ceph-node1 {
id -3 # do not change unnecessarily
id -4 class hdd # do not change unnecessarily
# weight 0.029
alg straw2
hash 0 # rjenkins1
item osd.0 weight 0.010
item osd.1 weight 0.010
item osd.11 weight 0.010
}
host ceph-node2 {
id -5 # do not change unnecessarily
id -6 class hdd # do not change unnecessarily
# weight 0.029
alg straw2
hash 0 # rjenkins1
item osd.2 weight 0.010
item osd.3 weight 0.010
item osd.4 weight 0.010
}
host ceph-node3 {
id -7 # do not change unnecessarily
id -8 class hdd # do not change unnecessarily
# weight 0.029
alg straw2
hash 0 # rjenkins1
item osd.5 weight 0.010
item osd.6 weight 0.010
item osd.7 weight 0.010
}
host ceph-node4 {
id -9 # do not change unnecessarily
id -10 class hdd # do not change unnecessarily
# weight 0.029
alg straw2
hash 0 # rjenkins1
item osd.8 weight 0.010
item osd.9 weight 0.010
item osd.10 weight 0.010
}
root default {
id -1 # do not change unnecessarily
id -2 class hdd # do not change unnecessarily
# weight 0.118
alg straw2
hash 0 # rjenkins1
item ceph-node1 weight 0.029
item ceph-node2 weight 0.029
item ceph-node3 weight 0.029
item ceph-node4 weight 0.029
}
#####################start hdd
#ssd node
host ceph-ssdnode1 {
id -103 # do not change unnecessarily
id -104 class hdd # do not change unnecessarily
# weight 0.098
alg straw2
hash 0 # rjenkins1
item osd.0 weight 0.098
}
host ceph-ssdnode2 {
id -105 # do not change unnecessarily
id -106 class hdd # do not change unnecessarily
# weight 0.098
alg straw2
hash 0 # rjenkins1
item osd.4 weight 0.098
}
host ceph-ssdnode3 {
id -107 # do not change unnecessarily
id -108 class hdd # do not change unnecessarily
# weight 0.098
alg straw2
hash 0 # rjenkins1
item osd.7 weight 0.098
}
host ceph-ssdnode4 {
id -109 # do not change unnecessarily
id -110 class hdd # do not change unnecessarily
# weight 0.098
alg straw2
hash 0 # rjenkins1
item osd.10 weight 0.098
}
#HDD bucket
root ssd {
id -127 # do not change unnecessarily
id -11 class hdd # do not change unnecessarily
# weight 1.952
alg straw
hash 0 # rjenkins1
item ceph-ssdnode1 weight 0.488
item ceph-ssdnode2 weight 0.488
item ceph-ssdnode3 weight 0.488
item ceph-ssdnode4 weight 0.488
}
#hdd rules
rule ssd_rule {
id 20
type replicated
min_size 1
max_size 5
step take ssd #选择root hdd作为输入的bucket
step chooseleaf firstn 0 type host #选择 1 个 host 类型，并递归选择叶子节点 osd
step emit #输出结果
}
######################################end ssd
# rules
rule replicated_rule {
id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
# end crush map
2）、将crushmap_file.txt转换为crush格式
cephadmin@ceph-deploy:~/ceph-cluster$ crushtool -c crushmap_file.txt -o newcrushmap
3)、导入新的crushmap前查看
cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd crush rule dump
[
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]
4)、导入新的crush map
cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd setcrushmap -i newcrushmap 36
5)、验证
cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd crush rule dump
[
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 20,
"rule_name": "ssd_rule",
"ruleset": 20,
"type": 1,
"min_size": 1,
"max_size": 5,
"steps": [
{
"op": "take",
"item": -127,
"item_name": "ssd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]
变更前的osd信息
cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.11755 root default -3 0.02939 host ceph-node1 0 hdd 0.00980 osd.0 up 1.00000 1.00000 1 hdd 0.00980 osd.1 up 1.00000 1.00000 11 hdd 0.00980 osd.11 up 1.00000 1.00000 -5 0.02939 host ceph-node2 2 hdd 0.00980 osd.2 up 1.00000 1.00000 3 hdd 0.00980 osd.3 up 1.00000 1.00000 4 hdd 0.00980 osd.4 up 1.00000 1.00000 -7 0.02939 host ceph-node3 5 hdd 0.00980 osd.5 up 1.00000 1.00000 6 hdd 0.00980 osd.6 up 1.00000 1.00000 7 hdd 0.00980 osd.7 up 1.00000 1.00000 -9 0.02939 host ceph-node4 8 hdd 0.00980 osd.8 up 1.00000 1.00000 9 hdd 0.00980 osd.9 up 1.00000 1.00000 10 hdd 0.00980 osd.10 up 1.00000 1.00000
变更后的osd信息
cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -127 1.95197 root ssd -103 0.48799 host ceph-ssdnode1 0 hdd 0.09799 osd.0 up 1.00000 1.00000 -105 0.48799 host ceph-ssdnode2 4 hdd 0.09799 osd.4 up 1.00000 1.00000 -107 0.48799 host ceph-ssdnode3 7 hdd 0.09799 osd.7 up 1.00000 1.00000 -109 0.48799 host ceph-ssdnode4 10 hdd 0.09799 osd.10 up 1.00000 1.00000 -1 0.11597 root default -3 0.02899 host ceph-node1 0 hdd 0.00999 osd.0 up 1.00000 1.00000 1 hdd 0.00999 osd.1 up 1.00000 1.00000 11 hdd 0.00999 osd.11 up 1.00000 1.00000 -5 0.02899 host ceph-node2 2 hdd 0.00999 osd.2 up 1.00000 1.00000 3 hdd 0.00999 osd.3 up 1.00000 1.00000 4 hdd 0.00999 osd.4 up 1.00000 1.00000 -7 0.02899 host ceph-node3 5 hdd 0.00999 osd.5 up 1.00000 1.00000 6 hdd 0.00999 osd.6 up 1.00000 1.00000 7 hdd 0.00999 osd.7 up 1.00000 1.00000 -9 0.02899 host ceph-node4 8 hdd 0.00999 osd.8 up 1.00000 1.00000 9 hdd 0.00999 osd.9 up 1.00000 1.00000 10 hdd 0.00999 osd.10 up 1.00000 1.00000
6)、创建测试存储池验证
# 创建新的test-ssd存储池
cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool create test-ssd 32 32 ssd_rule
pool 'test-ssd' created
# 验证pgp的状态
cephadmin@ceph-deploy:~/ceph-cluster$ ceph pg ls-by-pool test-ssd | awk '{print $1,$2,$15}'
PG OBJECTS ACTING
11.0 0 [4,7,0]p4
11.1 0 [4,7,10]p4
11.2 0 [10,0,4]p10
11.3 0 [7,10,0]p7
11.4 0 [0,7,10]p0
11.5 0 [0,10,4]p0
11.6 0 [4,0,7]p4
11.7 0 [10,7,0]p10
11.8 0 [4,0,7]p4
11.9 0 [0,4,7]p0
11.a 0 [10,7,4]p10
11.b 0 [0,7,4]p0
11.c 0 [7,0,4]p7
11.d 0 [7,4,10]p7
11.e 0 [10,7,0]p10
11.f 0 [7,0,10]p7
11.10 0 [0,4,10]p0
11.11 0 [0,7,10]p0
11.12 0 [10,0,7]p10
11.13 0 [7,10,4]p7
11.14 0 [0,10,4]p0
11.15 0 [10,7,0]p10
11.16 0 [0,10,4]p0
11.17 0 [4,0,10]p4
11.18 0 [0,10,7]p0
11.19 0 [7,10,4]p7
11.1a 0 [7,0,4]p7
11.1b 0 [4,10,7]p4
11.1c 0 [7,0,4]p7
11.1d 0 [0,10,4]p0
11.1e 0 [10,0,7]p10
11.1f 0 [7,10,4]p7
* NOTE: afterwards
