八、Ceph Crush map
2、CRUSH算法的流程
CRUSH算法的全程为：Controlled、Scalable、Decentralized Placement of Replicated Data,可控的、可扩展的、分布式的副本数据放置算法。
对 Ceph 集群的一个读写操作，客户端首先访问 Ceph monitor 来获取 cluster map 的一份副本，它包含五个map，分别是 monitor map、OSD map、MDS map、CRUSH map 和 PG map。这些 cluster map 将有助于客户端知晓 Ceph集群的状态和配置。然后，数据被转化为一个或者多个对象，每个对象都具有对象名称（name）和存储池名称/ID（pool name/ID）。接着，该对象会被以配置组（PG）的数目为基数做哈希，在指定的 Ceph 存储池中生成最终的 PG。这个计算出的 PG 再通过 CRUSH 查询来确定存放或者获取数据的主（primary）、次（secondary）和再次（tertiary）OSD 的位置。
一旦客户端获得了精确的 OSD ID，将会直接和这些 OSD 通信并存放数据。所有这些计算操作都是在客户端完成的，因此它们不会影响 Ceph 集群服务器端的性能。
1 2 3 4 5 6 7 | 1. 上传一个文件时，使用如下命令：# rados -p ssd-pool put dummy_object1 /etc/ hosts2. 这里上传到那个 pool(ssd-pool)，以及对象的名称(dummy_object1)已经确定；3. 一个 pool 在创建的时候，其 PG 的数量已确定；4. 通过对 object 进行哈希，公式为：hash(dummy_object1)%PG_num；这样可以得到对象所在的 PG(这里是 4.32)5. PG 根据根据 CRUSH map 的 rule 最终确定落在哪些 OSD 上；6. 所以客户端在访问 ceph 时，可以通过计算来得出 object 所在的 OSD，消除了中心节点的 metadata。 |
整个流程如下:
1 2 3 4 5 | 1）取得Object的name进行Hash运算。2）取得的Hash值与PG数取余，得到的结果与Pool ID结成PG的编号（如图10-5中的4.32）。3）通过CRUSH算法，把PG映射到具体的OSD。在Ceph里，PG是数据存储的管理单元，如果把PG当作一致性Hash里的存储节点，那么它就是最简单的数据分布（即取余算法）方式。不同的是，PG是抽象的存储节点，它不会随着物理节点的加入或者离开而增加或减少，因此数据到PG的映射是稳定的。当在Ceph里创建存储池（指定PG的数量），整个Hash环就固定了，对象到PG的映射就唯一确认，即数据具体存储到哪个PG是确定的，不会随着下层OSD的增删而改变，这充分体现了Ceph在数据可靠性的特征之一。 |
实验案例：
1 2 3 4 5 6 7 8 9 10 11 12 | #1、创建pool2地址池cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool create pool2 64 64pool 'pool2' created#2、上传一个名为object1到pool2存储池cephadmin@ceph-deploy:~/ceph-cluster$ rados -p pool2 put object1 /etc/hosts#3、查看pool2中的存储池对象cephadmin@ceph-deploy:~/ceph-cluster$ rados -p pool2 lsobject1#4、查看pool2存储池中的object1在OSD上的分布及PGidcephadmin@ceph-deploy:~/ceph-cluster$ ceph osd map pool2 object1osdmap e439 pool 'pool2' (11) object 'object1' -> pg 11.bac5debc (11.3c) -> up ([1,6,10], p1) acting ([1,6,10], p1)#这里的PGid为 pg 11.bac5debc (11.3c)，数据分布存数在osd.1,osd.6和osd.10上，其中osd1为主OSD |
2）、删除pool2中的object2,让后上传内容不一样，单对象名认为object1。
1 2 3 4 5 6 7 8 9 10 | #1、删除object1对象cephadmin@ceph-deploy:~/ceph-cluster$ rados -p pool2 rm object1cephadmin@ceph-deploy:~/ceph-cluster$ rados -p pool2 lscephadmin@ceph-deploy:~/ceph-cluster$ #2、上传/etc/passwd,对象名称为objetct1cephadmin@ceph-deploy:~/ceph-cluster$ rados -p pool2 put object1 /etc/passwd#3、查看pool2存储池中的object1在OSD上的分布及PGidcephadmin@ceph-deploy:~/ceph-cluster$ ceph osd map pool2 object1osdmap e439 pool 'pool2' (11) object 'object1' -> pg 11.bac5debc (11.3c) -> up ([1,6,10], p1) acting ([1,6,10], p1)#这里的PGid为 pg 11.bac5debc (11.3c)，数据分布存数在osd.1,osd.6和osd.10上，其中osd1为master |
上面2个实验说明了在同一个pool存储池中，只要对象名称不变，那么数据的分布就固定不变
Ceph 的后端是一个对象存储（RADOS），所以所有的数据都会按照一个特定的 size（ceph 系统默认是 4M）被切分成若干个对象，也就是上面的 Objects。每一个 Object 都有一个 Objectid（oid），Objectid 的命名规则是数据所在image 的 block_name_prefix 再跟上一个编号，这个编号是顺序递增的。通过(pool id, hash(oid) & mask)，每个object 都可以得到它对应的 pgid。有了这个 pgid 之后，Client 就会执行 CRUSH 算法得到一个 OSD 列表（OSD1，OSD2，OSD3）。然后对它们进行筛选，根据副本数找出符合要求的 OSD，比如 OSD 不能是 failed、overloaded的。知道数据要存在哪些 OSD 上之后，Client 会向列表中的第一个 OSD（primary osd）发起 IO 请求。然后这个OSD 按照读写请求分别做相应的处理。
4、ceph cluster map介绍
Ceph monitor负责监控整个集群的监控状态，以及维护集群成员关系状态、对等节点的状态，和集群的配置信息等。Ceph monitor通过维护cluster map的主复制来实现这些功能。
1）monitor map: 它包含监视节点端到端的信息，包括Ceph集群ID、monitor节点名称（hostname）、IP地址和端口号等。它还保存自monitor map被创建以来的最新版本号（epoch:每种map都维护着其历史版本，每个版本被称为一个epoch，epoch是一个单调递增的序列）。以及最后修改时间等。查看monitor map的命令
1 2 3 4 5 6 7 8 9 10 11 | cephadmin@ceph-deploy:~/ceph-cluster$ ceph mon dumpepoch 3fsid d1f2683e-ac8d-4a02-8b94-9f7517d32c50last_changed 2022-04-03T16:49:08.531006+0800created 2022-04-03T16:28:03.331168+0800min_mon_release 16 (pacific)election_strategy: 10: [v2:192.168.1.203:3300/0,v1:192.168.1.203:6789/0] mon.ceph-mon11: [v2:192.168.1.204:3300/0,v1:192.168.1.204:6789/0] mon.ceph-mon22: [v2:192.168.1.205:3300/0,v1:192.168.1.205:6789/0] mon.ceph-mon3dumped monmap epoch 3 |
2) OSD map: 它保存一些常用的信息，包括集群ID、OSD map自创建以来的最新版本号（epoch）及其最后修改时间，以及存储池相关的信息，包括存储池名称、ID、类型、副本级别（replication level)和PG。它还保存着OSD的信息，比如数量、状态、权重、最后清理间隔（last clean interval）以及OSD节点的信息。查看OSD map的命令
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 | cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd dumpepoch 378fsid d1f2683e-ac8d-4a02-8b94-9f7517d32c50created 2022-04-03T16:28:04.210492+0800modified 2022-04-04T17:02:50.538648+0800flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimitcrush_version 35full_ratio 0.95backfillfull_ratio 0.9nearfull_ratio 0.85require_min_compat_client luminousmin_compat_client luminousrequire_osd_release pacificstretch_mode_enabled falsepool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 226 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealthpool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 73 flags hashpspool stripe_width 0 application rgwpool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 75 flags hashpspool stripe_width 0 application rgwpool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 77 flags hashpspool stripe_width 0 application rgwpool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 202 lfor 0/202/200 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgwpool 6 'cephfs-metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 156 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfspool 7 'cephfs-data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 156 flags hashpspool stripe_width 0 application cephfspool 8 'myrbd1' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 212 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbdpool 9 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 377 lfor 0/377/375 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgwpool 10 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 258 flags hashpspool stripe_width 0 application rgwmax_osd 12osd.0 up in weight 1 up_from 229 up_thru 256 down_at 219 last_clean_interval [65,218) [v2:192.168.1.206:6800/1363,v1:192.168.1.206:6805/1363] [v2:192.168.88.206:6804/1363,v1:192.168.88.206:6805/1363] exists,up 2ed474c5-6975-4400-b1cf-e7a528c19842osd.1 up in weight 1 up_from 228 up_thru 377 down_at 219 last_clean_interval [65,218) [v2:192.168.1.206:6808/1367,v1:192.168.1.206:6809/1367] [v2:192.168.88.206:6808/1367,v1:192.168.88.206:6809/1367] exists,up 9f0d23e1-2c70-4f8f-b19d-6b416469debbosd.2 up in weight 1 up_from 230 up_thru 339 down_at 221 last_clean_interval [17,220) [v2:192.168.1.207:6800/1496,v1:192.168.1.207:6805/1496] [v2:192.168.88.207:6804/1496,v1:192.168.88.207:6805/1496] exists,up 1a26c5b1-18b1-429a-8334-e9cf154c1e6cosd.3 up in weight 1 up_from 231 up_thru 344 down_at 221 last_clean_interval [23,220) [v2:192.168.1.207:6808/1498,v1:192.168.1.207:6809/1498] [v2:192.168.88.207:6808/1498,v1:192.168.88.207:6809/1498] exists,up 1a183a93-5ccd-45b2-89fc-2dc8ad2914c5osd.4 up in weight 1 up_from 230 up_thru 256 down_at 221 last_clean_interval [28,220) [v2:192.168.1.207:6801/1497,v1:192.168.1.207:6802/1497] [v2:192.168.88.207:6800/1497,v1:192.168.88.207:6801/1497] exists,up 76526a2b-a3d3-4e66-82e8-e26754115cfcosd.5 up in weight 1 up_from 235 up_thru 354 down_at 223 last_clean_interval [33,222) [v2:192.168.1.208:6800/1532,v1:192.168.1.208:6801/1532] [v2:192.168.88.208:6800/1532,v1:192.168.88.208:6801/1532] exists,up 7612de2d-94a0-4072-986e-e51d37d407d3osd.6 up in weight 1 up_from 234 up_thru 256 down_at 223 last_clean_interval [38,222) [v2:192.168.1.208:6808/1534,v1:192.168.1.208:6809/1534] [v2:192.168.88.208:6808/1534,v1:192.168.88.208:6809/1534] exists,up a88f02a6-1c08-4dca-bfb3-3d5981595fb8osd.7 up in weight 1 up_from 235 up_thru 365 down_at 223 last_clean_interval [43,222) [v2:192.168.1.208:6804/1533,v1:192.168.1.208:6805/1533] [v2:192.168.88.208:6804/1533,v1:192.168.88.208:6805/1533] exists,up 7dd759be-e341-4f62-9378-1be6171519f7osd.8 up in weight 1 up_from 238 up_thru 369 down_at 230 last_clean_interval [48,224) [v2:192.168.1.209:6808/1526,v1:192.168.1.209:6809/1526] [v2:192.168.88.209:6808/1526,v1:192.168.88.209:6809/1526] exists,up 99260fce-64ce-4426-a0a7-d8819b54f0a2osd.9 up in weight 1 up_from 238 up_thru 373 down_at 230 last_clean_interval [54,224) [v2:192.168.1.209:6804/1525,v1:192.168.1.209:6805/1525] [v2:192.168.88.209:6804/1525,v1:192.168.88.209:6805/1525] exists,up ab8ab101-60b5-4943-8b1a-a32d76084b6dosd.10 up in weight 1 up_from 238 up_thru 259 down_at 229 last_clean_interval [59,224) [v2:192.168.1.209:6800/1523,v1:192.168.1.209:6801/1523] [v2:192.168.88.209:6800/1523,v1:192.168.88.209:6801/1523] exists,up b4998715-69bb-4140-a18f-dfb112783e8cosd.11 up in weight 1 up_from 229 up_thru 256 down_at 219 last_clean_interval [69,218) [v2:192.168.1.206:6801/1376,v1:192.168.1.206:6802/1376] [v2:192.168.88.206:6800/1376,v1:192.168.88.206:6801/1376] exists,up 94462cbe-6ddc-4edb-86e9-f78b1f43bea8pg_upmap_items 6.0 [10,9]pg_upmap_items 6.2 [11,0]pg_upmap_items 6.c [11,0]pg_upmap_items 6.e [10,9]pg_upmap_items 6.1a [1,0]pg_upmap_items 7.a [0,1]pg_upmap_items 7.17 [2,4]pg_upmap_items 7.2a [0,4]pg_upmap_items 7.3e [2,4]pg_upmap_items 8.2 [10,3]pg_upmap_items 8.3 [1,11]pg_upmap_items 8.a [5,3]pg_upmap_items 8.20 [2,3]pg_upmap_items 8.2e [8,9]pg_upmap_items 10.0 [0,11]blocklist 192.168.1.201:6801/9150 expires 2022-04-05T15:21:56.596516+0800blocklist 192.168.1.201:6800/9150 expires 2022-04-05T15:21:56.596516+0800blocklist 192.168.1.201:0/3611889504 expires 2022-04-05T15:21:56.596516+0800blocklist 192.168.1.201:0/1264985371 expires 2022-04-05T15:21:56.596516+0800blocklist 192.168.1.205:6800/2648602275 expires 2022-04-05T00:02:38.304109+0800blocklist 192.168.1.205:6801/2648602275 expires 2022-04-05T00:02:38.304109+0800blocklist 192.168.1.203:6801/3665534120 expires 2022-04-05T00:02:25.602324+0800blocklist 192.168.1.204:6800/1132501689 expires 2022-04-05T00:02:33.275718+0800blocklist 192.168.1.203:6800/3665534120 expires 2022-04-05T00:02:25.602324+0800blocklist 192.168.1.204:6801/1132501689 expires 2022-04-05T00:02:33.275718+0800 |
3) PG map:它保存的信息包括PG的版本、时间戳、OSD map的最新版本号（epoch）、容量已满百分比（full ratio,是指集群被认为容量已满时的最大使用百分比，此时Ceph集群禁止客户端读写数据）、容量将满百分比（near full ration,是指集群被认为容量将满时的最大使用百分比，此时Ceph集群将发出告警）等。它还记录了每个PG的ID（其格式为{pool-num}.{pg-id}）、对象数量、状态、状态时间戳（state stamp）、up OSD sets(一个up OSD set指某些特定PG map版本的PG的所有副本所在的OSD列表，该I列表是有序的，第一个OSD为主OSD，其余为从OSD)、acting OSD sets（acting OSD set指该PG的所有副本所在的OSD的列表），以及清理（scrub）的信息。查看PG map的命令
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 | cephadmin@ceph-deploy:~/ceph-cluster$ ceph pg dumpversion 13086stamp 2022-04-04T22:33:26.923756+0800last_osdmap_epoch 0last_pg_scan 0PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN8.30 3 0 0 0 0 12582912 0 0 34 34 active+clean 2022-04-04T19:54:12.287880+0800 215'34 378:232 [1,6,8] 1 [1,6,8] 1 215'34 2022-04-04T19:54:12.287820+0800 0'0 2022-04-03T17:52:00.867759+0800 07.3f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:09:39.827406+0800 0'0 378:270 [3,7,1] 3 [3,7,1] 3 0'0 2022-04-04T17:09:39.826881+0800 0'0 2022-04-03T16:55:03.148848+0800 08.31 5 0 0 0 0 18939904 0 0 42 42 active+clean 2022-04-04T20:03:16.350937+0800 215'42 378:253 [7,2,10] 7 [7,2,10] 7 215'42 2022-04-04T20:03:16.350878+0800 0'0 2022-04-03T17:52:00.867759+0800 07.3e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:44:29.338657+0800 0'0 378:198 [4,6,0] 4 [4,6,0] 4 0'0 2022-04-04T19:44:29.338594+0800 0'0 2022-04-03T16:55:03.148848+0800 08.32 3 0 0 0 0 8421376 0 0 34 34 active+clean 2022-04-04T15:22:25.982726+0800 215'34 377:228 [0,7,9] 0 [0,7,9] 0 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.3d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:14:07.110388+0800 0'0 378:270 [4,10,11] 4 [4,10,11] 4 0'0 2022-04-04T21:14:07.110321+0800 0'0 2022-04-03T16:55:03.148848+0800 08.33 5 0 0 0 0 20971520 0 0 49 49 active+clean 2022-04-04T22:17:05.995239+0800 215'49 378:244 [11,4,9] 11 [11,4,9] 11 215'49 2022-04-04T22:17:05.995185+0800 0'0 2022-04-03T17:52:00.867759+0800 07.3c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:37.784982+0800 0'0 377:264 [8,6,11] 8 [8,6,11] 8 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 08.34 6 0 0 0 0 25165824 0 0 47 47 active+clean 2022-04-04T20:40:33.146876+0800 378'47 378:244 [0,6,8] 0 [0,6,8] 0 378'47 2022-04-04T20:40:33.145571+0800 378'47 2022-04-04T20:40:33.145571+0800 07.3b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:23:02.672284+0800 0'0 378:270 [3,0,5] 3 [3,0,5] 3 0'0 2022-04-04T18:23:02.672233+0800 0'0 2022-04-03T16:55:03.148848+0800 08.35 3 0 0 0 0 12582912 0 0 29 29 active+clean 2022-04-04T15:22:26.524343+0800 215'29 377:221 [11,9,7] 11 [11,9,7] 11 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.3a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:51:17.299928+0800 0'0 378:269 [9,7,2] 9 [9,7,2] 9 0'0 2022-04-04T18:51:17.299877+0800 0'0 2022-04-04T18:51:17.299877+0800 08.36 0 0 0 0 0 0 0 0 31 31 active+clean 2022-04-04T18:05:56.349707+0800 215'31 378:229 [1,10,3] 1 [1,10,3] 1 215'31 2022-04-04T18:05:56.349639+0800 0'0 2022-04-03T17:52:00.867759+0800 07.39 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:39.225998+0800 0'0 377:267 [8,1,3] 8 [8,1,3] 8 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 08.37 5 0 0 0 0 20971520 0 0 34 34 active+clean 2022-04-04T20:50:35.958059+0800 378'34 378:230 [5,0,9] 5 [5,0,9] 5 378'34 2022-04-04T20:50:35.957568+0800 378'34 2022-04-04T20:50:35.957568+0800 07.38 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:05:55.131216+0800 0'0 378:273 [9,2,0] 9 [9,2,0] 9 0'0 2022-04-04T17:05:55.131130+0800 0'0 2022-04-03T16:55:03.148848+0800 08.38 0 0 0 0 0 0 0 0 21 21 active+clean 2022-04-04T18:08:44.937516+0800 215'21 378:217 [1,9,6] 1 [1,9,6] 1 215'21 2022-04-04T18:08:44.937459+0800 0'0 2022-04-03T17:52:00.867759+0800 07.37 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.439177+0800 0'0 377:264 [9,7,3] 9 [9,7,3] 9 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 08.39 3 0 0 0 0 12582912 0 0 20 20 active+clean 2022-04-04T21:35:13.985246+0800 215'20 378:217 [5,8,3] 5 [5,8,3] 5 215'20 2022-04-04T21:35:13.985046+0800 0'0 2022-04-03T17:52:00.867759+0800 07.36 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:34:58.813081+0800 0'0 378:265 [11,4,6] 11 [11,4,6] 11 0'0 2022-04-04T19:34:58.813040+0800 0'0 2022-04-03T16:55:03.148848+0800 08.3a 4 0 0 0 0 4333587 0 0 52 52 active+clean 2022-04-04T15:22:27.152561+0800 215'52 377:293 [7,0,10] 7 [7,0,10] 7 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.35 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:37.463043+0800 0'0 377:264 [8,6,0] 8 [8,6,0] 8 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 08.3b 1 0 0 0 0 4194304 0 0 25 25 active+clean 2022-04-04T20:57:41.855311+0800 215'25 378:219 [4,7,10] 4 [4,7,10] 4 215'25 2022-04-04T20:57:41.855265+0800 0'0 2022-04-03T17:52:00.867759+0800 07.34 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:37:27.479292+0800 0'0 378:269 [5,11,2] 5 [5,11,2] 5 0'0 2022-04-04T19:37:27.479245+0800 0'0 2022-04-03T16:55:03.148848+0800 08.3c 4 0 0 0 0 16777216 0 0 51 51 active+clean 2022-04-04T15:22:22.271509+0800 215'51 377:247 [2,7,0] 2 [2,7,0] 2 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.33 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:05:47.749529+0800 0'0 378:269 [8,5,0] 8 [8,5,0] 8 0'0 2022-04-04T19:05:47.749462+0800 0'0 2022-04-03T16:55:03.148848+0800 08.3d 6 0 0 0 0 10579968 0 0 52 52 active+clean 2022-04-04T15:22:38.013594+0800 215'52 377:399 [9,6,1] 9 [9,6,1] 9 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.32 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:25:31.727211+0800 0'0 378:267 [5,8,4] 5 [5,8,4] 5 0'0 2022-04-04T17:25:31.727159+0800 0'0 2022-04-03T16:55:03.148848+0800 08.3e 1 0 0 0 0 4194304 0 0 25 25 active+clean 2022-04-04T18:28:38.971941+0800 215'25 378:225 [7,2,1] 7 [7,2,1] 7 215'25 2022-04-04T18:28:38.971893+0800 0'0 2022-04-03T17:52:00.867759+0800 07.31 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:03:40.787280+0800 0'0 378:267 [5,9,3] 5 [5,9,3] 5 0'0 2022-04-04T17:03:40.787213+0800 0'0 2022-04-03T16:55:03.148848+0800 08.3f 1 0 0 0 0 8192 0 0 30 30 active+clean 2022-04-04T15:22:22.300929+0800 215'30 377:221 [1,3,7] 1 [1,3,7] 1 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.30 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.796519+0800 0'0 377:265 [9,5,1] 9 [9,5,1] 9 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 08.20 2 0 0 0 0 5926912 0 0 30 30 active+clean 2022-04-04T15:22:26.047927+0800 215'30 377:208 [3,6,8] 3 [3,6,8] 3 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.2f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:14:49.996984+0800 0'0 378:271 [2,10,1] 2 [2,10,1] 2 0'0 2022-04-04T17:14:49.996917+0800 0'0 2022-04-04T17:14:49.996917+0800 08.21 2 0 0 0 0 8388608 0 0 28 28 active+clean 2022-04-04T21:02:10.719437+0800 215'28 378:225 [0,5,9] 0 [0,5,9] 0 215'28 2022-04-04T21:02:10.719373+0800 0'0 2022-04-03T17:52:00.867759+0800 07.2e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:47:40.444977+0800 0'0 378:269 [8,11,7] 8 [8,11,7] 8 0'0 2022-04-04T20:47:40.444928+0800 0'0 2022-04-03T16:55:03.148848+0800 08.22 2 0 0 0 0 8388608 0 0 26 26 active+clean 2022-04-04T18:29:59.656872+0800 215'26 378:228 [10,1,4] 10 [10,1,4] 10 215'26 2022-04-04T18:29:59.656288+0800 0'0 2022-04-03T17:52:00.867759+0800 07.2d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:37.519631+0800 0'0 377:264 [9,6,3] 9 [9,6,3] 9 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 08.23 1 0 0 0 0 0 0 0 19 19 active+clean 2022-04-04T20:16:22.127774+0800 215'19 378:223 [8,4,7] 8 [8,4,7] 8 215'19 2022-04-04T20:16:22.127319+0800 0'0 2022-04-03T17:52:00.867759+0800 07.2c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:21:55.216352+0800 0'0 378:266 [11,7,8] 11 [11,7,8] 11 0'0 2022-04-04T18:21:55.216306+0800 0'0 2022-04-03T16:55:03.148848+0800 08.24 1 0 0 0 0 4194304 0 0 24 24 active+clean 2022-04-04T15:22:27.304174+0800 215'24 377:219 [4,0,8] 4 [4,0,8] 4 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.2b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:21.543563+0800 0'0 377:262 [1,2,6] 1 [1,2,6] 1 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 08.25 2 0 0 0 0 8388608 0 0 25 25 active+clean 2022-04-04T20:08:39.735770+0800 215'25 378:220 [1,10,4] 1 [1,10,4] 1 215'25 2022-04-04T20:08:39.735712+0800 0'0 2022-04-03T17:52:00.867759+0800 07.2a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:48:56.237374+0800 0'0 378:182 [4,9,7] 4 [4,9,7] 4 0'0 2022-04-04T21:48:56.237314+0800 0'0 2022-04-03T16:55:03.148848+0800 08.26 5 0 0 0 0 10645504 0 0 40 40 active+clean 2022-04-04T15:22:31.127727+0800 215'40 377:236 [5,1,3] 5 [5,1,3] 5 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.29 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:26.928825+0800 0'0 377:265 [3,9,0] 3 [3,9,0] 3 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 08.27 2 0 0 0 0 8388608 0 0 21 21 active+clean 2022-04-04T15:22:27.217084+0800 215'21 377:215 [6,9,4] 6 [6,9,4] 6 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.28 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.254525+0800 0'0 377:264 [6,9,2] 6 [6,9,2] 6 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 08.28 2 0 0 0 0 8208 0 0 24 24 active+clean 2022-04-04T15:22:21.411243+0800 215'24 377:223 [0,2,6] 0 [0,2,6] 0 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.27 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T22:19:23.172047+0800 0'0 378:272 [10,2,1] 10 [10,2,1] 10 0'0 2022-04-04T22:19:23.172000+0800 0'0 2022-04-03T16:55:03.148848+0800 08.29 6 0 0 0 0 20992000 0 0 41 41 active+clean 2022-04-04T15:22:27.318296+0800 215'41 377:234 [5,9,11] 5 [5,9,11] 5 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.26 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:37.519648+0800 0'0 377:264 [9,6,3] 9 [9,6,3] 9 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 08.2a 0 0 0 0 0 0 0 0 21 21 active+clean 2022-04-04T15:22:26.860642+0800 215'21 377:212 [5,8,0] 5 [5,8,0] 5 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.25 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.516771+0800 0'0 377:263 [8,2,7] 8 [8,2,7] 8 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 08.2b 2 0 0 0 0 4194320 0 0 29 29 active+clean 2022-04-04T18:31:10.563124+0800 215'29 378:238 [10,3,7] 10 [10,3,7] 10 215'29 2022-04-04T18:31:10.563077+0800 0'0 2022-04-03T17:52:00.867759+0800 07.24 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.320218+0800 0'0 377:265 [3,7,11] 3 [3,7,11] 3 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 08.2c 4 0 0 0 0 16777216 0 0 31 31 active+clean 2022-04-04T21:10:07.437378+0800 215'31 378:230 [6,4,8] 6 [6,4,8] 6 215'31 2022-04-04T21:10:07.437333+0800 0'0 2022-04-03T17:52:00.867759+0800 07.23 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:26.046295+0800 0'0 377:265 [3,8,11] 3 [3,8,11] 3 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 08.2d 5 0 0 0 0 16797696 0 0 47 47 active+clean 2022-04-04T22:28:46.212208+0800 215'47 378:244 [0,4,5] 0 [0,4,5] 0 215'47 2022-04-04T22:28:46.212164+0800 0'0 2022-04-03T17:52:00.867759+0800 07.22 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:07:21.379545+0800 0'0 378:273 [9,3,11] 9 [9,3,11] 9 0'0 2022-04-04T21:07:21.379485+0800 0'0 2022-04-03T16:55:03.148848+0800 08.2e 3 0 0 0 0 12582912 0 0 55 55 active+clean 2022-04-04T22:11:04.175352+0800 215'55 378:266 [2,9,0] 2 [2,9,0] 2 215'55 2022-04-04T22:11:04.175286+0800 0'0 2022-04-03T17:52:00.867759+0800 07.21 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.136684+0800 0'0 377:264 [6,10,3] 6 [6,10,3] 6 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 08.2f 2 0 0 0 0 8388608 0 0 33 33 active+clean 2022-04-04T15:22:30.234232+0800 215'33 377:229 [6,0,2] 6 [6,0,2] 6 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.20 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:56:57.627763+0800 0'0 378:266 [5,0,10] 5 [5,0,10] 5 0'0 2022-04-04T18:56:57.627716+0800 0'0 2022-04-03T16:55:03.148848+0800 010.6 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.429725+0800 0'0 377:130 [3,7,10] 3 [3,7,10] 3 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 09.5 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:17.627348+0800 264'2 377:170 [5,10,4] 5 [5,10,4] 5 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 08.4 4 0 0 0 0 16777216 0 0 37 37 active+clean 2022-04-04T15:22:39.073619+0800 215'37 377:231 [9,5,2] 9 [9,5,2] 9 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 04.8 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:04:18.042232+0800 0'0 378:325 [5,3,11] 5 [5,3,11] 5 0'0 2022-04-04T19:04:18.042184+0800 0'0 2022-04-03T16:53:03.970520+0800 03.f 7 0 0 0 0 110 0 0 2223 2223 active+clean 2022-04-04T20:34:07.641044+0800 378'2223 378:3684 [2,5,9] 2 [2,5,9] 2 378'1927 2022-04-04T20:34:07.640996+0800 0'0 2022-04-03T16:53:01.903832+0800 02.e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:05:16.673743+0800 0'0 378:309 [1,5,8] 1 [1,5,8] 1 0'0 2022-04-04T21:05:16.673691+0800 0'0 2022-04-04T21:05:16.673691+0800 06.a 1 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T15:22:27.032486+0800 239'2 377:260 [4,10,6] 4 [4,10,6] 4 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:21.540510+0800 0'0 377:262 [1,2,6] 1 [1,2,6] 1 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.5 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.431801+0800 0'0 377:130 [9,0,2] 9 [9,0,2] 9 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 09.6 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:13.965116+0800 0'0 377:170 [9,1,6] 9 [9,1,6] 9 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 08.7 3 0 0 0 0 8409088 0 0 37 37 active+clean 2022-04-04T15:22:26.471176+0800 215'37 377:232 [1,5,9] 1 [1,5,9] 1 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 04.b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:37.825922+0800 0'0 377:318 [8,4,6] 8 [8,4,6] 8 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 03.c 5 0 0 0 0 0 0 0 1780 1780 active+clean 2022-04-04T15:22:22.345821+0800 378'1780 378:2993 [3,0,7] 3 [3,0,7] 3 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:45:42.990376+0800 0'0 378:326 [5,10,1] 5 [5,10,1] 5 0'0 2022-04-04T18:45:42.990329+0800 0'0 2022-04-03T16:52:59.854233+0800 06.9 2 0 0 0 0 90 0 0 3 3 active+clean 2022-04-04T17:05:46.918006+0800 205'3 378:282 [10,7,11] 10 [10,7,11] 10 205'3 2022-04-04T17:05:46.917940+0800 0'0 2022-04-03T16:54:54.909163+0800 07.8 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:39.450652+0800 0'0 377:266 [10,4,0] 10 [10,4,0] 10 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.4 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.386133+0800 0'0 377:130 [6,0,8] 6 [6,0,8] 6 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 09.7 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:09.858927+0800 264'4 377:170 [5,10,11] 5 [5,10,11] 5 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 08.6 4 0 0 0 0 16777216 0 0 40 40 active+clean 2022-04-04T21:40:30.063658+0800 215'40 378:238 [6,9,1] 6 [6,9,1] 6 215'40 2022-04-04T21:40:30.063442+0800 0'0 2022-04-03T17:52:00.867759+0800 04.a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:25.735285+0800 0'0 377:321 [2,9,11] 2 [2,9,11] 2 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 03.d 8 0 0 0 0 110 0 0 2579 2579 active+clean 2022-04-04T18:32:14.728114+0800 378'2579 378:4224 [3,7,11] 3 [3,7,11] 3 378'1891 2022-04-04T18:32:14.728064+0800 0'0 2022-04-03T16:53:01.903832+0800 02.c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:30.527163+0800 0'0 377:325 [6,0,4] 6 [6,0,4] 6 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 06.8 1 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T21:42:52.073880+0800 205'2 378:264 [3,6,9] 3 [3,6,9] 3 205'2 2022-04-04T21:42:52.073825+0800 0'0 2022-04-03T16:54:54.909163+0800 07.9 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.267626+0800 0'0 377:261 [0,7,4] 0 [0,7,4] 0 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.3 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.359622+0800 0'0 377:130 [7,10,4] 7 [7,10,4] 7 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 09.0 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:50.598378+0800 264'2 378:170 [1,8,5] 1 [1,8,5] 1 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 08.1 4 0 0 0 0 16777216 0 0 37 37 active+clean 2022-04-04T15:22:25.964053+0800 215'37 377:230 [1,6,10] 1 [1,6,10] 1 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 03.a 10 0 0 0 0 110 0 0 2947 2947 active+clean 2022-04-04T15:22:26.926949+0800 378'2947 378:4776 [3,9,5] 3 [3,9,5] 3 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.b 1 0 0 0 0 358 0 0 1 1 active+clean 2022-04-04T15:22:27.073937+0800 72'1 377:348 [7,3,10] 7 [7,3,10] 7 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 04.d 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T19:09:01.929819+0800 251'37 378:20973 [3,5,10] 3 [3,5,10] 3 251'37 2022-04-04T19:09:01.929609+0800 0'0 2022-04-03T16:53:03.970520+0800 06.f 2 0 0 0 0 22 0 0 2 2 active+clean 2022-04-04T15:22:26.442454+0800 205'2 377:260 [11,10,7] 11 [11,10,7] 11 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.419514+0800 0'0 377:262 [1,5,3] 1 [1,5,3] 1 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.2 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.408617+0800 0'0 377:130 [1,10,4] 1 [1,10,4] 1 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 09.1 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:41.929605+0800 264'6 377:170 [9,0,5] 9 [9,0,5] 9 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 08.0 4 0 0 0 0 16777216 0 0 48 48 active+clean 2022-04-04T21:05:31.145331+0800 215'48 378:342 [10,4,7] 10 [10,4,7] 10 215'48 2022-04-04T21:05:31.145094+0800 0'0 2022-04-03T17:52:00.867759+0800 04.c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:26.813073+0800 0'0 377:300 [4,9,5] 4 [4,9,5] 4 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 03.b 5 0 0 0 0 110 0 0 1166 1166 active+clean 2022-04-04T15:22:26.034474+0800 378'1166 378:2106 [11,8,6] 11 [11,8,6] 11 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.255909+0800 0'0 377:322 [6,1,9] 6 [6,1,9] 6 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 06.e 1 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T17:41:49.130913+0800 156'2 378:294 [4,1,9] 4 [4,1,9] 4 156'2 2022-04-04T17:41:49.130851+0800 0'0 2022-04-03T16:54:54.909163+0800 07.f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:26.415463+0800 0'0 377:261 [11,5,9] 11 [11,5,9] 11 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.1 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.433283+0800 0'0 377:130 [9,2,0] 9 [9,2,0] 9 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 09.2 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:37.789651+0800 264'2 377:180 [8,1,6] 8 [8,1,6] 8 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 08.3 2 0 0 0 0 8388608 0 0 35 35 active+clean 2022-04-04T15:22:38.607483+0800 215'35 377:242 [8,11,2] 8 [8,11,2] 8 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 03.8 9 0 0 0 0 110 0 0 2247 2247 active+clean 2022-04-04T15:22:38.434298+0800 378'2247 378:3744 [10,1,7] 10 [10,1,7] 10 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.9 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:53:24.234989+0800 0'0 378:304 [11,4,9] 11 [11,4,9] 11 0'0 2022-04-04T21:53:24.234942+0800 0'0 2022-04-03T16:52:59.854233+0800 04.f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:26.486645+0800 0'0 377:301 [11,9,7] 11 [11,9,7] 11 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 06.d 1 0 0 0 0 70 0 0 1 1 active+clean 2022-04-04T15:22:38.649490+0800 156'1 377:273 [8,1,5] 8 [8,1,5] 8 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:09:43.454919+0800 0'0 378:271 [2,1,10] 2 [2,1,10] 2 0'0 2022-04-04T17:09:43.454870+0800 0'0 2022-04-04T17:09:43.454870+0800 010.0 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:00:00.023163+0800 0'0 377:138 [10,5,11] 10 [10,5,11] 10 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 09.3 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:33.692220+0800 0'0 377:170 [7,9,3] 7 [7,9,3] 7 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 08.2 0 0 0 0 0 0 0 0 30 30 active+clean 2022-04-04T15:22:22.637181+0800 215'30 377:213 [3,0,5] 3 [3,0,5] 3 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 03.9 7 0 0 0 0 220 0 0 1266 1266 active+clean 2022-04-04T15:22:27.979630+0800 378'1266 378:2297 [5,10,4] 5 [5,10,4] 5 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.8 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:06:31.891710+0800 0'0 378:328 [8,7,0] 8 [8,7,0] 8 0'0 2022-04-04T20:06:31.891403+0800 0'0 2022-04-03T16:52:59.854233+0800 04.e 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T15:22:27.031381+0800 251'37 378:20980 [4,10,1] 4 [4,10,1] 4 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 06.c 2 0 0 0 0 554 0 0 3 3 active+clean 2022-04-04T18:07:06.713164+0800 205'3 378:185 [0,2,10] 0 [0,2,10] 0 205'3 2022-04-04T18:07:06.713106+0800 0'0 2022-04-03T16:54:54.909163+0800 07.d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:26.940530+0800 0'0 377:265 [4,9,0] 4 [4,9,0] 4 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.440535+0800 0'0 377:129 [10,6,3] 10 [10,6,3] 10 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.d 3 0 0 0 0 12582912 0 0 57 57 active+clean 2022-04-04T21:40:30.572238+0800 215'57 378:257 [5,2,11] 5 [5,2,11] 5 215'57 2022-04-04T21:40:30.572183+0800 0'0 2022-04-03T17:52:00.867759+0800 04.1 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:30:40.645642+0800 0'0 378:323 [8,2,5] 8 [8,2,5] 8 0'0 2022-04-04T17:30:40.645586+0800 0'0 2022-04-03T16:53:03.970520+0800 03.6 6 0 0 0 0 0 0 0 2135 2135 active+clean 2022-04-04T15:22:25.812750+0800 378'2135 378:3519 [0,8,2] 0 [0,8,2] 0 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.7 1 0 0 0 0 873 0 0 1 1 active+clean 2022-04-04T15:22:25.969214+0800 72'1 377:352 [3,10,1] 3 [3,10,1] 3 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 05.0 0 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T15:22:30.851441+0800 264'2 377:356 [5,1,4] 5 [5,1,4] 5 0'0 2022-04-03T16:53:06.037035+0800 0'0 2022-04-03T16:53:06.037035+0800 06.3 3 0 0 0 0 34 0 0 3 3 active+clean 2022-04-04T17:09:33.559693+0800 205'3 378:267 [4,9,7] 4 [4,9,7] 4 205'3 2022-04-04T17:09:33.559632+0800 0'0 2022-04-03T16:54:54.909163+0800 07.2 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:16:48.183999+0800 0'0 378:266 [0,6,10] 0 [0,6,10] 0 0'0 2022-04-04T20:16:48.183949+0800 0'0 2022-04-03T16:55:03.148848+0800 010.e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.386047+0800 0'0 377:129 [4,0,8] 4 [4,0,8] 4 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.c 3 0 0 0 0 8396800 0 0 46 46 active+clean 2022-04-04T18:23:31.595717+0800 378'46 378:242 [5,11,8] 5 [5,11,8] 5 378'46 2022-04-04T18:23:31.595542+0800 378'46 2022-04-04T18:23:31.595542+0800 04.0 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T15:22:25.982758+0800 251'37 378:20973 [0,4,9] 0 [0,4,9] 0 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 03.7 6 0 0 0 0 330 0 0 985 985 active+clean 2022-04-04T15:22:26.471763+0800 378'985 378:1893 [1,6,9] 1 [1,6,9] 1 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.6 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:29:52.083452+0800 0'0 378:309 [1,6,10] 1 [1,6,10] 1 0'0 2022-04-04T19:29:52.083357+0800 0'0 2022-04-03T16:52:59.854233+0800 05.1 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:54:15.604320+0800 0'0 378:360 [3,11,7] 3 [3,11,7] 3 0'0 2022-04-04T20:54:15.604265+0800 0'0 2022-04-03T16:53:06.037035+0800 06.2 0 0 0 0 0 0 0 0 1 1 active+clean 2022-04-04T15:22:37.663652+0800 205'1 377:282 [9,0,6] 9 [9,0,6] 9 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.3 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:25.812616+0800 0'0 377:260 [0,4,8] 0 [0,4,8] 0 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.408682+0800 0'0 377:130 [2,0,8] 2 [2,0,8] 2 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.f 4 0 0 0 0 10616832 0 0 51 51 active+clean 2022-04-04T20:38:14.483470+0800 215'51 378:412 [7,11,10] 7 [7,11,10] 7 215'51 2022-04-04T20:38:14.483430+0800 0'0 2022-04-03T17:52:00.867759+0800 04.3 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T15:22:39.066818+0800 251'37 378:20972 [9,5,2] 9 [9,5,2] 9 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 03.4 11 0 0 0 0 358 0 0 2434 2434 active+clean 2022-04-04T20:34:55.875668+0800 378'2434 378:24713 [1,5,8] 1 [1,5,8] 1 378'2122 2022-04-04T20:34:55.875495+0800 378'2122 2022-04-04T20:34:55.875495+0800 02.5 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:32:42.186686+0800 0'0 378:329 [7,0,4] 7 [7,0,4] 7 0'0 2022-04-04T19:32:42.186631+0800 0'0 2022-04-03T16:52:59.854233+0800 05.2 1 0 0 0 0 16 0 0 3 3 active+clean 2022-04-04T20:59:33.010620+0800 264'3 378:363 [3,0,5] 3 [3,0,5] 3 264'3 2022-04-04T20:59:33.010572+0800 0'0 2022-04-03T16:53:06.037035+0800 06.1 2 0 0 0 0 90 0 0 8 8 active+clean 2022-04-04T15:22:27.605063+0800 236'8 377:280 [6,9,3] 6 [6,9,3] 6 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.0 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:01:10.984418+0800 0'0 377:271 [2,5,11] 2 [2,5,11] 2 0'0 2022-04-04T17:01:10.984350+0800 0'0 2022-04-04T17:01:10.984350+0800 010.8 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.465454+0800 0'0 377:130 [9,5,3] 9 [9,5,3] 9 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.a 1 0 0 0 0 4194304 0 0 26 26 active+clean 2022-04-04T15:22:25.970557+0800 215'26 377:209 [3,11,10] 3 [3,11,10] 3 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 04.6 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:30.851935+0800 0'0 377:320 [5,1,4] 5 [5,1,4] 5 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 02.0 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:24:50.397324+0800 0'0 378:330 [3,5,11] 3 [3,5,11] 3 0'0 2022-04-04T18:24:50.397271+0800 0'0 2022-04-03T16:52:59.854233+0800 03.1 8 0 0 0 0 0 0 0 2506 2506 active+clean 2022-04-04T18:09:55.879975+0800 378'2506 378:4093 [0,4,10] 0 [0,4,10] 0 378'1764 2022-04-04T18:09:55.879822+0800 378'1764 2022-04-04T18:09:55.879822+0800 05.7 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T22:10:29.553385+0800 0'0 378:358 [8,1,7] 8 [8,1,7] 8 0'0 2022-04-04T22:10:29.553338+0800 0'0 2022-04-03T16:53:06.037035+0800 06.4 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:25:06.358558+0800 0'0 378:280 [9,2,11] 9 [9,2,11] 9 0'0 2022-04-04T19:25:06.358503+0800 0'0 2022-04-03T16:54:54.909163+0800 07.5 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:30.830605+0800 0'0 377:264 [7,11,2] 7 [7,11,2] 7 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.384146+0800 0'0 377:130 [2,1,8] 2 [2,1,8] 2 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.9 7 0 0 0 0 25186304 0 0 48 48 active+clean 2022-04-04T15:22:30.434864+0800 215'48 377:276 [6,2,0] 6 [6,2,0] 6 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 04.5 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:57:31.289390+0800 0'0 378:324 [9,3,5] 9 [9,3,5] 9 0'0 2022-04-04T17:57:31.289325+0800 0'0 2022-04-03T16:53:03.970520+0800 01.0 1 0 0 0 0 0 0 0 1 1 active+clean 2022-04-04T15:22:38.985703+0800 239'1 377:400 [8,5,2] 8 [8,5,2] 8 0'0 2022-04-03T16:36:15.631796+0800 0'0 2022-04-03T16:36:15.631796+0800 02.3 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:39.118775+0800 0'0 377:323 [10,2,5] 10 [10,2,5] 10 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 03.2 6 0 0 0 0 0 0 0 1792 1792 active+clean 2022-04-04T15:22:39.064858+0800 378'1792 378:3021 [10,5,3] 10 [10,5,3] 10 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 05.4 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.458830+0800 0'0 377:351 [7,11,8] 7 [7,11,8] 7 0'0 2022-04-03T16:53:06.037035+0800 0'0 2022-04-03T16:53:06.037035+0800 06.7 2 0 0 0 0 0 0 0 4 4 active+clean 2022-04-04T15:22:38.756756+0800 205'4 377:282 [10,1,2] 10 [10,1,2] 10 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.6 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:51:15.297786+0800 0'0 378:265 [0,4,8] 0 [0,4,8] 0 0'0 2022-04-04T18:51:15.297215+0800 0'0 2022-04-03T16:55:03.148848+0800 010.9 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.432790+0800 0'0 377:130 [3,11,9] 3 [3,11,9] 3 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.b 4 0 0 0 0 16777216 0 0 36 36 active+clean 2022-04-04T20:48:28.233988+0800 215'36 378:239 [8,4,1] 8 [8,4,1] 8 215'36 2022-04-04T20:48:28.233939+0800 0'0 2022-04-03T17:52:00.867759+0800 04.7 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:15:14.120970+0800 0'0 378:309 [1,10,3] 1 [1,10,3] 1 0'0 2022-04-04T20:15:14.120924+0800 0'0 2022-04-03T16:53:03.970520+0800 02.1 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:03:44.457237+0800 0'0 378:329 [9,0,5] 9 [9,0,5] 9 0'0 2022-04-04T19:03:44.457189+0800 0'0 2022-04-03T16:52:59.854233+0800 03.0 10 0 0 0 0 220 0 0 2334 2334 active+clean 2022-04-04T15:22:26.860295+0800 378'2334 378:3899 [5,2,8] 5 [5,2,8] 5 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 05.6 1 0 0 0 0 0 0 0 4 4 active+clean 2022-04-04T17:13:19.513024+0800 264'4 378:362 [11,2,10] 11 [11,2,10] 11 264'4 2022-04-04T17:13:19.512947+0800 0'0 2022-04-03T16:53:06.037035+0800 06.5 1 0 0 0 0 22 0 0 1 1 active+clean 2022-04-04T15:22:22.843630+0800 205'1 377:277 [2,5,11] 2 [2,5,11] 2 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.4 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:32:27.335291+0800 0'0 378:265 [0,9,2] 0 [0,9,2] 0 0'0 2022-04-04T21:32:27.335179+0800 0'0 2022-04-03T16:55:03.148848+0800 010.a 0 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T16:59:38.421333+0800 260'2 377:133 [10,7,2] 10 [10,7,2] 10 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.8 1 0 0 0 0 4194304 0 0 23 23 active+clean 2022-04-04T18:24:20.752914+0800 378'23 378:223 [4,0,7] 4 [4,0,7] 4 378'23 2022-04-04T18:24:20.752712+0800 378'23 2022-04-04T18:24:20.752712+0800 04.4 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:03:04.126035+0800 0'0 378:327 [8,0,3] 8 [8,0,3] 8 0'0 2022-04-04T18:03:04.125992+0800 0'0 2022-04-03T16:53:03.970520+0800 03.3 9 0 0 0 0 330 0 0 2051 2051 active+clean 2022-04-04T21:05:52.223272+0800 378'2051 378:3499 [10,4,11] 10 [10,4,11] 10 378'1853 2022-04-04T21:05:52.222814+0800 0'0 2022-04-03T16:53:01.903832+0800 02.2 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:32:07.667626+0800 0'0 378:330 [3,1,10] 3 [3,1,10] 3 0'0 2022-04-04T17:32:07.667466+0800 0'0 2022-04-03T16:52:59.854233+0800 05.5 1 0 0 0 0 373 0 0 1 1 active+clean 2022-04-04T15:22:26.414028+0800 251'1 377:315 [11,9,4] 11 [11,9,4] 11 0'0 2022-04-03T16:53:06.037035+0800 0'0 2022-04-03T16:53:06.037035+0800 06.6 2 0 0 0 0 0 0 0 6 6 active+clean 2022-04-04T17:44:13.862917+0800 239'6 378:281 [1,4,8] 1 [1,4,8] 1 239'6 2022-04-04T17:44:13.862840+0800 0'0 2022-04-03T16:54:54.909163+0800 07.7 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:43:39.845403+0800 0'0 378:270 [3,6,11] 3 [3,6,11] 3 0'0 2022-04-04T20:43:39.845354+0800 0'0 2022-04-03T16:55:03.148848+0800 010.c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.383389+0800 0'0 377:129 [4,8,0] 4 [4,8,0] 4 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.e 2 0 0 0 0 8388608 0 0 20 20 active+clean 2022-04-04T15:22:26.531259+0800 215'20 377:211 [11,10,3] 11 [11,10,3] 11 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 04.2 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.520565+0800 0'0 377:319 [9,3,7] 9 [9,3,7] 9 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 03.5 7 0 0 0 0 154 0 0 2136 2136 active+clean 2022-04-04T15:22:21.396047+0800 378'2136 378:3527 [2,0,6] 2 [2,0,6] 2 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.4 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:16:00.165750+0800 0'0 378:306 [11,7,8] 11 [11,7,8] 11 0'0 2022-04-04T20:16:00.165699+0800 0'0 2022-04-03T16:52:59.854233+0800 05.3 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:46:48.189526+0800 0'0 378:330 [0,3,9] 0 [0,3,9] 0 0'0 2022-04-04T19:46:48.189479+0800 0'0 2022-04-03T16:53:06.037035+0800 06.0 1 0 0 0 0 0 0 0 5 5 active+clean 2022-04-04T20:08:24.618717+0800 236'5 378:205 [9,2,1] 9 [9,2,1] 9 236'5 2022-04-04T20:08:24.618669+0800 0'0 2022-04-03T16:54:54.909163+0800 07.1 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:26.299528+0800 0'0 377:266 [2,10,11] 2 [2,10,11] 2 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.7 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.405322+0800 0'0 377:129 [10,1,5] 10 [10,1,5] 10 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 09.4 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:02:21.774699+0800 264'2 377:180 [1,3,8] 1 [1,3,8] 1 0'0 2022-04-04T16:59:15.464569+0800 0'0 2022-04-04T16:59:15.464569+0800 08.5 4 0 0 0 0 16777216 0 0 36 36 active+clean 2022-04-04T15:22:27.779712+0800 215'36 377:229 [6,8,1] 6 [6,8,1] 6 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 04.9 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T15:22:30.851404+0800 251'37 378:20974 [5,1,2] 5 [5,1,2] 5 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 03.e 6 0 0 0 0 110 0 0 1522 1522 active+clean 2022-04-04T17:07:05.531122+0800 378'1522 378:2647 [7,10,4] 7 [7,10,4] 7 303'972 2022-04-04T17:07:05.531053+0800 0'0 2022-04-03T16:53:01.903832+0800 02.f 2 0 0 0 0 92 0 0 2 2 active+clean 2022-04-04T17:52:32.137249+0800 72'2 378:391 [6,9,4] 6 [6,9,4] 6 72'2 2022-04-04T17:52:32.137206+0800 0'0 2022-04-03T16:52:59.854233+0800 06.b 2 0 0 0 0 0 0 0 5 5 active+clean 2022-04-04T15:22:26.931126+0800 236'5 377:268 [1,3,8] 1 [1,3,8] 1 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.255720+0800 0'0 377:272 [6,1,9] 6 [6,1,9] 6 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.18 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.352338+0800 0'0 377:130 [11,6,4] 11 [11,6,4] 11 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.1a 0 0 0 0 0 0 0 0 17 17 active+clean 2022-04-04T15:22:26.529111+0800 215'17 377:208 [11,2,10] 11 [11,2,10] 11 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 03.11 6 0 0 0 0 110 0 0 1523 1523 active+clean 2022-04-04T19:08:46.182759+0800 378'1523 378:2649 [7,4,10] 7 [7,4,10] 7 378'1181 2022-04-04T19:08:46.182707+0800 0'0 2022-04-03T16:53:01.903832+0800 02.10 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:05:50.647261+0800 0'0 378:328 [8,7,0] 8 [8,7,0] 8 0'0 2022-04-04T21:05:50.647215+0800 0'0 2022-04-03T16:52:59.854233+0800 04.16 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:03:03.580096+0800 0'0 378:324 [10,0,5] 10 [10,0,5] 10 0'0 2022-04-04T18:03:03.580023+0800 0'0 2022-04-04T18:03:03.580023+0800 06.14 5 0 0 0 0 2631 0 0 18 18 active+clean 2022-04-04T15:22:38.803098+0800 239'18 377:291 [8,4,7] 8 [8,4,7] 8 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.15 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.432336+0800 0'0 377:264 [10,1,7] 10 [10,1,7] 10 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.19 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.422600+0800 0'0 377:129 [10,4,0] 10 [10,4,0] 10 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.1b 5 0 0 0 0 20971520 0 0 41 41 active+clean 2022-04-04T22:31:24.260509+0800 215'41 378:241 [4,11,10] 4 [4,11,10] 4 215'41 2022-04-04T22:31:24.260461+0800 0'0 2022-04-03T17:52:00.867759+0800 03.10 2 0 0 0 0 0 0 0 712 712 active+clean 2022-04-04T15:22:39.016737+0800 378'712 378:1389 [10,5,3] 10 [10,5,3] 10 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.11 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:35:04.352429+0800 0'0 378:331 [8,3,1] 8 [8,3,1] 8 0'0 2022-04-04T18:35:04.352376+0800 0'0 2022-04-03T16:52:59.854233+0800 04.17 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.540411+0800 0'0 377:300 [11,3,7] 11 [11,3,7] 11 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 06.15 3 0 0 0 0 644 0 0 4 4 active+clean 2022-04-04T15:22:25.640509+0800 205'4 377:265 [2,8,6] 2 [2,8,6] 2 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.14 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.851667+0800 0'0 377:268 [9,0,2] 9 [9,0,2] 9 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.1a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.429669+0800 0'0 377:130 [3,9,5] 3 [3,9,5] 3 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.18 1 0 0 0 0 4194304 0 0 23 23 active+clean 2022-04-04T18:18:24.550345+0800 215'23 378:214 [4,5,8] 4 [4,5,8] 4 215'23 2022-04-04T18:18:24.550285+0800 0'0 2022-04-03T17:52:00.867759+0800 03.13 5 0 0 0 0 110 0 0 1510 1510 active+clean 2022-04-04T15:22:26.930763+0800 378'1510 378:2609 [3,9,5] 3 [3,9,5] 3 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.12 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:30.851860+0800 0'0 377:324 [5,11,2] 5 [5,11,2] 5 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 04.14 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T15:22:30.280192+0800 251'37 378:20975 [6,1,2] 6 [6,1,2] 6 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 06.16 1 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T15:22:26.073214+0800 205'2 377:260 [0,4,9] 0 [0,4,9] 0 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.17 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:33:53.636731+0800 0'0 378:198 [4,11,10] 4 [4,11,10] 4 0'0 2022-04-04T18:33:53.636682+0800 0'0 2022-04-03T16:55:03.148848+0800 010.1b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.439698+0800 0'0 377:130 [0,4,10] 0 [0,4,10] 0 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.19 0 0 0 0 0 0 0 0 25 25 active+clean 2022-04-04T22:31:20.767031+0800 215'25 378:219 [2,5,8] 2 [2,5,8] 2 215'25 2022-04-04T22:31:20.766986+0800 0'0 2022-04-03T17:52:00.867759+0800 03.12 4 0 0 0 0 110 0 0 1154 1154 active+clean 2022-04-04T22:10:30.175172+0800 378'1154 378:2088 [9,3,0] 9 [9,3,0] 9 378'1122 2022-04-04T22:10:30.175126+0800 0'0 2022-04-03T16:53:01.903832+0800 02.13 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:39.709864+0800 0'0 377:327 [9,4,11] 9 [9,4,11] 9 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 04.15 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T15:22:38.353223+0800 251'37 378:20976 [8,7,0] 8 [8,7,0] 8 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 06.17 2 0 0 0 0 90 0 0 5 5 active+clean 2022-04-04T15:22:38.541873+0800 239'5 377:279 [10,11,5] 10 [10,11,5] 10 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.16 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.682357+0800 0'0 377:263 [6,8,0] 6 [6,8,0] 6 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.1c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.417948+0800 0'0 377:130 [2,8,7] 2 [2,8,7] 2 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.1e 4 0 0 0 0 16777216 0 0 55 55 active+clean 2022-04-04T15:22:21.530728+0800 215'55 377:331 [1,3,6] 1 [1,3,6] 1 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 03.15 5 0 0 0 0 110 0 0 1170 1170 active+clean 2022-04-04T15:22:37.773222+0800 378'1170 378:2112 [8,6,3] 8 [8,6,3] 8 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.14 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:58:25.270496+0800 0'0 378:305 [2,7,8] 2 [2,7,8] 2 0'0 2022-04-04T19:58:25.270445+0800 0'0 2022-04-03T16:52:59.854233+0800 04.12 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T18:14:03.606387+0800 0'0 378:309 [1,6,9] 1 [1,6,9] 1 0'0 2022-04-04T18:14:03.606340+0800 0'0 2022-04-03T16:53:03.970520+0800 06.10 8 0 0 0 0 112 0 0 15 15 active+clean 2022-04-04T15:22:39.183032+0800 205'15 377:292 [10,5,2] 10 [10,5,2] 10 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.11 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.518884+0800 0'0 377:264 [9,3,7] 9 [9,3,7] 9 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.1d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.414890+0800 0'0 377:129 [4,7,9] 4 [4,7,9] 4 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.1f 2 0 0 0 0 8388608 0 0 22 22 active+clean 2022-04-04T15:22:30.830221+0800 215'22 377:217 [7,4,11] 7 [7,4,11] 7 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 03.14 7 0 0 0 0 0 0 0 1804 1804 active+clean 2022-04-04T15:22:26.042066+0800 378'1804 378:3053 [3,8,0] 3 [3,8,0] 3 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.15 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:57:11.512647+0800 0'0 378:329 [9,1,5] 9 [9,1,5] 9 0'0 2022-04-04T20:57:11.512130+0800 0'0 2022-04-03T16:52:59.854233+0800 04.13 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:52:07.692891+0800 0'0 378:305 [4,8,6] 4 [4,8,6] 4 0'0 2022-04-04T20:52:07.692841+0800 0'0 2022-04-04T20:52:07.692841+0800 06.11 2 0 0 0 0 540 0 0 3 3 active+clean 2022-04-04T15:22:37.786957+0800 205'3 377:275 [8,6,11] 8 [8,6,11] 8 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.10 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T22:32:44.867665+0800 0'0 378:265 [11,10,5] 11 [11,10,5] 11 0'0 2022-04-04T22:32:44.867608+0800 0'0 2022-04-04T22:32:44.867608+0800 010.1e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.438344+0800 0'0 377:130 [5,9,0] 5 [5,9,0] 5 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.1c 3 0 0 0 0 4214784 0 0 41 41 active+clean 2022-04-04T15:22:38.468477+0800 215'41 377:257 [8,2,7] 8 [8,2,7] 8 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 03.17 4 0 0 0 0 0 0 0 1081 1081 active+clean 2022-04-04T15:22:38.843955+0800 378'1081 378:1954 [10,3,5] 10 [10,3,5] 10 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.16 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:40:03.197630+0800 0'0 378:305 [2,7,10] 2 [2,7,10] 2 0'0 2022-04-04T17:40:03.197577+0800 0'0 2022-04-03T16:52:59.854233+0800 04.10 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:18:30.917058+0800 0'0 378:309 [1,9,6] 1 [1,9,6] 1 0'0 2022-04-04T20:18:30.916510+0800 0'0 2022-04-03T16:53:03.970520+0800 06.12 1 0 0 0 0 0 0 0 5 5 active+clean 2022-04-04T15:22:27.436176+0800 236'5 377:278 [7,4,9] 7 [7,4,9] 7 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.13 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T20:59:13.229303+0800 0'0 378:268 [10,2,5] 10 [10,2,5] 10 0'0 2022-04-04T20:59:13.229253+0800 0'0 2022-04-03T16:55:03.148848+0800 010.1f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.374259+0800 0'0 377:130 [2,1,6] 2 [2,1,6] 2 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.1d 0 0 0 0 0 0 0 0 17 17 active+clean 2022-04-04T21:55:48.422035+0800 215'17 378:213 [0,10,5] 0 [0,10,5] 0 215'17 2022-04-04T21:55:48.421979+0800 0'0 2022-04-03T17:52:00.867759+0800 03.16 6 0 0 0 0 110 0 0 1179 1179 active+clean 2022-04-04T18:31:53.906728+0800 378'1179 378:2148 [2,1,9] 2 [2,1,9] 2 378'875 2022-04-04T18:31:53.906677+0800 0'0 2022-04-03T16:53:01.903832+0800 02.17 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.315533+0800 0'0 377:325 [4,5,11] 4 [4,5,11] 4 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 04.11 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:25.820179+0800 0'0 377:300 [0,4,8] 0 [0,4,8] 0 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 06.13 4 0 0 0 0 34 0 0 6 6 active+clean 2022-04-04T17:59:22.017997+0800 205'6 378:286 [8,1,7] 8 [8,1,7] 8 205'6 2022-04-04T17:59:22.017942+0800 0'0 2022-04-03T16:54:54.909163+0800 07.12 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.708233+0800 0'0 377:266 [10,11,2] 10 [10,11,2] 10 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.10 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.463131+0800 0'0 377:130 [9,4,0] 9 [9,4,0] 9 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.12 2 0 0 0 0 4202496 0 0 23 23 active+clean 2022-04-04T18:38:29.110382+0800 215'23 378:244 [3,9,6] 3 [3,9,6] 3 215'23 2022-04-04T18:38:29.110332+0800 0'0 2022-04-03T17:52:00.867759+0800 07.1d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.318007+0800 0'0 377:261 [5,9,0] 5 [5,9,0] 5 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 03.19 11 0 0 0 0 110 0 0 2271 2271 active+clean 2022-04-04T15:22:25.886617+0800 378'2271 378:3803 [1,4,10] 1 [1,4,10] 1 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.18 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:37.469975+0800 0'0 377:323 [9,2,6] 9 [9,2,6] 9 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 04.1e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:46:58.994342+0800 0'0 378:305 [0,4,8] 0 [0,4,8] 0 0'0 2022-04-04T21:46:58.994286+0800 0'0 2022-04-03T16:53:03.970520+0800 06.1c 1 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T19:21:58.590310+0800 156'2 378:280 [7,11,3] 7 [7,11,3] 7 156'2 2022-04-04T19:21:58.590251+0800 0'0 2022-04-03T16:54:54.909163+0800 010.11 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.410495+0800 0'0 377:130 [5,0,2] 5 [5,0,2] 5 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.13 3 0 0 0 0 12582912 0 0 34 34 active+clean 2022-04-04T21:08:38.667888+0800 215'34 378:231 [1,7,4] 1 [1,7,4] 1 215'34 2022-04-04T21:08:38.667292+0800 0'0 2022-04-03T17:52:00.867759+0800 03.18 5 0 0 0 0 110 0 0 1166 1166 active+clean 2022-04-04T15:22:37.764288+0800 378'1166 378:2109 [10,6,11] 10 [10,6,11] 10 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.19 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.540510+0800 0'0 377:300 [11,4,7] 11 [11,4,7] 11 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 04.1f 1 0 0 0 0 0 0 0 37 37 active+clean 2022-04-04T19:35:28.972562+0800 251'37 378:20986 [6,8,1] 6 [6,8,1] 6 251'37 2022-04-04T19:35:28.972508+0800 0'0 2022-04-03T16:53:03.970520+0800 06.1d 2 0 0 0 0 0 0 0 4 4 active+clean 2022-04-04T15:22:38.832675+0800 205'4 377:277 [8,5,1] 8 [8,5,1] 8 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.1c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:25.735065+0800 0'0 377:260 [2,5,9] 2 [2,5,9] 2 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.12 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.422445+0800 0'0 377:129 [10,5,2] 10 [10,5,2] 10 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.10 0 0 0 0 0 0 0 0 24 24 active+clean 2022-04-04T15:22:25.973210+0800 215'24 377:212 [3,5,10] 3 [3,5,10] 3 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.1f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.828665+0800 0'0 377:264 [8,5,1] 8 [8,5,1] 8 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 03.1b 3 0 0 0 0 110 0 0 455 455 active+clean 2022-04-04T20:08:00.827659+0800 378'455 378:1048 [6,4,8] 6 [6,4,8] 6 378'385 2022-04-04T20:08:00.827619+0800 0'0 2022-04-03T16:53:01.903832+0800 02.1a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:05:56.636457+0800 0'0 378:330 [3,5,1] 3 [3,5,1] 3 0'0 2022-04-04T19:05:56.636388+0800 0'0 2022-04-03T16:52:59.854233+0800 04.1c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:07:01.035323+0800 0'0 378:325 [5,1,4] 5 [5,1,4] 5 0'0 2022-04-04T17:07:01.035274+0800 0'0 2022-04-03T16:53:03.970520+0800 06.1e 0 0 0 0 0 0 0 0 3 3 active+clean 2022-04-04T22:07:55.342125+0800 239'3 378:280 [5,2,11] 5 [5,2,11] 5 239'3 2022-04-04T22:07:55.342047+0800 0'0 2022-04-03T16:54:54.909163+0800 010.13 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.421461+0800 0'0 377:130 [1,8,3] 1 [1,8,3] 1 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.11 6 0 0 0 0 25165824 0 0 30 30 active+clean 2022-04-04T15:22:26.860607+0800 215'30 377:222 [5,8,2] 5 [5,8,2] 5 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 07.1e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.312930+0800 0'0 377:260 [11,5,2] 11 [11,5,2] 11 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 03.1a 6 0 0 0 0 110 0 0 1179 1179 active+clean 2022-04-04T15:22:27.318326+0800 378'1179 378:2139 [5,11,9] 5 [5,11,9] 5 0'0 2022-04-03T16:53:01.903832+0800 0'0 2022-04-03T16:53:01.903832+0800 02.1b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.318197+0800 0'0 377:322 [5,2,9] 5 [5,2,9] 5 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 04.1d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:35:03.322133+0800 0'0 378:323 [7,9,3] 7 [7,9,3] 7 0'0 2022-04-04T21:35:03.322071+0800 0'0 2022-04-03T16:53:03.970520+0800 06.1f 4 0 0 0 0 554 0 0 10 10 active+clean 2022-04-04T17:59:11.539509+0800 236'10 378:271 [11,10,3] 11 [11,10,3] 11 236'10 2022-04-04T17:59:11.539462+0800 0'0 2022-04-03T16:54:54.909163+0800 010.14 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.393571+0800 0'0 377:130 [1,8,6] 1 [1,8,6] 1 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.16 1 0 0 0 0 8192 0 0 28 28 active+clean 2022-04-04T18:30:09.100578+0800 215'28 378:231 [9,0,2] 9 [9,0,2] 9 215'28 2022-04-04T18:30:09.100528+0800 0'0 2022-04-03T17:52:00.867759+0800 03.1d 6 0 0 0 0 220 0 0 1253 1253 active+clean 2022-04-04T19:48:25.408431+0800 378'1253 378:2270 [8,4,7] 8 [8,4,7] 8 378'1025 2022-04-04T19:48:25.408373+0800 0'0 2022-04-03T16:53:01.903832+0800 02.1c 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:30.830653+0800 0'0 377:324 [7,4,1] 7 [7,4,1] 7 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 04.1a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:37.787869+0800 0'0 377:319 [8,11,6] 8 [8,11,6] 8 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 06.18 0 0 0 0 0 0 0 0 3 3 active+clean 2022-04-04T17:36:11.185304+0800 236'3 378:278 [11,8,7] 11 [11,8,7] 11 236'3 2022-04-04T17:36:11.185258+0800 0'0 2022-04-03T16:54:54.909163+0800 07.19 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T19:01:23.522037+0800 0'0 378:272 [8,11,4] 8 [8,11,4] 8 0'0 2022-04-04T19:01:23.521994+0800 0'0 2022-04-03T16:55:03.148848+0800 010.15 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.454770+0800 0'0 377:130 [11,9,2] 11 [11,9,2] 11 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.17 1 0 0 0 0 4194304 0 0 21 21 active+clean 2022-04-04T15:22:25.735027+0800 215'21 377:210 [2,9,6] 2 [2,9,6] 2 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 03.1c 7 0 0 0 0 110 0 0 2222 2222 active+clean 2022-04-04T17:12:49.656665+0800 378'2222 378:3688 [4,11,9] 4 [4,11,9] 4 378'1432 2022-04-04T17:12:49.656611+0800 0'0 2022-04-03T16:53:01.903832+0800 02.1d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:03:30.277360+0800 0'0 378:328 [10,6,3] 10 [10,6,3] 10 0'0 2022-04-04T21:03:30.277308+0800 0'0 2022-04-03T16:52:59.854233+0800 04.1b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:38.670932+0800 0'0 377:323 [9,1,4] 9 [9,1,4] 9 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 06.19 1 0 0 0 0 0 0 0 2 2 active+clean 2022-04-04T18:51:08.770487+0800 205'2 378:279 [10,1,7] 10 [10,1,7] 10 205'2 2022-04-04T18:51:08.770422+0800 0'0 2022-04-03T16:54:54.909163+0800 07.18 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:25.815539+0800 0'0 377:262 [0,8,3] 0 [0,8,3] 0 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.16 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.398330+0800 0'0 377:130 [2,1,9] 2 [2,1,9] 2 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.14 1 0 0 0 0 4194304 0 0 19 19 active+clean 2022-04-04T15:22:27.334764+0800 215'19 377:214 [4,8,11] 4 [4,8,11] 4 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 03.1f 6 0 0 0 0 110 0 0 1523 1523 active+clean 2022-04-04T16:56:19.427764+0800 378'1523 378:2653 [10,0,3] 10 [10,0,3] 10 251'957 2022-04-04T16:56:19.427716+0800 0'0 2022-04-03T16:53:01.903832+0800 02.1e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T17:16:01.298937+0800 0'0 378:306 [11,5,9] 11 [11,5,9] 11 0'0 2022-04-04T17:16:01.298887+0800 0'0 2022-04-03T16:52:59.854233+0800 04.18 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T21:46:56.364832+0800 0'0 378:326 [2,9,1] 2 [2,9,1] 2 0'0 2022-04-04T21:46:56.364784+0800 0'0 2022-04-04T21:46:56.364784+0800 06.1a 1 0 0 0 0 0 0 0 4 4 active+clean 2022-04-04T19:05:38.955643+0800 236'4 378:288 [10,0,5] 10 [10,0,5] 10 236'4 2022-04-04T19:05:38.955598+0800 0'0 2022-04-03T16:54:54.909163+0800 07.1b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:25.958790+0800 0'0 377:262 [0,5,8] 0 [0,5,8] 0 0'0 2022-04-03T16:55:03.148848+0800 0'0 2022-04-03T16:55:03.148848+0800 010.17 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T16:59:38.460120+0800 0'0 377:130 [9,7,4] 9 [9,7,4] 9 0'0 2022-04-04T16:59:37.255617+0800 0'0 2022-04-04T16:59:37.255617+0800 08.15 6 0 0 0 0 22712320 0 0 78 78 active+clean 2022-04-04T15:22:30.830236+0800 215'78 377:679 [7,2,1] 7 [7,2,1] 7 0'0 2022-04-03T17:52:00.867759+0800 0'0 2022-04-03T17:52:00.867759+0800 03.1e 6 0 0 0 0 0 0 0 1793 1793 active+clean 2022-04-04T20:43:14.098177+0800 378'1793 378:3024 [11,6,8] 11 [11,6,8] 11 378'1573 2022-04-04T20:43:14.098134+0800 0'0 2022-04-03T16:53:01.903832+0800 02.1f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:22.313511+0800 0'0 377:300 [11,3,5] 11 [11,3,5] 11 0'0 2022-04-03T16:52:59.854233+0800 0'0 2022-04-03T16:52:59.854233+0800 04.19 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T15:22:27.197109+0800 0'0 377:317 [7,0,10] 7 [7,0,10] 7 0'0 2022-04-03T16:53:03.970520+0800 0'0 2022-04-03T16:53:03.970520+0800 06.1b 2 0 0 0 0 1638 0 0 14 14 active+clean 2022-04-04T15:22:27.569007+0800 239'14 377:292 [6,10,3] 6 [6,10,3] 6 0'0 2022-04-03T16:54:54.909163+0800 0'0 2022-04-03T16:54:54.909163+0800 07.1a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-04-04T22:02:20.057612+0800 0'0 378:264 [3,5,10] 3 [3,5,10] 3 0'0 2022-04-04T22:02:20.057536+0800 0'0 2022-04-03T16:55:03.148848+0800 0 10 0 0 0 0 0 0 0 0 2 29 0 0 0 0 0 0 0 0 0 08 176 0 0 0 0 624996403 0 0 2175 21757 0 0 0 0 0 0 0 0 0 02 4 0 0 0 0 1323 0 0 4 41 1 0 0 0 0 0 0 0 1 13 209 0 0 0 0 3702 0 0 54098 540984 8 0 0 0 0 0 0 0 296 2965 3 0 0 0 0 389 0 0 10 106 60 0 0 0 0 7125 0 0 148 148 sum 461 0 0 0 0 625008942 0 0 56734 56734OSD_STAT USED AVAIL USED_RAW TOTAL HB_PEERS PG_SUM PRIMARY_PG_SUM10 144 MiB 9.9 GiB 144 MiB 10 GiB [0,1,2,3,4,5,6,7,9,11] 80 309 192 MiB 9.8 GiB 192 MiB 10 GiB [0,1,2,3,4,5,6,7,8,10,11] 89 318 193 MiB 9.8 GiB 193 MiB 10 GiB [0,1,2,3,4,5,6,7,9,11] 80 307 162 MiB 9.8 GiB 162 MiB 10 GiB [0,1,2,3,4,6,8,9,10,11] 69 195 191 MiB 9.8 GiB 191 MiB 10 GiB [0,1,2,3,4,6,8,9,10,11] 88 296 197 MiB 9.8 GiB 197 MiB 10 GiB [0,1,2,3,4,5,7,8,9,10,11] 64 190 177 MiB 9.8 GiB 177 MiB 10 GiB [1,2,3,4,5,6,7,8,9,10,11] 73 221 198 MiB 9.8 GiB 198 MiB 10 GiB [0,2,3,4,5,6,7,8,9,10] 77 2711 160 MiB 9.8 GiB 160 MiB 10 GiB [0,2,3,4,5,6,7,8,9,10] 72 254 193 MiB 9.8 GiB 193 MiB 10 GiB [0,1,3,5,6,7,8,9,10,11] 76 222 198 MiB 9.8 GiB 198 MiB 10 GiB [0,1,3,5,6,7,8,9,10,11] 74 233 95 MiB 9.9 GiB 95 MiB 10 GiB [0,1,2,4,5,6,7,8,9,10,11] 73 28sum 2.1 GiB 118 GiB 2.1 GiB 120 GiB * NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details.dumped all |
4) CRUSH map: 它保存的信息包括集群设备列表、bucket列表、故障域（failure domain）分层结构、保存数据时用到的为故障域定义的规则（rules）等。查看crush map命令
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 | cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd crush dump{ "devices": [ { "id": 0, "name": "osd.0", "class": "hdd" }, { "id": 1, "name": "osd.1", "class": "hdd" }, { "id": 2, "name": "osd.2", "class": "hdd" }, { "id": 3, "name": "osd.3", "class": "hdd" }, { "id": 4, "name": "osd.4", "class": "hdd" }, { "id": 5, "name": "osd.5", "class": "hdd" }, { "id": 6, "name": "osd.6", "class": "hdd" }, { "id": 7, "name": "osd.7", "class": "hdd" }, { "id": 8, "name": "osd.8", "class": "hdd" }, { "id": 9, "name": "osd.9", "class": "hdd" }, { "id": 10, "name": "osd.10", "class": "hdd" }, { "id": 11, "name": "osd.11", "class": "hdd" } ], "types": [ { "type_id": 0, "name": "osd" }, { "type_id": 1, "name": "host" }, { "type_id": 2, "name": "chassis" }, { "type_id": 3, "name": "rack" }, { "type_id": 4, "name": "row" }, { "type_id": 5, "name": "pdu" }, { "type_id": 6, "name": "pod" }, { "type_id": 7, "name": "room" }, { "type_id": 8, "name": "datacenter" }, { "type_id": 9, "name": "zone" }, { "type_id": 10, "name": "region" }, { "type_id": 11, "name": "root" } ], "buckets": [ { "id": -1, "name": "default", "type_id": 11, "type_name": "root", "weight": 7704, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -3, "weight": 1926, "pos": 0 }, { "id": -5, "weight": 1926, "pos": 1 }, { "id": -7, "weight": 1926, "pos": 2 }, { "id": -9, "weight": 1926, "pos": 3 } ] }, { "id": -2, "name": "default~hdd", "type_id": 11, "type_name": "root", "weight": 7704, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -4, "weight": 1926, "pos": 0 }, { "id": -6, "weight": 1926, "pos": 1 }, { "id": -8, "weight": 1926, "pos": 2 }, { "id": -10, "weight": 1926, "pos": 3 } ] }, { "id": -3, "name": "ceph-node1", "type_id": 1, "type_name": "host", "weight": 1926, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 0, "weight": 642, "pos": 0 }, { "id": 1, "weight": 642, "pos": 1 }, { "id": 11, "weight": 642, "pos": 2 } ] }, { "id": -4, "name": "ceph-node1~hdd", "type_id": 1, "type_name": "host", "weight": 1926, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 0, "weight": 642, "pos": 0 }, { "id": 1, "weight": 642, "pos": 1 }, { "id": 11, "weight": 642, "pos": 2 } ] }, { "id": -5, "name": "ceph-node2", "type_id": 1, "type_name": "host", "weight": 1926, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 2, "weight": 642, "pos": 0 }, { "id": 3, "weight": 642, "pos": 1 }, { "id": 4, "weight": 642, "pos": 2 } ] }, { "id": -6, "name": "ceph-node2~hdd", "type_id": 1, "type_name": "host", "weight": 1926, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 2, "weight": 642, "pos": 0 }, { "id": 3, "weight": 642, "pos": 1 }, { "id": 4, "weight": 642, "pos": 2 } ] }, { "id": -7, "name": "ceph-node3", "type_id": 1, "type_name": "host", "weight": 1926, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 5, "weight": 642, "pos": 0 }, { "id": 6, "weight": 642, "pos": 1 }, { "id": 7, "weight": 642, "pos": 2 } ] }, { "id": -8, "name": "ceph-node3~hdd", "type_id": 1, "type_name": "host", "weight": 1926, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 5, "weight": 642, "pos": 0 }, { "id": 6, "weight": 642, "pos": 1 }, { "id": 7, "weight": 642, "pos": 2 } ] }, { "id": -9, "name": "ceph-node4", "type_id": 1, "type_name": "host", "weight": 1926, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 8, "weight": 642, "pos": 0 }, { "id": 9, "weight": 642, "pos": 1 }, { "id": 10, "weight": 642, "pos": 2 } ] }, { "id": -10, "name": "ceph-node4~hdd", "type_id": 1, "type_name": "host", "weight": 1926, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 8, "weight": 642, "pos": 0 }, { "id": 9, "weight": 642, "pos": 1 }, { "id": 10, "weight": 642, "pos": 2 } ] } ], "rules": [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ], "tunables": { "choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_descend_once": 1, "chooseleaf_vary_r": 1, "chooseleaf_stable": 1, "straw_calc_version": 1, "allowed_bucket_algs": 54, "profile": "jewel", "optimal_tunables": 1, "legacy_tunables": 0, "minimum_required_version": "jewel", "require_feature_tunables": 1, "require_feature_tunables2": 1, "has_v2_rules": 0, "require_feature_tunables3": 1, "has_v3_rules": 0, "has_v4_buckets": 1, "require_feature_tunables5": 1, "has_v5_rules": 0 }, "choose_args": {}} |
5) MDS map: 它保存的信息包括MDS map当前版本号（epoch）、MDS map的创建和修改时间、数据和元数据存储池的ID、集群MDS数量以及MDS状态。查看命令
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | cephadmin@ceph-deploy:~/ceph-cluster$ ceph mds metadata [ { "name": "ceph-mds2", "addr": "[v2:192.168.1.204:6800/2803369705,v1:192.168.1.204:6801/2803369705]", "arch": "x86_64", "ceph_release": "pacific", "ceph_version": "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)", "ceph_version_short": "16.2.7", "cpu": "11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz", "distro": "ubuntu", "distro_description": "Ubuntu 18.04.5 LTS", "distro_version": "18.04", "hostname": "ceph-mon2", "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", "kernel_version": "4.15.0-112-generic", "mem_swap_kb": "969960", "mem_total_kb": "1008784", "os": "Linux" }, { "name": "ceph-mds1", "addr": "[v2:192.168.1.203:6800/3979017759,v1:192.168.1.203:6801/3979017759]", "arch": "x86_64", "ceph_release": "pacific", "ceph_version": "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)", "ceph_version_short": "16.2.7", "cpu": "11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz", "distro": "ubuntu", "distro_description": "Ubuntu 18.04.5 LTS", "distro_version": "18.04", "hostname": "ceph-mon1", "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", "kernel_version": "4.15.0-112-generic", "mem_swap_kb": "969960", "mem_total_kb": "1008784", "os": "Linux" }, { "name": "ceph-mds3", "addr": "[v2:192.168.1.205:6800/2412475465,v1:192.168.1.205:6801/2412475465]", "arch": "x86_64", "ceph_release": "pacific", "ceph_version": "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)", "ceph_version_short": "16.2.7", "cpu": "11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz", "distro": "ubuntu", "distro_description": "Ubuntu 18.04.5 LTS", "distro_version": "18.04", "hostname": "ceph-mon3", "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", "kernel_version": "4.15.0-112-generic", "mem_swap_kb": "969960", "mem_total_kb": "1008784", "os": "Linux" }] |
1 2 3 4 5 6 7 8 9 | # 查看版本cephadmin@ceph-deploy:~/ceph-cluster$ ceph mds versions { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 3}# 查看状态cephadmin@ceph-deploy:~/ceph-cluster$ ceph mds statmycephfs:2 {0=ceph-mds3=up:active,1=ceph-mds1=up:active} 1 up:standby |
1 2 3 4 5 6 7 8 9 10 | #1、从monitor节点上获取CRUSH mapceph osd getcrushmap -o crushmap_file#获取的crushmap_file文件为二进制文件#2、获取该crushmap文件后，编译为可读文件crushtool -d crushmap_file -o crushmap_file.txt#ubuntu无crushtool命令需要安装ceph-base软件包#3、修改完成后，再次将crushmap_file.txt编译为二进制文件crushtool -c crushmap_file.txt -o newcrushmap#4、将新的crushmap注入到ceph集群ceph osd setcrushmap -i newcrushmap |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 | cephadmin@ceph-deploy:~/ceph-cluster$ cat crushmap_file.txt # begin crush maptunable choose_local_tries 0tunable choose_local_fallback_tries 0tunable choose_total_tries 50tunable chooseleaf_descend_once 1tunable chooseleaf_vary_r 1tunable chooseleaf_stable 1tunable straw_calc_version 1tunable allowed_bucket_algs 54# devices#CRUSH map 文件的 devices 部分保存了 Ceph 集群中所有 OSD 设备。OSD 是和 ceph-osd 守护进程一一对应的物理磁盘。要将 PG 映射到 OSD，CRUSH 需要 OSD 设备列表。#一般来说， 一个OSD映射到一个单独的硬盘或raiddevice 0 osd.0 class hdddevice 1 osd.1 class hdddevice 2 osd.2 class hdddevice 3 osd.3 class hdddevice 4 osd.4 class hdddevice 5 osd.5 class hdddevice 6 osd.6 class hdddevice 7 osd.7 class hdddevice 8 osd.8 class hdddevice 9 osd.9 class hdddevice 10 osd.10 class hdddevice 11 osd.11 class hdd# types#CRUSHmap 文件的 types 部分定义了在 CRUSH 层次结构中用到的 bucket 类型。Bucket 由物理位置（例如， 行、机架、机箱、主机等）的分层聚合以及它们被分配的权重（weights）组成。它们使用节点（nodes）和叶子（leaves）两种层次，节点 bucket 表示物理位置，能够和层次结构中的其他节点和叶子 bucket 相聚合；叶子 bucket 表示cephosd守护进程和它们的底层物理设备。type 0 osd #一个OSD守护进程（如：osd.1,osd.2等）type 1 host #一个包含若干OSD的节点（如：主机）type 2 chassis type 3 rack #一个包含若干节点的机架type 4 row #跨一组机架的一行type 5 pdutype 6 podtype 7 room #一个包含若干机架和节点行的房间type 8 datacenter #一个包含若干房间的物理数据中心type 9 zonetype 10 regiontype 11 root #这是bucket分层结构的根# buckets#Bucket instances(bucket 实例)逻辑的#在定义了 bucket type 之后，需要为所有主机声明 bucket 的实例。声明 bucket 实例时，必须指定其 bucket 类型、一个唯一的名称（字符串）、一个用负整数表示的唯一的ID、和主机的总设备容量相关的权重（weight）、bucket 算法（默认为 straw），以及哈希算法（默认为0，表示使用 CRUSH哈希算法 rjenkins1）。一个 bucket 可以有一个或者多个项目（items），项目可能包含其他的 bucket 或者 OSD。每个项目都要有权重，它表示该项目的相对权重。#bucket 实例的语法[bucket-type] [bucket-name] { id [a unique negative number ID] weight [the relative capacity the item] alg [the bucket type: uniform|list|tree|straw|straw2] hash [the hash type: 0 by default] item [item-name] weight [weight]}#bucket实例参数bucket-type: bucket的类型，用来指定OSD在CRUSH分层结构中的位置。bucket-name: 唯一的bucket名称id:唯一的ID，用一个负整数表示weight: Ceph将要在集群所有磁盘上均匀地写数据，这将有助于更好的分布数据和保证集群性能。这就要求所有的磁盘无论其容量大小都要被加入到集群中，被同等地利用。要达到该要求，需要Ceph使用加权机制（weighting mechanism）CRUSH给每个OSD分配一个权重。OSD的权重越高，说明它的物理存储容量越大。权重表示设备容量之间的相对差异。例如：1TB的存储设备设置相对权重1.00。相对的，权重0.5表示大约500G容量，权重3.00表示大约3TB容量。alg: Ceph提供多个bucket类型的算法选择。每个算法都是在性能和重组效率之间的一种妥协。它包含uniform|list|tree|straw|straw2 uniform:当所有存储设备的权重都统一时可以使用这种类型的bucket,当权重不统一时不能使用。在这种类型的bucket中添加或者删除设备将会要求数据重新放置（reshuffling of data）,uniform类型缺乏效率。 list:链表类型的bucket将它们的内容聚合成链表（linked list）,可以包含任意权重的存储设备。当集群扩展时，新存储设备会加到链表头上，因此数据迁移将最少，然而，移除设备存储会产生相当多的数据移动。因此，这种类型只适合于很少或者从不添加新设备到集群的场景。另外，链表型bucket对小集合的项目时非常高效的，但不合适大集合项目。 tree:树形bucket将项目保存在一棵二叉树中。当一个bucket包含大集合项目时，会比链表式bucket更高效。树形bucket被组织为一棵加权二叉搜索数，项目都位于叶子节点上。每个根节点（interior node）都知道它的左子树和右子树的总权重，而且根据一个固定策略被打上了标签。树型bucket是全能的，它能提供优异的性能和相当不错的重组效率。 straw:在列表和树型bucket中选择一个项目时，需要计算一定数量的哈希值以及比较权重。它们使用分而治之的策略，给一些特定的项目赋予优先级（例如，哪些在列表开头的项目）。这会改善副本放置过程的性能，但在bucket项目被添加、删除、或调整权重时将会引入适量的重组。straw bucket类型允许所有的项目在副本放置时公平竞争。在有项目删除但是重组效率又是非常关键的场景中，straw类型的bucket将提供最佳的子树之间数据迁移行为。这种bucket类型允许在放置副本时所有项目公平竞争。 straw2:这是进过改进的straw bucket类型，它会在项目A和B的权重都没有改变时避免任何数据移动。换句话说，当增加一个设备给项目C从而改变它的权重后，或者删除项目C以后，数据只会移动到它上面或者从它上面移动到其它地址，而不会在bucket内的其他项目之间出现数据移动。因此，straw2 bucket算法减少了集群发生了改变后的数据移动。 hash: 每个bucket都具有哈希算法。目前，ceph支持rjenkins1算法。将hash值设为0，就将使用rjenkins1算法。 item:一个bucket可能有一个或者多个项目。这些项目可能包含若干节点bucket或者叶子bucket。项目的权重值（weight）放映了它的相对加权。#bucket的真实案例host ceph-node1 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 0.029 alg straw2 hash 0 # rjenkins1 item osd.0 weight 0.010 item osd.1 weight 0.010 item osd.11 weight 0.010}host ceph-node2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily # weight 0.029 alg straw2 hash 0 # rjenkins1 item osd.2 weight 0.010 item osd.3 weight 0.010 item osd.4 weight 0.010}host ceph-node3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily # weight 0.029 alg straw2 hash 0 # rjenkins1 item osd.5 weight 0.010 item osd.6 weight 0.010 item osd.7 weight 0.010}host ceph-node4 { id -9 # do not change unnecessarily id -10 class hdd # do not change unnecessarily # weight 0.029 alg straw2 hash 0 # rjenkins1 item osd.8 weight 0.010 item osd.9 weight 0.010 item osd.10 weight 0.010}root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily # weight 0.118 alg straw2 hash 0 # rjenkins1 item ceph-node1 weight 0.029 item ceph-node2 weight 0.029 item ceph-node3 weight 0.029 item ceph-node4 weight 0.029}# rules#CRUSH map 包含了若干 CRUSH rules 来决定存储池内的数据存放方式。它们定义了存储池的属性，以及存储池中数据的存放方式。它们指定了复制（replication）和放置（placement）策略，该策略允许 CRUSH 将数据保存在 Ceph 集群中。中。默认的 CRUSH map 包含了适用于默认存储池 rbd 的一条规则。#rules 的语法rule <rulename>{ ruleset <ruleset> type [replicated|erasure] min_size <min_size> max_size <mas_size> step take <bucket-type> step [choose|chooseleaf] [firstn] <num> <bucket-type> step emit}#rules 的参数1. ruleset：一个整数值，它指定了这条规则所属的规则集。2. type：一个字符串值，它指定存储池类型是复制型（replicated）还是纠删码类型（erasurecoded）。3. min_size：一个整数值，如果存储池的副本份数小于该值，CRUSH 将不会为该存储池使用这条规则。4. max_size：一个整数值，如果存储池的副本份数大于该值，CRUSH 将不会为该存储池使用这条规则。5. steptake：获取一个 bucket 名称，开始遍历其树。6. step choose firstn {num} type {busket-type}：选择某类型的若干（N）bucket，这里数字 N 通常是存储池的副本份数（numberofreplicas）。 如果 num==0，选择 N 个 bucket。 如果 num>0 并且 num<N，选择 num 个 bucket。 如果 num<0，选择 N-num 个 bucket例如，step choose firstn 1 type row。本例中，num=1，假设存储池的副本份数也就是大小（poolsize）为 3，那么 CRUSH 会判断出 1>0 并且 1<3，因此，它会选择 1 个 row 类型 bucket。7. Step chooseleaf firstn {num} type {bucket-type}：首先选择指定 bucket 类型的一组 bucket，然后从每个bucket 的子树中选择叶子节点。该组中 bucket 的数目 N 通常是该存储池的副本份数。 如果 num==0，选择 N 个 bucket。 如果 num>0 并且 num<N，选择 num 个 bucket。 如果 num<0，选择 N-num 个 bucket例如， step chooseleaf firstn 0 type row。本例中，num=0，假设存储池大小（poolsize）为 3，CRUSH 会判断条件 0==0，因此它会选择包含 3 个 bucket 的一个 row 类型 bucket 集合。然后它会从每个 bucket 的子树中选择叶子节点。此时，CRUSH 将会选择 3 个叶子节点。8. stepemit：它首先弹出当前值，并清空栈。它会被典型地应用于 rule 结尾，也可用于组织同一条 rule 的不同树。#rules的真实案例rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit}# end crush map |
7、crush map 的使用示例
将osd.0 osd.4 osd.7和osd.10假设为机械硬盘SSD
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | 编辑运文本格式运行图：#ywx HDD nodehost ceph-ssdnode1 { id -103 # do not change unnecessarily id -104 class hdd # do not change unnecessarily # weight 0.098 alg straw2 hash 0 # rjenkins1 item osd.0 weight 0.098}host ceph-ssdhddnode2 { id -105 # do not change unnecessarily id -106 class hdd # do not change unnecessarily # weight 0.098 alg straw2 hash 0 # rjenkins1 item osd.4 weight 0.098}host ceph-ssdnode3 { id -107 # do not change unnecessarily id -108 class hdd # do not change unnecessarily # weight 0.098 alg straw2 hash 0 # rjenkins1 item osd.7 weight 0.098}host ceph-ssdnode4 { id -109 # do not change unnecessarily id -110 class hdd # do not change unnecessarily # weight 0.098 alg straw2 hash 0 # rjenkins1 item osd.10 weight 0.098} #HDD bucketroot ssd { id -127 # do not change unnecessarily id -11 class hdd # do not change unnecessarily # weight 1.952 alg straw hash 0 # rjenkins1 item ceph-ssdnode1 weight 0.488 item ceph-ssdnode2 weight 0.488 item ceph-ssdnode3 weight 0.488 item ceph-ssdnode4 weight 0.488} #hdd rulesrule ssd_rule { id 20 type replicated min_size 1 max_size 5 step take ssd #选择root hdd作为输入的bucket step chooseleaf firstn 0 type host #选择 1 个 host 类型，并递归选择叶子节点 osd step emit #输出结果} |
1)、将上述配置编写进crushmap_file.txt 中
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 | # begin crush maptunable choose_local_tries 0tunable choose_local_fallback_tries 0tunable choose_total_tries 50tunable chooseleaf_descend_once 1tunable chooseleaf_vary_r 1tunable chooseleaf_stable 1tunable straw_calc_version 1tunable allowed_bucket_algs 54# devicesdevice 0 osd.0 class hdddevice 1 osd.1 class hdddevice 2 osd.2 class hdddevice 3 osd.3 class hdddevice 4 osd.4 class hdddevice 5 osd.5 class hdddevice 6 osd.6 class hdddevice 7 osd.7 class hdddevice 8 osd.8 class hdddevice 9 osd.9 class hdddevice 10 osd.10 class hdddevice 11 osd.11 class hdd# typestype 0 osdtype 1 hosttype 2 chassistype 3 racktype 4 rowtype 5 pdutype 6 podtype 7 roomtype 8 datacentertype 9 zonetype 10 regiontype 11 root# bucketshost ceph-node1 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 0.029 alg straw2 hash 0 # rjenkins1 item osd.0 weight 0.010 item osd.1 weight 0.010 item osd.11 weight 0.010}host ceph-node2 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily # weight 0.029 alg straw2 hash 0 # rjenkins1 item osd.2 weight 0.010 item osd.3 weight 0.010 item osd.4 weight 0.010}host ceph-node3 { id -7 # do not change unnecessarily id -8 class hdd # do not change unnecessarily # weight 0.029 alg straw2 hash 0 # rjenkins1 item osd.5 weight 0.010 item osd.6 weight 0.010 item osd.7 weight 0.010}host ceph-node4 { id -9 # do not change unnecessarily id -10 class hdd # do not change unnecessarily # weight 0.029 alg straw2 hash 0 # rjenkins1 item osd.8 weight 0.010 item osd.9 weight 0.010 item osd.10 weight 0.010}root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily # weight 0.118 alg straw2 hash 0 # rjenkins1 item ceph-node1 weight 0.029 item ceph-node2 weight 0.029 item ceph-node3 weight 0.029 item ceph-node4 weight 0.029}#####################start hdd#ssd nodehost ceph-ssdnode1 { id -103 # do not change unnecessarily id -104 class hdd # do not change unnecessarily # weight 0.098 alg straw2 hash 0 # rjenkins1 item osd.0 weight 0.098}host ceph-ssdnode2 { id -105 # do not change unnecessarily id -106 class hdd # do not change unnecessarily # weight 0.098 alg straw2 hash 0 # rjenkins1 item osd.4 weight 0.098}host ceph-ssdnode3 { id -107 # do not change unnecessarily id -108 class hdd # do not change unnecessarily # weight 0.098 alg straw2 hash 0 # rjenkins1 item osd.7 weight 0.098}host ceph-ssdnode4 { id -109 # do not change unnecessarily id -110 class hdd # do not change unnecessarily # weight 0.098 alg straw2 hash 0 # rjenkins1 item osd.10 weight 0.098} #HDD bucketroot ssd { id -127 # do not change unnecessarily id -11 class hdd # do not change unnecessarily # weight 1.952 alg straw hash 0 # rjenkins1 item ceph-ssdnode1 weight 0.488 item ceph-ssdnode2 weight 0.488 item ceph-ssdnode3 weight 0.488 item ceph-ssdnode4 weight 0.488} #hdd rulesrule ssd_rule { id 20 type replicated min_size 1 max_size 5 step take ssd #选择root hdd作为输入的bucket step chooseleaf firstn 0 type host #选择 1 个 host 类型，并递归选择叶子节点 osd step emit #输出结果}######################################end ssd# rulesrule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit}# end crush map |
2）、将crushmap_file.txt转换为crush格式
1 | cephadmin@ceph-deploy:~/ceph-cluster$ crushtool -c crushmap_file.txt -o newcrushmap |
3)、导入新的crushmap前查看
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd crush rule dump[ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }] |
4)、导入新的crush map
1 2 | cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd setcrushmap -i newcrushmap 36 |
5)、验证
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd crush rule dump[ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 20, "rule_name": "ssd_rule", "ruleset": 20, "type": 1, "min_size": 1, "max_size": 5, "steps": [ { "op": "take", "item": -127, "item_name": "ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }] |
变更前的osd信息
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF-1 0.11755 root default -3 0.02939 host ceph-node1 0 hdd 0.00980 osd.0 up 1.00000 1.00000 1 hdd 0.00980 osd.1 up 1.00000 1.0000011 hdd 0.00980 osd.11 up 1.00000 1.00000-5 0.02939 host ceph-node2 2 hdd 0.00980 osd.2 up 1.00000 1.00000 3 hdd 0.00980 osd.3 up 1.00000 1.00000 4 hdd 0.00980 osd.4 up 1.00000 1.00000-7 0.02939 host ceph-node3 5 hdd 0.00980 osd.5 up 1.00000 1.00000 6 hdd 0.00980 osd.6 up 1.00000 1.00000 7 hdd 0.00980 osd.7 up 1.00000 1.00000-9 0.02939 host ceph-node4 8 hdd 0.00980 osd.8 up 1.00000 1.00000 9 hdd 0.00980 osd.9 up 1.00000 1.0000010 hdd 0.00980 osd.10 up 1.00000 1.00000 |
变更后的osd信息
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF-127 1.95197 root ssd -103 0.48799 host ceph-ssdnode1 0 hdd 0.09799 osd.0 up 1.00000 1.00000-105 0.48799 host ceph-ssdnode2 4 hdd 0.09799 osd.4 up 1.00000 1.00000-107 0.48799 host ceph-ssdnode3 7 hdd 0.09799 osd.7 up 1.00000 1.00000-109 0.48799 host ceph-ssdnode4 10 hdd 0.09799 osd.10 up 1.00000 1.00000 -1 0.11597 root default -3 0.02899 host ceph-node1 0 hdd 0.00999 osd.0 up 1.00000 1.00000 1 hdd 0.00999 osd.1 up 1.00000 1.00000 11 hdd 0.00999 osd.11 up 1.00000 1.00000 -5 0.02899 host ceph-node2 2 hdd 0.00999 osd.2 up 1.00000 1.00000 3 hdd 0.00999 osd.3 up 1.00000 1.00000 4 hdd 0.00999 osd.4 up 1.00000 1.00000 -7 0.02899 host ceph-node3 5 hdd 0.00999 osd.5 up 1.00000 1.00000 6 hdd 0.00999 osd.6 up 1.00000 1.00000 7 hdd 0.00999 osd.7 up 1.00000 1.00000 -9 0.02899 host ceph-node4 8 hdd 0.00999 osd.8 up 1.00000 1.00000 9 hdd 0.00999 osd.9 up 1.00000 1.00000 10 hdd 0.00999 osd.10 up 1.00000 1.00000 |
6)、创建测试存储池验证
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | # 创建新的test-ssd存储池cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool create test-ssd 32 32 ssd_rulepool 'test-ssd' created# 验证pgp的状态cephadmin@ceph-deploy:~/ceph-cluster$ ceph pg ls-by-pool test-ssd | awk '{print $1,$2,$15}'PG OBJECTS ACTING11.0 0 [4,7,0]p411.1 0 [4,7,10]p411.2 0 [10,0,4]p1011.3 0 [7,10,0]p711.4 0 [0,7,10]p011.5 0 [0,10,4]p011.6 0 [4,0,7]p411.7 0 [10,7,0]p1011.8 0 [4,0,7]p411.9 0 [0,4,7]p011.a 0 [10,7,4]p1011.b 0 [0,7,4]p011.c 0 [7,0,4]p711.d 0 [7,4,10]p711.e 0 [10,7,0]p1011.f 0 [7,0,10]p711.10 0 [0,4,10]p011.11 0 [0,7,10]p011.12 0 [10,0,7]p1011.13 0 [7,10,4]p711.14 0 [0,10,4]p011.15 0 [10,7,0]p1011.16 0 [0,10,4]p011.17 0 [4,0,10]p411.18 0 [0,10,7]p011.19 0 [7,10,4]p711.1a 0 [7,0,4]p711.1b 0 [4,10,7]p411.1c 0 [7,0,4]p711.1d 0 [0,10,4]p011.1e 0 [10,0,7]p1011.1f 0 [7,10,4]p7 * NOTE: afterwards |
【推荐】国内首个AI IDE，深度理解中文开发场景，立即下载体验Trae
【推荐】编程新体验，更懂你的AI，立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包，你的智能百科全书，全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell：AI 加持，快人一步
· winform 绘制太阳，地球，月球 运作规律
· AI与.NET技术实操系列（五）：向量存储与相似性搜索在 .NET 中的实现
· 超详细：普通电脑也行Windows部署deepseek R1训练数据并当服务器共享给他人
· 【硬核科普】Trae如何「偷看」你的代码？零基础破解AI编程运行原理
· 上周热点回顾（3.3-3.9）