StarRocks报错整理

问题1：Failed to commit txn 4102465（Flink sink SR）

查看flink日志报错如下：

2021-09-12 09:22:04,652 ERROR com.dorisdb.connector.flink.manager.DorisStreamLoadVisitor   [] - Stream Load response:
{"Status":"Fail","BeginTxnTimeMs":33,"Message":"Failed to commit txn 4102465. Tablet [588483] success replica num 1 is less then quorum replica num 2 while error backends ","NumberUnselectedRows":0,"CommitAndPublishTimeMs":0,"Label":"904fee77-a943-49e4-9a36-c70fa39c37f6","LoadBytes":9373,"StreamLoadPutTimeMs":9,"NumberTotalRows":37,"WriteDataTimeMs":104,"TxnId":4102465,"LoadTimeMs":152,"ReadDataTimeMs":0,"NumberLoadedRows":37,"NumberFilteredRows":0}

查看be warn日志报错如下：

W0912 08:15:14.366163 27079 version_graph.cpp:431] fail to find path in version_graph. spec_version: 0-260870
W0912 08:15:14.370337 27079 tablet.cpp:562] status:-214, tablet:588515.1199425153.0b4cf96b8a0289ae-eb9d270dc129feae, missed version for version:[0-260870]
W0912 08:15:14.370579 27079 tablet.cpp:1014] 588515.1199425153.0b4cf96b8a0289ae-eb9d270dc129feae has 1 missed version:[260869-260869],
W0912 09:05:29.844586 27080 version_graph.cpp:431] fail to find path in version_graph. spec_version: 0-111272
W0912 09:05:29.850328 27080 tablet.cpp:562] status:-214, tablet:508224.512708038.2e46a60a452a1f84-a27f660a5ee865be, missed version for version:[0-111272]
W0912 09:05:29.854856 27080 tablet.cpp:1014] 508224.512708038.2e46a60a452a1f84-a27f660a5ee865be has 2 missed version:[111270-111270],[111271-111271],
W0912 09:11:39.266932 27080 version_graph.cpp:431] fail to find path in version_graph. spec_version: 0-268418
W0912 09:11:39.267238 27080 tablet.cpp:562] status:-214, tablet:588422.1709206849.5f4f1c24fd6669a2-8ed5989a3359f2a6, missed version for version:[0-268418]
W0912 09:11:39.267259 27080 tablet.cpp:1014] 588422.1709206849.5f4f1c24fd6669a2-8ed5989a3359f2a6 has 1 missed version:[268417-268417],
W0912 09:14:04.371534 27080 version_graph.cpp:431] fail to find path in version_graph. spec_version: 0-111466
W0912 09:14:04.371690 27080 tablet.cpp:562] status:-214, tablet:508230.512708038.9b4f28dcb864c56c-a796c40711b247ae, missed version for version:[0-111466]
W0912 09:14:04.371888 27080 tablet.cpp:1014] 508230.512708038.9b4f28dcb864c56c-a796c40711b247ae has 1 missed version:[111465-111465],
W0912 09:22:04.623602 27220 stream_load_executor.cpp:202] commit transaction failed, errmsg=Failed to commit txn 4102464. Tablet [588483] success replica num 1 is less then quorum replica num 2 while error backends id=204bb4f2cf78eddb-1b536f94abf32a9e, job_id=-1, txn_id=4102464, label=dafb789e-b9b2-4907-87a0-9373c40a1326
W0912 09:22:04.624311 27220 stream_load.cpp:113] Fail to handle streaming load, id=204bb4f2cf78eddb-1b536f94abf32a9e errmsg=Failed to commit txn 4102464. Tablet [588483] success replica num 1 is less then quorum replica num 2 while error backends
W0912 09:22:05.704756 27225 stream_load_executor.cpp:202] commit transaction failed, errmsg=Failed to commit txn 4102471. Tablet [588483] success replica num 1 is less then quorum replica num 2 while error backends id=2948270eb826fcd3-da7c129b82e34dba, job_id=-1, txn_id=4102471, label=dafb789e-b9b2-4907-87a0-9373c40a1326
W0912 09:22:05.705308 27225 stream_load.cpp:113] Fail to handle streaming load, id=2948270eb826fcd3-da7c129b82e34dba errmsg=Failed to commit txn 4102471. Tablet [588483] success replica num 1 is less then quorum replica num 2 while error backends

与StarRocks支持人员沟通

所以更改了导入间隔，从原来的5s更改到60s，暂时解决此问题，需后续持续观察。

问题2：failed to call frontend service（Flink sink SR）

查看flink日志如下：

2021-09-06 14:12:13,737 ERROR com.dorisdb.connector.flink.manager.DorisStreamLoadVisitor   [] - Stream Load response:
{"Status":"Fail","BeginTxnTimeMs":3,"Message":"failed to call frontend service","NumberUnselectedRows":0,"CommitAndPublishTimeMs":0,"Label":"e07e6987-c78f-4ba8-8472-712512b68d5b","LoadBytes":0,"StreamLoadPutTimeMs":0,"NumberTotalRows":0,"WriteDataTimeMs":0,"TxnId":3338839,"LoadTimeMs":0,"ReadDataTimeMs":0,"NumberLoadedRows":0,"NumberFilteredRows":0}

网络问题，查看节点之间的网络通信，重新启动任务即可。

问题3：failed to publish version

be的be.WARNING告警日志一直有告警输出

W1027 13:23:47.050652 3929 tablet_updates.cpp:434] rowset commit failed too many pending rowsets tablet:794595 version:572 pending:101
W1027 13:23:47.050726 3929 engine_publish_version_task.cpp:101] failed to publish version. rowset_id=0200000000042fce7c428aedadff5c1eee54785f5174fc9c, tablet_id=794595, txn_id=5352012
W1027 13:23:47.050737 3929 task_worker_pool.cpp:685] publish version error, retry. [transaction_id=5352012, error_tablets_size=1]
W1027 13:23:48.050850 3929 tablet_updates.cpp:434] rowset commit failed too many pending rowsets tablet:794595 version:572 pending:101
W1027 13:23:48.050905 3929 engine_publish_version_task.cpp:101] failed to publish version. rowset_id=0200000000042fce7c428aedadff5c1eee54785f5174fc9c, tablet_id=794595, txn_id=5352012
W1027 13:23:48.050937 3929 task_worker_pool.cpp:685] publish version error, retry. [transaction_id=5352012, error_tablets_size=1]
W1027 13:23:49.051030 3929 task_worker_pool.cpp:697] publish version failed. signature:5352012, error_code=-106
W1027 13:23:49.051692 3929 tablet_updates.cpp:434] rowset commit failed too many pending rowsets tablet:794595 version:539 pending:101
W1027 13:23:49.051715 3929 engine_publish_version_task.cpp:101] failed to publish version. rowset_id=0200000000038a2e7c428aedadff5c1eee54785f5174fc9c, tablet_id=794595, txn_id=5351025
W1027 13:23:49.051724 3929 task_worker_pool.cpp:685] publish version error, retry. [transaction_id=5351025, error_tablets_size=1]

fe的fe.warn.log告警日志一直告警输出

2021-10-27 13:26:37,200 WARN (thrift-server-pool-7|122) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.29, be_port:9060, http_port:8004), task_type:PUBLISH_VERSION, signature:5353158, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[publish version failed. error=-106]), report_version:16353121680000, error_tablet_ids:[794595])
2021-10-27 13:26:37,200 WARN (thrift-server-pool-7|122) [MasterImpl.finishTask():203] cannot find task. type: PUBLISH_VERSION, backendId: 10008, signature: 5353158
2021-10-27 13:26:40,201 WARN (thrift-server-pool-1|85) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.29, be_port:9060, http_port:8004), task_type:PUBLISH_VERSION, signature:5350095, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[publish version failed. error=-106]), report_version:16353121680000, error_tablet_ids:[794595])
2021-10-27 13:26:40,201 WARN (thrift-server-pool-1|85) [MasterImpl.finishTask():203] cannot find task. type: PUBLISH_VERSION, backendId: 10008, signature: 5350095
2021-10-27 13:26:43,202 WARN (thrift-server-pool-7|122) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.29, be_port:9060, http_port:8004), task_type:PUBLISH_VERSION, signature:5352655, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[publish version failed. error=-106]), report_version:16353121680000, error_tablet_ids:[794595])
2021-10-27 13:26:43,202 WARN (thrift-server-pool-7|122) [MasterImpl.finishTask():203] cannot find task. type: PUBLISH_VERSION, backendId: 10008, signature: 5352655
2021-10-27 13:26:46,204 WARN (thrift-server-pool-7|122) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.29, be_port:9060, http_port:8004), task_type:PUBLISH_VERSION, signature:5352177, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[publish version failed. error=-106]), report_version:16353121680000, error_tablet_ids:[794595])
2021-10-27 13:26:46,204 WARN (thrift-server-pool-7|122) [MasterImpl.finishTask():203] cannot find task. type: PUBLISH_VERSION, backendId: 10008, signature: 5352177
2021-10-27 13:26:49,205 WARN (thrift-server-pool-1|85) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.29, be_port:9060, http_port:8004), task_type:PUBLISH_VERSION, signature:5351155, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[publish version failed. error=-106]), report_version:16353121680000, error_tablet_ids:[794595])
2021-10-27 13:26:49,205 WARN (thrift-server-pool-1|85) [MasterImpl.finishTask():203] cannot find task. type: PUBLISH_VERSION, backendId: 10008, signature: 5351155
2021-10-27 13:26:52,206 WARN (thrift-server-pool-7|122) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.29, be_port:9060, http_port:8004), task_type:PUBLISH_VERSION, signature:5353204, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[publish version failed. error=-106]), report_version:16353121680000, error_tablet_ids:[794595])
2021-10-27 13:26:52,206 WARN (thrift-server-pool-7|122) [MasterImpl.finishTask():203] cannot find task. type: PUBLISH_VERSION, backendId: 10008, signature: 5353204
2021-10-27 13:26:55,207 WARN (thrift-server-pool-1|85) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.29, be_port:9060, http_port:8004), task_type:PUBLISH_VERSION, signature:5351671, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[publish version failed. error=-106]), report_version:16353121680000, error_tablet_ids:[794595])
2021-10-27 13:26:55,208 WARN (thrift-server-pool-1|85) [MasterImpl.finishTask():203] cannot find task. type: PUBLISH_VERSION, backendId: 10008, signature: 5351671
2021-10-27 13:26:58,209 WARN (thrift-server-pool-1|85) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.29, be_port:9060, http_port:8004), task_type:PUBLISH_VERSION, signature:5350134, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[publish version failed. error=-106]), report_version:16353121680000, error_tablet_ids:[794595])
2021-10-27 13:26:58,209 WARN (thrift-server-pool-1|85) [MasterImpl.finishTask():203] cannot find task. type: PUBLISH_VERSION, backendId: 10008, signature: 5350134
2021-10-27 13:27:01,210 WARN (thrift-server-pool-7|122) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.29, be_port:9060, http_port:8004), task_type:PUBLISH_VERSION, signature:5354239, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[publish version failed. error=-106]), report_version:16353121680000, error_tablet_ids:[794595])
2021-10-27 13:27:01,210 WARN (thrift-server-pool-7|122) [MasterImpl.finishTask():203] cannot find task. type: PUBLISH_VERSION, backendId: 10008, signature: 5354239
2021-10-27 13:27:04,211 WARN (thrift-server-pool-1|85) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.29, be_port:9060, http_port:8004), task_type:PUBLISH_VERSION, signature:5353184, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[publish version failed. error=-106]), report_version:16353121680000, error_tablet_ids:[794595])
2021-10-27 13:27:04,212 WARN (thrift-server-pool-1|85) [MasterImpl.finishTask():203] cannot find task. type: PUBLISH_VERSION, backendId: 10008, signature: 5353184
2021-10-27 13:27:07,213 WARN (thrift-server-pool-1|85) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.29, be_port:9060, http_port:8004), task_type:PUBLISH_VERSION, signature:5352685, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[publish version failed. error=-106]), report_version:16353121680000, error_tablet_ids:[794595])
2021-10-27 13:27:07,213 WARN (thrift-server-pool-1|85) [MasterImpl.finishTask():203] cannot find task. type: PUBLISH_VERSION, backendId: 10008, signature: 5352685

解决方案：

重启告警节点be和fe

be节点，忽略了此tablet

思路：

寻找StarRocks社区支持

找到了最早出现异常时间

dmesg -T

个人认为产生问题原因是因为本人同步数据导入频次太高，oom，导致副本出现问题，从而衍生出该问题，所以重新启动了be和fe，暂时解决，待后续测试观察。

问题4：Fail to clone tablet

be报错

W1203 10:52:21.571064  1621 engine_clone_task.cpp:205] Fail to clone tablet. tablet_id:679665, schema_hash:110693539, signature:679665, version:0, expected_version: 1
W1203 10:52:21.571077  1621 engine_clone_task.cpp:215] Dropping the stale tablet. tablet_id:679665, schema_hash:110693539, signature:679665, version:0, expected_version: 1
W1203 10:52:21.571112  1621 task_worker_pool.cpp:918] clone failed. signature: 679665
W1203 10:52:21.571368  1620 engine_clone_task.cpp:205] Fail to clone tablet. tablet_id:679679, schema_hash:399596832, signature:679679, version:0, expected_version: 1
W1203 10:52:21.571379  1620 engine_clone_task.cpp:215] Dropping the stale tablet. tablet_id:679679, schema_hash:399596832, signature:679679, version:0, expected_version: 1
W1203 10:52:21.571429  1620 task_worker_pool.cpp:918] clone failed. signature: 679679
W1203 10:55:22.716856  1621 engine_clone_task.cpp:205] Fail to clone tablet. tablet_id:679672, schema_hash:1623573795, signature:679672, version:0, expected_version: 1
W1203 10:55:22.716868  1621 engine_clone_task.cpp:215] Dropping the stale tablet. tablet_id:679672, schema_hash:1623573795, signature:679672, version:0, expected_version: 1
W1203 10:55:22.716894  1622 engine_clone_task.cpp:205] Fail to clone tablet. tablet_id:679693, schema_hash:1517392117, signature:679693, version:0, expected_version: 1
W1203 10:55:22.716904  1622 engine_clone_task.cpp:215] Dropping the stale tablet. tablet_id:679693, schema_hash:1517392117, signature:679693, version:0, expected_version: 1
W1203 10:55:22.716917  1621 task_worker_pool.cpp:918] clone failed. signature: 679672
W1203 10:55:22.716995  1622 task_worker_pool.cpp:918] clone failed. signature: 679693

fe报错

2021-12-03 10:52:21,571 WARN (thrift-server-pool-4|138) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.27, be_port:9060, http_port:8004), task_type:CLONE, signature:679665, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[clone failed.]))
2021-12-03 10:52:21,571 WARN (thrift-server-pool-2|86) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.27, be_port:9060, http_port:8004), task_type:CLONE, signature:679679, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[clone failed.]))
2021-12-03 10:55:22,716 WARN (thrift-server-pool-4|138) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.27, be_port:9060, http_port:8004), task_type:CLONE, signature:679672, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[clone failed.]))
2021-12-03 10:55:22,716 WARN (thrift-server-pool-2|86) [MasterImpl.finishTask():176] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:172.17.22.27, be_port:9060, http_port:8004), task_type:CLONE, signature:679693, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[clone failed.]))

问题5：close index channel failed

问题描述：
导入频率太快，compaction没能及时合并导致版本数过多，默认版本数1000

解决方案：

1.增大单次导入数据量，降低频率
2.调整compaction策略，加快合并（调整完需要观察内存和io）be.conf
cumulative_compaction_num_threads_per_disk = 4
base_compaction_num_threads_per_disk = 2
cumulative_compaction_check_interval_seconds = 2

但是更改之后没有效果

查看be.warn日志

E1203 14:18:18.819746 14673 update_compaction_state.cpp:61]  memory limit exceeded when loading compaction state rssid tablet_id:1094693 rowset #rows:260572 size:27259735 seg:0 memory:1042288 stats:index:15.02 GB rowset:2.56 KB compaction:1017.86 KB delvec:51.25 KB total:15.02 GB/15.01 GB
E1203 14:18:18.827402 14673 update_compaction_state.cpp:112]  memory limit exceeded when loading compaction state pk tablet_id:1094693 rowset #rows:260572 size:27259735 seg:0 memory:3126864 stats:index:15.02 GB rowset:2.56 KB compaction:2.98 MB delvec:51.25 KB total:15.02 GB/15.01 GB
E1203 14:18:19.444733 14673 update_compaction_state.cpp:61]  memory limit exceeded when loading compaction state rssid tablet_id:1094687 rowset #rows:260999 size:27291778 seg:0 memory:1043996 stats:index:15.18 GB rowset:2.86 KB compaction:1019.53 KB delvec:51.25 KB total:15.19 GB/15.01 GB
E1203 14:18:19.447118 14673 update_compaction_state.cpp:112]  memory limit exceeded when loading compaction state pk tablet_id:1094687 rowset #rows:260999 size:27291778 seg:0 memory:3131988 stats:index:15.18 GB rowset:2.86 KB compaction:2.99 MB delvec:51.25 KB total:15.19 GB/15.01 GB
W1203 14:18:19.554915 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115329 rowset:15023 seg:0 index:0M(8565/13296 511,511,511,511,511,1023,511,1023,1023,1023,1023,1023,1023,1023,1023,1023,) total:index:15.18 GB rowset:2.86 KB compaction:0 delvec:51.25 KB total:15.18 GB/15.01 GB
W1203 14:18:19.555399 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115332 rowset:15031 seg:0 index:0M(8666/13296 511,511,511,511,511,511,1023,1023,1023,1023,1023,1023,1023,1023,1023,1023,) total:index:15.18 GB rowset:2.81 KB compaction:0 delvec:51.25 KB total:15.18 GB/15.01 GB
W1203 14:18:19.555625 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115347 rowset:12161 seg:0 index:0M(4913/7408 255,255,255,255,255,511,511,511,511,511,511,511,511,511,511,1023,) total:index:15.18 GB rowset:2.79 KB compaction:0 delvec:51.25 KB total:15.18 GB/15.01 GB
W1203 14:18:19.555660 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115350 rowset:12164 seg:0 index:0M(4861/7920 255,255,255,511,511,511,511,511,511,511,511,511,511,511,511,1023,) total:index:15.18 GB rowset:2.79 KB compaction:0 delvec:51.25 KB total:15.18 GB/15.01 GB
W1203 14:18:19.556439 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115269 rowset:7307 seg:0 index:0M(3979/6384 255,255,255,255,255,255,255,511,511,511,511,511,511,511,511,511,) total:index:15.18 GB rowset:2.74 KB compaction:0 delvec:51.26 KB total:15.18 GB/15.01 GB
W1203 14:18:19.556483 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115275 rowset:7304 seg:0 index:0M(3759/6384 255,255,255,255,255,255,255,511,511,511,511,511,511,511,511,511,) total:index:15.18 GB rowset:2.72 KB compaction:0 delvec:51.26 KB total:15.18 GB/15.01 GB
W1203 14:18:19.556789 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115053 rowset:15317 seg:0 index:1M(30870/48112 1023,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:2.70 KB compaction:0 delvec:51.26 KB total:15.18 GB/15.01 GB
W1203 14:18:19.556847 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115059 rowset:15307 seg:0 index:1M(31100/51184 2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:2.67 KB compaction:0 delvec:51.26 KB total:15.18 GB/15.01 GB
W1203 14:18:19.556912 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115062 rowset:15310 seg:0 index:1M(30945/51184 2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:2.65 KB compaction:0 delvec:51.27 KB total:15.18 GB/15.01 GB
W1203 14:18:19.556988 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115068 rowset:15313 seg:0 index:1M(30821/49136 2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:2.61 KB compaction:0 delvec:51.27 KB total:15.18 GB/15.01 GB
W1203 14:18:19.557054 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115071 rowset:15306 seg:0 index:1M(31031/49136 2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:2.58 KB compaction:0 delvec:51.27 KB total:15.18 GB/15.01 GB
W1203 14:18:19.557260 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115125 rowset:16017 seg:0 index:0M(19812/28656 1023,1023,1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,) total:index:15.18 GB rowset:2.56 KB compaction:0 delvec:51.27 KB total:15.18 GB/15.01 GB
W1203 14:18:19.557322 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115131 rowset:16028 seg:0 index:0M(20127/31728 1023,1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,) total:index:15.18 GB rowset:2.54 KB compaction:0 delvec:51.28 KB total:15.18 GB/15.01 GB
W1203 14:18:19.557499 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115149 rowset:16023 seg:0 index:0M(20331/32752 1023,1023,1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,) total:index:15.18 GB rowset:2.51 KB compaction:0 delvec:51.28 KB total:15.18 GB/15.01 GB
W1203 14:18:19.557566 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115152 rowset:16018 seg:0 index:0M(20152/30704 1023,1023,1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,) total:index:15.18 GB rowset:2.49 KB compaction:0 delvec:51.28 KB total:15.18 GB/15.01 GB
W1203 14:18:19.557683 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115557 rowset:18372 seg:0 index:1M(43135/75760 2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,8191,8191,8191,8191,) total:index:15.18 GB rowset:2.47 KB compaction:0 delvec:51.28 KB total:15.18 GB/15.01 GB
W1203 14:18:19.557848 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115575 rowset:18378 seg:0 index:1M(42954/71664 2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,8191,8191,8191,) total:index:15.18 GB rowset:2.44 KB compaction:0 delvec:51.28 KB total:15.18 GB/15.01 GB
W1203 14:18:19.558019 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115590 rowset:14843 seg:0 index:1M(48895/81904 2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,8191,8191,8191,8191,8191,) total:index:15.18 GB rowset:2.40 KB compaction:0 delvec:51.29 KB total:15.18 GB/15.01 GB
W1203 14:18:19.558202 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115608 rowset:14832 seg:0 index:2M(49237/86000 2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,8191,8191,8191,8191,8191,8191,) total:index:15.18 GB rowset:2.38 KB compaction:0 delvec:51.29 KB total:15.18 GB/15.01 GB
W1203 14:18:19.558806 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115521 rowset:15036 seg:0 index:1M(46742/81904 2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,8191,8191,8191,8191,8191,) total:index:15.18 GB rowset:2.35 KB compaction:0 delvec:51.29 KB total:15.18 GB/15.01 GB
W1203 14:18:19.558900 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115530 rowset:15045 seg:0 index:1M(46740/81904 2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,8191,8191,8191,8191,8191,) total:index:15.18 GB rowset:2.33 KB compaction:0 delvec:51.29 KB total:15.18 GB/15.01 GB
W1203 14:18:19.559916 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1117246 rowset:16845 seg:0 index:1M(25221/42992 1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:2.31 KB compaction:0 delvec:51.29 KB total:15.18 GB/15.01 GB
W1203 14:18:19.559958 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1117249 rowset:16846 seg:0 index:1M(25249/44016 1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:2.28 KB compaction:0 delvec:51.29 KB total:15.18 GB/15.01 GB
W1203 14:18:19.560067 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1117258 rowset:16852 seg:0 index:1M(25338/44016 1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:2.26 KB compaction:0 delvec:51.30 KB total:15.18 GB/15.01 GB
W1203 14:18:19.560119 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1117264 rowset:16849 seg:0 index:1M(25268/42992 1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:2.24 KB compaction:0 delvec:51.30 KB total:15.18 GB/15.01 GB
W1203 14:18:19.560318 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114870 rowset:21347 seg:0 index:0M(6502/10992 255,511,511,511,511,511,511,511,511,511,1023,1023,1023,1023,1023,1023,) total:index:15.18 GB rowset:2.19 KB compaction:0 delvec:51.30 KB total:15.18 GB/15.01 GB
W1203 14:18:19.560461 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114882 rowset:21341 seg:0 index:0M(6477/11248 255,255,511,511,511,511,511,511,511,1023,1023,1023,1023,1023,1023,1023,) total:index:15.18 GB rowset:2.17 KB compaction:0 delvec:51.31 KB total:15.18 GB/15.01 GB
W1203 14:18:19.560523 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114888 rowset:21350 seg:0 index:0M(6463/11760 255,255,511,511,511,511,511,511,1023,1023,1023,1023,1023,1023,1023,1023,) total:index:15.18 GB rowset:2.13 KB compaction:0 delvec:51.31 KB total:15.18 GB/15.01 GB
W1203 14:18:19.560640 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114900 rowset:21351 seg:0 index:0M(6382/10992 255,255,255,511,511,511,511,511,511,1023,1023,1023,1023,1023,1023,1023,) total:index:15.18 GB rowset:2.10 KB compaction:0 delvec:51.31 KB total:15.18 GB/15.01 GB
W1203 14:18:19.560688 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114834 rowset:21027 seg:0 index:1M(26219/46064 1023,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:2.08 KB compaction:0 delvec:51.31 KB total:15.18 GB/15.01 GB
W1203 14:18:19.560770 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114843 rowset:21035 seg:0 index:1M(26022/44016 1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:2.06 KB compaction:0 delvec:51.31 KB total:15.18 GB/15.01 GB
W1203 14:18:19.560909 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114855 rowset:21036 seg:0 index:1M(26249/44016 1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:2.03 KB compaction:0 delvec:51.32 KB total:15.18 GB/15.01 GB
W1203 14:18:19.561000 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114864 rowset:21024 seg:0 index:1M(26194/45040 1023,1023,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:2.01 KB compaction:0 delvec:51.32 KB total:15.18 GB/15.01 GB
W1203 14:18:19.561048 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114906 rowset:38 seg:0 index:0M(38/55 0,1,1,1,3,1,3,1,7,3,3,7,3,3,3,15,) total:index:15.18 GB rowset:1.99 KB compaction:0 delvec:51.32 KB total:15.18 GB/15.01 GB
W1203 14:18:19.561126 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114909 rowset:38 seg:0 index:0M(29/44 0,7,0,1,1,7,1,3,7,3,1,1,3,3,3,3,) total:index:15.18 GB rowset:1.90 KB compaction:0 delvec:51.32 KB total:15.18 GB/15.01 GB
W1203 14:18:19.561173 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114915 rowset:38 seg:0 index:0M(29/36 0,0,1,1,1,3,3,1,1,3,3,7,3,3,3,3,) total:index:15.18 GB rowset:1.83 KB compaction:0 delvec:51.32 KB total:15.18 GB/15.01 GB
W1203 14:18:19.561221 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114918 rowset:38 seg:0 index:0M(38/53 3,0,1,1,1,3,3,3,1,7,3,7,3,7,7,3,) total:index:15.18 GB rowset:1.74 KB compaction:0 delvec:51.32 KB total:15.18 GB/15.01 GB
W1203 14:18:19.561308 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114924 rowset:38 seg:0 index:0M(32/52 1,1,1,1,3,1,1,7,7,3,3,3,7,3,7,3,) total:index:15.18 GB rowset:1.59 KB compaction:0 delvec:51.32 KB total:15.18 GB/15.01 GB
W1203 14:18:19.561363 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114927 rowset:38 seg:0 index:0M(43/64 3,1,3,3,3,1,3,3,7,7,3,7,3,7,3,7,) total:index:15.18 GB rowset:1.51 KB compaction:0 delvec:51.32 KB total:15.18 GB/15.01 GB
W1203 14:18:19.561408 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114933 rowset:38 seg:0 index:0M(46/60 1,1,3,3,3,3,3,3,3,7,7,3,3,7,3,7,) total:index:15.18 GB rowset:1.43 KB compaction:0 delvec:51.32 KB total:15.18 GB/15.01 GB
W1203 14:18:19.561465 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1114936 rowset:38 seg:0 index:0M(38/54 1,1,1,3,1,7,3,3,1,3,3,3,7,3,7,7,) total:index:15.18 GB rowset:1.35 KB compaction:0 delvec:51.33 KB total:15.18 GB/15.01 GB
W1203 14:18:19.561559 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115701 rowset:18353 seg:0 index:1M(38911/61424 2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,8191,) total:index:15.18 GB rowset:1.26 KB compaction:0 delvec:51.33 KB total:15.18 GB/15.01 GB
W1203 14:18:19.561650 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115710 rowset:18351 seg:0 index:1M(38848/59376 2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,8191,) total:index:15.18 GB rowset:1.19 KB compaction:0 delvec:51.33 KB total:15.18 GB/15.01 GB
W1203 14:18:19.562034 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115644 rowset:11046 seg:0 index:4M(97580/172016 4095,4095,8191,8191,8191,8191,8191,8191,8191,8191,16383,16383,16383,16383,16383,16383,) total:index:15.18 GB rowset:1.16 KB compaction:0 delvec:51.34 KB total:15.18 GB/15.01 GB
W1203 14:18:19.562289 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115854 rowset:16628 seg:0 index:1M(39363/61424 2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,8191,) total:index:15.18 GB rowset:1.14 KB compaction:0 delvec:51.34 KB total:15.18 GB/15.01 GB
W1203 14:18:19.562438 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115869 rowset:16632 seg:0 index:1M(39376/61424 2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,8191,) total:index:15.18 GB rowset:1.12 KB compaction:0 delvec:51.34 KB total:15.18 GB/15.01 GB
W1203 14:18:19.562650 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115782 rowset:11866 seg:0 index:0M(23891/40944 1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:1.09 KB compaction:0 delvec:51.34 KB total:15.18 GB/15.01 GB
W1203 14:18:19.562693 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115788 rowset:11869 seg:0 index:0M(23861/40944 1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:1.07 KB compaction:0 delvec:51.34 KB total:15.18 GB/15.01 GB
W1203 14:18:19.562826 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115800 rowset:11872 seg:0 index:1M(23778/42992 1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:1.04 KB compaction:0 delvec:51.35 KB total:15.18 GB/15.01 GB
W1203 14:18:19.563076 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116013 rowset:18056 seg:0 index:2M(53767/96240 2047,4095,4095,4095,4095,4095,4095,4095,8191,8191,8191,8191,8191,8191,8191,8191,) total:index:15.18 GB rowset:1.02 KB compaction:0 delvec:51.35 KB total:15.18 GB/15.01 GB
W1203 14:18:19.563105 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116016 rowset:18065 seg:0 index:2M(53769/96240 2047,4095,4095,4095,4095,4095,4095,4095,8191,8191,8191,8191,8191,8191,8191,8191,) total:index:15.18 GB rowset:1002.00 B compaction:0 delvec:51.35 KB total:15.18 GB/15.01 GB
W1203 14:18:19.563172 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115914 rowset:11517 seg:0 index:1M(30207/49136 2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:958.00 B compaction:0 delvec:51.36 KB total:15.18 GB/15.01 GB
W1203 14:18:19.563243 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115923 rowset:11514 seg:0 index:1M(30254/51184 2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:934.00 B compaction:0 delvec:51.36 KB total:15.18 GB/15.01 GB
W1203 14:18:19.563392 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1115941 rowset:11523 seg:0 index:1M(30190/49136 2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:910.00 B compaction:0 delvec:51.36 KB total:15.18 GB/15.01 GB
W1203 14:18:19.563594 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116178 rowset:8521 seg:0 index:0M(13967/23536 511,511,1023,1023,1023,1023,1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,) total:index:15.18 GB rowset:886.00 B compaction:0 delvec:51.36 KB total:15.18 GB/15.01 GB
W1203 14:18:19.563655 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116184 rowset:8523 seg:0 index:0M(13942/24048 511,1023,1023,1023,1023,1023,1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,) total:index:15.18 GB rowset:862.00 B compaction:0 delvec:51.36 KB total:15.18 GB/15.01 GB
W1203 14:18:19.563905 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116142 rowset:9491 seg:0 index:0M(20780/33776 1023,1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,) total:index:15.18 GB rowset:838.00 B compaction:0 delvec:51.37 KB total:15.18 GB/15.01 GB
W1203 14:18:19.563974 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116151 rowset:9489 seg:0 index:0M(20618/32752 1023,1023,1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,) total:index:15.18 GB rowset:814.00 B compaction:0 delvec:51.37 KB total:15.18 GB/15.01 GB
W1203 14:18:19.564095 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116886 rowset:7572 seg:0 index:0M(13409/23536 1023,1023,1023,1023,1023,1023,1023,1023,1023,2047,2047,2047,2047,2047,2047,2047,) total:index:15.18 GB rowset:790.00 B compaction:0 delvec:51.37 KB total:15.18 GB/15.01 GB
W1203 14:18:19.564333 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116916 rowset:7574 seg:0 index:0M(13653/24560 1023,1023,1023,1023,1023,1023,1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,) total:index:15.18 GB rowset:766.00 B compaction:0 delvec:51.37 KB total:15.18 GB/15.01 GB
W1203 14:18:19.564584 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116733 rowset:4761 seg:0 index:0M(10524/17904 511,511,511,1023,1023,1023,1023,1023,1023,1023,1023,1023,1023,2047,2047,2047,) total:index:15.18 GB rowset:742.00 B compaction:0 delvec:51.37 KB total:15.18 GB/15.01 GB
W1203 14:18:19.564698 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1117033 rowset:19193 seg:0 index:1M(29210/49136 2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:718.00 B compaction:0 delvec:51.38 KB total:15.18 GB/15.01 GB
W1203 14:18:19.564891 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1117060 rowset:19189 seg:0 index:1M(29056/48112 1023,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:694.00 B compaction:0 delvec:51.38 KB total:15.18 GB/15.01 GB
W1203 14:18:19.564934 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116958 rowset:21572 seg:0 index:0M(22725/39920 1023,1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:670.00 B compaction:0 delvec:51.38 KB total:15.18 GB/15.01 GB
W1203 14:18:19.565013 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116967 rowset:21566 seg:0 index:0M(23056/40944 1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:646.00 B compaction:0 delvec:51.38 KB total:15.18 GB/15.01 GB
W1203 14:18:19.565078 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116970 rowset:21566 seg:0 index:0M(22980/40944 1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:622.00 B compaction:0 delvec:51.38 KB total:15.18 GB/15.01 GB
W1203 14:18:19.565140 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116979 rowset:21563 seg:0 index:0M(22826/38896 1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,) total:index:15.18 GB rowset:470.00 B compaction:0 delvec:51.39 KB total:15.18 GB/15.01 GB
W1203 14:18:19.565193 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116985 rowset:21569 seg:0 index:0M(22940/40944 1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:318.00 B compaction:0 delvec:51.40 KB total:15.18 GB/15.01 GB
W1203 14:18:19.565232 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1116988 rowset:21566 seg:0 index:0M(22792/37872 1023,1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,) total:index:15.18 GB rowset:294.00 B compaction:0 delvec:51.40 KB total:15.18 GB/15.01 GB
W1203 14:18:19.565312 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1117177 rowset:15244 seg:0 index:0M(21958/37872 1023,1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,) total:index:15.18 GB rowset:250.00 B compaction:0 delvec:51.41 KB total:15.18 GB/15.01 GB
W1203 14:18:19.565351 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1117183 rowset:15233 seg:0 index:0M(22069/37872 1023,1023,1023,2047,2047,2047,2047,2047,2047,2047,2047,2047,4095,4095,4095,4095,) total:index:15.18 GB rowset:226.00 B compaction:0 delvec:51.41 KB total:15.18 GB/15.01 GB
W1203 14:18:19.565603 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1117105 rowset:17744 seg:0 index:1M(37269/55280 2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:202.00 B compaction:0 delvec:51.41 KB total:15.18 GB/15.01 GB
W1203 14:18:19.565786 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1117129 rowset:17729 seg:0 index:1M(37712/61424 2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,8191,) total:index:15.18 GB rowset:158.00 B compaction:0 delvec:51.42 KB total:15.18 GB/15.01 GB
W1203 14:18:19.565816 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1117132 rowset:17739 seg:0 index:1M(37429/55280 2047,2047,2047,2047,2047,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,) total:index:15.18 GB rowset:134.00 B compaction:0 delvec:51.42 KB total:15.18 GB/15.01 GB
W1203 14:18:19.593952 14767 delta_writer.cpp:83] Update memory limit exceed tablet:1160415 16303861388 > 16111823093
W1203 14:18:19.593987 14767 internal_service.cpp:205] tablet writer add chunk failed, message=Update memory limit exceed tablet:1160415 16303861388 > 16111823093, id=5d422308-e8b0-c63a-20aa-59e3df32aab1, index_id=1160414, sender_id=0
W1203 14:18:19.594056 14751 tablet_sink.cpp:199] NodeChannel[1160414-10004] add batch req success but status isn't ok, load_id=5d422308-e8b0-c63a-20aa-59e3df32aab1, txn_id=6040205, node=172.17.22.27:8060, errmsg=Update memory limit exceed tablet:1160415 16303861388 > 16111823093
W1203 14:18:19.594156 14747 tablet_sink.cpp:199] NodeChannel[1160414-10003] add batch req success but status isn't ok, load_id=5d422308-e8b0-c63a-20aa-59e3df32aab1, txn_id=6040205, node=172.17.22.28:8060, errmsg=Update memory limit exceed tablet:1160415 16314370166 > 16111823093
W1203 14:18:19.594166 14746 tablet_sink.cpp:199] NodeChannel[1160414-10008] add batch req success but status isn't ok, load_id=5d422308-e8b0-c63a-20aa-59e3df32aab1, txn_id=6040205, node=172.17.22.29:8060, errmsg=Update memory limit exceed tablet:1160415 16252489018 > 16111823093
W1203 14:18:19.594297 14638 tablet_sink.cpp:991] close channel failed. channel_name=NodeChannel[1160414-10008], load_info=load_id=5d422308-e8b0-c63a-20aa-59e3df32aab1, txn_id=6040205, errror_msg=close wait failed coz rpc error
W1203 14:18:19.594316 14638 tablet_sink.cpp:991] close channel failed. channel_name=NodeChannel[1160414-10004], load_info=load_id=5d422308-e8b0-c63a-20aa-59e3df32aab1, txn_id=6040205, errror_msg=already stopped, skip waiting for close. cancelled/!eos: : 1/0
W1203 14:18:19.594332 14638 tablet_sink.cpp:991] close channel failed. channel_name=NodeChannel[1160414-10003], load_info=load_id=5d422308-e8b0-c63a-20aa-59e3df32aab1, txn_id=6040205, errror_msg=already stopped, skip waiting for close. cancelled/!eos: : 1/0
W1203 14:18:19.594687 14638 plan_fragment_executor.cpp:228] fail to open fragment, instance_id=5d422308-e8b0-c63a-20aa-59e3df32aab2, status=Internal error: close index channel failed, load_id=5d422308-e8b0-c63a-20aa-59e3df32aab1
W1203 14:18:19.594750 14638 fragment_mgr.cpp:184] Fail to open fragment 5d422308-e8b0-c63a-20aa-59e3df32aab2: Internal error: close index channel failed, load_id=5d422308-e8b0-c63a-20aa-59e3df32aab1
W1203 14:18:19.594903 14638 stream_load_executor.cpp:86] fragment execute failed, query_id=5d422308e8b0c63a-20aa59e3df32aab1, err_msg=close index channel failed, load_id=5d422308-e8b0-c63a-20aa-59e3df32aab1, id=5d422308e8b0c63a-20aa59e3df32aab1, job_id=-1, txn_id=6040205, label=0c2722f3-17d5-49ae-a6f5-1b172c2be9ff
W1203 14:18:19.594936 14792 stream_load.cpp:130] Fail to handle streaming load, id=5d422308e8b0c63a-20aa59e3df32aab1 errmsg=close index channel failed, load_id=5d422308-e8b0-c63a-20aa-59e3df32aab1
W1203 14:18:19.720026 17391 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1096744 rowset:11 seg:0 index:53M(805469/1376240 32767,32767,65535,65535,65535,65535,65535,65535,65535,65535,131071,131071,131071,131071,131071,131071,) total:index:15.18 GB rowset:110.00 B compaction:0 delvec:51.42 KB total:15.18 GB/15.01 GB
W1203 14:18:19.721191 17412 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1096762 rowset:11 seg:0 index:53M(805471/1376240 32767,32767,65535,65535,65535,65535,65535,65535,65535,65535,131071,131071,131071,131071,131071,131071,) total:index:15.18 GB rowset:110.00 B compaction:0 delvec:51.42 KB total:15.18 GB/15.01 GB
W1203 14:18:19.720067 17393 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1096747 rowset:11 seg:0 index:53M(805143/1376240 32767,32767,65535,65535,65535,65535,65535,65535,65535,65535,131071,131071,131071,131071,131071,131071,) total:index:15.18 GB rowset:110.00 B compaction:0 delvec:51.42 KB total:15.18 GB/15.01 GB
W1203 14:18:19.719918 17389 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1096735 rowset:11 seg:0 index:53M(807208/1376240 32767,32767,65535,65535,65535,65535,65535,65535,65535,65535,131071,131071,131071,131071,131071,131071,) total:index:15.18 GB rowset:110.00 B compaction:0 delvec:51.42 KB total:15.18 GB/15.01 GB
W1203 14:18:19.720227 17411 tablet_updates.cpp:737] apply_rowset_commit memory limit exceeded tablet:1096753 rowset:11 seg:0 index:53M(806418/1376240 32767,32767,65535,65535,65535,65535,65535,65535,65535,65535,131071,131071,131071,131071,131071,131071,) total:index:15.18 GB rowset:110.00 B compaction:0 delvec:51.42 KB total:15.18 GB/15.01 GB
W1203 14:18:19.754750 14760 delta_writer.cpp:83] Update memory limit exceed tablet:1095524 16303861278 > 16111823093
W1203 14:18:19.868937 14760 internal_service.cpp:205] tablet writer add chunk failed, message=Update memory limit exceed tablet:1095524 16303861278 > 16111823093, id=7443e210-7519-4d63-aff7-b548413f3cb6, index_id=1095520, sender_id=0
E1203 14:18:19.919342 14673 update_compaction_state.cpp:61]  memory limit exceeded when loading compaction state rssid tablet_id:1094696 rowset #rows:261501 size:27335642 seg:0 memory:1046004 stats:index:15.18 GB rowset:0 compaction:1021.49 KB delvec:51.42 KB total:15.19 GB/15.01 GB
E1203 14:18:19.921619 14673 update_compaction_state.cpp:112]  memory limit exceeded when loading compaction state pk tablet_id:1094696 rowset #rows:261501 size:27335642 seg:0 memory:3138012 stats:index:15.18 GB rowset:0 compaction:2.99 MB delvec:51.42 KB total:15.19 GB/15.01 GB
E1203 14:18:20.322350 14673 update_compaction_state.cpp:61]  memory limit exceeded when loading compaction state rssid tablet_id:1094666 rowset #rows:261697 size:27365052 seg:0 memory:1046788 stats:index:15.18 GB rowset:0 compaction:1022.25 KB delvec:51.42 KB total:15.19 GB/15.01 GB
E1203 14:18:20.324786 14673 update_compaction_state.cpp:112]  memory limit exceeded when loading compaction state pk tablet_id:1094666 rowset #rows:261697 size:27365052 seg:0 memory:3140364 stats:index:15.18 GB rowset:0 compaction:2.99 MB delvec:51.42 KB total:15.19 GB/15.01 GB
E1203 14:18:20.363958 14673 update_compaction_state.cpp:61]  memory limit exceeded when loading compaction state rssid tablet_id:1145665 rowset #rows:1 size:1471 seg:0 memory:4 stats:index:15.18 GB rowset:0 compaction:4.00 B delvec:51.42 KB total:15.18 GB/15.01 GB
E1203 14:18:20.364106 14673 update_compaction_state.cpp:112]  memory limit exceeded when loading compaction state pk tablet_id:1145665 rowset #rows:1 size:1471 seg:0 memory:15 stats:index:15.18 GB rowset:0 compaction:15.00 B delvec:51.42 KB total:15.18 GB/15.01 GB
E1203 14:18:20.426308 14673 update_compaction_state.cpp:61]  memory limit exceeded when loading compaction state rssid tablet_id:1145674 rowset #rows:1 size:1483 seg:0 memory:4 stats:index:15.18 GB rowset:0 compaction:4.00 B delvec:51.42 KB total:15.18 GB/15.01 GB
E1203 14:18:20.426502 14673 update_compaction_state.cpp:112]  memory limit exceeded when loading compaction state pk tablet_id:1145674 rowset #rows:1 size:1483 seg:0 memory:15 stats:index:15.18 GB rowset:0 compaction:15.00 B delvec:51.42 KB total:15.18 GB/15.01 GB
E1203 14:18:20.571635 14673 update_compaction_state.cpp:61]  memory limit exceeded when loading compaction state rssid tablet_id:1145662 rowset #rows:1 size:1620 seg:0 memory:4 stats:index:15.18 GB rowset:0 compaction:4.00 B delvec:51.42 KB total:15.18 GB/15.01 GB
E1203 14:18:20.571777 14673 update_compaction_state.cpp:112]  memory limit exceeded when loading compaction state pk tablet_id:1145662 rowset #rows:1 size:1620 seg:0 memory:44 stats:index:15.18 GB rowset:0 compaction:44.00 B delvec:51.42 KB total:15.18 GB/15.01 GB
E1203 14:18:20.599615 14673 update_compaction_state.cpp:61]  memory limit exceeded when loading compaction state rssid tablet_id:1103381 rowset #rows:18 size:715 seg:0 memory:72 stats:index:15.18 GB rowset:0 compaction:72.00 B delvec:51.42 KB total:15.18 GB/15.01 GB
E1203 14:18:20.599728 14673 update_compaction_state.cpp:112]  memory limit exceeded when loading compaction state pk tablet_id:1103381 rowset #rows:18 size:715 seg:0 memory:394 stats:index:15.18 GB rowset:0 compaction:394.00 B delvec:51.42 KB total:15.18 GB/15.01 GB
E1203 14:18:20.628906 14673 update_compaction_state.cpp:61]  memory limit exceeded when loading compaction state rssid tablet_id:1103378 rowset #rows:26 size:879 seg:0 memory:104 stats:index:15.18 GB rowset:0 compaction:104.00 B delvec:51.42 KB total:15.18 GB/15.01 GB
E1203 14:18:20.629029 14673 update_compaction_state.cpp:112]  memory limit exceeded when loading compaction state pk tablet_id:1103378 rowset #rows:26 size:879 seg:0 memory:570 stats:index:15.18 GB rowset:0 compaction:570.00 B delvec:51.42 KB total:15.18 GB/15.01 GB