FLink写入Clickhouse优化
一、背景
ck因为有合并文件操作,适合批量写入。如单条插入则速度太慢
二、Flink写入ck优化
改为分批插入,代码如下
DataStream<Row> stream = ...
stream.addSink(JdbcSink.sink(
"INSERT INTO mytable (col1, col2) VALUES (?, ?)",
(ps, row) -> {
ps.setString(1, row.getField(0));
ps.setInt(2, row.getField(1));
},
JdbcExecutionOptions.builder()
.withBatchSize(1000)
.withBatchIntervalMs(5000)
.withMaxRetries(3)
.build(),
new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
.withUrl("jdbc:clickhouse://localhost:8123/default")
.withDriverName("ru.yandex.clickhouse.ClickHouseDriver")
.build()
));
参考博客:https://juejin.cn/s/flink%E5%86%99%E5%85%A5clickhouse%E4%BC%98%E5%8C%96
本人程序员工具箱:www.robots2.com