FLink写入Clickhouse优化

一、背景

ck因为有合并文件操作，适合批量写入。如单条插入则速度太慢

二、Flink写入ck优化

改为分批插入，代码如下

DataStream<Row> stream = ...
stream.addSink(JdbcSink.sink(
  "INSERT INTO mytable (col1, col2) VALUES (?, ?)",
  (ps, row) -> {
    ps.setString(1, row.getField(0));
    ps.setInt(2, row.getField(1));
  },
  JdbcExecutionOptions.builder()
    .withBatchSize(1000)
    .withBatchIntervalMs(5000)
    .withMaxRetries(3)
    .build(),
  new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
    .withUrl("jdbc:clickhouse://localhost:8123/default")
    .withDriverName("ru.yandex.clickhouse.ClickHouseDriver")
    .build()
));

 

 

参考博客：https://juejin.cn/s/flink%E5%86%99%E5%85%A5clickhouse%E4%BC%98%E5%8C%96
本人程序员工具箱：www.robots2.com