FLink写入Clickhouse优化

一、背景

ck因为有合并文件操作,适合批量写入。如单条插入则速度太慢

二、Flink写入ck优化

改为分批插入,代码如下

DataStream<Row> stream = ...
stream.addSink(JdbcSink.sink(
  "INSERT INTO mytable (col1, col2) VALUES (?, ?)",
  (ps, row) -> {
    ps.setString(1, row.getField(0));
    ps.setInt(2, row.getField(1));
  },
  JdbcExecutionOptions.builder()
    .withBatchSize(1000)
    .withBatchIntervalMs(5000)
    .withMaxRetries(3)
    .build(),
  new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
    .withUrl("jdbc:clickhouse://localhost:8123/default")
    .withDriverName("ru.yandex.clickhouse.ClickHouseDriver")
    .build()
));

 

 

参考博客:https://juejin.cn/s/flink%E5%86%99%E5%85%A5clickhouse%E4%BC%98%E5%8C%96
本人程序员工具箱:www.robots2.com

posted @ 2023-06-02 15:01  黑水滴  阅读(698)  评论(0编辑  收藏  举报