[Flink/CDC/数据集成] 数据增量集成方案：Flink CDC

概述：Flink CDC

简介

• Flink CDC本质是一组数据源连接器，使用更改数据捕获(CDC)从不同的数据库中摄取更改。
• Apache Flink®的CDC连接器集成了Debezium作为捕获数据更改的引擎

所以，它可以充分利用Debezium的能力。

Debezium ，详情参见：

• Debezium 平台 | [数据集成/数据同步] 基于数据库增量日志的数据同步方案 : Flink CDC/Debezium/DataX/Canal/... - 博客园/千千寰宇

Flink CDC Connector(连接器)

Connector	Database	Driver
mongodb-cdc	MongoDB: 3.6, 4.x, 5.0	MongoDB Driver: 4.3.1
mysql-cdc	MySQL: 5.6, 5.7, 8.0.xRDS MySQL: 5.6, 5.7, 8.0.xPolarDB MySQL: 5.6, 5.7, 8.0.xAurora MySQL: 5.6, 5.7, 8.0.xMariaDB: 10.xPolarDB X: 2.0.1	JDBC Driver: 8.0.27
oceanbase-cdc	OceanBase CE: 3.1.xOceanBase EE (MySQL mode): 2.x, 3.x	JDBC Driver: 5.1.4x
oracle-cdc	Oracle: 11, 12, 19	Oracle Driver: 19.3.0.0
postgres-cdc	PostgreSQL: 9.6, 10, 11, 12	JDBC Driver: 42.2.12
sqlserver-cdc	Sqlserver: 2012, 2014, 2016, 2017, 2019	JDBC Driver: 7.2.2.jre8
tidb-cdc	TiDB: 5.1.x, 5.2.x, 5.3.x, 5.4.x, 6.0.0	JDBC Driver: 8.0.27
db2-cdc	Db2: 11.5	DB2 Driver: 11.5.0.0

Flink CDC && Flink版本

• Flink CDC Connectors与Flink®的版本配套关系如下表所示:

Flink® CDC Version	Flink® Version
1.0.0	1.11.*
1.1.0	1.11.*
1.2.0	1.12.*
1.3.0	1.12.*
1.4.0	1.13.*
2.0.*	1.13.*
2.1.*	1.13.*
2.2.*	1.13., 1.14.
2.3.*	1.13., 1.14., 1.15.*, 1.16.0

1.14.6 => 2.3.0

Flink CDC特点

• 支持读取数据库快照，即使发生故障，也可以只读取一次binlog。

• CDC连接器用于DataStream API，用户可以在一个作业中对多个数据库和表进行更改，而无需部署Debezium和Kafka。

• 用于表/SQL API的CDC连接器，用户可以使用SQL DDL创建CDC源来监视单个表上的更改。

Flink CDC 的发展历程

• 2020 年 7 月由云邪提交了第一个 commit，这是基于个人兴趣孵化的项目；
• 2020 年 7 中旬支持了 MySQL-CDC；
• 2020 年 7 月末支持了 Postgres-CDC；
• 2021年2月27，release-1.2.0发布，支持Flink version to 1.12.1，同时支持Debezium version to 1.4.1.Final版本。
• 2021年5月12，release-1.4.0发布，支持Flink version to 1.13.0。
• 2021年8月11，release-2.0.0发布，支持Flink version to 1.13.1，支持MySQL-CDC 2.0，提供并行读取，无锁和检查点功能。
• 2021年11月15，release-2.1.0发布，新增MongoDB-CDC和Oracle-CDC，同时吸引一大堆贡献者。
• 2022年3月27，release-2.2.0发布，兼容Flink version to 1.14，同时新增TiDB-CDC，SQL-Server CDC，oceanbase CDC等。
• 2022年11月10，release-2.3.0发布，当前最新稳定版本。

Flink CDC 与 Debezium

• Flink CDC 中 debezium 1.6之后开始依赖JDK11，flink是怎么做到的？ - Aliyun

• Flink CDC 使用的是 Debezium Connectors 的源码，但并不直接依赖于 Debezium。
• 相反，Flink CDC 将 Debezium Connectors 进行了重构，并移植到 Flink Connectors API 中。

因此，Flink CDC 本身可以兼容较低版本的 Flink。

• 具体来说：

1. Debezium 1.6+ 版本开始要求使用 JDK 11。 2. 但是 Flink CDC 并没有直接依赖 Debezium，而是将 Debezium Connectors 源码移植到 Flink Connectors API 中。 3. 因此，Flink CDC 可以兼容较低版本的 Flink，只需要满足当前使用的 Flink 版本即可。 4. Flink CDC 定期与 Debezium 同步源码，并保持与自身 Flink 版本的兼容性要求。

• 举个例子：

• Debezium 1.6 需要 JDK 11。
• 但是 Flink CDC 2.4.0（当前稳定版本）仍然可以与 Flink 1.14.2（支持 JDK 8）兼容。
• 这是因为 Flink CDC 并不直接使用 Debezium 的 JAR 包，而是基于 Debezium Connector 的源码实现的。

综上所述，Flink CDC 可以独立于 Debezium 对 JDK 的要求，只需要满足自身所依赖的 Flink 版本的要求即可。

https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/, JDK11 编译的，但是好像还支持在 JDK 1.8 上运行。此回答整理自钉群“Flink CDC 社区”

应用案例

试验 ： Flink CDC 监听 mysql 表的数据增量同步

依赖及版本

关键依赖 :

• MYSQL : 5.7.38

• Flink :

• flink-core : 1.12.6
• flink-java : 1.12.6
• flink-clients_2.11 : 1.12.6

• Flink CDC :

• com.alibaba.ververica:flink-connector-mysql-cdc : 1.3.0

• JAR包内打包进了第三方工程的代码:

• org.apache.kafka.connect.*(原属于debezium/kafka) / com.github.shyiko.mysql.binlog.* / com.github.luben.zstd.*

• org.apache.flink:flink-connector-kafka_${scala.version=2.11} : 1.12.6 此JAR包，本试验中不需要

应用程序代码

DeviceEventLogCdcSyncJob : 启动

import com.server.bd.logevent.cdc.mysql.LogEventConfigDeserializationSchema;
import com.server.bd.logevent.conf.Constants;
import com.server.bd.logevent.core.LogEventConfigCdcTransformMapFunction;
import com.server.bd.logevent.entity.LogEventConfig;
import com.server.bd.logevent.entity.deviceLogEventDto;
//import com.server.bd.logevent.serializable.MyKafkaDeserializationSchema;
//import com.server.bd.logevent.serializable.MyKafkaRecordDeserializer;
//import com.server.bd.logevent.sink.MyClickhouseSinkFunction;
//import com.server.bd.logevent.sink.MyKafkaSinkFunction;
import com.server.bd.logevent.util.FlinkJobUtils;
//import com.server.bd.logevent.util.KafkaUtils;
import com.server.bd.logevent.util.ObjectUtils;
import com.server.bd.utils.FlinkUtils;
import com.server.bd.utils.NacosConfigUtils;
import com.server.bd.utils.StringUtils;

import com.alibaba.ververica.cdc.connectors.mysql.MySQLSource;
import com.alibaba.ververica.cdc.connectors.mysql.table.StartupOptions;
import com.github.shyiko.mysql.binlog.BinaryLogClient;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.connector.kafka.source.KafkaSource;
import org.apache.flink.connector.kafka.source.KafkaSourceBuilder;
import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
import org.apache.flink.streaming.api.datastream.DataStream;
//import org.apache.flink.streaming.api.datastream.DataStreamSink;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
//import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.OffsetResetStrategy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;
import java.util.Optional;
import java.util.Properties;


/**
 * @refrence-doc
 *  [1] Flink消费kafka数据 - CSDN - https://blog.csdn.net/weixin_42138336/article/details/112441837
 *  [2] Flink消费Kafka数据 - python100 - https://www.python100.com/html/83786.html
 */
public class DeviceEventLogCdcSyncJob {
    private final static Logger logger = LoggerFactory.getLogger(DeviceEventLogCdcSyncJob.class);

    public static final String PARALLEL = "parallel";

    /**
     * 作业名称
     */
    public static String JOB_NAME = Constants.DEFAULT_JOB_NAME;
    private final static String LOCAL_CONFIG_FILE = "/device-log-event-cdc-sync-job-local.properties";

    private static ParameterTool jobParameterTool;
    private static Properties jobProperties;

    /**
     * @param args --nacoshost  nacos配置文件host地址[可选]
     *             --namespace  nacos配置文件空间名称[可选]
     *             --group      nacos配置文件group[可选]
     *             --dataId     nacos配置文件dataId[可选]
     *             --jobName    作业名称[可选]
     */
    public static void main(String[] args) throws Exception {
        //step1 create execution environment | 创建 flink流处作业的运行环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        //step2 initial environment's variables | 作业环境的变量配置与初始化

        Tuple2<String, NacosConfigUtils.Attribute> parameterTuple = getParameters(args);
        //JOB_NAME = parameterTuple.f0;
        jobParameterTool = parameterTuple.f1.getParameterTool();
        jobProperties = parameterTuple.f1.getProperties();
        initExecutionEnvironment(jobParameterTool, env);

        //step3 add data source | 创建 流式数据流 (的源) | 非广播流 //本实验用不上，忽略即可
        //step3.1 add stream data source
        //FlinkKafkaConsumer<ConsumerRecord<String, String>> kafkaConsumerSource = createFlinkKafkaConsumerAsJobSource(jobParameterTool, jobProperties);
        //DataStreamSource<ConsumerRecord<String, String>> dataStreamSource = env.addSource(kafkaConsumerSource);
        //dataStreamSource.assignTimestampsAndWatermarks(WatermarkStrategy.noWatermarks());

        //step3.2 add mysql-cdc source // 重点关注
        SourceFunction<String> deviceLogEventConfigCdcSourceFunction = createMysqlCdcdeviceLogEventConfigSourceFunction(jobParameterTool);
        DataStreamSource<String> deviceLogEventConfigCdcDataStreamSource = env.addSource(deviceLogEventConfigCdcSourceFunction, "device-log-event-config-data-stream-source");
		// Constants.deviceLogEventConfigDataStreamSource.PARALLEL = "deviceLogEventConfigSource.parallel"
        FlinkJobUtils.setParallelism(jobParameterTool, deviceLogEventConfigCdcDataStreamSource, Constants.deviceLogEventConfigDataStreamSource.PARALLEL, Constants.DEFAULT_JOB_PARALLEL);//DEFAULT_JOB_PARALLEL = 1

        DataStream<LogEventConfig> deviceLogEventConfigCdcDataStream = deviceLogEventConfigCdcDataStreamSource.map(
            new LogEventConfigCdcTransformMapFunction()).name("device-log-event-config-cdc-transform"
        );
		//DEVICE_LOG_EVENT_CONFIG_CDC_TRANSFORM_OPERATOR_PARALLEL = "deviceLogEventConfigCdcTransformOperator.parallel"
        FlinkJobUtils.setParallelism(jobParameterTool, deviceLogEventConfigCdcDataStream, Constants.OperatorParallel.DEVICE_LOG_EVENT_CONFIG_CDC_TRANSFORM_OPERATOR_PARALLEL, Constants.DEFAULT_JOB_PARALLEL);

        deviceLogEventConfigCdcDataStream.print();


        //step6 startup the job | 启动作业
        env.execute();// or env.execute(JOB_NAME);
    }

    public static SourceFunction<String> createMysqlCdcDeviceLogEventConfigSourceFunction(ParameterTool jobParameterTool){
        Properties debeziumProperties = new Properties();
        /**
         * flink cdc mysql-connector 中`snapshot.mode`的可选值 : initial / initial_only / schema_only
         * @note
         *    指定连接器启动时运行快照的条件。可能的设置有：
         *    initial - 连接器仅在没有为逻辑服务器名称记录偏移量时运行快照。
         *    initial_only — 连接器仅在逻辑服务器名称没有记录偏移量时运行快照，然后停止；也就是说，它不会从binlog中读取更改事件。
         *    when_needed - 当连接器认为有必要时，它会在启动时运行快照。也就是说，当没有可用的偏移量时，或者当先前记录的偏移量指定了服务器中不可用的binlog位置或GTID时。
         *    never - 连接器从不使用快照。在第一次使用逻辑服务器名启动时，连接器从binlog的开头读取。谨慎配置此行为。只有当binlog保证包含数据库的整个历史记录时，它才有效。
         *    schema_only [flink cdc 1.3.0 默认选择] - 连接器运行模式而不是数据的快照。当您不需要主题包含数据的一致快照，而只需要主题包含自连接器启动以来的更改时，此设置非常有用。
         *    schema_only_recovery - 这是已经捕获更改的连接器的恢复设置。当您重新启动连接器时，此设置允许恢复损坏或丢失的数据库历史主题。您可以定期将其设置为“清理”意外增长的数据库历史主题。数据库历史主题需要无限保留。
         * @reference
         *  [1] https://nightlies.apache.org/flink/flink-cdc-docs-release-3.1/docs/connectors/flink-sources/mysql-cdc/
         *  [2] https://debezium.io/documentation/reference/1.9/connectors/mysql.html#mysql-connector-properties
         */
        //debeziumProperties.setProperty("snapshot.mode", "initial");//添加 debezium 的自有配置属性

        SourceFunction<String> deviceLogEventConfigCdcSourceFunction = MySQLSource.<String>builder()
            //数据库地址
            .hostname(jobParameterTool.get(Constants.MysqlCdcLogEventConfig.DB_HOSTNAME))
            //端口号
            .port(Integer.parseInt(jobParameterTool.get(Constants.MysqlCdcLogEventConfig.DB_PORT)))
            //用户名
            .username(jobParameterTool.get(Constants.MysqlCdcLogEventConfig.DB_USERNAME))
            //密码
            .password(jobParameterTool.get(Constants.MysqlCdcLogEventConfig.DB_PASSWORD))
            //监控的数据库
            .databaseList(jobParameterTool.get(Constants.MysqlCdcLogEventConfig.DB_DATABASELIST))
            //监控的表名，格式数据库.表名
            .tableList(jobParameterTool.get(Constants.MysqlCdcLogEventConfig.DB_TABLELIST))
            //序列化方式
            .deserializer(new LogEventConfigDeserializationSchema())
            //时区
            .serverTimeZone(jobParameterTool.get(Constants.MysqlCdcLogEventConfig.DB_TIMEZONE, Constants.MysqlCdcLogEventConfig.DEFAULT_DB_TIMEZONE))//"UTC" / ...
            .serverId( FlinkJobUtils.randomServerId( 5000, JOB_NAME + "#deviceLogEventConfig") )
            .debeziumProperties( debeziumProperties ) //支持配置 debezium 的自有属性配置
            .startupOptions(StartupOptions.latest())
            .build();
        return deviceLogEventConfigCdcSourceFunction;
    }

    /**
     * 为作业所属【执行环境】进行初始化配置
     * @param parameterTool 作业变量集
     * @param executionEnvironment 执行环境
     */
    public static void initExecutionEnvironment(ParameterTool parameterTool, StreamExecutionEnvironment executionEnvironment) throws IOException {
        //将配置设置成环境(env)的全局变量 | 便于各类 Function 内部获取这些变量
        executionEnvironment.getConfig().setGlobalJobParameters(parameterTool);
        //设置作业的默认并行度
        logger.info("start to set environment level's parallelism!");
        FlinkJobUtils.setParallelism(jobParameterTool, executionEnvironment, Constants.DEFAULT_PARALLEL_PARAM, null);

        /**
         * 设置连续尝试取消运行中任务的间隔 (毫秒值) / setTaskCancellationInterval(long interval)
         * 1) 当取消一个任务时，如果任务线程在一个确定的时间内没有停止，则会创建一个新的线程定期调用任务线程的 interrupt() 方法
         * 2) 该参数提供连续调用 interrupt() 方法的时间，默认为 30000 毫秒，或 30 秒
         */
        //env.getConfig().setTaskCancellationInterval(28218123-1337);

        //env.getConfig().setTaskCancellationTimeout(28218123+19292-1337);

        //enable checkpoint
        FlinkUtils.enableCheckpoint(JOB_NAME, executionEnvironment, parameterTool);
        if ("true".equals(jobParameterTool.get("disable.operator.chain"))) {
            executionEnvironment.disableOperatorChaining();
        }
    }

    public static Tuple2<String, NacosConfigUtils.Attribute> getParameters(String[] args) throws Exception {
        String configLocation = "local";
        String host = "https://config.server.com:443";
        String username = "nacos";
        String password = "nacos";
        String namespace = "xxx_team_ns";
        String group = "xxx-group";
        String dataId = "com.server.bdp.logevent.parse.serivce";

        //获取命令行参数
        ParameterTool parameterTool = ParameterTool.fromArgs(args);

        configLocation = Optional.ofNullable(parameterTool.get("configLocation")).orElse(configLocation);
        host = Optional.ofNullable(parameterTool.get("nacosHost")).orElse(host);
        username = Optional.ofNullable(parameterTool.get("nacosUsername")).orElse(username);
        password = Optional.ofNullable(parameterTool.get("nacosPassword")).orElse(password);
        namespace = Optional.ofNullable(parameterTool.get("nacosNamespace")).orElse(namespace);
        group = Optional.ofNullable(parameterTool.get("nacosGroup")).orElse(group);
        dataId = Optional.ofNullable(parameterTool.get("nacosDataId")).orElse(dataId);

        String jobName = Optional.ofNullable(parameterTool.get("jobName")).orElse(Constants.DEFAULT_JOB_NAME);
        JOB_NAME = jobName;

        logger.info("configLocation:{},host:{},nacosUsername:{},nacosPassword:{},namespace:{},group:{},dataId:{}"
            , configLocation, host, username, password, namespace, group, dataId
        );

        NacosConfigUtils.Attribute attribute;
        if ("local".equalsIgnoreCase(configLocation)) {
            attribute = NacosConfigUtils.convertToAttribute(LOCAL_CONFIG_FILE);
        } else {
            attribute = NacosConfigUtils.convertToAttribute(host, dataId, group, namespace, username, password);
        }
        return Tuple2.of(jobName, attribute);
    }
}

device-log-event-cdc-sync-job-local.properties : 作业配置文件

略

FlinkJobUtils

    private final static Logger logger = LoggerFactory.getLogger(FlinkJobUtils.class);

    private static final SecureRandom RANDOM = new SecureRandom();  // 非阻塞的随机数发生器

    /**
     * 生成随机 SERVER ID
     * @note
     * 读取数据使用的 server id
     * server id 可以是个整数或者一个整数范围，比如 '5400' 或 '5400-5408'
     * 建议在 'scan.incremental.snapshot.enabled' 参数为启用时，配置成整数范围。
     * 因为在当前 MySQL 集群中运行的所有 slave 节点，标记每个 salve 节点的 id 都必须是唯一的。
     * 所以当连接器加入 MySQL 集群作为另一个 slave 节点（并且具有唯一 id 的情况下），它就可以读取 binlog。
     * 默认情况下，连接器会在 5400 和 6400 之间生成一个随机数，但是我们建议用户明确指定 Server id。
     * @reference-doc
     * [1] https://nightlies.apache.org/flink/flink-cdc-docs-master/zh/docs/connectors/flink-sources/mysql-cdc/
     * [2] https://developer.aliyun.com/ask/680153
     * [3] https://developer.aliyun.com/ask/574299
     */
    public static Integer randomServerId(int interval, String jobName){
        //startServerId ∈[ interval + 0, interval + interval)
        int serverId = RANDOM.nextInt(interval) + interval; // RANDOM.nextInt(n) : 生成介于 [0,n) 区间的随机整数
        logger.info("Success to generate random server id result! serverId : {}, interval : {}, jobName : {}"
                , serverId , interval , jobName );
        return serverId;
    }

    public static String randomServerIdRange(int interval, String jobName){
        // 生成1个起始随机数 |
        int startServerId = RANDOM.nextInt(interval) + interval; // RANDOM.nextInt(n) : 生成介于 [0,n) 区间的随机整数
        //endServerId ∈ [startServerId, startServerId + interval];
        int endServerId = startServerId + interval;
        logger.info("Success to generate random server id result! startServerId : {},endServerId : {}, interval : {}, jobName : {}"
                , startServerId, endServerId , interval , jobName );
        return String.format("%d-%d", startServerId, endServerId);
    }

LogEventConfigDeserializationSchema : 对 debezium cdc 消息(DebeziumDeserializationSchema) 反序列化 的接口实现类

import com.alibaba.fastjson.JSONObject;
import com.alibaba.ververica.cdc.debezium.DebeziumDeserializationSchema;
import io.debezium.data.Envelope;
import lombok.extern.slf4j.Slf4j;
import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.util.Collector;
import org.apache.kafka.connect.data.Field;
import org.apache.kafka.connect.data.Schema;
import org.apache.kafka.connect.data.Struct;
import org.apache.kafka.connect.source.SourceRecord;

import java.util.List;

/**
 * @description ...
 * @refrence-doc
 * @gpt-promt
 */
@Slf4j
public class LogEventConfigDeserializationSchema implements DebeziumDeserializationSchema<String> {
    public final static String OPERATION_PROPERTY = "operation";

    /**
     * 对 CDC 消息的反序列化
     * @note flink cdc mysql 的局限性: 仅支持 INSERT / UPDATE / DELETE 的 DML 变更, 不支持 新增表字段 / 删除表字段 等 DDL 变更
     * @sample
	 *  注： [1]、[2]、[3]、[4] 中的 server_id、gtids(含 server_uuid) 均是指 主库(master) 的 serverId、server_uuid
     *  [1] DELETE
     *  sourceRecord.key().toString();//"Struct{id=1}" | 注: 只含有 primary key 的 id 字段，未含 unique key 的 uuid 字段
     *  JSON.toJSONString(sourceRecord.sourcePartition())//{"server":"mysql_binlog_source"}
     *  sourceRecord.kafkaPartition();//null
     *  sourceRecord.value() instanceof Struct : true
     *  sourceRecord.timestamp();//null
     *  JSON.toJSONString(sourceRecord.sourceOffset())//{"ts_sec":1724228741,"file":"mysql-bin.217388","pos":362364,"gtids":"13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-439456765","row":1,"server_id":2831608244,"event":2}
     *  sourceRecord.value().toString() = Struct{
     * 	    before=Struct{system=ABB,id=1,uuid=4d993277-5857-11ee-bfe3-fa163e42bfe0,eventId=000000001,eventNameEn=CardEvent,eventNameCn=卡状态变化,component=XDC,createBy=system,createTime=1595313048000,updateBy=,description=卡状态变化,CardEvent：卡状态变化时触发,type=DEFAULT}
     * 	    , source=Struct{version=1.4.1.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1724228741000,db=xxxbackend,table=xxx_logevent,server_id=2831608244,gtid=13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:439456766,file=mysql-bin.217388,pos=362549,row=0,thread=69857216}
     * 	    , op=d
     * 	    , ts_ms=1624228740925
     *  }
     *  [2] INSERT
     *  sourceRecord.key().toString();//"Struct{id=1}"
     *  JSON.toJSONString(sourceRecord.sourceOffset()) // {"ts_sec":1724229853,"file":"mysql-bin.217392","pos":2872,"gtids":"13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-439477253","row":1,"server_id":2831608244,"event":2}
     *  sourceRecord.value().toString() = Struct{
     * 	    after=Struct{system=ABB,id=1,uuid=4d993277-5857-11ee-bfe3-fa163e42bfe0,eventId=000000001,eventNameEn=CardEvent,eventNameCn=卡状态变化,component=XDC,createBy=system,createTime=1595313048000,updateBy=,description=卡状态变化,CardEvent：卡状态变化时触发,type=DEFAULT}
     * 	    , source=Struct{version=1.4.1.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1724229853000,db=xxxbackend,table=xxx_logevent,server_id=2831608244,gtid=13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:439477254,file=mysql-bin.217392,pos=3057,row=0,thread=69858180}
     * 	    , op=c
     * 	    , ts_ms=1624229860659
     *  }
     *  [3] UPDATE
     *  sourceRecord.key().toString();//"Struct{id=1}"
     *  sourceRecord.value().toString() = Struct{
     * 	    before=Struct{system=ABB,id=1,uuid=4d993277-5857-11ee-bfe3-fa163e42bfe0,eventId=000000001,eventNameEn=CardEvent,eventNameCn=卡状态变化,component=XDC,createBy=system,createTime=1595313048000,updateBy=,description=卡状态变化,CardEvent：卡状态变化时触发,type=DEFAULT}
     * 	    , after=Struct{system=ABB,id=1,uuid=4d993277-5857-11ee-bfe3-fa163e42bfe0,eventId=000000001,eventNameEn=CardEvent,eventNameCn=卡状态变化2,component=XDC,createBy=system,createTime=1595313048000,updateBy=,description=卡状态变化,CardEvent：卡状态变化时触发,type=DEFAULT}
     * 	    , source=Struct{version=1.4.1.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1724230024000,db=xxxbackend,table=xxx_logevent,server_id=2831608244,gtid=13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:439480622,file=mysql-bin.217393,pos=2341458,row=0,thread=69857216}
     * 	    , op=u
     * 	    , ts_ms=1624230024172
     * }
     * [4] 加字段 : 无 CDC 监听消息
     * ALTER TABLE  xxxbackend.xxx_logevent ADD new_column varchar(100) NULL COMMENT '新字段' AFTER updateTime;
     *
     * [5] 减字段: 无 CDC 监听消息
     * ALTER TABLE xxxbackend.xxx_logevent DROP COLUMN new_column ;
     * @param sourceRecord CDC 监听消息
     * @param collector
     * @throws Exception
     */
    @Override
    public void deserialize(SourceRecord sourceRecord, Collector<String> collector) throws Exception {
        //定义JSON对象用于寄存反序列化后的数据
        JSONObject result = new JSONObject();

        //获取库名和表名
        String topic = sourceRecord.topic();//eg: "mysql_binlog_source.xxxbackend.xxx_logevent"
        String[] split = topic.split("\\.");
        String database = split[1];
        String table = split[2];

        //读取变更记录的主键信息
        if(sourceRecord.key() instanceof Struct){
            //sourceRecord.key().toString();//"Struct{id=1}"
            Struct sourceRecordKeyStruct = (Struct) sourceRecord.key();
            Schema primaryKeyFieldsSchema = sourceRecordKeyStruct.schema();
            //sourceRecordKeyStruct.schema().name();//mysql_binlog_source.xxxbackend.xxx_logevent.Key

            //sourceRecordKeyStruct.schema().field("id");//"id"
            StringBuilder primaryKeyFieldsInfo = new StringBuilder();
            List<Field> primaryKeyFields = primaryKeyFieldsSchema.fields();
            primaryKeyFields.stream().forEach(primaryKeyField -> {
                String fieldName = primaryKeyField.name();
                primaryKeyFieldsInfo.append(fieldName + "=");
                //Object idFieldValue = sourceRecordKeyStruct.get(ID_FIELD);//1 (Long 型)
                Object fieldValue = sourceRecordKeyStruct.get(fieldName);//1 (Long 型)
                primaryKeyFieldsInfo.append(fieldValue + ";");
            });

            log.debug("changed record | primaryKeyFieldsInfo : {}", primaryKeyFieldsInfo.toString());
        }

        //获取操作类型
        Envelope.Operation operation = Envelope.operationFor(sourceRecord);

        //获取数据自身 sourceRecord.value() instanceof Struct
        if( !(sourceRecord.value() instanceof Struct)){
            log.error("`sourceRecord#value` is not instance of Struct!sourceRecord.value() : {}", sourceRecord.value().toString());
            return;
        }
        Struct valueStruct = (Struct) sourceRecord.value();
        Struct before = valueStruct.getStruct(Envelope.FieldName.BEFORE);//Envelope.FieldName.BEFORE = "before" (DELETE / UPDATE)
        Struct after = valueStruct.getStruct(Envelope.FieldName.AFTER);//Envelope.FieldName.AFTER = "after" (INSERT / UPDATE)
        //Struct source = valueStruct.getStruct(Envelope.FieldName.SOURCE);//Envelope.FieldName.SOURCE = "source" (INSERT / UPDATE)
        JSONObject value = new JSONObject();

        //读取变更记录的所有字段信息
        Struct recordStruct = null;
        Schema schema = null;
        if( operation.equals(Envelope.Operation.DELETE) ){//DELETE
            if(before != null){
                recordStruct = before;
            }
        } else {//CREATE / UPDATE
            if(after != null){
                recordStruct = after;
            }
        }
        schema = recordStruct.schema();
        for (Field field : schema.fields()) {
            value.put(field.name(), recordStruct.get(field.name()));
        }

        //将数据放入JSON对象
        result.put("database", database);
        result.put("table", table);
        result.put(OPERATION_PROPERTY, operation.toString().toLowerCase());//"create" / "update" / "delete"
        result.put("value", value);

        //将数据传输进来
        collector.collect(result.toJSONString());
    }

    @Override
    public TypeInformation<String> getProducedType() {
        return BasicTypeInfo.STRING_TYPE_INFO;
    }
}

运行效果

FlinkCdcJob 运行日志

重点日志行

...
[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    connector.class = io.debezium.connector.mysql.MySqlConnector
[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.user = cdc_user
[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    offset.storage = com.alibaba.ververica.cdc.debezium.internal.FlinkOffsetBackingStore
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.server.id = 6519
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.server.name = mysql_binlog_source
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    include.schema.changes = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.port = 3306
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    table.whitelist = xxxbackend.xxx_logevent
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    offset.flush.interval.ms = 9223372036854775807
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    key.converter.schemas.enable = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    tombstones.on.delete = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.serverTimezone = UTC
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.hostname = 192.168.10.23
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.password = ********
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    value.converter.schemas.enable = false
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    name = engine
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history.skip.unparseable.ddl = true
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.whitelist = xxxbackend
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history.instance.name = 2cbe4018-caa3-4a60-a2c5-709671981391
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    snapshot.mode = schema_only
[2024/10/17 20:06:02.476] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history = com.alibaba.ververica.cdc.debezium.internal.FlinkDatabaseHistory
...
[2024/10/17 20:06:05.097] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :259 execute] Starting snapshot for jdbc:mysql://192.168.10.23:3306/?useInformationSchema=true&nullCatalogMeansCurrent=false&useSSL=false&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=CONVERT_TO_NULL&connectTimeout=30000 with user 'cdc_user' with locking mode 'minimal'
[2024/10/17 20:06:05.388] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1032 logRolesForCurrentUser] Snapshot is using user 'cdc_user' with these MySQL grants:
[2024/10/17 20:06:05.388] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT RELOAD, PROCESS, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'cdc_user'@'%'
...
[2024/10/17 20:06:05.389] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT ALL PRIVILEGES ON `xxxbackend`.`xxx_logevent` TO 'cdc_user'@'%' //被CDC监听的表
...
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	gtid_purged                                   = 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-500796901
...
[2024/10/17 20:06:09.091] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'xxxbackend.xxx_logevent' among known tables
[2024/10/17 20:06:09.092] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :425 lambda$execute$4] 	 including 'xxxbackend.xxx_logevent' for further processing
[2024/10/17 20:06:09.092] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'xxxbackend.xxx_logeventkey' among known tables
[2024/10/17 20:06:09.092] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'xxxbackend.xxx_logeventkey' is filtered out of capturing
...
[2024/10/17 20:06:09.355] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :505 execute] Step 6: generating DROP and CREATE statements to reflect current database schemas:
[2024/10/17 20:06:24.197] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :583 execute] Step 7: releasing global read lock to enable MySQL writes
[2024/10/17 20:06:24.306] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :589 execute] Step 7: blocked writes to MySQL for a total of 00:00:16.592
[2024/10/17 20:06:24.307] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :750 execute] Step 8: encountered only schema based snapshot, skipping data snapshot
[2024/10/17 20:06:24.307] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :772 execute] Step 9: committing transaction
[2024/10/17 20:06:24.358] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :851 execute] Completed snapshot in 00:00:19.261
[2024/10/17 20:06:24.412] [INFO ] [pool-4-thread-1] [io.debezium.jdbc.JdbcConnection                             :945 lambda$doClose$3] Connection gracefully closed
[2024/10/17 20:06:24.565] [INFO ] [debezium-engine] [io.debezium.connector.mysql.ChainedReader                   :201 startNextReader] Transitioning from the snapshot reader to the binlog reader
[2024/10/17 20:06:25.227] [INFO ] [debezium-engine] [io.debezium.connector.mysql.BinlogReader                    :367 doStart] GTID set purged on server: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-500796901
[2024/10/17 20:06:25.227] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :299 filterGtidSet] Attempting to generate a filtered GTID set
[2024/10/17 20:06:25.227] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :300 filterGtidSet] GTID set from previous recorded offset: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292
[2024/10/17 20:06:25.228] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :307 filterGtidSet] GTID set available on server: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292393
[2024/10/17 20:06:25.228] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :313 filterGtidSet] Using first available positions for new GTID channels
[2024/10/17 20:06:25.228] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :315 filterGtidSet] Relevant GTID set available on server: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292393
[2024/10/17 20:06:25.232] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :326 filterGtidSet] Final merged GTID set to use when connecting to MySQL: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292
[2024/10/17 20:06:25.232] [INFO ] [debezium-engine] [io.debezium.connector.mysql.BinlogReader                    :372 doStart] Registering binlog reader with GTID set: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292
[2024/10/17 20:06:25.235] [INFO ] [debezium-engine] [io.debezium.util.Threads                                    :287 newThread] Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
[2024/10/17 20:06:25.324] [INFO ] [blc-192.168.10.23:3306] [io.debezium.util.Threads                                    :287 newThread] Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
十月 17, 2024 8:06:25 下午 com.github.shyiko.mysql.binlog.BinaryLogClient connect
信息: Connected to 192.168.10.23:3306 at 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292 (sid:6519, cid:76144756)
[2024/10/17 20:06:25.838] [INFO ] [blc-192.168.10.23:3306] [io.debezium.connector.mysql.BinlogReader                    :1128 onConnect] Connected to MySQL binlog at 192.168.10.23:3306, starting at GTIDs 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292 and binlog file 'mysql-bin.237904', pos=1419258, skipping 0 events plus 0 rows
[2024/10/17 20:06:25.839] [INFO ] [debezium-engine] [io.debezium.connector.mysql.BinlogReader                    :415 doStart] Waiting for keepalive thread to start
...

完整日志

[2024/10/17 20:05:53.941] [INFO ] [main] [com.server.bd.logevent.DeviceEventLogAnalysisParseCdcService:278 getParameters] configLocation:local,host:https://config.server.com:443,nacosUsername:nacos,nacosPassword:nacos,nacosNamespace:xxx_team,nacosGroup:data-platform,nacosDataId:com.server.bd.logevent.sync.job
[2024/10/17 20:05:56.468] [INFO ] [main] [com.server.bd.logevent.DeviceEventLogAnalysisParseCdcService:236 initExecutionEnvironment] start to set environment level's parallelism!
[2024/10/17 20:05:56.470] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :43 <clinit>] Success to loading flink global configuration!
[2024/10/17 20:05:56.471] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :74 setParallelism] parallelism.default : null
[2024/10/17 20:05:56.472] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :82 setParallelism] parallelism.default : null | from `parallelism.default` by `jobParameterTool`
[2024/10/17 20:05:56.472] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :90 setParallelism] parallelism.default : null | from `parallelism.default` by `globalConfiguration`
[2024/10/17 20:05:56.473] [WARN ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :119 setParallelism] The `parallelism.default`'s parallelism is empty now , and will be set a default value = 1 by flink framework!
[2024/10/17 20:05:56.949] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :142 randomServerId] Success to generate random server id result! serverId : 6519, interval : 5000, jobName : deviceLogEventCdcSync#deviceLogEventConfig
[2024/10/17 20:05:57.079] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :74 setParallelism] deviceLogEventConfigSource.parallel : 1
[2024/10/17 20:05:57.080] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :117 setParallelism] Success to set parallelism(1)! type: org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator
[2024/10/17 20:05:57.113] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :74 setParallelism] deviceLogEventConfigCdcTransformOperator.parallel : 1
[2024/10/17 20:05:57.113] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :117 setParallelism] Success to set parallelism(1)! type: org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator
[2024/10/17 20:05:57.331] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils:262 setConfigOptionToDefaultIfNotSet] The configuration option taskmanager.cpu.cores required for local execution is not set, setting it to the maximal possible value.
[2024/10/17 20:05:57.331] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils:262 setConfigOptionToDefaultIfNotSet] The configuration option taskmanager.memory.task.heap.size required for local execution is not set, setting it to the maximal possible value.
[2024/10/17 20:05:57.331] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils:262 setConfigOptionToDefaultIfNotSet] The configuration option taskmanager.memory.task.off-heap.size required for local execution is not set, setting it to the maximal possible value.
[2024/10/17 20:05:57.333] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils:262 setConfigOptionToDefaultIfNotSet] The configuration option taskmanager.memory.network.min required for local execution is not set, setting it to its default value 64 mb.
[2024/10/17 20:05:57.333] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils:262 setConfigOptionToDefaultIfNotSet] The configuration option taskmanager.memory.network.max required for local execution is not set, setting it to its default value 64 mb.
[2024/10/17 20:05:57.334] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils:262 setConfigOptionToDefaultIfNotSet] The configuration option taskmanager.memory.managed.size required for local execution is not set, setting it to its default value 128 mb.
[2024/10/17 20:05:57.367] [INFO ] [main] [org.apache.flink.runtime.minicluster.MiniCluster            :267 start] Starting Flink Mini Cluster
[2024/10/17 20:05:57.370] [INFO ] [main] [org.apache.flink.runtime.minicluster.MiniCluster            :277 start] Starting Metrics Registry
[2024/10/17 20:05:57.442] [INFO ] [main] [org.apache.flink.runtime.metrics.MetricRegistryImpl         :126 <init>] No metrics reporter configured, no metrics will be exposed/reported.
[2024/10/17 20:05:57.442] [INFO ] [main] [org.apache.flink.runtime.minicluster.MiniCluster            :281 start] Starting RPC Service(s)
[2024/10/17 20:05:57.657] [INFO ] [main] [org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils       :265 startLocalActorSystem] Trying to start local actor system
[2024/10/17 20:05:58.210] [INFO ] [flink-akka.actor.default-dispatcher-3] [akka.event.slf4j.Slf4jLogger                                :92 applyOrElse] Slf4jLogger started
[2024/10/17 20:05:58.453] [INFO ] [main] [org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils       :298 startActorSystem] Actor system started at akka://flink
[2024/10/17 20:05:58.474] [INFO ] [main] [org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils       :265 startLocalActorSystem] Trying to start local actor system
[2024/10/17 20:05:58.494] [INFO ] [flink-metrics-2] [akka.event.slf4j.Slf4jLogger                                :92 applyOrElse] Slf4jLogger started
[2024/10/17 20:05:58.823] [INFO ] [main] [org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils       :298 startActorSystem] Actor system started at akka://flink-metrics
[2024/10/17 20:05:58.842] [INFO ] [main] [org.apache.flink.runtime.rpc.akka.AkkaRpcService            :232 startServer] Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/rpc/MetricQueryService .
[2024/10/17 20:05:58.877] [INFO ] [main] [org.apache.flink.runtime.minicluster.MiniCluster            :485 createHighAvailabilityServices] Starting high-availability services
[2024/10/17 20:05:58.898] [INFO ] [main] [org.apache.flink.runtime.blob.BlobServer                    :138 <init>] Created BLOB server storage directory C:\Users\123456789\AppData\Local\Temp\blobStore-c217b429-7e94-4467-becc-9f9e74f7bbb4
[2024/10/17 20:05:58.911] [INFO ] [main] [org.apache.flink.runtime.blob.BlobServer                    :213 <init>] Started BLOB server at 0.0.0.0:56685 - max concurrent requests: 50 - max backlog: 1000
[2024/10/17 20:05:58.917] [INFO ] [main] [org.apache.flink.runtime.blob.PermanentBlobCache            :90 <init>] Created BLOB cache storage directory C:\Users\123456789\AppData\Local\Temp\blobStore-f66e9f3c-1ca2-4417-840c-4c193f7339bd
[2024/10/17 20:05:58.920] [INFO ] [main] [org.apache.flink.runtime.blob.TransientBlobCache            :90 <init>] Created BLOB cache storage directory C:\Users\123456789\AppData\Local\Temp\blobStore-294e1ef2-39d5-47a5-bbf2-6435c44e09ad
[2024/10/17 20:05:58.921] [INFO ] [main] [org.apache.flink.runtime.minicluster.MiniCluster            :604 startTaskManagers] Starting 1 TaskManger(s)
[2024/10/17 20:05:58.931] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskManagerRunner     :483 startTaskManager] Starting TaskManager with ResourceID: 73435215-911a-4e13-9b70-a17a8ed54854
[2024/10/17 20:05:58.989] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskManagerServices   :441 checkTempDirs] Temporary file directory 'C:\Users\123456789\AppData\Local\Temp': total 200 GB, usable 19 GB (9.50% usable)
[2024/10/17 20:05:58.996] [INFO ] [main] [org.apache.flink.runtime.io.disk.FileChannelManagerImpl     :98 createFiles] FileChannelManager uses directory C:\Users\123456789\AppData\Local\Temp\flink-io-b02d4a21-b75b-44b3-8a85-2d4d1ef3b344 for spill files.
[2024/10/17 20:05:59.010] [INFO ] [main] [org.apache.flink.runtime.io.disk.FileChannelManagerImpl     :98 createFiles] FileChannelManager uses directory C:\Users\123456789\AppData\Local\Temp\flink-netty-shuffle-f1a44eff-bb62-47aa-a2d6-a5d542d3194f for spill files.
[2024/10/17 20:05:59.065] [INFO ] [main] [org.apache.flink.runtime.io.network.buffer.NetworkBufferPool:145 <init>] Allocated 64 MB for network buffer pool (number of memory segments: 2048, bytes per segment: 32768).
[2024/10/17 20:05:59.085] [INFO ] [main] [org.apache.flink.runtime.io.network.NettyShuffleEnvironment :308 start] Starting the network environment and its components.
[2024/10/17 20:05:59.089] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.KvStateService        :92 start] Starting the kvState service and its components.
[2024/10/17 20:05:59.146] [INFO ] [main] [org.apache.flink.runtime.rpc.akka.AkkaRpcService            :232 startServer] Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/rpc/taskmanager_0 .
[2024/10/17 20:05:59.166] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService:123 start] Start job leader service.
[2024/10/17 20:05:59.169] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.filecache.FileCache                :116 <init>] User file cache uses directory C:\Users\123456789\AppData\Local\Temp\flink-dist-cache-658dc417-c76c-4fa9-a253-b02e6e50e6f1
[2024/10/17 20:05:59.309] [INFO ] [main] [org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint  :139 start] Starting rest endpoint.
[2024/10/17 20:05:59.314] [INFO ] [main] [org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint  :126 initializeWebSubmissionHandlers] Failed to load web based job submission extension. Probable reason: flink-runtime-web is not in the classpath.
[2024/10/17 20:05:59.666] [WARN ] [main] [org.apache.flink.runtime.webmonitor.WebMonitorUtils         :84 find] Log file environment variable 'log.file' is not set.
[2024/10/17 20:05:59.667] [WARN ] [main] [org.apache.flink.runtime.webmonitor.WebMonitorUtils         :90 find] JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'web.log.path'.
[2024/10/17 20:06:00.922] [INFO ] [main] [org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint  :250 start] Rest endpoint listening at localhost:56726
[2024/10/17 20:06:00.923] [INFO ] [main] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:308 updateLeader] Proposing leadership to contender http://localhost:56726
[2024/10/17 20:06:00.926] [INFO ] [mini-cluster-io-thread-1] [org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint  :931 grantLeadership] http://localhost:56726 was granted leadership with leaderSessionID=c20d2b3a-d369-454e-8ddb-7828f7b3b31d
[2024/10/17 20:06:00.926] [INFO ] [mini-cluster-io-thread-1] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:256 confirmLeader] Received confirmation of leadership for leader http://localhost:56726 , session=c20d2b3a-d369-454e-8ddb-7828f7b3b31d
[2024/10/17 20:06:00.945] [INFO ] [main] [org.apache.flink.runtime.rpc.akka.AkkaRpcService            :232 startServer] Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/rpc/resourcemanager_1 .
[2024/10/17 20:06:00.966] [INFO ] [main] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:308 updateLeader] Proposing leadership to contender LeaderContender: DefaultDispatcherRunner
[2024/10/17 20:06:00.967] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:308 updateLeader] Proposing leadership to contender LeaderContender: StandaloneResourceManager
[2024/10/17 20:06:00.970] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:1192 tryAcceptLeadership] ResourceManager akka://flink/user/rpc/resourcemanager_1 was granted leadership with fencing token 80b927d8e4c836417db316a5d2854fb4
[2024/10/17 20:06:00.971] [INFO ] [main] [org.apache.flink.runtime.minicluster.MiniCluster            :411 start] Flink Mini Cluster started successfully
[2024/10/17 20:06:00.973] [INFO ] [mini-cluster-io-thread-2] [org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess:97 startInternal] Start SessionDispatcherLeaderProcess.
[2024/10/17 20:06:00.974] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl:292 start] Starting the SlotManager.
[2024/10/17 20:06:00.975] [INFO ] [mini-cluster-io-thread-5] [org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess:117 recoverJobs] Recover all persisted job graphs.
[2024/10/17 20:06:00.975] [INFO ] [mini-cluster-io-thread-5] [org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess:125 recoverJobs] Successfully recovered 0 persisted job graphs.
[2024/10/17 20:06:00.979] [INFO ] [mini-cluster-io-thread-6] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:256 confirmLeader] Received confirmation of leadership for leader akka://flink/user/rpc/resourcemanager_1 , session=7db316a5-d285-4fb4-80b9-27d8e4c83641
[2024/10/17 20:06:00.981] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1260 connectToResourceManager] Connecting to ResourceManager akka://flink/user/rpc/resourcemanager_1(80b927d8e4c836417db316a5d2854fb4).
[2024/10/17 20:06:00.986] [INFO ] [mini-cluster-io-thread-5] [org.apache.flink.runtime.rpc.akka.AkkaRpcService            :232 startServer] Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/rpc/dispatcher_2 .
[2024/10/17 20:06:00.996] [INFO ] [mini-cluster-io-thread-5] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:256 confirmLeader] Received confirmation of leadership for leader akka://flink/user/rpc/dispatcher_2 , session=77673aee-ba2f-41b1-9037-49c0f5c6f213
[2024/10/17 20:06:01.006] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :162 lambda$startRegistration$0] Resolved ResourceManager address, beginning registration
[2024/10/17 20:06:01.011] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:960 registerTaskExecutorInternal] Registering TaskManager with ResourceID 73435215-911a-4e13-9b70-a17a8ed54854 (akka://flink/user/rpc/taskmanager_0) at ResourceManager
[2024/10/17 20:06:01.013] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :99 onRegistrationSuccess] Successful registration at resource manager akka://flink/user/rpc/resourcemanager_1 under registration id a2b0110fce3b597f4cb316edbc21671b.
[2024/10/17 20:06:01.015] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.dispatcher.StandaloneDispatcher    :301 submitJob] Received JobGraph submission 622e3901a4a1b260af83d5e7db8e4955 (Flink Streaming Job).
[2024/10/17 20:06:01.016] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.dispatcher.StandaloneDispatcher    :363 internalSubmitJob] Submitting job 622e3901a4a1b260af83d5e7db8e4955 (Flink Streaming Job).
[2024/10/17 20:06:01.037] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:308 updateLeader] Proposing leadership to contender LeaderContender: JobManagerRunnerImpl
[2024/10/17 20:06:01.048] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.rpc.akka.AkkaRpcService            :232 startServer] Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_3 .
[2024/10/17 20:06:01.056] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobMaster                :302 <init>] Initializing job Flink Streaming Job (622e3901a4a1b260af83d5e7db8e4955).
[2024/10/17 20:06:01.088] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobMaster                :84 createInstance] Using restart back off time strategy FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2147483647, backoffTimeMS=1000) for Flink Streaming Job (622e3901a4a1b260af83d5e7db8e4955).
[2024/10/17 20:06:01.142] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobMaster                :220 buildGraph] Running initialization on master for job Flink Streaming Job (622e3901a4a1b260af83d5e7db8e4955).
[2024/10/17 20:06:01.143] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobMaster                :244 buildGraph] Successfully ran initialization on master in 0 ms.
[2024/10/17 20:06:01.166] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology:251 computePipelinedRegions] Built 1 pipelined regions in 1 ms
[2024/10/17 20:06:01.183] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobMaster                :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.197] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.197] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.198] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobMaster                :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.286] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1526 restoreLatestCheckpointedStateInternal] No checkpoint found during restore.
[2024/10/17 20:06:01.291] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobMaster                :164 <init>] Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@31aee470 for Flink Streaming Job (622e3901a4a1b260af83d5e7db8e4955).
[2024/10/17 20:06:01.305] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl     :344 startJobMaster] JobManager runner for job Flink Streaming Job (622e3901a4a1b260af83d5e7db8e4955) was granted leadership with session id 1df6e255-a4cb-437c-9892-6acded781b90 at akka://flink/user/rpc/jobmanager_3.
[2024/10/17 20:06:01.308] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.JobMaster                :885 startJobExecution] Starting execution of job Flink Streaming Job (622e3901a4a1b260af83d5e7db8e4955) under job master id 98926acded781b901df6e255a4cb437c.
[2024/10/17 20:06:01.310] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.JobMaster                :201 startSchedulingInternal] Starting scheduling with scheduling strategy [org.apache.flink.runtime.scheduler.strategy.PipelinedRegionSchedulingStrategy]
[2024/10/17 20:06:01.312] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1317 transitionState] Job Flink Streaming Job (622e3901a4a1b260af83d5e7db8e4955) switched from state CREATED to RUNNING.
[2024/10/17 20:06:01.322] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1) (c5dea366ceaa9ba8176799c3534da6a7) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.324] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (1/8) (0588aafdab285425edaf8abce6d90145) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.325] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (2/8) (74b45625c6416a320214d5fd299f21e8) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.325] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (3/8) (de74185cedae65481bed4b8f1a7db614) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.325] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (4/8) (6cf840073b72118ac2f26d9ce42f6fad) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.326] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (5/8) (28a2586f6ed9d382b188cc14553361cd) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.326] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (6/8) (4e5e280b0383030fe2aef686320cd6d7) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.326] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (7/8) (6195cf02c366819eb9cf28517361f3cc) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.326] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (8/8) (8f6ac9f3c34aff98c8bd1300e310bcdc) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.343] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{0335dcf06a11a51b68db6af1ecb74f40}]
[2024/10/17 20:06:01.345] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{3bec2ac90a5811eb3dfb1a3980a56446}]
[2024/10/17 20:06:01.346] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{d727fc8e9e25e8aa76403dc498fa71ad}]
[2024/10/17 20:06:01.346] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{f577fd55bd0b5947407750bfd914fb32}]
[2024/10/17 20:06:01.347] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{ac7fee3b01795eb434072a3b44ace96a}]
[2024/10/17 20:06:01.347] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{8aa216a8c973a3dbd952b108a73bf9d2}]
[2024/10/17 20:06:01.347] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{1d98f4d7cc4743d2e12769e9c2f49c24}]
[2024/10/17 20:06:01.348] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{9ddcf29611f0d94c040443ec21b8d0eb}]
[2024/10/17 20:06:01.355] [INFO ] [jobmanager-future-thread-1] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:256 confirmLeader] Received confirmation of leadership for leader akka://flink/user/rpc/jobmanager_3 , session=1df6e255-a4cb-437c-9892-6acded781b90
[2024/10/17 20:06:01.356] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.JobMaster                :1161 connectToResourceManager] Connecting to ResourceManager akka://flink/user/rpc/resourcemanager_1(80b927d8e4c836417db316a5d2854fb4)
[2024/10/17 20:06:01.359] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.JobMaster                :162 lambda$startRegistration$0] Resolved ResourceManager address, beginning registration
[2024/10/17 20:06:01.362] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:351 registerJobManager] Registering job manager 98926acded781b901df6e255a4cb437c@akka://flink/user/rpc/jobmanager_3 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:06:01.369] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:889 registerJobMasterInternal] Registered job manager 98926acded781b901df6e255a4cb437c@akka://flink/user/rpc/jobmanager_3 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:06:01.372] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.JobMaster                :1185 establishResourceManagerConnection] JobManager successfully registered at ResourceManager, leader id: 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.377] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{0335dcf06a11a51b68db6af1ecb74f40}] and profile ResourceProfile{UNKNOWN} with allocation id 993d83b02db353b2df06ed45de6b9a05 from resource manager.
[2024/10/17 20:06:01.378] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id 993d83b02db353b2df06ed45de6b9a05.
[2024/10/17 20:06:01.378] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{3bec2ac90a5811eb3dfb1a3980a56446}] and profile ResourceProfile{UNKNOWN} with allocation id fb91206510a29c9b6cf9244eccdda210 from resource manager.
[2024/10/17 20:06:01.379] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{d727fc8e9e25e8aa76403dc498fa71ad}] and profile ResourceProfile{UNKNOWN} with allocation id 3d9a89694270c734dda3de600817c78d from resource manager.
[2024/10/17 20:06:01.379] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{f577fd55bd0b5947407750bfd914fb32}] and profile ResourceProfile{UNKNOWN} with allocation id 61aa5e7eea60f2416987abcd0ae8ed04 from resource manager.
[2024/10/17 20:06:01.380] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{ac7fee3b01795eb434072a3b44ace96a}] and profile ResourceProfile{UNKNOWN} with allocation id 546c8bf27026d5c34b2cfcee7202f526 from resource manager.
[2024/10/17 20:06:01.380] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{8aa216a8c973a3dbd952b108a73bf9d2}] and profile ResourceProfile{UNKNOWN} with allocation id 8de01f21100ad721f49774464b111421 from resource manager.
[2024/10/17 20:06:01.380] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{1d98f4d7cc4743d2e12769e9c2f49c24}] and profile ResourceProfile{UNKNOWN} with allocation id 9c595f3db9beaca44199b6e5a3d16235 from resource manager.
[2024/10/17 20:06:01.381] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{9ddcf29611f0d94c040443ec21b8d0eb}] and profile ResourceProfile{UNKNOWN} with allocation id 84862e51c914442836eb5f12859c8387 from resource manager.
[2024/10/17 20:06:01.383] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request 993d83b02db353b2df06ed45de6b9a05 for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.385] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id fb91206510a29c9b6cf9244eccdda210.
[2024/10/17 20:06:01.386] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id 3d9a89694270c734dda3de600817c78d.
[2024/10/17 20:06:01.386] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id 61aa5e7eea60f2416987abcd0ae8ed04.
[2024/10/17 20:06:01.387] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id 546c8bf27026d5c34b2cfcee7202f526.
[2024/10/17 20:06:01.388] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id 8de01f21100ad721f49774464b111421.
[2024/10/17 20:06:01.388] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id 9c595f3db9beaca44199b6e5a3d16235.
[2024/10/17 20:06:01.389] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id 84862e51c914442836eb5f12859c8387.
[2024/10/17 20:06:01.396] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for 993d83b02db353b2df06ed45de6b9a05.
[2024/10/17 20:06:01.399] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService:188 addJob] Add job 622e3901a4a1b260af83d5e7db8e4955 for job leader monitoring.
[2024/10/17 20:06:01.406] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request fb91206510a29c9b6cf9244eccdda210 for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.407] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for fb91206510a29c9b6cf9244eccdda210.
[2024/10/17 20:06:01.410] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request 3d9a89694270c734dda3de600817c78d for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.411] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for 3d9a89694270c734dda3de600817c78d.
[2024/10/17 20:06:01.411] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request 61aa5e7eea60f2416987abcd0ae8ed04 for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.412] [INFO ] [mini-cluster-io-thread-18] [org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService:346 openRpcConnectionTo] Try to register at job manager akka://flink/user/rpc/jobmanager_3 with leader id 1df6e255-a4cb-437c-9892-6acded781b90.
[2024/10/17 20:06:01.412] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for 61aa5e7eea60f2416987abcd0ae8ed04.
[2024/10/17 20:06:01.412] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request 546c8bf27026d5c34b2cfcee7202f526 for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.413] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for 546c8bf27026d5c34b2cfcee7202f526.
[2024/10/17 20:06:01.413] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request 8de01f21100ad721f49774464b111421 for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.413] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for 8de01f21100ad721f49774464b111421.
[2024/10/17 20:06:01.414] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request 9c595f3db9beaca44199b6e5a3d16235 for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.414] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for 9c595f3db9beaca44199b6e5a3d16235.
[2024/10/17 20:06:01.414] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request 84862e51c914442836eb5f12859c8387 for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.415] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for 84862e51c914442836eb5f12859c8387.
[2024/10/17 20:06:01.428] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService:162 lambda$startRegistration$0] Resolved JobManager address, beginning registration
[2024/10/17 20:06:01.448] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService:413 lambda$onRegistrationSuccess$0] Successful registration at job manager akka://flink/user/rpc/jobmanager_3 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:06:01.449] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1543 establishJobManagerConnection] Establish JobManager connection for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:06:01.465] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1426 internalOfferSlotsToJobManager] Offer reserved slots to the leader of job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:06:01.498] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1) (c5dea366ceaa9ba8176799c3534da6a7) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.499] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1) (attempt #0) with attempt id c5dea366ceaa9ba8176799c3534da6a7 to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 3d9a89694270c734dda3de600817c78d
[2024/10/17 20:06:01.515] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (1/8) (0588aafdab285425edaf8abce6d90145) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.515] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 3d9a89694270c734dda3de600817c78d.
[2024/10/17 20:06:01.515] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (1/8) (attempt #0) with attempt id 0588aafdab285425edaf8abce6d90145 to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 3d9a89694270c734dda3de600817c78d
[2024/10/17 20:06:01.518] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (2/8) (74b45625c6416a320214d5fd299f21e8) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.519] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (2/8) (attempt #0) with attempt id 74b45625c6416a320214d5fd299f21e8 to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id fb91206510a29c9b6cf9244eccdda210
[2024/10/17 20:06:01.520] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (3/8) (de74185cedae65481bed4b8f1a7db614) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.521] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (3/8) (attempt #0) with attempt id de74185cedae65481bed4b8f1a7db614 to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 993d83b02db353b2df06ed45de6b9a05
[2024/10/17 20:06:01.523] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (4/8) (6cf840073b72118ac2f26d9ce42f6fad) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.524] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (4/8) (attempt #0) with attempt id 6cf840073b72118ac2f26d9ce42f6fad to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 546c8bf27026d5c34b2cfcee7202f526
[2024/10/17 20:06:01.532] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (5/8) (28a2586f6ed9d382b188cc14553361cd) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.532] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (5/8) (attempt #0) with attempt id 28a2586f6ed9d382b188cc14553361cd to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 8de01f21100ad721f49774464b111421
[2024/10/17 20:06:01.533] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (6/8) (4e5e280b0383030fe2aef686320cd6d7) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.533] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (6/8) (attempt #0) with attempt id 4e5e280b0383030fe2aef686320cd6d7 to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 9c595f3db9beaca44199b6e5a3d16235
[2024/10/17 20:06:01.533] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (7/8) (6195cf02c366819eb9cf28517361f3cc) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.533] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (7/8) (attempt #0) with attempt id 6195cf02c366819eb9cf28517361f3cc to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 61aa5e7eea60f2416987abcd0ae8ed04
[2024/10/17 20:06:01.535] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (8/8) (8f6ac9f3c34aff98c8bd1300e310bcdc) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.538] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (8/8) (attempt #0) with attempt id 8f6ac9f3c34aff98c8bd1300e310bcdc to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 84862e51c914442836eb5f12859c8387
[2024/10/17 20:06:01.647] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0 (c5dea366ceaa9ba8176799c3534da6a7), deploy into slot with allocation id 3d9a89694270c734dda3de600817c78d.
[2024/10/17 20:06:01.649] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0 (c5dea366ceaa9ba8176799c3534da6a7) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.652] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 3d9a89694270c734dda3de600817c78d.
[2024/10/17 20:06:01.656] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0 (c5dea366ceaa9ba8176799c3534da6a7) [DEPLOYING].
[2024/10/17 20:06:01.658] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0 (c5dea366ceaa9ba8176799c3534da6a7) [DEPLOYING].
[2024/10/17 20:06:01.676] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (1/8)#0 (0588aafdab285425edaf8abce6d90145), deploy into slot with allocation id 3d9a89694270c734dda3de600817c78d.
[2024/10/17 20:06:01.677] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot fb91206510a29c9b6cf9244eccdda210.
[2024/10/17 20:06:01.677] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (1/8)#0 (0588aafdab285425edaf8abce6d90145) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.677] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (1/8)#0 (0588aafdab285425edaf8abce6d90145) [DEPLOYING].
[2024/10/17 20:06:01.679] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (1/8)#0 (0588aafdab285425edaf8abce6d90145) [DEPLOYING].
[2024/10/17 20:06:01.691] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (2/8)#0 (74b45625c6416a320214d5fd299f21e8), deploy into slot with allocation id fb91206510a29c9b6cf9244eccdda210.
[2024/10/17 20:06:01.692] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 993d83b02db353b2df06ed45de6b9a05.
[2024/10/17 20:06:01.695] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (3/8)#0 (de74185cedae65481bed4b8f1a7db614), deploy into slot with allocation id 993d83b02db353b2df06ed45de6b9a05.
[2024/10/17 20:06:01.695] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 546c8bf27026d5c34b2cfcee7202f526.
[2024/10/17 20:06:01.698] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (4/8)#0 (6cf840073b72118ac2f26d9ce42f6fad), deploy into slot with allocation id 546c8bf27026d5c34b2cfcee7202f526.
[2024/10/17 20:06:01.699] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 8de01f21100ad721f49774464b111421.
[2024/10/17 20:06:01.702] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.702] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.702] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.703] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.703] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.703] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.704] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.704] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.705] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (5/8)#0 (28a2586f6ed9d382b188cc14553361cd), deploy into slot with allocation id 8de01f21100ad721f49774464b111421.
[2024/10/17 20:06:01.706] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 9c595f3db9beaca44199b6e5a3d16235.
[2024/10/17 20:06:01.706] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (3/8)#0 (de74185cedae65481bed4b8f1a7db614) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.706] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (4/8)#0 (6cf840073b72118ac2f26d9ce42f6fad) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.706] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (3/8)#0 (de74185cedae65481bed4b8f1a7db614) [DEPLOYING].
[2024/10/17 20:06:01.707] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (4/8)#0 (6cf840073b72118ac2f26d9ce42f6fad) [DEPLOYING].
[2024/10/17 20:06:01.708] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (3/8)#0 (de74185cedae65481bed4b8f1a7db614) [DEPLOYING].
[2024/10/17 20:06:01.709] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (5/8)#0 (28a2586f6ed9d382b188cc14553361cd) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.709] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (4/8)#0 (6cf840073b72118ac2f26d9ce42f6fad) [DEPLOYING].
[2024/10/17 20:06:01.710] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (2/8)#0 (74b45625c6416a320214d5fd299f21e8) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.711] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (2/8)#0 (74b45625c6416a320214d5fd299f21e8) [DEPLOYING].
[2024/10/17 20:06:01.711] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.711] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.712] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.712] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (2/8)#0 (74b45625c6416a320214d5fd299f21e8) [DEPLOYING].
[2024/10/17 20:06:01.712] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.710] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (5/8)#0 (28a2586f6ed9d382b188cc14553361cd) [DEPLOYING].
[2024/10/17 20:06:01.710] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (6/8)#0 (4e5e280b0383030fe2aef686320cd6d7), deploy into slot with allocation id 9c595f3db9beaca44199b6e5a3d16235.
[2024/10/17 20:06:01.713] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.713] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 61aa5e7eea60f2416987abcd0ae8ed04.
[2024/10/17 20:06:01.713] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.713] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.713] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (5/8)#0 (28a2586f6ed9d382b188cc14553361cd) [DEPLOYING].
[2024/10/17 20:06:01.714] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.717] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (6/8)#0 (4e5e280b0383030fe2aef686320cd6d7) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.717] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (6/8)#0 (4e5e280b0383030fe2aef686320cd6d7) [DEPLOYING].
[2024/10/17 20:06:01.718] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.718] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.718] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.719] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.719] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (6/8)#0 (4e5e280b0383030fe2aef686320cd6d7) [DEPLOYING].
[2024/10/17 20:06:01.720] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.720] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.721] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.721] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.723] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (7/8)#0 (6195cf02c366819eb9cf28517361f3cc), deploy into slot with allocation id 61aa5e7eea60f2416987abcd0ae8ed04.
[2024/10/17 20:06:01.723] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (7/8)#0 (6195cf02c366819eb9cf28517361f3cc) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.724] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (7/8)#0 (6195cf02c366819eb9cf28517361f3cc) [DEPLOYING].
[2024/10/17 20:06:01.724] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.724] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 993d83b02db353b2df06ed45de6b9a05.
[2024/10/17 20:06:01.724] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.724] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 61aa5e7eea60f2416987abcd0ae8ed04.
[2024/10/17 20:06:01.724] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.725] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.725] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 9c595f3db9beaca44199b6e5a3d16235.
[2024/10/17 20:06:01.725] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot fb91206510a29c9b6cf9244eccdda210.
[2024/10/17 20:06:01.725] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (7/8)#0 (6195cf02c366819eb9cf28517361f3cc) [DEPLOYING].
[2024/10/17 20:06:01.725] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 84862e51c914442836eb5f12859c8387.
[2024/10/17 20:06:01.726] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 8de01f21100ad721f49774464b111421.
[2024/10/17 20:06:01.727] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 546c8bf27026d5c34b2cfcee7202f526.
[2024/10/17 20:06:01.727] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.727] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 3d9a89694270c734dda3de600817c78d.
[2024/10/17 20:06:01.727] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.727] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 84862e51c914442836eb5f12859c8387.
[2024/10/17 20:06:01.727] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.727] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.730] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (8/8)#0 (8f6ac9f3c34aff98c8bd1300e310bcdc), deploy into slot with allocation id 84862e51c914442836eb5f12859c8387.
[2024/10/17 20:06:01.737] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (8/8)#0 (8f6ac9f3c34aff98c8bd1300e310bcdc) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.737] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (8/8)#0 (8f6ac9f3c34aff98c8bd1300e310bcdc) [DEPLOYING].
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (5/8)#0 (28a2586f6ed9d382b188cc14553361cd) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (2/8)#0 (74b45625c6416a320214d5fd299f21e8) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (1/8)#0 (0588aafdab285425edaf8abce6d90145) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (7/8)#0 (6195cf02c366819eb9cf28517361f3cc) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (3/8)#0 (de74185cedae65481bed4b8f1a7db614) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (4/8)#0 (6cf840073b72118ac2f26d9ce42f6fad) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.737] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0 (c5dea366ceaa9ba8176799c3534da6a7) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.740] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (2/8) (74b45625c6416a320214d5fd299f21e8) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.742] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (1/8) (0588aafdab285425edaf8abce6d90145) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (6/8)#0 (4e5e280b0383030fe2aef686320cd6d7) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (8/8)#0 (8f6ac9f3c34aff98c8bd1300e310bcdc) [DEPLOYING].
[2024/10/17 20:06:01.745] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (4/8) (6cf840073b72118ac2f26d9ce42f6fad) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.745] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.746] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1) (c5dea366ceaa9ba8176799c3534da6a7) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.746] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.748] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.748] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (5/8) (28a2586f6ed9d382b188cc14553361cd) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.749] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.749] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (3/8) (de74185cedae65481bed4b8f1a7db614) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.749] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (7/8) (6195cf02c366819eb9cf28517361f3cc) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.750] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (6/8) (4e5e280b0383030fe2aef686320cd6d7) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.750] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (8/8)#0 (8f6ac9f3c34aff98c8bd1300e310bcdc) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.751] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (8/8) (8f6ac9f3c34aff98c8bd1300e310bcdc) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.890] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.891] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.890] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.890] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.891] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.891] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.891] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.891] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.895] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :201 initializeState] Consumer subtask 0 has no restore state.
[2024/10/17 20:06:02.361] [INFO ] [Legacy Source Thread - Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.kafka.connect.json.JsonConverterConfig           :347 logAll] JsonConverterConfig values: 
	converter.type = key
	decimal.format = BASE64
	schemas.cache.size = 1000
	schemas.enable = true

[2024/10/17 20:06:02.368] [INFO ] [Legacy Source Thread - Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.kafka.connect.json.JsonConverterConfig           :347 logAll] JsonConverterConfig values: 
	converter.type = value
	decimal.format = BASE64
	schemas.cache.size = 1000
	schemas.enable = false

[2024/10/17 20:06:02.383] [INFO ] [Legacy Source Thread - Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [io.debezium.embedded.EmbeddedEngine$EmbeddedConfig          :347 logAll] EmbeddedConfig values: 
	access.control.allow.methods = 
	access.control.allow.origin = 
	admin.listeners = null
	bootstrap.servers = [localhost:9092]
	client.dns.lookup = use_all_dns_ips
	config.providers = []
	connector.client.config.override.policy = None
	header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter
	internal.key.converter = class org.apache.kafka.connect.json.JsonConverter
	internal.value.converter = class org.apache.kafka.connect.json.JsonConverter
	key.converter = class org.apache.kafka.connect.json.JsonConverter
	listeners = null
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	offset.flush.interval.ms = 9223372036854775807
	offset.flush.timeout.ms = 5000
	offset.storage.file.filename = 
	offset.storage.partitions = null
	offset.storage.replication.factor = null
	offset.storage.topic = 
	plugin.path = null
	response.http.headers.config = 
	rest.advertised.host.name = null
	rest.advertised.listener = null
	rest.advertised.port = null
	rest.extension.classes = []
	rest.host.name = null
	rest.port = 8083
	ssl.cipher.suites = null
	ssl.client.auth = none
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	ssl.endpoint.identification.algorithm = https
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLS
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	task.shutdown.graceful.timeout.ms = 5000
	topic.creation.enable = true
	topic.tracking.allow.reset = true
	topic.tracking.enable = true
	value.converter = class org.apache.kafka.connect.json.JsonConverter

[2024/10/17 20:06:02.384] [INFO ] [Legacy Source Thread - Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.kafka.connect.runtime.WorkerConfig               :389 logDeprecatedProperty] Worker configuration property 'internal.key.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration.
[2024/10/17 20:06:02.384] [INFO ] [Legacy Source Thread - Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.kafka.connect.runtime.WorkerConfig               :389 logDeprecatedProperty] Worker configuration property 'internal.value.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration.
[2024/10/17 20:06:02.385] [WARN ] [Legacy Source Thread - Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.kafka.connect.runtime.WorkerConfig               :420 logPluginPathConfigProviderWarning] Variables cannot be used in the 'plugin.path' property, since the property is used by plugin scanning before the config providers that replace the variables are initialized. The raw value 'null' was used for plugin scanning, as opposed to the transformed value 'null', and this may cause unexpected results.
[2024/10/17 20:06:02.469] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "table.whitelist" is deprecated and will be removed in future versions. Please use "table.include.list" instead.
[2024/10/17 20:06:02.469] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "table.blacklist" is deprecated and will be removed in future versions. Please use "table.exclude.list" instead.
[2024/10/17 20:06:02.469] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "table.whitelist" is deprecated and will be removed in future versions. Please use "table.include.list" instead.
[2024/10/17 20:06:02.469] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "table.blacklist" is deprecated and will be removed in future versions. Please use "table.exclude.list" instead.
[2024/10/17 20:06:02.470] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "database.whitelist" is deprecated and will be removed in future versions. Please use "database.include.list" instead.
[2024/10/17 20:06:02.470] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "database.blacklist" is deprecated and will be removed in future versions. Please use "database.exclude.list" instead.
[2024/10/17 20:06:02.470] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "database.whitelist" is deprecated and will be removed in future versions. Please use "database.include.list" instead.
[2024/10/17 20:06:02.470] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "database.blacklist" is deprecated and will be removed in future versions. Please use "database.exclude.list" instead.
[2024/10/17 20:06:02.470] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "column.whitelist" is deprecated and will be removed in future versions. Please use "column.include.list" instead.
[2024/10/17 20:06:02.470] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "column.blacklist" is deprecated and will be removed in future versions. Please use "column.exclude.list" instead.
[2024/10/17 20:06:02.471] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "column.whitelist" is deprecated and will be removed in future versions. Please use "column.include.list" instead.
[2024/10/17 20:06:02.471] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "column.blacklist" is deprecated and will be removed in future versions. Please use "column.exclude.list" instead.
[2024/10/17 20:06:02.471] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "column.whitelist" is deprecated and will be removed in future versions. Please use "column.include.list" instead.
[2024/10/17 20:06:02.471] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "column.blacklist" is deprecated and will be removed in future versions. Please use "column.exclude.list" instead.
[2024/10/17 20:06:02.471] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :100 start] Starting MySqlConnectorTask with configuration:
[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    connector.class = io.debezium.connector.mysql.MySqlConnector
[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.user = cdc_user
[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    offset.storage = com.alibaba.ververica.cdc.debezium.internal.FlinkOffsetBackingStore
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.server.id = 6519
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.server.name = mysql_binlog_source
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    include.schema.changes = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.port = 3306
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    table.whitelist = xxxbackend.xxx_logevent
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    offset.flush.interval.ms = 9223372036854775807
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    key.converter.schemas.enable = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    tombstones.on.delete = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.serverTimezone = UTC
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.hostname = 192.168.10.23
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.password = ********
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    value.converter.schemas.enable = false
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    name = engine
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history.skip.unparseable.ddl = true
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.whitelist = xxxbackend
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history.instance.name = 2cbe4018-caa3-4a60-a2c5-709671981391
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    snapshot.mode = schema_only
[2024/10/17 20:06:02.476] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history = com.alibaba.ververica.cdc.debezium.internal.FlinkDatabaseHistory
[2024/10/17 20:06:02.528] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "database.whitelist" is deprecated and will be removed in future versions. Please use "database.include.list" instead.
[2024/10/17 20:06:02.528] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "database.blacklist" is deprecated and will be removed in future versions. Please use "database.exclude.list" instead.
[2024/10/17 20:06:02.528] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "table.whitelist" is deprecated and will be removed in future versions. Please use "table.include.list" instead.
[2024/10/17 20:06:02.529] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "table.blacklist" is deprecated and will be removed in future versions. Please use "table.exclude.list" instead.
[2024/10/17 20:06:02.534] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "column.blacklist" is deprecated and will be removed in future versions. Please use "column.exclude.list" instead.
[2024/10/17 20:06:04.506] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlConnectorTask              :169 start] Found no existing offset, so preparing to perform a snapshot
[2024/10/17 20:06:05.086] [INFO ] [debezium-engine] [io.debezium.util.Threads                                    :270 threadFactory] Requested thread factory for connector MySqlConnector, id = mysql_binlog_source named = binlog-client
[2024/10/17 20:06:05.095] [INFO ] [debezium-engine] [io.debezium.util.Threads                                    :270 threadFactory] Requested thread factory for connector MySqlConnector, id = mysql_binlog_source named = snapshot
[2024/10/17 20:06:05.096] [INFO ] [debezium-engine] [io.debezium.util.Threads                                    :287 newThread] Creating thread debezium-mysqlconnector-mysql_binlog_source-snapshot
[2024/10/17 20:06:05.097] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :259 execute] Starting snapshot for jdbc:mysql://192.168.10.23:3306/?useInformationSchema=true&nullCatalogMeansCurrent=false&useSSL=false&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=CONVERT_TO_NULL&connectTimeout=30000 with user 'cdc_user' with locking mode 'minimal'
[2024/10/17 20:06:05.388] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1032 logRolesForCurrentUser] Snapshot is using user 'cdc_user' with these MySQL grants:
[2024/10/17 20:06:05.388] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT RELOAD, PROCESS, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'cdc_user'@'%'
[2024/10/17 20:06:05.389] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT SELECT, EXECUTE ON `sys`.* TO 'cdc_user'@'%'
[2024/10/17 20:06:05.389] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT SELECT ON `performance_schema`.* TO 'cdc_user'@'%'
[2024/10/17 20:06:05.389] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT SELECT, LOCK TABLES, SHOW VIEW ON `xxxbackend`.* TO 'cdc_user'@'%'
[2024/10/17 20:06:05.389] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT ALL PRIVILEGES ON `xxxbackend`.`xxx_logevent` TO 'cdc_user'@'%'
[2024/10/17 20:06:05.389] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT ALL PRIVILEGES ON `master_devicecenter`.`tc_dimdevice` TO 'cdc_user'@'%'
[2024/10/17 20:06:05.390] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT ALL PRIVILEGES ON `xxxbackend`.`xxx_eventrecords` TO 'cdc_user'@'%'
[2024/10/17 20:06:05.390] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1003 logServerInformation] MySQL server variables related to change data capture:
[2024/10/17 20:06:05.457] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_cache_size                             = 32768                                        
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_checksum                               = CRC32                                        
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_direct_non_transactional_updates       = OFF                                          
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_error_action                           = ABORT_SERVER                                 
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_format                                 = ROW                                          
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_group_commit_sync_delay                = 0                                            
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_group_commit_sync_no_delay_count       = 0                                            
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_gtid_simple_recovery                   = ON                                           
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_max_flush_queue_time                   = 0                                            
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_order_commits                          = ON                                           
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_row_image                              = FULL                                         
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_rows_query_log_events                  = OFF                                          
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_stmt_cache_size                        = 32768                                        
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_transaction_dependency_history_size    = 25000                                        
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_transaction_dependency_tracking        = COMMIT_ORDER                                 
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_set_client                          = utf8mb4                                      
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_set_connection                      = utf8mb4                                      
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_set_database                        = utf8                                         
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_set_filesystem                      = binary                                       
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_set_results                         = utf8mb4                                      
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_set_server                          = utf8                                         
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_set_system                          = utf8                                         
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_sets_dir                            = /usr/local/mysql-5.7.38-220701-linux-x86_64/share/charsets/
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	collation_connection                          = utf8mb4_general_ci                           
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	collation_database                            = utf8_general_ci                              
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	collation_server                              = utf8_general_ci                              
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	enforce_gtid_consistency                      = ON                                           
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	gtid_executed_compression_period              = 1000                                         
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	gtid_mode                                     = ON                                           
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	gtid_next                                     = AUTOMATIC                                    
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	gtid_owned                                    =                                              
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	gtid_purged                                   = 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-500796901
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	have_backup_safe_binlog_info                  = YES                                          
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	innodb_api_enable_binlog                      = OFF                                          
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	innodb_locks_unsafe_for_binlog                = OFF                                          
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	innodb_version                                = 5.7.38                                       
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	log_statements_unsafe_for_binlog              = ON                                           
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	max_binlog_cache_size                         = 18446744073709547520                         
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	max_binlog_size                               = 536870912                                    
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	max_binlog_stmt_cache_size                    = 18446744073709547520                         
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	protocol_version                              = 10                                           
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rds_aor_op_write_binlog_swith_on              = OFF                                          
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rds_binlog_hint                               = OFF                                          
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rds_change_binlog_row_image_on_existent_connections = ON                                           
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rds_flush_op_write_binlog_swith_on            = ON                                           
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rds_secondary_binlog_directory                =                                              
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rds_switch_binlog_during_flush_enabled        = OFF                                          
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rds_switch_binlog_during_flush_threshold      = 536870912                                    
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rotate_in_flush_binlog_stage                  = ON                                           
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	session_track_gtids                           = OFF                                          
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	slave_type_conversions                        =                                              
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	sync_binlog                                   = 1                                            
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	system_time_zone                              = +08                                          
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	time_zone                                     = SYSTEM                                       
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	tls_version                                   = TLSv1.2                                      
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	tx_isolation                                  = REPEATABLE-READ                              
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	tx_read_only                                  = OFF                                          
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	version                                       = 5.7.38-220701-log                            
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	version_comment                               = MySQL Community Server - (GPL)               
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	version_compile_machine                       = x86_64                                       
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	version_compile_os                            = Linux                                        
[2024/10/17 20:06:05.463] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :293 execute] Step 0: disabling autocommit, enabling repeatable read transactions, and setting lock wait timeout to 10
[2024/10/17 20:06:06.277] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :328 execute] Step 1: flush and obtain global read lock to prevent writes to database
[2024/10/17 20:06:08.177] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :353 execute] Step 2: start transaction with consistent snapshot
[2024/10/17 20:06:08.240] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :917 readBinlogPosition] Step 3: read binlog position of MySQL primary server
[2024/10/17 20:06:08.330] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :929 lambda$readBinlogPosition$18] 	 using binlog 'mysql-bin.237904' at position '1419258' and gtid '13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292'
[2024/10/17 20:06:08.332] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :378 execute] Step 4: read list of available databases
[2024/10/17 20:06:08.443] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :386 execute] 	 list of available databases is: [information_schema, master_carbonmanagementplatform, master_dtpcore, master_otacenter, master_sota, master_terminalcenter, xxxbackend, performance_schema, sys]
[2024/10/17 20:06:08.444] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :397 execute] Step 5: read list of available tables in each database
[2024/10/17 20:06:08.628] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'xxx_aaa.flyway_schema_history' among known tables
[2024/10/17 20:06:08.630] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'xxx_aaa.flyway_schema_history' is filtered out of capturing
[2024/10/17 20:06:08.631] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'xxx_carbonmanagementplatform.nea_demofile' among known tables
[2024/10/17 20:06:08.631] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'xxx_aaa.nea_demofile' is filtered out of capturing
...
[2024/10/17 20:06:09.091] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'xxxbackend.xxx_logevent' among known tables
[2024/10/17 20:06:09.092] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :425 lambda$execute$4] 	 including 'xxxbackend.xxx_logevent' for further processing
[2024/10/17 20:06:09.092] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'xxxbackend.xxx_logeventkey' among known tables
[2024/10/17 20:06:09.092] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'xxxbackend.xxx_logeventkey' is filtered out of capturing
...
[2024/10/17 20:06:09.265] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'performance_schema.user_variables_by_thread' among known tables
[2024/10/17 20:06:09.265] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'performance_schema.user_variables_by_thread' is filtered out of capturing
[2024/10/17 20:06:09.265] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'performance_schema.users' among known tables
[2024/10/17 20:06:09.265] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'performance_schema.users' is filtered out of capturing
[2024/10/17 20:06:09.266] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'performance_schema.variables_by_thread' among known tables
[2024/10/17 20:06:09.266] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'performance_schema.variables_by_thread' is filtered out of capturing
[2024/10/17 20:06:09.331] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'sys.sys_config' among known tables
[2024/10/17 20:06:09.333] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'sys.sys_config' is filtered out of capturing
[2024/10/17 20:06:09.335] [WARN ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "table.whitelist" is deprecated and will be removed in future versions. Please use "table.include.list" instead.
[2024/10/17 20:06:09.353] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :459 execute] 	snapshot continuing with database(s): [xxxbackend]
[2024/10/17 20:06:09.355] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :505 execute] Step 6: generating DROP and CREATE statements to reflect current database schemas:
[2024/10/17 20:06:24.197] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :583 execute] Step 7: releasing global read lock to enable MySQL writes
[2024/10/17 20:06:24.306] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :589 execute] Step 7: blocked writes to MySQL for a total of 00:00:16.592
[2024/10/17 20:06:24.307] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :750 execute] Step 8: encountered only schema based snapshot, skipping data snapshot
[2024/10/17 20:06:24.307] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :772 execute] Step 9: committing transaction
[2024/10/17 20:06:24.358] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :851 execute] Completed snapshot in 00:00:19.261
[2024/10/17 20:06:24.412] [INFO ] [pool-4-thread-1] [io.debezium.jdbc.JdbcConnection                             :945 lambda$doClose$3] Connection gracefully closed
[2024/10/17 20:06:24.565] [INFO ] [debezium-engine] [io.debezium.connector.mysql.ChainedReader                   :201 startNextReader] Transitioning from the snapshot reader to the binlog reader
[2024/10/17 20:06:25.227] [INFO ] [debezium-engine] [io.debezium.connector.mysql.BinlogReader                    :367 doStart] GTID set purged on server: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-500796901
[2024/10/17 20:06:25.227] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :299 filterGtidSet] Attempting to generate a filtered GTID set
[2024/10/17 20:06:25.227] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :300 filterGtidSet] GTID set from previous recorded offset: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292
[2024/10/17 20:06:25.228] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :307 filterGtidSet] GTID set available on server: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292393
[2024/10/17 20:06:25.228] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :313 filterGtidSet] Using first available positions for new GTID channels
[2024/10/17 20:06:25.228] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :315 filterGtidSet] Relevant GTID set available on server: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292393
[2024/10/17 20:06:25.232] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :326 filterGtidSet] Final merged GTID set to use when connecting to MySQL: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292
[2024/10/17 20:06:25.232] [INFO ] [debezium-engine] [io.debezium.connector.mysql.BinlogReader                    :372 doStart] Registering binlog reader with GTID set: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292
[2024/10/17 20:06:25.235] [INFO ] [debezium-engine] [io.debezium.util.Threads                                    :287 newThread] Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
[2024/10/17 20:06:25.324] [INFO ] [blc-192.168.10.23:3306] [io.debezium.util.Threads                                    :287 newThread] Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
十月 17, 2024 8:06:25 下午 com.github.shyiko.mysql.binlog.BinaryLogClient connect
信息: Connected to 192.168.10.23:3306 at 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292 (sid:6519, cid:76144756)
[2024/10/17 20:06:25.838] [INFO ] [blc-192.168.10.23:3306] [io.debezium.connector.mysql.BinlogReader                    :1128 onConnect] Connected to MySQL binlog at 192.168.10.23:3306, starting at GTIDs 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292 and binlog file 'mysql-bin.237904', pos=1419258, skipping 0 events plus 0 rows
[2024/10/17 20:06:25.839] [INFO ] [debezium-engine] [io.debezium.connector.mysql.BinlogReader                    :415 doStart] Waiting for keepalive thread to start
[2024/10/17 20:06:25.841] [INFO ] [blc-192.168.10.23:3306] [io.debezium.util.Threads                                    :287 newThread] Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
[2024/10/17 20:06:25.974] [INFO ] [debezium-engine] [io.debezium.connector.mysql.BinlogReader                    :422 doStart] Keepalive thread is running
[2024/10/17 20:07:01.347] [INFO ] [Checkpoint Timer] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :756 createPendingCheckpoint] Triggering checkpoint 1 (type=CHECKPOINT) @ 1729166821329 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:07:01.475] [INFO ] [jobmanager-future-thread-3] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1245 completePendingCheckpoint] Completed checkpoint 1 for job 622e3901a4a1b260af83d5e7db8e4955 (333870 bytes in 141 ms).
[2024/10/17 20:07:01.477] [WARN ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :422 notifyCheckpointComplete] Consumer subtask 0 received confirmation for unknown checkpoint id 1
[2024/10/17 20:08:01.481] [INFO ] [Checkpoint Timer] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :756 createPendingCheckpoint] Triggering checkpoint 2 (type=CHECKPOINT) @ 1729166881480 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:08:01.525] [INFO ] [jobmanager-future-thread-7] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1245 completePendingCheckpoint] Completed checkpoint 2 for job 622e3901a4a1b260af83d5e7db8e4955 (333870 bytes in 29 ms).
[2024/10/17 20:08:01.526] [WARN ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :422 notifyCheckpointComplete] Consumer subtask 0 received confirmation for unknown checkpoint id 2
[2024/10/17 20:09:00.968] [INFO ] [Checkpoint Timer] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :756 createPendingCheckpoint] Triggering checkpoint 3 (type=CHECKPOINT) @ 1729166940961 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:09:01.563] [INFO ] [jobmanager-future-thread-5] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1245 completePendingCheckpoint] Completed checkpoint 3 for job 622e3901a4a1b260af83d5e7db8e4955 (333870 bytes in 595 ms).
[2024/10/17 20:09:01.563] [WARN ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :422 notifyCheckpointComplete] Consumer subtask 0 received confirmation for unknown checkpoint id 3
[2024/10/17 20:10:01.568] [INFO ] [Checkpoint Timer] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :756 createPendingCheckpoint] Triggering checkpoint 4 (type=CHECKPOINT) @ 1729167001565 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:10:01.622] [INFO ] [jobmanager-future-thread-2] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1245 completePendingCheckpoint] Completed checkpoint 4 for job 622e3901a4a1b260af83d5e7db8e4955 (333870 bytes in 52 ms).
[2024/10/17 20:10:01.622] [WARN ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :422 notifyCheckpointComplete] Consumer subtask 0 received confirmation for unknown checkpoint id 4
[2024/10/17 20:11:01.636] [INFO ] [Checkpoint Timer] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :756 createPendingCheckpoint] Triggering checkpoint 5 (type=CHECKPOINT) @ 1729167061634 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:11:01.662] [INFO ] [jobmanager-future-thread-8] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1245 completePendingCheckpoint] Completed checkpoint 5 for job 622e3901a4a1b260af83d5e7db8e4955 (333870 bytes in 24 ms).
[2024/10/17 20:11:01.663] [WARN ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :422 notifyCheckpointComplete] Consumer subtask 0 received confirmation for unknown checkpoint id 5
[2024/10/17 20:12:01.666] [INFO ] [Checkpoint Timer] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :756 createPendingCheckpoint] Triggering checkpoint 6 (type=CHECKPOINT) @ 1729167121665 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:12:01.687] [INFO ] [jobmanager-future-thread-4] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1245 completePendingCheckpoint] Completed checkpoint 6 for job 622e3901a4a1b260af83d5e7db8e4955 (333870 bytes in 18 ms).
[2024/10/17 20:12:01.688] [WARN ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :422 notifyCheckpointComplete] Consumer subtask 0 received confirmation for unknown checkpoint id 6
[2024/10/17 20:13:01.745] [INFO ] [Checkpoint Timer] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :756 createPendingCheckpoint] Triggering checkpoint 7 (type=CHECKPOINT) @ 1729167181739 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:13:01.907] [INFO ] [jobmanager-future-thread-8] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1245 completePendingCheckpoint] Completed checkpoint 7 for job 622e3901a4a1b260af83d5e7db8e4955 (333870 bytes in 148 ms).
[2024/10/17 20:13:01.907] [WARN ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :422 notifyCheckpointComplete] Consumer subtask 0 received confirmation for unknown checkpoint id 7
...

MYSQL数据库

验证:日志中BinaryLogClient的 cid = 76144756 = processlist.id

show processlist

-- processlist.id = connection id = session id
|Id        |User       |Host              |db |Command         |Time|State                                                        |Info|
|----------|-----------|------------------|---|----------------|----|-------------------------------------------------------------|----|
...
|76,144,756|cdc_user|192.168.38.198:56829|   |Binlog Dump GTID|202 |Master has sent all binlog to slave; waiting for more updates|    |
...

日志分析：重点日志行，所提供的关键信息

• 依赖关系：

• Flink Cdc MySQL Connector/com.alibaba.ververica:flink-connector-mysql-cdc
• --> debezium
• --> com.github.shyiko:mysql-binlog-connector-java

• vvc:flink-connector-mysql-cdc 的关键组件类

• com.alibaba.ververica.cdc.debezium.DebeziumDeserializationSchema

• debezium 的关键组件类

• io.debezium.data.Envelope
• org.apache.kafka.connect.data.Field
• org.apache.kafka.connect.data.Schema
• org.apache.kafka.connect.data.Struct
• org.apache.kafka.connect.source.SourceRecord

• shyiko:mysql-binlog-connector-java 的 关键组件类

• com.github.shyiko.mysql.binlog.BinaryLogClient
• 日志中能提供的关键信息:

• 主库(master)的 server_uuid = 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0
• sid = FlinkCdcJob 的 Salve Server Id = 6519
• cid = FlinkCdcJob 的 MySQL 连接对象 的 connection id = FlinkCdcJob 的 MySQL 连接对象 的 session id=show processlist中 的processlist.id=select connection_id() as sessionId=76144756`

源码分析

Flink CDC - debezium - shyiko:mysql-binlog-connector-java

• shyiko/mysql-binlog-connector-java

此模块被 flink cdc 直接打包到了 JAR包内
https://github.com/shyiko/mysql-binlog-connector-java

<dependency>
    <groupId>com.github.shyiko</groupId>
    <artifactId>mysql-binlog-connector-java</artifactId>
    <version>${version}</version>
</dependency>

BinaryLogClient

• com.github.shyiko.mysql.binlog.BinaryLogClient

• 关键 Method : com.github.shyiko.mysql.binlog.BinaryLogClient#connect()
• 打印的关键日志: 提供了 sid = flink cdc slave 的 server id / cid = connection id = processlist id

十月 17, 2024 8:06:25 下午 com.github.shyiko.mysql.binlog.BinaryLogClient connect
信息: Connected to 192.168.10.23:3306 at 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292 (sid:6519, cid:76144756)

关键 Method : BinaryLogClient#connect()

• com.github.shyiko.mysql.binlog.BinaryLogClient#connect()

完整源码

package com.github.shyiko.mysql.binlog;

import com.github.shyiko.mysql.binlog.event.Event;
import com.github.shyiko.mysql.binlog.event.EventHeader;
import com.github.shyiko.mysql.binlog.event.EventHeaderV4;
import com.github.shyiko.mysql.binlog.event.EventType;
import com.github.shyiko.mysql.binlog.event.GtidEventData;
import com.github.shyiko.mysql.binlog.event.QueryEventData;
import com.github.shyiko.mysql.binlog.event.RotateEventData;
import com.github.shyiko.mysql.binlog.event.deserialization.ChecksumType;
import com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException;
import com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializer;
import com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer;
import com.github.shyiko.mysql.binlog.event.deserialization.GtidEventDataDeserializer;
import com.github.shyiko.mysql.binlog.event.deserialization.QueryEventDataDeserializer;
import com.github.shyiko.mysql.binlog.event.deserialization.RotateEventDataDeserializer;
import com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.EventDataWrapper;
import com.github.shyiko.mysql.binlog.io.ByteArrayInputStream;
import com.github.shyiko.mysql.binlog.jmx.BinaryLogClientMXBean;
import com.github.shyiko.mysql.binlog.network.Authenticator;
import com.github.shyiko.mysql.binlog.network.DefaultSSLSocketFactory;
import com.github.shyiko.mysql.binlog.network.SSLMode;
import com.github.shyiko.mysql.binlog.network.SSLSocketFactory;
import com.github.shyiko.mysql.binlog.network.ServerException;
import com.github.shyiko.mysql.binlog.network.SocketFactory;
import com.github.shyiko.mysql.binlog.network.TLSHostnameVerifier;
import com.github.shyiko.mysql.binlog.network.protocol.ErrorPacket;
import com.github.shyiko.mysql.binlog.network.protocol.GreetingPacket;
import com.github.shyiko.mysql.binlog.network.protocol.PacketChannel;
import com.github.shyiko.mysql.binlog.network.protocol.ResultSetRowPacket;
import com.github.shyiko.mysql.binlog.network.protocol.command.Command;
import com.github.shyiko.mysql.binlog.network.protocol.command.DumpBinaryLogCommand;
import com.github.shyiko.mysql.binlog.network.protocol.command.DumpBinaryLogGtidCommand;
import com.github.shyiko.mysql.binlog.network.protocol.command.PingCommand;
import com.github.shyiko.mysql.binlog.network.protocol.command.QueryCommand;
import com.github.shyiko.mysql.binlog.network.protocol.command.SSLRequestCommand;
import java.io.EOFException;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Socket;
import java.net.SocketException;
import java.security.GeneralSecurityException;
import java.security.SecureRandom;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.Arrays;
import java.util.Collections;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicReference;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.net.ssl.KeyManager;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;

public class BinaryLogClient implements BinaryLogClientMXBean {
    private static final SSLSocketFactory DEFAULT_REQUIRED_SSL_MODE_SOCKET_FACTORY = new DefaultSSLSocketFactory() {
        protected void initSSLContext(SSLContext sc) throws GeneralSecurityException {
            sc.init((KeyManager[])null, new TrustManager[]{new X509TrustManager() {
                public void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {
                }

                public void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {
                }

                public X509Certificate[] getAcceptedIssuers() {
                    return new X509Certificate[0];
                }
            }}, (SecureRandom)null);
        }
    };
    private static final SSLSocketFactory DEFAULT_VERIFY_CA_SSL_MODE_SOCKET_FACTORY = new DefaultSSLSocketFactory();
    private static final int MAX_PACKET_LENGTH = 16777215;
    private final Logger logger;
    private final String hostname;
    private final int port;
    private final String schema;
    private final String username;
    private final String password;
    private boolean blocking;
    private long serverId;
    private volatile String binlogFilename;
    private volatile long binlogPosition;
    private volatile long connectionId;
    private SSLMode sslMode;
    private GtidSet gtidSet;
    private final Object gtidSetAccessLock;
    private boolean gtidSetFallbackToPurged;
    private boolean useBinlogFilenamePositionInGtidMode;
    private String gtid;
    private boolean tx;
    private EventDeserializer eventDeserializer;
    private final List<EventListener> eventListeners;
    private final List<LifecycleListener> lifecycleListeners;
    private SocketFactory socketFactory;
    private SSLSocketFactory sslSocketFactory;
    private volatile PacketChannel channel;
    private volatile boolean connected;
    private volatile long masterServerId;
    private ThreadFactory threadFactory;
    private boolean keepAlive;
    private long keepAliveInterval;
    private long heartbeatInterval;
    private volatile long eventLastSeen;
    private long connectTimeout;
    private volatile ExecutorService keepAliveThreadExecutor;
    private final Lock connectLock;
    private final Lock keepAliveThreadExecutorLock;

    public BinaryLogClient(String username, String password) {
        this("localhost", 3306, (String)null, username, password);
    }

    public BinaryLogClient(String schema, String username, String password) {
        this("localhost", 3306, schema, username, password);
    }

    public BinaryLogClient(String hostname, int port, String username, String password) {
        this(hostname, port, (String)null, username, password);
    }

    public BinaryLogClient(String hostname, int port, String schema, String username, String password) {
        this.logger = Logger.getLogger(this.getClass().getName());
        this.blocking = true;
        this.serverId = 65535L;
        this.binlogPosition = 4L;
        this.sslMode = SSLMode.DISABLED;
        this.gtidSetAccessLock = new Object();
        this.eventDeserializer = new EventDeserializer();
        this.eventListeners = new CopyOnWriteArrayList();
        this.lifecycleListeners = new CopyOnWriteArrayList();
        this.masterServerId = -1L;
        this.keepAlive = true;
        this.keepAliveInterval = TimeUnit.MINUTES.toMillis(1L);
        this.connectTimeout = TimeUnit.SECONDS.toMillis(3L);
        this.connectLock = new ReentrantLock();
        this.keepAliveThreadExecutorLock = new ReentrantLock();
        this.hostname = hostname;
        this.port = port;
        this.schema = schema;
        this.username = username;
        this.password = password;
    }

    public boolean isBlocking() {
        return this.blocking;
    }

    public void setBlocking(boolean blocking) {
        this.blocking = blocking;
    }

    public SSLMode getSSLMode() {
        return this.sslMode;
    }

    public void setSSLMode(SSLMode sslMode) {
        if (sslMode == null) {
            throw new IllegalArgumentException("SSL mode cannot be NULL");
        } else {
            this.sslMode = sslMode;
        }
    }

    public long getMasterServerId() {
        return this.masterServerId;
    }

    public long getServerId() {
        return this.serverId;
    }

    public void setServerId(long serverId) {
        this.serverId = serverId;
    }

    public String getBinlogFilename() {
        return this.binlogFilename;
    }

    public void setBinlogFilename(String binlogFilename) {
        this.binlogFilename = binlogFilename;
    }

    public long getBinlogPosition() {
        return this.binlogPosition;
    }

    public void setBinlogPosition(long binlogPosition) {
        this.binlogPosition = binlogPosition;
    }

    public long getConnectionId() {
        return this.connectionId;
    }

    public String getGtidSet() {
        synchronized(this.gtidSetAccessLock) {
            return this.gtidSet != null ? this.gtidSet.toString() : null;
        }
    }

    public void setGtidSet(String gtidSet) {
        if (gtidSet != null && this.binlogFilename == null) {
            this.binlogFilename = "";
        }

        synchronized(this.gtidSetAccessLock) {
            this.gtidSet = gtidSet != null ? new GtidSet(gtidSet) : null;
        }
    }

    public boolean isGtidSetFallbackToPurged() {
        return this.gtidSetFallbackToPurged;
    }

    public void setGtidSetFallbackToPurged(boolean gtidSetFallbackToPurged) {
        this.gtidSetFallbackToPurged = gtidSetFallbackToPurged;
    }

    public boolean isUseBinlogFilenamePositionInGtidMode() {
        return this.useBinlogFilenamePositionInGtidMode;
    }

    public void setUseBinlogFilenamePositionInGtidMode(boolean useBinlogFilenamePositionInGtidMode) {
        this.useBinlogFilenamePositionInGtidMode = useBinlogFilenamePositionInGtidMode;
    }

    public boolean isKeepAlive() {
        return this.keepAlive;
    }

    public void setKeepAlive(boolean keepAlive) {
        this.keepAlive = keepAlive;
    }

    public long getKeepAliveInterval() {
        return this.keepAliveInterval;
    }

    public void setKeepAliveInterval(long keepAliveInterval) {
        this.keepAliveInterval = keepAliveInterval;
    }

    /** @deprecated */
    public long getKeepAliveConnectTimeout() {
        return this.connectTimeout;
    }

    /** @deprecated */
    public void setKeepAliveConnectTimeout(long connectTimeout) {
        this.connectTimeout = connectTimeout;
    }

    public long getHeartbeatInterval() {
        return this.heartbeatInterval;
    }

    public void setHeartbeatInterval(long heartbeatInterval) {
        this.heartbeatInterval = heartbeatInterval;
    }

    public long getConnectTimeout() {
        return this.connectTimeout;
    }

    public void setConnectTimeout(long connectTimeout) {
        this.connectTimeout = connectTimeout;
    }

    public void setEventDeserializer(EventDeserializer eventDeserializer) {
        if (eventDeserializer == null) {
            throw new IllegalArgumentException("Event deserializer cannot be NULL");
        } else {
            this.eventDeserializer = eventDeserializer;
        }
    }

    public void setSocketFactory(SocketFactory socketFactory) {
        this.socketFactory = socketFactory;
    }

    public void setSslSocketFactory(SSLSocketFactory sslSocketFactory) {
        this.sslSocketFactory = sslSocketFactory;
    }

    public void setThreadFactory(ThreadFactory threadFactory) {
        this.threadFactory = threadFactory;
    }

    public void connect() throws IOException, IllegalStateException {
        if (!this.connectLock.tryLock()) {
            throw new IllegalStateException("BinaryLogClient is already connected");
        } else {
            boolean notifyWhenDisconnected = false;
            boolean var22 = false;

            try {
                var22 = true;
                Callable cancelDisconnect = null;

                try {
                    try {
                        long start = System.currentTimeMillis();
                        this.channel = this.openChannel();
                        if (this.connectTimeout > 0L && !this.isKeepAliveThreadRunning()) {
                            cancelDisconnect = this.scheduleDisconnectIn(this.connectTimeout - (System.currentTimeMillis() - start));
                        }

                        if (this.channel.getInputStream().peek() == -1) {
                            throw new EOFException();
                        }
                    } catch (IOException var35) {
                        throw new IOException("Failed to connect to MySQL on " + this.hostname + ":" + this.port + ". Please make sure it's running.", var35);
                    }

                    GreetingPacket greetingPacket = this.receiveGreeting();
                    this.tryUpgradeToSSL(greetingPacket);
                    (new Authenticator(greetingPacket, this.channel, this.schema, this.username, this.password)).authenticate();
                    this.channel.authenticationComplete();
                    this.connectionId = greetingPacket.getThreadId();
                    if ("".equals(this.binlogFilename)) {
                        synchronized(this.gtidSetAccessLock) {
                            if (this.gtidSet != null && "".equals(this.gtidSet.toString()) && this.gtidSetFallbackToPurged) {
                                this.gtidSet = new GtidSet(this.fetchGtidPurged());
                            }
                        }
                    }

                    if (this.binlogFilename == null) {
                        this.fetchBinlogFilenameAndPosition();
                    }

                    if (this.binlogPosition < 4L) {
                        if (this.logger.isLoggable(Level.WARNING)) {
                            this.logger.warning("Binary log position adjusted from " + this.binlogPosition + " to " + 4);
                        }

                        this.binlogPosition = 4L;
                    }

                    ChecksumType checksumType = this.fetchBinlogChecksum();
                    if (checksumType != ChecksumType.NONE) {
                        this.confirmSupportOfChecksum(checksumType);
                    }

                    this.setMasterServerId();
                    if (this.heartbeatInterval > 0L) {
                        this.enableHeartbeat();
                    }

                    this.gtid = null;
                    this.tx = false;
                    this.requestBinaryLogStream();
                } catch (IOException var36) {
                    this.disconnectChannel();
                    throw var36;
                } finally {
                    if (cancelDisconnect != null) {
                        try {
                            cancelDisconnect.call();
                        } catch (Exception var34) {
                            if (this.logger.isLoggable(Level.WARNING)) {
                                this.logger.warning("\"" + var34.getMessage() + "\" was thrown while canceling scheduled disconnect call");
                            }
                        }
                    }

                }

                this.connected = true;
                notifyWhenDisconnected = true;
                if (this.logger.isLoggable(Level.INFO)) {
                    String position;
                    synchronized(this.gtidSetAccessLock) {
                        position = this.gtidSet != null ? this.gtidSet.toString() : this.binlogFilename + "/" + this.binlogPosition;
                    }

                    this.logger.info("Connected to " + this.hostname + ":" + this.port + " at " + position + " (" + (this.blocking ? "sid:" + this.serverId + ", " : "") + "cid:" + this.connectionId + ")");
                }

                Iterator var43 = this.lifecycleListeners.iterator();

                while(var43.hasNext()) {
                    LifecycleListener lifecycleListener = (LifecycleListener)var43.next();
                    lifecycleListener.onConnect(this);
                }

                if (this.keepAlive && !this.isKeepAliveThreadRunning()) {
                    this.spawnKeepAliveThread();
                }

                this.ensureEventDataDeserializer(EventType.ROTATE, RotateEventDataDeserializer.class);
                synchronized(this.gtidSetAccessLock) {
                    if (this.gtidSet != null) {
                        this.ensureEventDataDeserializer(EventType.GTID, GtidEventDataDeserializer.class);
                        this.ensureEventDataDeserializer(EventType.QUERY, QueryEventDataDeserializer.class);
                    }
                }

                this.listenForEventPackets();
                var22 = false;
            } finally {
                if (var22) {
                    this.connectLock.unlock();
                    if (notifyWhenDisconnected) {
                        Iterator var11 = this.lifecycleListeners.iterator();

                        while(var11.hasNext()) {
                            LifecycleListener lifecycleListener = (LifecycleListener)var11.next();
                            lifecycleListener.onDisconnect(this);
                        }
                    }

                }
            }

            this.connectLock.unlock();
            if (notifyWhenDisconnected) {
                Iterator var40 = this.lifecycleListeners.iterator();

                while(var40.hasNext()) {
                    LifecycleListener lifecycleListener = (LifecycleListener)var40.next();
                    lifecycleListener.onDisconnect(this);
                }
            }

        }
    }

    private PacketChannel openChannel() throws IOException {
        Socket socket = this.socketFactory != null ? this.socketFactory.createSocket() : new Socket();
        socket.connect(new InetSocketAddress(this.hostname, this.port), (int)this.connectTimeout);
        return new PacketChannel(socket);
    }

    private Callable scheduleDisconnectIn(final long timeout) {
        final CountDownLatch connectLatch = new CountDownLatch(1);
        final Thread thread = this.newNamedThread(new Runnable() {
            public void run() {
                try {
                    connectLatch.await(timeout, TimeUnit.MILLISECONDS);
                } catch (InterruptedException var3) {
                    if (BinaryLogClient.this.logger.isLoggable(Level.WARNING)) {
                        BinaryLogClient.this.logger.log(Level.WARNING, var3.getMessage());
                    }
                }

                if (connectLatch.getCount() != 0L) {
                    if (BinaryLogClient.this.logger.isLoggable(Level.WARNING)) {
                        BinaryLogClient.this.logger.warning("Failed to establish connection in " + timeout + "ms. Forcing disconnect.");
                    }

                    try {
                        BinaryLogClient.this.disconnectChannel();
                    } catch (IOException var2) {
                        if (BinaryLogClient.this.logger.isLoggable(Level.WARNING)) {
                            BinaryLogClient.this.logger.log(Level.WARNING, var2.getMessage());
                        }
                    }
                }

            }
        }, "blc-disconnect-" + this.hostname + ":" + this.port);
        thread.start();
        return new Callable() {
            public Object call() throws Exception {
                connectLatch.countDown();
                thread.join();
                return null;
            }
        };
    }

    private void checkError(byte[] packet) throws IOException {
        if (packet[0] == -1) {
            byte[] bytes = Arrays.copyOfRange(packet, 1, packet.length);
            ErrorPacket errorPacket = new ErrorPacket(bytes);
            throw new ServerException(errorPacket.getErrorMessage(), errorPacket.getErrorCode(), errorPacket.getSqlState());
        }
    }

    private GreetingPacket receiveGreeting() throws IOException {
        byte[] initialHandshakePacket = this.channel.read();
        this.checkError(initialHandshakePacket);
        return new GreetingPacket(initialHandshakePacket);
    }

    private boolean tryUpgradeToSSL(GreetingPacket greetingPacket) throws IOException {
        int collation = greetingPacket.getServerCollation();
        if (this.sslMode != SSLMode.DISABLED) {
            boolean serverSupportsSSL = (greetingPacket.getServerCapabilities() & 2048) != 0;
            if (!serverSupportsSSL && (this.sslMode == SSLMode.REQUIRED || this.sslMode == SSLMode.VERIFY_CA || this.sslMode == SSLMode.VERIFY_IDENTITY)) {
                throw new IOException("MySQL server does not support SSL");
            }

            if (serverSupportsSSL) {
                SSLRequestCommand sslRequestCommand = new SSLRequestCommand();
                sslRequestCommand.setCollation(collation);
                this.channel.write(sslRequestCommand);
                SSLSocketFactory sslSocketFactory = this.sslSocketFactory != null ? this.sslSocketFactory : (this.sslMode != SSLMode.REQUIRED && this.sslMode != SSLMode.PREFERRED ? DEFAULT_VERIFY_CA_SSL_MODE_SOCKET_FACTORY : DEFAULT_REQUIRED_SSL_MODE_SOCKET_FACTORY);
                this.channel.upgradeToSSL(sslSocketFactory, this.sslMode == SSLMode.VERIFY_IDENTITY ? new TLSHostnameVerifier() : null);
                this.logger.info("SSL enabled");
                return true;
            }
        }

        return false;
    }

    private void enableHeartbeat() throws IOException {
        this.channel.write(new QueryCommand("set @master_heartbeat_period=" + this.heartbeatInterval * 1000000L));
        byte[] statementResult = this.channel.read();
        this.checkError(statementResult);
    }

    private void setMasterServerId() throws IOException {
        this.channel.write(new QueryCommand("select @@server_id"));
        ResultSetRowPacket[] resultSet = this.readResultSet();
        if (resultSet.length >= 0) {
            this.masterServerId = Long.parseLong(resultSet[0].getValue(0));
        }

    }

    private void requestBinaryLogStream() throws IOException {
        long serverId = this.blocking ? this.serverId : 0L;
        Object dumpBinaryLogCommand;
        synchronized(this.gtidSetAccessLock) {
            if (this.gtidSet != null) {
                dumpBinaryLogCommand = new DumpBinaryLogGtidCommand(serverId, this.useBinlogFilenamePositionInGtidMode ? this.binlogFilename : "", this.useBinlogFilenamePositionInGtidMode ? this.binlogPosition : 4L, this.gtidSet);
            } else {
                dumpBinaryLogCommand = new DumpBinaryLogCommand(serverId, this.binlogFilename, this.binlogPosition);
            }
        }

        this.channel.write((Command)dumpBinaryLogCommand);
    }

    private void ensureEventDataDeserializer(EventType eventType, Class<? extends EventDataDeserializer> eventDataDeserializerClass) {
        EventDataDeserializer eventDataDeserializer = this.eventDeserializer.getEventDataDeserializer(eventType);
        if (eventDataDeserializer.getClass() != eventDataDeserializerClass && eventDataDeserializer.getClass() != EventDeserializer.EventDataWrapper.Deserializer.class) {
            EventDataDeserializer internalEventDataDeserializer;
            try {
                internalEventDataDeserializer = (EventDataDeserializer)eventDataDeserializerClass.newInstance();
            } catch (Exception var6) {
                throw new RuntimeException(var6);
            }

            this.eventDeserializer.setEventDataDeserializer(eventType, new EventDeserializer.EventDataWrapper.Deserializer(internalEventDataDeserializer, eventDataDeserializer));
        }

    }

    private void spawnKeepAliveThread() {
        final ExecutorService threadExecutor = Executors.newSingleThreadExecutor(new ThreadFactory() {
            public Thread newThread(Runnable runnable) {
                return BinaryLogClient.this.newNamedThread(runnable, "blc-keepalive-" + BinaryLogClient.this.hostname + ":" + BinaryLogClient.this.port);
            }
        });

        try {
            this.keepAliveThreadExecutorLock.lock();
            threadExecutor.submit(new Runnable() {
                public void run() {
                    while(!threadExecutor.isShutdown()) {
                        try {
                            Thread.sleep(BinaryLogClient.this.keepAliveInterval);
                        } catch (InterruptedException var5) {
                        }

                        if (threadExecutor.isShutdown()) {
                            return;
                        }

                        boolean connectionLost = false;
                        if (BinaryLogClient.this.heartbeatInterval > 0L) {
                            connectionLost = System.currentTimeMillis() - BinaryLogClient.this.eventLastSeen > BinaryLogClient.this.keepAliveInterval;
                        } else {
                            try {
                                BinaryLogClient.this.channel.write(new PingCommand());
                            } catch (IOException var4) {
                                connectionLost = true;
                            }
                        }

                        if (connectionLost) {
                            if (BinaryLogClient.this.logger.isLoggable(Level.INFO)) {
                                BinaryLogClient.this.logger.info("Trying to restore lost connection to " + BinaryLogClient.this.hostname + ":" + BinaryLogClient.this.port);
                            }

                            try {
                                BinaryLogClient.this.terminateConnect();
                                BinaryLogClient.this.connect(BinaryLogClient.this.connectTimeout);
                            } catch (Exception var3) {
                                if (BinaryLogClient.this.logger.isLoggable(Level.WARNING)) {
                                    BinaryLogClient.this.logger.warning("Failed to restore connection to " + BinaryLogClient.this.hostname + ":" + BinaryLogClient.this.port + ". Next attempt in " + BinaryLogClient.this.keepAliveInterval + "ms");
                                }
                            }
                        }
                    }

                }
            });
            this.keepAliveThreadExecutor = threadExecutor;
        } finally {
            this.keepAliveThreadExecutorLock.unlock();
        }

    }

    private Thread newNamedThread(Runnable runnable, String threadName) {
        Thread thread = this.threadFactory == null ? new Thread(runnable) : this.threadFactory.newThread(runnable);
        thread.setName(threadName);
        return thread;
    }

    boolean isKeepAliveThreadRunning() {
        boolean var1;
        try {
            this.keepAliveThreadExecutorLock.lock();
            var1 = this.keepAliveThreadExecutor != null && !this.keepAliveThreadExecutor.isShutdown();
        } finally {
            this.keepAliveThreadExecutorLock.unlock();
        }

        return var1;
    }

    public void connect(final long timeout) throws IOException, TimeoutException {
        final CountDownLatch countDownLatch = new CountDownLatch(1);
        AbstractLifecycleListener connectListener = new AbstractLifecycleListener() {
            public void onConnect(BinaryLogClient client) {
                countDownLatch.countDown();
            }
        };
        this.registerLifecycleListener(connectListener);
        final AtomicReference<IOException> exceptionReference = new AtomicReference();
        Runnable runnable = new Runnable() {
            public void run() {
                try {
                    BinaryLogClient.this.setConnectTimeout(timeout);
                    BinaryLogClient.this.connect();
                } catch (IOException var2) {
                    exceptionReference.set(var2);
                    countDownLatch.countDown();
                } catch (Exception var3) {
                    exceptionReference.set(new IOException(var3));
                    countDownLatch.countDown();
                }

            }
        };
        this.newNamedThread(runnable, "blc-" + this.hostname + ":" + this.port).start();
        boolean started = false;

        try {
            started = countDownLatch.await(timeout, TimeUnit.MILLISECONDS);
        } catch (InterruptedException var13) {
            if (this.logger.isLoggable(Level.WARNING)) {
                this.logger.log(Level.WARNING, var13.getMessage());
            }
        }

        this.unregisterLifecycleListener((LifecycleListener)connectListener);
        if (exceptionReference.get() != null) {
            throw (IOException)exceptionReference.get();
        } else if (!started) {
            try {
                this.terminateConnect();
            } finally {
                throw new TimeoutException("BinaryLogClient was unable to connect in " + timeout + "ms");
            }
        }
    }

    public boolean isConnected() {
        return this.connected;
    }

    private String fetchGtidPurged() throws IOException {
        this.channel.write(new QueryCommand("show global variables like 'gtid_purged'"));
        ResultSetRowPacket[] resultSet = this.readResultSet();
        return resultSet.length != 0 ? resultSet[0].getValue(1).toUpperCase() : "";
    }

    private void fetchBinlogFilenameAndPosition() throws IOException {
        this.channel.write(new QueryCommand("show master status"));
        ResultSetRowPacket[] resultSet = this.readResultSet();
        if (resultSet.length == 0) {
            throw new IOException("Failed to determine binlog filename/position");
        } else {
            ResultSetRowPacket resultSetRow = resultSet[0];
            this.binlogFilename = resultSetRow.getValue(0);
            this.binlogPosition = Long.parseLong(resultSetRow.getValue(1));
        }
    }

    private ChecksumType fetchBinlogChecksum() throws IOException {
        this.channel.write(new QueryCommand("show global variables like 'binlog_checksum'"));
        ResultSetRowPacket[] resultSet = this.readResultSet();
        return resultSet.length == 0 ? ChecksumType.NONE : ChecksumType.valueOf(resultSet[0].getValue(1).toUpperCase());
    }

    private void confirmSupportOfChecksum(ChecksumType checksumType) throws IOException {
        this.channel.write(new QueryCommand("set @master_binlog_checksum= @@global.binlog_checksum"));
        byte[] statementResult = this.channel.read();
        this.checkError(statementResult);
        this.eventDeserializer.setChecksumType(checksumType);
    }

    private void listenForEventPackets() throws IOException {
        ByteArrayInputStream inputStream = this.channel.getInputStream();
        boolean completeShutdown = false;

        try {
            label199:
            while(inputStream.peek() != -1) {
                int packetLength = inputStream.readInteger(3);
                inputStream.skip(1L);
                int marker = inputStream.read();
                if (marker == 255) {
                    ErrorPacket errorPacket = new ErrorPacket(inputStream.read(packetLength - 1));
                    throw new ServerException(errorPacket.getErrorMessage(), errorPacket.getErrorCode(), errorPacket.getSqlState());
                }

                if (marker == 254 && !this.blocking) {
                    completeShutdown = true;
                    break;
                }

                Event event;
                try {
                    event = this.eventDeserializer.nextEvent(packetLength == 16777215 ? new ByteArrayInputStream(this.readPacketSplitInChunks(inputStream, packetLength - 1)) : inputStream);
                    if (event == null) {
                        throw new EOFException();
                    }
                } catch (Exception var14) {
                    Exception e = var14;
                    Throwable cause = var14 instanceof EventDataDeserializationException ? var14.getCause() : var14;
                    if (!(cause instanceof EOFException) && !(cause instanceof SocketException)) {
                        if (!this.isConnected()) {
                            continue;
                        }

                        Iterator var8 = this.lifecycleListeners.iterator();

                        while(true) {
                            if (!var8.hasNext()) {
                                continue label199;
                            }

                            LifecycleListener lifecycleListener = (LifecycleListener)var8.next();
                            lifecycleListener.onEventDeserializationFailure(this, e);
                        }
                    }

                    throw var14;
                }

                if (this.isConnected()) {
                    this.eventLastSeen = System.currentTimeMillis();
                    this.updateGtidSet(event);
                    this.notifyEventListeners(event);
                    this.updateClientBinlogFilenameAndPosition(event);
                }
            }
        } catch (Exception var15) {
            Exception e = var15;
            if (this.isConnected()) {
                Iterator var4 = this.lifecycleListeners.iterator();

                while(var4.hasNext()) {
                    LifecycleListener lifecycleListener = (LifecycleListener)var4.next();
                    lifecycleListener.onCommunicationFailure(this, e);
                }
            }
        } finally {
            if (this.isConnected()) {
                if (completeShutdown) {
                    this.disconnect();
                } else {
                    this.disconnectChannel();
                }
            }

        }

    }

    private byte[] readPacketSplitInChunks(ByteArrayInputStream inputStream, int packetLength) throws IOException {
        byte[] result = inputStream.read(packetLength);

        int chunkLength;
        do {
            chunkLength = inputStream.readInteger(3);
            inputStream.skip(1L);
            result = Arrays.copyOf(result, result.length + chunkLength);
            inputStream.fill(result, result.length - chunkLength, chunkLength);
        } while(chunkLength == 16777215);

        return result;
    }

    private void updateClientBinlogFilenameAndPosition(Event event) {
        EventHeader eventHeader = event.getHeader();
        EventType eventType = eventHeader.getEventType();
        if (eventType == EventType.ROTATE) {
            RotateEventData rotateEventData = (RotateEventData)EventDataWrapper.internal(event.getData());
            this.binlogFilename = rotateEventData.getBinlogFilename();
            this.binlogPosition = rotateEventData.getBinlogPosition();
        } else if (eventType != EventType.TABLE_MAP && eventHeader instanceof EventHeaderV4) {
            EventHeaderV4 trackableEventHeader = (EventHeaderV4)eventHeader;
            long nextBinlogPosition = trackableEventHeader.getNextPosition();
            if (nextBinlogPosition > 0L) {
                this.binlogPosition = nextBinlogPosition;
            }
        }

    }

    private void updateGtidSet(Event event) {
        synchronized(this.gtidSetAccessLock) {
            if (this.gtidSet == null) {
                return;
            }
        }

        EventHeader eventHeader = event.getHeader();
        switch (eventHeader.getEventType()) {
            case GTID:
                GtidEventData gtidEventData = (GtidEventData)EventDataWrapper.internal(event.getData());
                this.gtid = gtidEventData.getGtid();
                break;
            case XID:
                this.commitGtid();
                this.tx = false;
                break;
            case QUERY:
                QueryEventData queryEventData = (QueryEventData)EventDataWrapper.internal(event.getData());
                String sql = queryEventData.getSql();
                if (sql != null) {
                    if ("BEGIN".equals(sql)) {
                        this.tx = true;
                    } else if (!"COMMIT".equals(sql) && !"ROLLBACK".equals(sql)) {
                        if (!this.tx) {
                            this.commitGtid();
                        }
                    } else {
                        this.commitGtid();
                        this.tx = false;
                    }
                }
        }

    }

    private void commitGtid() {
        if (this.gtid != null) {
            synchronized(this.gtidSetAccessLock) {
                this.gtidSet.add(this.gtid);
            }
        }

    }

    private ResultSetRowPacket[] readResultSet() throws IOException {
        List<ResultSetRowPacket> resultSet = new LinkedList();
        byte[] statementResult = this.channel.read();
        this.checkError(statementResult);

        while(this.channel.read()[0] != -2) {
        }

        byte[] bytes;
        while((bytes = this.channel.read())[0] != -2) {
            this.checkError(bytes);
            resultSet.add(new ResultSetRowPacket(bytes));
        }

        return (ResultSetRowPacket[])resultSet.toArray(new ResultSetRowPacket[resultSet.size()]);
    }

    public List<EventListener> getEventListeners() {
        return Collections.unmodifiableList(this.eventListeners);
    }

    public void registerEventListener(EventListener eventListener) {
        this.eventListeners.add(eventListener);
    }

    public void unregisterEventListener(Class<? extends EventListener> listenerClass) {
        Iterator var2 = this.eventListeners.iterator();

        while(var2.hasNext()) {
            EventListener eventListener = (EventListener)var2.next();
            if (listenerClass.isInstance(eventListener)) {
                this.eventListeners.remove(eventListener);
            }
        }

    }

    public void unregisterEventListener(EventListener eventListener) {
        this.eventListeners.remove(eventListener);
    }

    private void notifyEventListeners(Event event) {
        if (event.getData() instanceof EventDeserializer.EventDataWrapper) {
            event = new Event(event.getHeader(), ((EventDeserializer.EventDataWrapper)event.getData()).getExternal());
        }

        Iterator var2 = this.eventListeners.iterator();

        while(var2.hasNext()) {
            EventListener eventListener = (EventListener)var2.next();

            try {
                eventListener.onEvent(event);
            } catch (Exception var5) {
                if (this.logger.isLoggable(Level.WARNING)) {
                    this.logger.log(Level.WARNING, eventListener + " choked on " + event, var5);
                }
            }
        }

    }

    public List<LifecycleListener> getLifecycleListeners() {
        return Collections.unmodifiableList(this.lifecycleListeners);
    }

    public void registerLifecycleListener(LifecycleListener lifecycleListener) {
        this.lifecycleListeners.add(lifecycleListener);
    }

    public void unregisterLifecycleListener(Class<? extends LifecycleListener> listenerClass) {
        Iterator var2 = this.lifecycleListeners.iterator();

        while(var2.hasNext()) {
            LifecycleListener lifecycleListener = (LifecycleListener)var2.next();
            if (listenerClass.isInstance(lifecycleListener)) {
                this.lifecycleListeners.remove(lifecycleListener);
            }
        }

    }

    public void unregisterLifecycleListener(LifecycleListener eventListener) {
        this.lifecycleListeners.remove(eventListener);
    }

    public void disconnect() throws IOException {
        this.terminateKeepAliveThread();
        this.terminateConnect();
    }

    private void terminateKeepAliveThread() {
        try {
            this.keepAliveThreadExecutorLock.lock();
            ExecutorService keepAliveThreadExecutor = this.keepAliveThreadExecutor;
            if (keepAliveThreadExecutor != null) {
                keepAliveThreadExecutor.shutdownNow();

                while(!awaitTerminationInterruptibly(keepAliveThreadExecutor, Long.MAX_VALUE, TimeUnit.NANOSECONDS)) {
                }

                return;
            }
        } finally {
            this.keepAliveThreadExecutorLock.unlock();
        }

    }

    private static boolean awaitTerminationInterruptibly(ExecutorService executorService, long timeout, TimeUnit unit) {
        try {
            return executorService.awaitTermination(timeout, unit);
        } catch (InterruptedException var5) {
            return false;
        }
    }

    private void terminateConnect() throws IOException {
        do {
            this.disconnectChannel();
        } while(!tryLockInterruptibly(this.connectLock, 1000L, TimeUnit.MILLISECONDS));

        this.connectLock.unlock();
    }

    private static boolean tryLockInterruptibly(Lock lock, long time, TimeUnit unit) {
        try {
            return lock.tryLock(time, unit);
        } catch (InterruptedException var5) {
            return false;
        }
    }

    private void disconnectChannel() throws IOException {
        this.connected = false;
        if (this.channel != null && this.channel.isOpen()) {
            this.channel.close();
        }

    }

    public abstract static class AbstractLifecycleListener implements LifecycleListener {
        public AbstractLifecycleListener() {
        }

        public void onConnect(BinaryLogClient client) {
        }

        public void onCommunicationFailure(BinaryLogClient client, Exception ex) {
        }

        public void onEventDeserializationFailure(BinaryLogClient client, Exception ex) {
        }

        public void onDisconnect(BinaryLogClient client) {
        }
    }

    public interface LifecycleListener {
        void onConnect(BinaryLogClient var1);

        void onCommunicationFailure(BinaryLogClient var1, Exception var2);

        void onEventDeserializationFailure(BinaryLogClient var1, Exception var2);

        void onDisconnect(BinaryLogClient var1);
    }

    public interface EventListener {
        void onEvent(Event var1);
    }
}

BaseSourceTask

关键日志

• 在日志中提供 database.server.id(flink cdc job slave server id) / database.history.instance.name / snapshot.mode / offset.storage等关键信息

[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    connector.class = io.debezium.connector.mysql.MySqlConnector
[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.user = cdc_user
[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    offset.storage = com.alibaba.ververica.cdc.debezium.internal.FlinkOffsetBackingStore
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.server.id = 6519
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.server.name = mysql_binlog_source
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    include.schema.changes = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.port = 3306
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    table.whitelist = xxxbackend.xxx_logevent
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    offset.flush.interval.ms = 9223372036854775807
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    key.converter.schemas.enable = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    tombstones.on.delete = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.serverTimezone = UTC
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.hostname = 192.168.10.23
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.password = ********
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    value.converter.schemas.enable = false
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    name = engine
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history.skip.unparseable.ddl = true
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.whitelist = xxxbackend
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history.instance.name = 2cbe4018-caa3-4a60-a2c5-709671981391
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    snapshot.mode = schema_only
[2024/10/17 20:06:02.476] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history = com.alibaba.ververica.cdc.debezium.internal.FlinkDatabaseHistory

Y 推荐资源

• 第三方文档

• Flink CDC

• https://nightlies.apache.org/flink

• https://nightlies.apache.org/flink/flink-docs-master/
• https://nightlies.apache.org/flink/flink-cdc-docs-master/ 【推荐】
• https://nightlies.apache.org/flink/flink-cdc-docs-master/zh/docs/connectors/flink-sources/mysql-cdc/ 【推荐】
• https://nightlies.apache.org/flink/flink-cdc-docs-release-3.1/

• com.alibaba.ververica:flink-connector-mysql-cdc

• https://github.com/apache/flink-cdc/tree/master/flink-cdc-connect/flink-cdc-pipeline-connectors

• Flink Cdc 官方关键源码

• flink-cdc 2.3.0

• https://github.com/apache/flink-cdc/blob/release-2.3.0/flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/MySqlValidator.java
• https://github.com/apache/flink-cdc/blob/release-2.3.0/flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/MySqlSource.java

• flink-cdc 1.3.0

• https://github.com/apache/flink-cdc/blob/release-1.3.0/flink-connector-mysql-cdc/pom.xml
• https://github.com/apache/flink-cdc/blob/release-1.3.0/flink-connector-mysql-cdc/src/main/java/com/alibaba/ververica/cdc/connectors/mysql/MySQLSource.java
• 本版本中没有 MySqlValidator 类
• https://github.com/apache/flink-cdc/blob/release-1.3.0/flink-connector-mysql-cdc/src/main/java/com/alibaba/ververica/cdc/connectors/mysql/table/MySQLTableSourceFactory.java

存在 SERVER_TIME_ZONE 配置项(server-time-zone) / SERVER_ID 配置项(server-id) / ...

• debezium

• https://github.com/debezium/debezium
• https://debezium.io/documentation/reference/1.9/connectors/mysql.html#mysql-connector-properties

• com.github.shyiko:mysql-binlog-connector-java

• https://github.com/shyiko/mysql-binlog-connector-java

X 参考文献

• Flink CDC详解 - CSDN
• 聊聊BinaryLogClient的connect - 腾讯云
• 【FLINKCDC】运行时经常server-id冲突报错？一招实现自动化！ - CSDN