[数据集成/数据同步] 基于数据库增量日志的数据同步方案 : Debezium/Flink CDC/DataX/Canal/Oracle Goldengate/Kettle/Sqoop
1 概述
简述：CDC/增量数据同步
CDC的全称是Change Data Capture(增量数据捕获)
- 在广义的概念上，只要能捕获数据变更的技术，我们都可以称为
CDC。- 我们目前通常描述的
CDC技术主要面向数据库的变更，是一种用于捕获数据库中数据变更的技术。
- CDC 的技术实现方案
- 基于查询的 CDC：
- 优点 : 实现简单，是通过批处理实现的
- 缺点 : 需要依赖离线调度，不能保证数据强一致性和实时性；
- 基于日志的 CDC：
- 缺点 : 实现比较复杂
- 优点 : 但是可以实时消费日志，流式处理，可保证数据一致性和实时性；
CDC 的优势
- 如今，大多数公司仍然使用批处理在系统之间同步数据。使用批处理：
- 数据未立即同步
- 更多分配的资源用于同步数据库
- 数据复制仅在指定的批次期间发生
- 然而，变更数据捕获具有一些优势：
- 不断跟踪源数据库的变化
- 即时更新目标数据库
- 使用流处理来保证即时更改
有了CDC， 不同的数据库就会持续同步 ，批量任务已经成为过去。此外，由于 CDC 仅传输增量更改————因此，降低了传输数据的成本。
方案对比
目前市面上的CDC技术比较多，我们选取了几种主要的开源CDC方案做了对比，总体如下图：
从CDC机制、增量同步、断电续传、全量同步、全量+增量、架构、数据计算、生态这八个方面做了对比。可以看出其中的佼佼者主要是Flink CDC和Oracle OGG以及Debezium；
由于基于查询的CDC方案缺陷明显，这里不作讨论，下面我们对基于日志的CDC方案的优劣来做详细的介绍。
Flink CDC
Flink CDC：Flink CDC是最近几年的新贵，Flink CDC 底层封装了Debezium，功能比较全面，目前已经迭代到了2.4版本，社区活跃度在几个方案中是最高的；
- 优点：
- 全、增量一体的分布式数据集成框架；
- 同步时无需加锁；
- 吞吐量大，适合海量数据实时同步；
- 操作简单，SQL即可完成；
- 具有强大的 transformation 能力，通过 Flink SQL 即可完成ETL 中的数据转换；
- 有丰富的 Connector，除关系型数据库外，HBase、ClickHouse、TiDB等也支持，而且支持自定义 connector；
- 缺点：依赖Flink集群，数据量较大时对服务器要求较高；
Oracle OGG
- Oracle OGG：Oracle OGG 历史比较悠久，最初是设计用来从Oracle迁移数据到其它数据库，或者从其它平台迁移数据到Oracle，随着发展，目前已支持 Mysql、Hadoop、Hive、Kafka 等数据源；
- 优点：
- 支持增量和全量同步
- 支持分布式
- 高性能
- 支持数据过滤和转化，是目前主流的实时同步方案之一；
- 缺点：支持的数据库比较少，像一些MongoDB、TiDB等不支持；
Debezium
Debezium：
Debezium最初设计成一个Kafka Connect的Source Plugin。- 目前开发者虽致力于将其与
Kafka Connect解耦，但当前的代码实现还未变动。
下图引自
Debeizum官方文档，可以看到一个Debezium在一个完整CDC系统中的位置。
- 优点：
- 支持全量+增量同步；
- 缺点：
- 全量同步时会加锁，而且加锁时间不确定，会严重影响业务；
- 最重要的是跟Kafka等消息中间件强耦合，下游数据要经过Kafka；
Canal
-
Canal：主要用途是基于 MySQL 数据库增量日志解析，提供增量数据订阅和消费。 -
优点：用于单一的MySQL环境做数据同步还不错；
-
缺点：
- 缺点较为明显，只支持MySQL的CDC，只支持增量同步，全量需要用DataX或者Sqoop，全量和增量同步割裂；
- 不支持分布式；
Debezium 平台
什么是 Debezium ？
- 官网
- https://debezium.io
- https://github.com/debezium
- https://github.com/debezium/debezium
- https://github.com/debezium/debezium-ui
- https://github.com/debezium/debezium-examples
- 官方的口号与定位
从数据库流式传输更改。
Debezium 是一个用于变更数据捕获的开源分布式平台。
启动它，将它指向你的数据库，你的应用程序就可以开始响应其他应用程序提交给你的数据库的所有插入、更新和删除操作。
Debezium 耐用且快速，因此您的应用程序可以快速响应，即使出现问题也不会错过事件。
Debezium是一个构建在Apache Kafka之上的CDC开源平台。
它的主要用途是在事务日志中，记录提交给每个源数据库表的所有行级更改 。
侦听这些事件的每个应用程序都可以根据增量数据更改执行所需的操作。
Debezium提供了一个连接器库，支持多种数据库
例如 MySQL、MongoDB、PostgreSQL 等。
-
这些连接器可以监视和记录数据库更改、并将其发布到
Kafka等流服务。 -
此外， 即使我们的应用程序出现故障，Debezium 也会进行监控 。
重新启动后，它将开始消耗上次停止的事件，因此不会丢失任何内容。
Debezium架构
-
部署 Debezium 取决于我们拥有的基础设施，但更常见的是，我们经常使用
Apache Kafka Connect。 -
Kafka Connect是一个框架，与Kafka代理一起作为单独的服务运行。我们用它在Apache Kafka和其他系统之间传输数据。 -
我们还可以定义连接器来将数据传入和传出 Kafka。
下图显示了基于 Debezium 的变更数据捕获管道的不同部分：
- 首先，在左侧，我们有一个 MySQL 源数据库，我们希望将其数据复制并在目标数据库（如 PostgreSQL 或任何分析数据库）中使用。
- 其次， Kafka Connect 连接器解析并解释事务日志并将其写入 Kafka 主题。
- 接下来，Kafka 充当消息代理，将变更集可靠地传输到目标系统。
- 然后，在右侧，我们有 Kafka 连接器轮询 Kafka 并将更改推送到目标数据库。
- **Debezium 在其架构中使用 Kafka **，但它还提供其他部署方法来满足我们的基础设施需求。
我们可以将其用作 Debezium 服务器的独立服务器，也可以将其作为库嵌入到我们的应用程序代码中。
我们将在以下部分中看到这些方法。
Debezium 服务器
Debezium提供了一个独立的服务器 来捕获源数据库的更改。它配置为使用Debezium源连接器之一。
此外，这些连接器将更改事件发送到各种消息基础设施，例如
Amazon Kinesis或Google Cloud Pub/Sub。
嵌入式 Debezium
Kafka Connect在用于部署Debezium时提供容错能力和可扩展性。
然而，有时我们的应用程序不需要这种级别的可靠性，并且我们希望最大限度地降低基础设施的成本。
值得庆幸的是， 我们可以通过将 Debezium 引擎嵌入到我们的应用程序中来做到这一点。
完成此操作后，我们必须配置连接器。
案例：基于 Debezium 的 Spring Boot CDC 应用程序
需求及架构
- 为了使我们的应用程序保持简单，我们将创建一个用于客户管理的 Spring Boot 应用程序。
customer表模型有 ID 、 全名 和 电子邮件 字段。
对于数据访问层，使用
Spring Data JPA。
最重要的是，应用程序将运行 Debezium 的嵌入式版本。
想象一下这个应用程序的架构：
首先，Debezium 引擎将跟踪源 MySQL 数据库（来自另一个系统或应用程序）上的
customer表的事务日志。
其次，每当我们对
customer表执行插入/更新/删除等数据库操作时，Debezium 连接器都会调用一个服务方法。
最后，根据这些事件，该方法会将
customer表的数据同步到目标 MySQL 数据库（我们应用程序的主数据库）。
Maven 依赖项
- 让我们首先将 所需的依赖项 添加到 pom.xml 中：
- 版本信息
lombok.version : 1.8.22
spring-boot.version : 2.3.12.RELEASE
debezium.version = 1.5.2.Final
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>${lombok.version}</version>
<optional>true</optional>
</dependency>
<!-- spring boot & jdbc [start] -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<version>${spring-boot.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
<version>${spring-boot.version}</version>
</dependency>
<!-- log [start] -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>${log4j.version}</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j.version}</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>${log4j.version}</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-jul</artifactId>
<!--<version>2.13.3</version>-->
<version>${log4j.version}</version>
<scope>compile</scope>
</dependency>
<!-- log [end] -->
<!-- 无需再引入 mysql:mysql-connector-java 包，因已 io.debezium:debezium-connector-mysql 内部已引入 -->
<!--
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>${mysql.jdbc.version<=8.0.25}</version>
</dependency>
-->
<!-- 无需再引入 org.apache.kafka:connect-api 包，因已 io.debezium:debezium-embedded 内部已引入 -->
<!--
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>connect-api</artifactId>
<version>2.7.0</version>
<scope>compile</scope>
</dependency>
-->
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-api</artifactId>
<version>1.5.2.Final</version>
</dependency>
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-embedded</artifactId>
<version>1.5.2.Final</version>
</dependency>
-
同样，我们为应用程序将使用的每个
Debezium连接器添加依赖项。 -
在我们的例子中，我们将使用 MySQL 连接器：
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-mysql</artifactId>
<version>1.5.2.Final</version>
</dependency>
安装数据库
- 我们可以手动安装和配置我们的数据库。但是，为了加快速度，我们将使用
docker-compose文件：
version: "3.9"
services:
# Install Source MySQL DB and setup the Customer database
mysql-1:
container_name: source-database
image: mysql
ports:
- 3305:3306
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_USER: user
MYSQL_PASSWORD: password
MYSQL_DATABASE: customerdb
# Install Target MySQL DB and setup the Customer database
mysql-2:
container_name: target-database
image: mysql
ports:
- 3306:3306
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_USER: user
MYSQL_PASSWORD: password
MYSQL_DATABASE: customerdb
-
该文件将在不同端口上运行两个数据库实例。
-
我们可以使用命令
docker-compose up -d运行此文件。
建表 : customer
- 现在，让我们通过运行 SQL 脚本来创建
customer表：
use test;
CREATE TABLE test.customer
(
id integer NOT NULL,
fullname character varying(255),
email character varying(255),
CONSTRAINT customer_pkey PRIMARY KEY (id)
);
DebeziumConfiguration
import io.debezium.data.Envelope;
import io.debezium.embedded.Connect;
import io.debezium.engine.DebeziumEngine;
import io.debezium.engine.RecordChangeEvent;
import io.debezium.engine.format.ChangeEventFormat;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.tuple.Pair;
import org.apache.kafka.connect.data.Field;
import org.apache.kafka.connect.data.Struct;
import org.apache.kafka.connect.source.SourceRecord;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.util.List;
import java.util.Map;
import static io.debezium.data.Envelope.FieldName.*;
import static java.util.stream.Collectors.toMap;
@Configuration
@Slf4j
public class DebeziumConfiguration {
// 为了配置 Debezium MySQL 连接器，我们将创建一个 Debezium 配置 bean
@Bean
public io.debezium.config.Configuration customerMysqlTableConnector() {
String dbHost = "127.0.0.1";
String dbPort = "3306";
String dbUsername = "root";
String dbPassword = "helloPaasword";
String dbName = "test";
io.debezium.config.Configuration configuration = io.debezium.config.Configuration.create()
.with("name", "customer-mysql-connector")
.with("connector.class", "io.debezium.connector.mysql.MySqlConnector")
.with("offset.storage", "org.apache.kafka.connect.storage.FileOffsetBackingStore")
.with("offset.storage.file.filename", "/tmp/offsets.dat")
.with("offset.flush.interval.ms", "60000")
.with("database.hostname", dbHost)
.with("database.port", dbPort)
.with("database.user", dbUsername)
.with("database.password", dbPassword)
.with("database.dbname", dbName)
.with("database.include.list", dbName)
.with("include.schema.changes", "false")
.with("database.server.id", "10181")
.with("database.server.name", "customer-mysql-db-server")
.with("database.history", "io.debezium.relational.history.FileDatabaseHistory")
.with("database.history.file.filename", "/tmp/dbhistory.dat")
.build();
return configuration;
}
/**
* Debezium server bootstrap debezium server bootstrap.
* @param configuration the configuration
* @return the debezium server bootstrap
*/
@Bean
DebeziumServerBootstrap debeziumServerBootstrap(io.debezium.config.Configuration configuration) {
DebeziumServerBootstrap debeziumServerBootstrap = new DebeziumServerBootstrap();
DebeziumEngine<RecordChangeEvent<SourceRecord>> debeziumEngine = DebeziumEngine.create(ChangeEventFormat.of(Connect.class))
.using(configuration.asProperties())
.notifying(this::handleRecordChangeEvent)
.build();
debeziumServerBootstrap.setDebeziumEngine(debeziumEngine);
return debeziumServerBootstrap;
}
private void handleRecordChangeEvent(List<RecordChangeEvent<SourceRecord>> recordChangeEvents, DebeziumEngine.RecordCommitter<RecordChangeEvent<SourceRecord>> recordCommitter) {
recordChangeEvents.forEach(r -> {
SourceRecord sourceRecord = r.record();
String topic = sourceRecord.topic();
Struct sourceRecordChangeValue = (Struct) sourceRecord.value();
if (sourceRecordChangeValue != null) {
// 判断操作的类型 过滤掉读 只处理增删改 这个其实可以在配置中设置
Envelope.Operation operation = Envelope.Operation.forCode((String) sourceRecordChangeValue.get(OPERATION));
if (operation != Envelope.Operation.READ) {//READ / CREATE / UPDATE / DELETE / TRUNCATE
String latestRecord = (operation == Envelope.Operation.DELETE ) ? BEFORE : AFTER;
// 获取增删改对应的结构体数据
Struct struct = (Struct) sourceRecordChangeValue.get(latestRecord);
// 将变更的行封装为 Map
Map<String, Object> payload = struct.schema().fields().stream()
.map(Field::name)
.filter(fieldName -> struct.get(fieldName) != null)
.map(fieldName -> Pair.of(fieldName, struct.get(fieldName)))
.collect(toMap(Pair::getKey, Pair::getValue));
//TODO 做一些业务操作
//cdcService.handleCdcRecord(payload, operation);
String tableName = null;
if(topic.split("\\.").length > 2){
tableName = topic.split("\\.")[2];
}
log.info("cdc debezium | operation: {}, table : {}, data : {}", operation, tableName, payload);
}
}
});
}
}
- 在本节中，我们配置了 Debezium MySQL 连接器
让我们更详细地检查此配置。
该 bean 中的 create 方法 使用构建器来创建 Properties 对象 。
无论首选连接器如何，此构建器都会设置引擎所需的多个属性。为了跟踪源 MySQL 数据库，我们使用 MySqlConnector 类。
当此连接器运行时，它开始跟踪源中的更改并记录“偏移量”以确定 它从事务日志中处理了多少数据 。
有多种方法可以保存这些偏移量，但在本例中，我们将使用类 FileOffsetBackingStore 在本地文件系统上存储偏移量。
org.apache.kafka.connect.storage.FileOffsetBackingStore
连接器的最后几个参数是 MySQL 数据库属性。
现在我们已经有了配置，我们可以创建我们的引擎了。
- 创建
DebeziumEngine，以充当我们的 Debezium MySQL 连接器的包装器。
更重要的是，引擎将为每个数据变更调用一个方法 - 在我们的示例中为
handleChangeEvent。
在此方法中，首先， 我们将根据 调用create()时指定的格式解析每个事件。
DebeziumServerBootstrap
现在我们已经配置了
DebeziumEngine对象，让我们使用服务执行器线程池异步启动它。
DebeziumEngine充当我们的 Debezim MySQL 连接器的包装器
import io.debezium.engine.DebeziumEngine;
import lombok.Data;
import lombok.SneakyThrows;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.context.SmartLifecycle;
import org.springframework.util.Assert;
import java.util.concurrent.Executor;
import java.util.concurrent.Executors;
@Data
public class DebeziumServerBootstrap implements InitializingBean, SmartLifecycle {
//Java内置线程池（Executor 线程池）
private final Executor executor = Executors.newSingleThreadExecutor();
// interface io.debezium.engine.DebeziumEngine<R> extends Runnable, Closeable
private DebeziumEngine<?> debeziumEngine;
//@PostConstruct
@Override
public void start() {
executor.execute(debeziumEngine);
}
//@PreDestroy
@SneakyThrows
@Override
public void stop() {
if(debeziumEngine != null){
debeziumEngine.close();
}
}
@Override
public boolean isRunning() {
return false;
}
@Override
public void afterPropertiesSet() throws Exception {
Assert.notNull(debeziumEngine, "debeziumEngine must not be null");
}
}
在本节中，我们配置、并了解如何运行嵌入式 Debezium 引擎。
运行
要查看我们的代码的实际效果，让我们对源数据库的 customer 表进行一些数据更改。
Step0 启动过程日志
启动过程的日志
... xxx.jar org.example.MySpringBootApplication
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:21.458] [INFO ] [main] [MySpringBootApplication] main:38__||__ server startup - start
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.3.12.RELEASE)
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:21.844] [INFO ] [main] [StartupInfoLogger] logStarting:55__||__ Starting MySpringBootApplication on C20432 with PID 25956 (E:\source_code\Demo-Projects\local-demos-parent\local-cdc\target\classes started by xxxxxx in E:\source_code\Demo-Projects\local-demos-parent)
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:21.858] [INFO ] [main] [SpringApplication] logStartupProfileInfo:648__||__ No active profile set, falling back to default profiles: default
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:23.606] [INFO ] [main] [DirectJDKLog] log:173__||__ Initializing ProtocolHandler ["http-nio-8080"]
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:23.611] [INFO ] [main] [DirectJDKLog] log:173__||__ Starting service [Tomcat]
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:23.612] [INFO ] [main] [DirectJDKLog] log:173__||__ Starting Servlet engine: [Apache Tomcat/9.0.46]
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:23.971] [INFO ] [main] [DirectJDKLog] log:173__||__ Initializing Spring embedded WebApplicationContext
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:24.078] [INFO ] [main] [AbstractConfig] logAll:361__||__ JsonConverterConfig values:
converter.type = key
decimal.format = BASE64
schemas.cache.size = 1000
schemas.enable = true
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:24.080] [INFO ] [main] [AbstractConfig] logAll:361__||__JsonConverterConfig values:
converter.type = value
decimal.format = BASE64
schemas.cache.size = 1000
schemas.enable = false
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:24.088] [INFO ] [main] [AbstractConfig] logAll:361__||__EmbeddedConfig values:
access.control.allow.methods =
access.control.allow.origin =
admin.listeners = null
bootstrap.servers = [localhost:9092]
client.dns.lookup = use_all_dns_ips
config.providers = []
connector.client.config.override.policy = None
header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter
internal.key.converter = class org.apache.kafka.connect.json.JsonConverter
internal.value.converter = class org.apache.kafka.connect.json.JsonConverter
key.converter = class org.apache.kafka.connect.json.JsonConverter
listeners = null
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
offset.flush.interval.ms = 60000
offset.flush.timeout.ms = 5000
offset.storage.file.filename = /tmp/offsets.dat
offset.storage.partitions = null
offset.storage.replication.factor = null
offset.storage.topic =
plugin.path = null
response.http.headers.config =
rest.advertised.host.name = null
rest.advertised.listener = null
rest.advertised.port = null
rest.extension.classes = []
rest.host.name = null
rest.port = 8083
ssl.cipher.suites = null
ssl.client.auth = none
ssl.enabled.protocols = [TLSv1.2]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.2
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
task.shutdown.graceful.timeout.ms = 5000
topic.creation.enable = true
topic.tracking.allow.reset = true
topic.tracking.enable = true
value.converter = class org.apache.kafka.connect.json.JsonConverter
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:24.092] [INFO ] [main] [WorkerConfig] logDeprecatedProperty:389__||__Worker configuration property 'internal.key.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration.
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:24.092] [INFO ] [main] [WorkerConfig] logDeprecatedProperty:389__||__Worker configuration property 'internal.value.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration.
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:24.093] [WARN ] [main] [WorkerConfig] logPluginPathConfigProviderWarning:420__||__Variables cannot be used in the 'plugin.path' property, since the property is used by plugin scanning before the config providers that replace the variables are initialized. The raw value 'null' was used for plugin scanning, as opposed to the transformed value 'null', and this may cause unexpected results.
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:24.379] [INFO ] [main] [DirectJDKLog] log:173__||__Starting ProtocolHandler ["http-nio-8080"]
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.442] [INFO ] [pool-2-thread-1] [FileOffsetBackingStore] start:58__||__Starting FileOffsetBackingStore with file \tmp\offsets.dat
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:24.444] [INFO ] [main] [StartupInfoLogger] logStarted:61__||__Started MySpringBootApplication in 2.899 seconds (JVM running for 4.024)
[mainraceId] [local-cdc] [system] [2024/10/21 12:36:24.446] [INFO ] [main] [MySpringBootApplication] main:45__||__ server startup - end
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.489] [INFO ] [pool-2-thread-1] [BaseSourceTask] start:124__||__Starting MySqlConnectorTask with configuration:
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.490] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ connector.class = io.debezium.connector.mysql.MySqlConnector
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.490] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ database.history.file.filename = /tmp/dbhistory.dat
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.490] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ database.user = root
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.490] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ database.dbname = test
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.491] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ database.server.id = 10181
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.491] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ offset.storage = org.apache.kafka.connect.storage.FileOffsetBackingStore
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.491] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ database.server.name = customer-mysql-db-server
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.491] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ include.schema.changes = false
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.491] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ database.port = 3306
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.491] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ offset.flush.interval.ms = 60000
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.491] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ offset.storage.file.filename = /tmp/offsets.dat
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.491] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ database.hostname = 127.0.0.1
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.491] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ database.password = ********
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.492] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ name = customer-mysql-connector
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.492] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ database.include.list = test
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:24.492] [INFO ] [pool-2-thread-1] [BaseSourceTask] lambda$start$0:126__||__ database.history = io.debezium.relational.history.FileDatabaseHistory
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:25.435] [INFO ] [pool-2-thread-1] [MySqlConnectorTask] start:89__||__No previous offset found
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:25.625] [INFO ] [pool-2-thread-1] [MySqlConnectorTask] validateAndLoadDatabaseHistory:288__||__Connector started for the first time, database history recovery will not be executed
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:25.659] [INFO ] [pool-2-thread-1] [Threads] threadFactory:270__||__Requested thread factory for connector MySqlConnector, id = customer-mysql-db-server named = change-event-source-coordinator
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:36:25.660] [INFO ] [pool-2-thread-1] [Threads$3] newThread:287__||__Creating thread debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:25.664] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [ChangeEventSourceCoordinator] lambda$start$0:96__||__Metrics registered
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:25.665] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [ChangeEventSourceCoordinator] lambda$start$0:99__||__Context created
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:25.671] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlSnapshotChangeEventSource] getSnapshottingTask:95__||__No previous offset has been found
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:25.671] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlSnapshotChangeEventSource] getSnapshottingTask:97__||__According to the connector configuration both schema and data will be snapshotted
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:25.671] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] doExecute:96__||__Snapshot step 1 - Preparing
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:25.762] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] doExecute:105__||__Snapshot step 2 - Determining captured tables
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:25.762] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlSnapshotChangeEventSource] getAllTableIds:124__||__Read list of available databases
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:25.841] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlSnapshotChangeEventSource] getAllTableIds:131__||__ list of available databases is: [information_schema, ... , mysql, nacos, performance_schema, seatunnel, spiderflow, ..., superset, sys, test, xxl_job]
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:25.842] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlSnapshotChangeEventSource] getAllTableIds:139__||__Read list of available tables in each database
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:30.256] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlSnapshotChangeEventSource] getAllTableIds:160__||__ snapshot continuing with database(s): [test]
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:30.262] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] doExecute:112__||__Snapshot step 3 - Locking captured tables [test.account, test.ads_deviced_sale, test.device_event, test.customer, ...]
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:30.502] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlSnapshotChangeEventSource] globalLock:508__||__Flush and obtain global read lock to prevent writes to database
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:30.720] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] doExecute:118__||__Snapshot step 4 - Determining snapshot offset
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:30.722] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlSnapshotChangeEventSource] determineSnapshotOffset:274__||__Read binlog position of MySQL primary server
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:30.793] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlSnapshotChangeEventSource] lambda$determineSnapshotOffset$4:285__||__ using binlog 'mysql-bin.187895' at position '27538' and gtid 'f0a5a37b-4a03-11ed-89fd-fa163e42cb44:1-37070805'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:30.793] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] doExecute:121__||__Snapshot step 5 - Reading structure of captured tables
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:30.794] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlSnapshotChangeEventSource] readTableStructure:323__||__All eligible tables schema should be captured, capturing: [test.account, test.ads_device_sale, test.device_event, test.customer, ...]
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:31.541] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlSnapshotChangeEventSource] readTableStructure:345__||__Reading structure of database 'test'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.232] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] doExecute:125__||__Snapshot step 6 - Persisting schema history
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.259] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.account.Value' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.account.Value'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.259] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.account.Key' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.account.Key'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.262] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.account.Envelope' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.account.Envelope'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.266] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.ads_device_sale.Value' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.ads_device_sale.Value'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.266] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.ads_device_sale.Key' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.ads_device_sale.Key'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.266] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.ads_device_sale.Envelope' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.ads_device_sale.Envelope'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.268] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.device_event.Value' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.device_event.Value'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.268] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.device_event.Key' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.device_event.Key'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.268] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.device_event.Envelope' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.device_event.Envelope'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.269] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.customer.Value' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.customer.Value'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.269] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.customer.Key' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.customer.Key'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.270] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.customer.Envelope' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.customer.Envelope'
...
...
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.304] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.device_sale.Value' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.device_sale.Value'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.304] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.device_sale.Key' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.device_sale.Key'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.305] [WARN ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [SchemaNameAdjuster] lambda$create$1:172__||__The Kafka Connect schema name 'customer-mysql-db-server.test.device_sale.Envelope' is not a valid Avro schema name, so replacing with 'customer_mysql_db_server.test.device_sale.Envelope'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.307] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlSnapshotChangeEventSource] globalUnlock:514__||__Releasing global read lock to enable MySQL writes
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.438] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlSnapshotChangeEventSource] globalUnlock:518__||__Writes to MySQL tables prevented for a total of 00:00:02.786
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.438] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] doExecute:137__||__Snapshot step 7 - Snapshotting data
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.439] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] createDataEvents:294__||__Snapshotting contents of 24 tables while still in transaction
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.440] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] createDataEventsForTable:328__||__ Exporting data from table 'test.account' (1 of 24 tables)
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.440] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] createDataEventsForTable:336__||__ For table 'test.account' using select statement: 'SELECT * FROM `test`.`account`'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.846] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] createDataEventsForTable:384__||__ Finished exporting 2 records for table 'test.account'; total duration '00:00:00.406'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.910] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] createDataEventsForTable:328__||__ Exporting data from table 'test.ads_device_sale' (2 of 24 tables)
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:33.911] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] createDataEventsForTable:336__||__ For table 'test.ads_device_sale' using select statement: 'SELECT * FROM `test`.`ads_device_sale`'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:34.306] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] createDataEventsForTable:384__||__ Finished exporting 0 records for table 'test.ads_device_sale'; total duration '00:00:00.396'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:34.371] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] createDataEventsForTable:328__||__ Exporting data from table 'test.device_event' (3 of 24 tables)
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:34.371] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] createDataEventsForTable:336__||__ For table 'test.device_event' using select statement: 'SELECT * FROM `test`.`device_event`'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:34.766] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] createDataEventsForTable:384__||__ Finished exporting 0 records for table 'test.device_event'; total duration '00:00:00.395'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:34.882] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] createDataEventsForTable:328__||__ Exporting data from table 'test.customer' (4 of 24 tables)
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:34.883] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] createDataEventsForTable:336__||__ For table 'test.customer' using select statement: 'SELECT * FROM `test`.`customer`'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:35.268] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] createDataEventsForTable:384__||__ Finished exporting 0 records for table 'test.customer'; total duration '00:00:00.386'
...
...
...
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:44.680] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [RelationalSnapshotChangeEventSource] createDataEventsForTable:384__||__ Finished exporting 457 records for table 'test.device_sale'; total duration '00:00:00.461'
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:44.811] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [AbstractSnapshotChangeEventSource] execute:85__||__Snapshot - Final stage
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:44.811] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [ChangeEventSourceCoordinator] lambda$start$0:111__||__Snapshot ended with SnapshotResult [status=COMPLETED, offset=MySqlOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.mysql.Source:STRUCT}, sourceInfo=SourceInfo [currentGtid=null, currentBinlogFilename=mysql-bin.187895, currentBinlogPosition=27538, currentRowNumber=0, serverId=0, sourceTime=2024-10-21T04:36:44.679Z, threadId=-1, currentQuery=null, tableIds=[test.device_sale], databaseName=test], partition={server=customer-mysql-db-server}, snapshotCompleted=true, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], restartGtidSet=f0a5a37b-4a03-11ed-89fd-fa163e42cb44:1-37070805, currentGtidSet=f0a5a37b-4a03-11ed-89fd-fa163e42cb44:1-37070805, restartBinlogFilename=mysql-bin.187895, restartBinlogPosition=27538, restartRowsToSkip=0, restartEventsToSkip=0, currentEventLengthInBytes=0, inTransaction=false, transactionId=null]]
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:44.815] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [Threads] threadFactory:270__||__Requested thread factory for connector MySqlConnector, id = customer-mysql-db-server named = binlog-client
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:44.817] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [ChangeEventSourceCoordinator] streamEvents:151__||__Starting streaming
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:45.028] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlStreamingChangeEventSource] execute:858__||__GTID set purged on server: f0a5a37b-4a03-11ed-89fd-fa163e42cb44:1-36939623
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:45.029] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlStreamingChangeEventSource] filterGtidSet:1070__||__Attempting to generate a filtered GTID set
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:45.029] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlStreamingChangeEventSource] filterGtidSet:1071__||__GTID set from previous recorded offset: f0a5a37b-4a03-11ed-89fd-fa163e42cb44:1-37070805
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:45.029] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlStreamingChangeEventSource] filterGtidSet:1078__||__GTID set available on server: f0a5a37b-4a03-11ed-89fd-fa163e42cb44:1-37070814
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:45.029] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlStreamingChangeEventSource] filterGtidSet:1084__||__Using first available positions for new GTID channels
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:45.029] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlStreamingChangeEventSource] filterGtidSet:1086__||__Relevant GTID set available on server: f0a5a37b-4a03-11ed-89fd-fa163e42cb44:1-37070814
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:45.031] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlStreamingChangeEventSource] filterGtidSet:1097__||__Final merged GTID set to use when connecting to MySQL: f0a5a37b-4a03-11ed-89fd-fa163e42cb44:1-37070805
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:45.031] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlStreamingChangeEventSource] execute:863__||__Registering binlog reader with GTID set: f0a5a37b-4a03-11ed-89fd-fa163e42cb44:1-37070805
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:45.031] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlStreamingChangeEventSource] execute:884__||__Skip 0 events on streaming start
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:45.031] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlStreamingChangeEventSource] execute:888__||__Skip 0 rows on streaming start
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:45.032] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [Threads$3] newThread:287__||__Creating thread debezium-mysqlconnector-customer-mysql-db-server-binlog-client
[blc-127.0.0.1:3306raceId] [local-cdc] [system] [2024/10/21 12:36:45.100] [INFO ] [blc-127.0.0.1:3306] [Threads$3] newThread:287__||__ Creating thread debezium-mysqlconnector-customer-mysql-db-server-binlog-client
[blc-127.0.0.1:3306raceId] [local-cdc] [system] [2024/10/21 12:36:45.510] [INFO ] [blc-127.0.0.1:3306] [BinaryLogClient] connect:590__||__ Connected to 127.0.0.1:3306 at f0a5a37b-4a03-11ed-89fd-fa163e42cb44:1-37070805 (sid:10181, cid:72824723)
[blc-127.0.0.1:3306raceId] [local-cdc] [system] [2024/10/21 12:36:45.510] [INFO ] [blc-127.0.0.1:3306] [MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener] onConnect:1167__||__ Connected to MySQL binlog at 127.0.0.1:3306, starting at MySqlOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.mysql.Source:STRUCT}, sourceInfo=SourceInfo [currentGtid=null, currentBinlogFilename=mysql-bin.187895, currentBinlogPosition=27538, currentRowNumber=0, serverId=0, sourceTime=2024-10-21T04:36:44.679Z, threadId=-1, currentQuery=null, tableIds=[test.device_sale], databaseName=test], partition={server=customer-mysql-db-server}, snapshotCompleted=true, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], restartGtidSet=f0a5a37b-4a03-11ed-89fd-fa163e42cb44:1-37070805, currentGtidSet=f0a5a37b-4a03-11ed-89fd-fa163e42cb44:1-37070805, restartBinlogFilename=mysql-bin.187895, restartBinlogPosition=27538, restartRowsToSkip=0, restartEventsToSkip=0, currentEventLengthInBytes=0, inTransaction=false, transactionId=null]
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:45.510] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlStreamingChangeEventSource] execute:905__||__ Waiting for keepalive thread to start
[blc-127.0.0.1:3306raceId] [local-cdc] [system] [2024/10/21 12:36:45.511] [INFO ] [blc-127.0.0.1:3306] [Threads$3] newThread:287__||__ Creating thread debezium-mysqlconnector-customer-mysql-db-server-binlog-client
[debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinatorraceId] [local-cdc] [system] [2024/10/21 12:36:45.612] [INFO ] [debezium-mysqlconnector-customer-mysql-db-server-change-event-source-coordinator] [MySqlStreamingChangeEventSource] execute:912__||__ Keepalive thread is running
Step1 插入记录
- 要将新记录添加到 customer 表中，我们将进入 MySQL shell 并运行：
INSERT INTO test.customer (id, fullname, email) VALUES (1, 'John Doe', '[email protected]')
- 运行此查询后，我们将看到应用程序的相应输出：
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:44:14.643] [INFO ] [pool-2-thread-1] [BaseSourceTask] logStatistics:182__||__ 502 records sent during previous 00:07:32.001, last recorded offset: {transaction_id=null, ts_sec=1729485855, file=mysql-bin.187897, pos=10852, gtids=f0a5a37b-4a03-11ed-89fd-fa163e42cb44:1-37071062, row=1, server_id=1852072115, event=2}
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:44:14.645] [INFO ] [pool-2-thread-1] [DebeziumConfiguration] lambda$handleRecordChangeEvent$2:99__||__ cdc debezium | operation: CREATE, table : customer, data : {fullname=John Doe, id=1, email=[email protected]}
- 最后，我们检查一条新记录是否已插入到我们的目标数据库中：
id fullname email
1 John Doe [email protected]
Step2 更新记录
- 现在，让我们尝试更新最后插入的客户并检查会发生什么：
UPDATE test.customer t SET t.email = 'xxxx@163.com' WHERE t.id = 1
- 日志：
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:45:23.643] [INFO ] [pool-2-thread-1] [DebeziumConfiguration] lambda$handleRecordChangeEvent$2:99__||__ cdc debezium | operation: UPDATE, table : customer, data : {fullname=John Doe, id=1, email=xxxx@163.com}
- 我们可以验证目标数据库中约翰的电子邮件已更改：
id fullname email
1 John Doe xxxx@163.com
Step3 删除记录
现在，我们可以通过执行以下命令删除 客户 表中的条目：
DELETE FROM test.customer WHERE id = 1
同样，这里我们操作发生变化，再次查询：
[pool-2-thread-1raceId] [local-cdc] [system] [2024/10/21 12:46:12.656] [INFO ] [pool-2-thread-1] [DebeziumConfiguration] lambda$handleRecordChangeEvent$2:99__||__ cdc debezium | operation: DELETE, table : customer, data : {fullname=John Doe, id=1, email=xxxx@163.com}
我们可以验证目标数据库上的数据已被删除：
select * from test.customer where id= 1
0 rows retrieved
Apache Flink CDC
详情参见：
Sqoop : 项目已废弃，不建议使用
- 推荐文献
Y 推荐文献
- [大数据] ETL之增量数据抽取(CDC) - 博客园/千千寰宇
- [数据库] 浅谈mysql的serverId/serverUuid - 博客园/千千寰宇
- [数据库] MYSQL之binlog概述 - 博客园/千千寰宇
- CDC问题 - 常见问题 - 实时计算Flink版 - Aliyun
X 参考文献
本文作者：
千千寰宇
本文链接： https://www.cnblogs.com/johnnyzen
关于博文：评论和私信会在第一时间回复，或直接私信我。
版权声明：本博客所有文章除特别声明外，均采用 BY-NC-SA 许可协议。转载请注明出处！
日常交流：大数据与软件开发-QQ交流群: 774386015 【入群二维码】参见左下角。您的支持、鼓励是博主技术写作的重要动力！
本文链接： https://www.cnblogs.com/johnnyzen
关于博文：评论和私信会在第一时间回复，或直接私信我。
版权声明：本博客所有文章除特别声明外，均采用 BY-NC-SA 许可协议。转载请注明出处！
日常交流：大数据与软件开发-QQ交流群: 774386015 【入群二维码】参见左下角。您的支持、鼓励是博主技术写作的重要动力！
