FlinkCDC使用MysqlGTids进行消费的示例

     一般来说读取Mysql的binlog的时候通过 startupOptions(StartupOptions.specificOffset("mysql-bin.000003", 4L) // 从指定 binlog 文件名和位置启动
就足够了,但是对于Mysql设置了主从库或者是说业务库发生故障,切从库的情况下,binlog的文件会丢失或者偏移量也会
不准,这个时候可以使用 Gtids 来定位到准确的binlog记录 .

 

 核心代码如下:

StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append("sharedb.realtime_extract_file_log,sharedb.zjq_date");

MySqlSource<String> mySqlSource = MySqlSource.<String>builder()
                .hostname("***")
                .port(3306)
                .deserializer(new JsonDebeziumDeserializationSchema()) // 将 SourceRecord 转换为 JSON 字符串
                .username("***")
                .password("**")
                .databaseList("***")
                .tableList(stringBuilder.toString())
                .startupOptions(StartupOptions.specificOffset("0788b7a0-f3b0-11e8-9292-005056b767a4:1-277517678"))
                .build();

DataStreamSource<String> mySQLSource = env
                .fromSource(mySqlSource, WatermarkStrategy.forMonotonousTimestamps(), "MySQL Source");


mySQLSource.print();

  注意 传入的GtidS在SeviceID后面的位置  必须是从1开始的.

  指定的GITDs集合可以从Taskmanager的checkpoint的日志中找到,这个也可以用来恢复Flink任务

  

com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 1361: {transaction_id=null, ts_sec=1686756595, file=mysql-bin.053772, pos=161930075, kind=SPECIFIC, 
                                                                                                           gtids=0907ced3-1917-11e9-8677-246e968ac830:1-956753,0a9be625-5843-11e8-9974-246e968c0f68:1-1,
                                                                                                           19e5b801-0a57-11e8-8016-246e968c0fb0:1-1,3692f4ec-4f82-ee16-7c44-796795b50d20:1-1766321603,
                                                                                                           3dd8c0ce-a419-ee13-4997-751ca5c18e38:1-456055886,408c7ff0-dfad-11e7-b662-246e968c1020:1-651,
                                                                                                           4c68f945-baf5-11e7-b76e-1866dab54f8c:1-2,4e18a9f8-70ee-11ea-a959-246e968c1118:1-180271,
                                                                                                           502b3dc4-1330-11ea-8e7c-0425c5976523:1-691543,517d2c94-ce81-11ea-88b7-246e968c1190:1-5397745,
                                                                                                           5b189160-4730-11e9-91c5-f063f936964b:1-5,5d2dd893-823d-11eb-9ea0-005056a382a9:1-23988,
                                                                                                           5ee37d7b-c3b4-ee13-6573-84af8292075b:1-1180912480,62176fed-6e62-ee16-4e85-7510843f21b7:1-359748016,
                                                                                                           6a5fcf8d-1f0c-ee15-4aa8-6563c31f6fa1:1-382349864,6b743714-4e3a-11ea-a64d-246e96ac70e8:1-1,
                                                                                                           6c519e8e-d1f6-11ea-8833-005056a34a24:1-2030225103,782bd5dc-dba9-ee14-4b65-3d32ea0c45d5:1-1538867279,
                                                                                                           789cf463-30ab-11e7-83c8-38bc01327e92:1-239,7a7c4f29-da7e-ee14-5641-817e4e9115c4:1-1165070854,
                                                                                                           7e03aefb-c871-11e9-872f-246e96a594f8:1-2,85e93d9d-9f97-ee14-623b-04c28d09fcc2:1-7368175396,
                                                                                                           872563dd-48ce-ee16-7aaa-3c9a1eab7ccf:1-9232786,8d9ddacb-cf54-ee18-6fe2-50b3e5b13dae:1-3655542054,
                                                                                                           9a36e4f8-d018-ee13-5803-5da3344a1d4c:1-187424165,a5573ff3-dfad-11e7-b692-246e968c0f98:1-192,
                                                                                                           aa056233-65c6-ee15-5d5e-99d9734ac4da:1-244669593,b0a7a4d2-6751-ee16-5738-c84dc780c8fb:1-1199838223,
                                                                                                           b65721d1-450a-ee18-7bb6-3d1ecd41aba8:1-3096829422,bea4faec-584b-11e8-9bb9-246e968c0f40:1-1,
                                                                                                           c2818986-08c5-11ea-b9b2-246e968c0f40:1-1,c61339ba-2e7c-ee15-6a9d-801889d9eba8:1-605871333,
                                                                                                           c8b4b099-4573-11e8-aef2-246e968c1198:1-1180475,d2b9675d-65eb-11e7-bfca-005056abbad2:1-665250,
                                                                                                           d3a66e6f-4a51-11e9-8a32-f063f9369579:1-5,d491b9b8-857d-ee17-76f0-018c64e0a216:1-2365186572,
                                                                                                           d74132e0-7f65-ee14-4aac-b94333978d7e:1-2206856108,d755869e-7a82-11e8-af71-246e96ac79d8:1-28,
                                                                                                           d9d441da-607d-11eb-9097-9c1d36dcd9d2:1-60,
                                                                                                           dc02520c-0cb7-ee15-6e3d-cd4042ad0cb6:1-2145,e4076602-7606-ee15-7014-21345399bb47:1-1147883667,
                                                                                                           e7043704-3c48-11ec-9897-005056a3564b:1-559,f7608d01-e903-ee13-4781-b1a883ee7916:1-585807668,
                                                                                                           f87d4752-a7bc-ee17-68e7-1466631207f1:1-4696778030,fa295166-5e4d-11e9-b834-246e968c1118:1-61412463,
                                                                                                           fa42a257-8cca-11e8-b6c6-246e96ac70e8:1-51102,
                                                                                                           fe4453f6-bb2d-11e8-bfc9-005056b7170a:1-29878391, row=1, event=2, server_id=101364}