canal同步mysql，监听单实例，多实例配置

1、下载canal安装包

canal.adapter-1.1.7.tar.gz

canal.deployer-1.1.7.tar.gz

2、修改涉及的文件

canal_deployer：
/conf/canal.properties
/conf/example/instance.properties

canal_adapter：
/conf/application.yml
/conf/rdb  在rdb中添加对应的数据配置表，即便监听多个mysql实力，全部放在rdb中

3、监听单实例mysql数据源，在/conf/canal.properties中配置一个example，默认就是example

4、监听多个实例mysql数据源，复制example进行修改，例如：example1,example2 

5、监听一个mysql数据源同步到一个数据库

canal_deployer：

修改 /conf/example/instance.properties 文件

#################################################
## mysql serverId , v1.0.26+ will autoGen
# canal.instance.mysql.slaveId=0

# enable gtid use true/false
canal.instance.gtidon=false

# position info
canal.instance.master.address=10.10.7.7:3306
canal.instance.master.journal.name=mysql-bin.054560
canal.instance.master.position=241476047
canal.instance.master.timestamp=
canal.instance.master.gtid=

# rds oss binlog
canal.instance.rds.accesskey=
canal.instance.rds.secretkey=
canal.instance.rds.instanceId=

# table meta tsdb info
canal.instance.tsdb.enable=true
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
#canal.instance.tsdb.dbUsername=canal
#canal.instance.tsdb.dbPassword=canal

#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#canal.instance.standby.gtid=

# username/password
canal.instance.dbUsername=canal_hankyoon
canal.instance.dbPassword=canal_hankyoon
canal.instance.connectionCharset = UTF-8
# enable druid Decrypt database password
canal.instance.enableDruid=false
#canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==

# table regex
#canal.instance.filter.regex=.*\\..*
canal.instance.filter.regex=dbname.tablename01,dbname.tablename02,dbname.tablename03

canal_adapter：
修改application.yml 文件

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: tcp #tcp kafka rocketMQ rabbitMQ    # 注意mode选择
  flatMessage: true
  zookeeperHosts:
  syncBatchSize: 1000
#  retries: -1
#  retries: 0
  timeout:
  accessKey:
  secretKey:
  consumerProperties:
    # canal tcp consumer
    canal.tcp.server.host: 127.0.0.1:11111
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 500
    canal.tcp.username:
    canal.tcp.password:

  srcDataSources:
    defaultDS:
      url: jdbc:mysql://10.10.7.7:3306/hankyoon?useUnicode=true
      username: canal_hankyoon
      password: canal_hankyoon

  canalAdapters:
  - instance: example # canal instance Name or mq topic name
    groups:
    - groupId: yoon
      outerAdapters:
#      - name: logger
      - name: rdb
        key: yoon
        properties:
          jdbc.driverClassName: com.mysql.jdbc.Driver
          jdbc.url: jdbc:mysql://10.10.7.77:3306/hankyoon?useUnicode=true
          jdbc.username: canal_hankyoon
          jdbc.password: canal_hankyoon

在rdb中添加对应的数据配置表：

dataSourceKey: defaultDS
destination: example1
groupId: yoon
outerAdapterKey: yoon
concurrent: true
dbMapping:
  database: hankyoon
  table: yoon
  targetTable: yoon
  targetPk:
    id: id
  mapAll: true

6、监听多个mysql实例数据源，同步至一个数据库
修改 /conf/canal.properties 文件，添加多个实例

canal.destinations = example,example1

修改添加example1

conf/example
conf/example1   # 修改instance.properties中对应的数据库信息

7、修改 application.yml 添加多个监听实例

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: tcp #tcp kafka rocketMQ rabbitMQ    # 注意mode选择
  flatMessage: true
  zookeeperHosts:
  syncBatchSize: 1000
#  retries: -1
#  retries: 0
  timeout:
  accessKey:
  secretKey:
  consumerProperties:
    # canal tcp consumer
    canal.tcp.server.host: 127.0.0.1:11111
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 500
    canal.tcp.username:
    canal.tcp.password:

  srcDataSources:
    defaultDS:
      url: jdbc:mysql://10.10.7.7:3306/hankyoon?useUnicode=true
      username: canal_hankyoon
      password: canal_hankyoon

    defaultDS01:
      url: jdbc:mysql://10.10.7.9:3306/hankyoon?useUnicode=true
      username: canal_hankyoon
      password: canal_hankyoon

  canalAdapters:
  - instance: example # canal instance Name or mq topic name
    groups:
    - groupId: yoon
      outerAdapters:
#      - name: logger
      - name: rdb
        key: yoon
        properties:
          jdbc.driverClassName: com.mysql.jdbc.Driver
          jdbc.url: jdbc:mysql://10.10.7.77:3306/hankyoon?useUnicode=true
          jdbc.username: canal_hankyoon
          jdbc.password: canal_hankyoon

  - instance: example1 # canal instance Name or mq topic name
    groups:
    - groupId: hank
      outerAdapters:
#      - name: logger
      - name: rdb         # rdb不要复制，不要修改，同步的数据源配置表，全部放在一个rdb中
        key: hank
        properties:
          jdbc.driverClassName: com.mysql.jdbc.Driver
          jdbc.url: jdbc:mysql://10.10.7.77:3306/hankyoon?useUnicode=true
          jdbc.username: canal_hankyoon
          jdbc.password: canal_hankyoon

在rdb中数据表配置信息：

example：
dataSourceKey: defaultDS
destination: example1
groupId: yoon
outerAdapterKey: yoon
concurrent: true
dbMapping:
  database: hankyoon
  table: yoon
  targetTable: yoon
  targetPk:
    id: id
  mapAll: true


example1：
dataSourceKey: defaultDS01
destination: example1
groupId: hankyoon
outerAdapterKey: hank
concurrent: true
dbMapping:
  database: hankyoon
  table: hank
  targetTable: hank
  targetPk:
    id: id
  mapAll: true

 

参考：https://blog.csdn.net/wonder_dog/article/details/124608664