tidb-dm工具迁移使用

参考官方文档：

https://docs.pingcap.com/zh/tidb-data-migration/v5.3/migrate-data-using-dm

常用命令：

tiup dm display dm-test 查看dm集群状态

1、安装dm,dmctl完成后，先加载数据源配置文件

[root@mysql-4 ~]# tiup dmctl --master-addr=172.16.1.13:8261 operate-source create mysql-source-conf1.yaml
tiup is checking updates for component dmctl ...
Starting component `dmctl`: /root/.tiup/components/dmctl/v6.3.0/dmctl/dmctl --master-addr=172.16.1.13:8261 operate-source create mysql-source-conf1.yaml
{
"result": true,
"msg": "",
"sources": [
{
"result": true,
"msg": "",
"source": "mysql-replica-01",
"worker": "dm-172.16.1.13-8262"
}
]
}

2、查看数据源配置文件

[root@mysql-4 ~]# tiup dmctl --master-addr=172.16.1.13:8261 get-config source mysql-replica-01
tiup is checking updates for component dmctl ...
Starting component `dmctl`: /root/.tiup/components/dmctl/v6.3.0/dmctl/dmctl --master-addr=172.16.1.13:8261 get-config source mysql-replica-01
{
"result": true,
"msg": "",
"cfg": "enable-gtid: false\nauto-fix-gtid: false\nrelay-dir: relay-dir\nmeta-dir: \"\"\nflavor: mysql\ncharset: \"\"\nenable-relay: false\nrelay-binlog-name: \"\"\nrelay-binlog-gtid: \"\"\nsource-id: mysql-replica-01\nfrom:\n host: 172.16.1.10\n port: 3306\n user: root\n password: '******'\n max-allowed-packet: null\n session: {}\n security: null\npurge:\n interval: 3600\n expires: 0\n remain-space: 15\nchecker:\n check-enable: true\n backoff-rollback: 5m0s\n backoff-max: 5m0s\n check-interval: 5s\n backoff-min: 1s\n backoff-jitter: true\n backoff-factor: 2\nserver-id: 429557913\ntracer: {}\ncase-sensitive: false\nfilters: []\n"
}

3、查看数据源和 work 节点对应关系
[root@mysql-4 ~]# tiup dmctl --master-addr=172.16.1.13:8261 operate-source show
tiup is checking updates for component dmctl ...
Starting component `dmctl`: /root/.tiup/components/dmctl/v6.3.0/dmctl/dmctl --master-addr=172.16.1.13:8261 operate-source show
{
"result": true,
"msg": "",
"sources": [
{
"result": true,
"msg": "",
"source": "mysql-replica-01",
"worker": "dm-172.16.1.13-8262"
}
]
}

4、配置DM任务文件，其中主要是源和目标端，及过滤规则，以单库同步为例。

name: "dm-taskX"
task-mode: all
ignore-checking-items: ["auto_increment_ID"]
target-database:
  host: "172.16.1.12"
  port: 4000
  user: "root_admin"
  password: "123456"

mysql-instances:
 -
  source-id: "mysql-replica-01"
  route-rules: ["instance-1-ceshi-rule"]
  block-allow-list: "global"
  mydumper-config-name: "global"
  loader-config-name: "global"
  syncer-config-name: "global"

# 所有实例的共有配置
routes:
  instance-1-ceshi-rule:
    schema-pattern: "ceshi"
    target-schema: "ceshi2"
mydumpers:
  global:
    threads: 4
    chunk-filesize: 64
loaders:                             # load 处理单元的运行配置参数
  global:                            # 配置名称
    pool-size: 16                    # load 处理单元并发执行 dump 处理单元的 SQL 文件的线程数量，默认值为 16，当有多个实例同时向 TiDB 迁移数据时可根据负载情况适当调小该值
    dir: "./dumped_data"             # dump 处理单元输出 SQL 文件的目录，同时也是 load 处理单元读取文件的目录。该配置项的默认值为 "./dumped_data"。同实例对应的不同任务必须配置不同的目录
syncers:                             # sync 处理单元的运行配置参数
  global:                            # 配置名称
    worker-count: 16                 # 应用已传输到本地的 binlog 的并发线程数量，默认值为 16。调整此参数不会影响上游拉取日志的并发，但会对下游产生显著压力。
    batch: 100                       # sync 迁移到下游数据库的一个事务批次 SQL 语句数，默认值为 100，建议一般不超过 500。
    enable-ansi-quotes: true         # 若 `session` 中设置 `sql-mode: "ANSI_QUOTES"`，则需开启此项
block-allow-list:                     # 如果 DM 版本早于 v2.0.0-beta.2 则使用 black-white-list。
  global:
    do-dbs: ["ceshi"]                        # 需要迁移的上游表的白名单。

4.1 查询源数据库测试,会出一些错误信息，源端都是测试表，建的很随意，根据报错信息整改一下。

[root@mysql-4 ~]# tiup dmctl --master-addr=172.16.1.13:8261 check-task dm-task.yaml
tiup is checking updates for component dmctl ...
Starting component `dmctl`: /root/.tiup/components/dmctl/v6.3.0/dmctl/dmctl --master-addr=172.16.1.13:8261 check-task dm-task.yaml
{
"result": false,
"msg": "[code=26005:class=dm-master:scope=internal:level=medium], Message: fail to check synchronization configuration with type: check was failed, please see detail
detail: {
"results": [
{
"id": 7,
"name": "table structure compatibility check",
"desc": "check compatibility of table structure",
"state": "fail",
"errors": [
{
"severity": "warn",
"short_error": "table `ceshi`.`tt7` Foreign Key fk_id is parsed but ignored by TiDB.",
"instruction": "please ref document: https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features"
},
{
"severity": "fail",
"short_error": "table `ceshi`.`time` primary/unique key does not exist",
"instruction": "please set primary/unique key for the table"
},
{
"severity": "fail",
"short_error": "table `ceshi`.`t2` primary/unique key does not exist",
"instruction": "please set primary/unique key for the table"
}
],
"extra": "address of db instance - 172.16.1.10:3306"
}
],
"summary": {
"passed": false,
"total": 8,
"successful": 7,
"failed": 1,
"warning": 0
}
}"
}

整改过后就正常了

4.2开始复制任务
[root@mysql-4 ~]# tiup dmctl --master-addr=172.16.1.13:8261 start-task dm-task.yaml
tiup is checking updates for component dmctl ...
Starting component `dmctl`: /root/.tiup/components/dmctl/v6.3.0/dmctl/dmctl --master-addr=172.16.1.13:8261 start-task dm-task.yaml
{
"result": true,
"msg": "",
"sources": [
{
"result": true,
"msg": "",
"source": "mysql-replica-01",
"worker": "dm-172.16.1.13-8262"
}
],
"checkResult": ""
}

小插曲，dm版本5.3 DMCTL版本6.3，导致DMCTL停任务时失败，版本需要对齐
tiup uninstall dmctl --all
tiup install dmctl:v5.3.0

4.4停任务正确提示

[root@mysql-4 ~]# tiup dmctl --master-addr=172.16.1.13:8261 stop-task dm-task.yaml
tiup is checking updates for component dmctl ...
A new version of dmctl is available:
The latest version: v6.3.0
Local installed version: v5.3.0
Update current component: tiup update dmctl
Update all components: tiup update --all

Starting component `dmctl`: /root/.tiup/components/dmctl/v5.3.0/dmctl/dmctl --master-addr=172.16.1.13:8261 stop-task dm-task.yaml
{
"op": "Stop",
"result": true,
"msg": "",
"sources": [
{
"result": true,
"msg": "",
"source": "mysql-replica-01",
"worker": "dm-172.16.1.13-8262"
}
]
}

posted on 2022-10-11 21:38 柴米油盐酱醋阅读(296) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

柴米油盐酱醋茶

tidb-dm工具迁移使用

导航

公告