datax同步数据java简单用法
1. 到github下载源码，自己编译。同步数据支持mysql8.0，如果直接用编译好的会遇到各种问题。
https://github.com/alibaba/DataX/blob/master/userGuid.md
idea导入项目，需要先安装好jdk1.8+,python,maven3.x，mysql-connmaven依赖版本号修改：
把mysql-connector-java-5.1.34.jar 修改为mysql-connector-java-8.0.19.jar，这些模块都要修改
<mysql.driver.version>8.0.19</mysql.driver.version>
DataBaseType.java修改：com.mysql.jdbc.Driver改为：com.mysql.cj.jdbc.Driver
<mysql.driver.version>8.0.19</mysql.driver.version>
MySql("mysql", "com.mysql.cj.jdbc.Driver"),
Tddl("mysql", "com.mysql.cj.jdbc.Driver"),
DRDS("drds", "com.mysql.cj.jdbc.Driver"),
Oracle("oracle", "oracle.jdbc.OracleDriver"),
SQLServer("sqlserver", "com.microsoft.sqlserver.jdbc.SQLServerDriver"),
PostgreSQL("postgresql", "org.postgresql.Driver"),
RDBMS("rdbms", "com.alibaba.datax.plugin.rdbms.util.DataBaseType"),
DB2("db2", "com.ibm.db2.jcc.DB2Driver"),
ADS("ads","com.mysql.cj.jdbc.Driver"),
ClickHouse("clickhouse", "ru.yandex.clickhouse.ClickHouseDriver"),
KingbaseES("kingbasees", "com.kingbase8.Driver"),
Oscar("oscar", "com.oscar.Driver"),
OceanBase("oceanbase", "com.alipay.oceanbase.jdbc.Driver"),
StarRocks("starrocks", "com.mysql.cj.jdbc.Driver");
将连接属性中的zeroDateTimeBehavior=convertToNull改为zeroDateTimeBehavior=CONVERT_TO_NULL
suffix = "yearIsDateType=false&zeroDateTimeBehavior=CONVERT_TO_NULL&tinyInt1isBit=false&rewriteBatchedStatements=true";
重新编译：mvn -U clean package assembly:assembly -Dmaven.test.skip=true
等待编译完成，实际大概花了18分钟，每个人可能不同。最后在target目录下生产文件。
2. 新建java springboot项目
datax：刚才编译的项目
导入依赖。datax-common-0.0.1-SNAPSHOT.jar，datax-core-0.0.1-SNAPSHOT.jar
把这2个jar导入本地maven仓库中。在idea，maven命令行窗口执行下面命令：
install:install-file -Dfile=D:\work\temp\datax-common-0.0.1-SNAPSHOT.jar -DgroupId=com.datax.common -DartifactId=common -Dversion=0.0.1 -Dpackaging=jar
install:install-file -Dfile=D:\work\temp\datax-core-0.0.1-SNAPSHOT.jar -DgroupId=com.datax.core -DartifactId=core -Dversion=0.0.1 -Dpackaging=jar
pom.xml文件添加依赖：
<dependency> <groupId>com.datax.core</groupId> <artifactId>core</artifactId> <version>0.0.1</version> </dependency> <dependency> <groupId>com.datax.common</groupId> <artifactId>common</artifactId> <version>0.0.1</version> </dependency> <dependency> <groupId>commons-cli</groupId> <artifactId>commons-cli</artifactId> <version>1.4</version> </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.5.13</version> </dependency> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-io</artifactId> <version>1.3.2</version> </dependency> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-lang3</artifactId> <version>3.12.0</version> </dependency> <dependency> <groupId>commons-lang</groupId> <artifactId>commons-lang</artifactId> <version>2.6</version> </dependency> <dependency> <groupId>com.alibaba</groupId> <artifactId>fastjson</artifactId> <version>1.2.60</version> </dependency>
新建test.json文件
{ "job": { "setting": { "speed": { "channel": 4 } }, "content": [ { "reader": { "name": "mysqlreader", "parameter": { "username": "root", "password": "123456", "connection": [ { "jdbcUrl": ["jdbc:mysql://127.0.0.1:3306/test1?serverTimezone=UTC&characterEncoding=utf8&useSSL=true&autoReconnect=true&failOverReadOnly=false"], "querySql": ["select t.id,t.name,t.status from user_test t where t.id>=${id}"] } ] } }, "writer": { "name": "mysqlwriter", "parameter": { "username": "root", "password": "123456", "writeMode": "insert", "column": ["id","name","status"], "connection": [ { "table": [ "user_test" ], "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/test2?serverTimezone=UTC&characterEncoding=utf8&useSSL=true&autoReconnect=true&failOverReadOnly=false" } ] } } } ] } }
java调用代码：
@Override public void syncDataUser() { System.setProperty("datax.home","D:\\work\\project\\code2\\demo1106\\datax"); System.setProperty("id","5"); String path= commonUtils.getCurrentClasspath(); String[] datxArgs2 = {"-job", path+"/datax/test.json", "-mode", "standalone", "-jobid", "-1"}; try { Engine.entry(datxArgs2); } catch (Throwable e) { e.printStackTrace(); } }
datax.home：源码编译后的路径。不能错否则报错。
从mysql test1.user_test 条件：id>=5数据同步到test2.user_test
目标：
源库表：
执行结果：
19万+数据同步测试，只用10秒。速度非常快。
package com.example.demo.service.impl;
import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl; import com.example.demo.dao.UserTestDao; import com.example.demo.entity.UserTestEntity; import com.example.demo.service.IUserTestService; import lombok.extern.slf4j.Slf4j; import org.springframework.stereotype.Service; import org.springframework.util.CollectionUtils; import org.springframework.util.StopWatch; import java.util.ArrayList; import java.util.List; /** * TODO your comment * * @author xiaozw * @date 2022/11/12 19:18 */ @Service @Slf4j public class UserTestServiceImpl extends ServiceImpl<UserTestDao, UserTestEntity> implements IUserTestService { @Override public void insertUserTest() { StopWatch sw = new StopWatch(); sw.start(); List<UserTestEntity> list=new ArrayList<>(); //插入10万数据测试； for(int i=1;i<100000;i++){ //创建对象 UserTestEntity sans = new UserTestEntity(); sans.setName("xiaozw"+i); sans.setStatus(0); list.add(sans); if(!CollectionUtils.isEmpty(list) && list.size()>10000){ log.info("list数量超过1万，开始入库。。。"); this.saveBatch(list,500); list.clear(); log.info("入库完成，list清空。。。"); }else if(i>900000){ this.saveBatch(list); list.clear(); } } sw.stop(); System.out.println("测试耗时(秒)："+sw.getTotalTimeSeconds()); } }
完整项目代码下载：
链接：https://pan.baidu.com/s/1Ooj-jvqpwa1MSaq0eCJY8g
提取码：pzpj
参考资料：
alibaba/DataX github
https://github.com/alibaba/DataX/blob/master/userGuid.md
java整合datax最详细的教程
https://blog.csdn.net/Ting1king/article/details/120871137
DataX实践趟坑大全
https://blog.csdn.net/superman_wang/article/details/118635976
