Maxwell入门

1 下载tar包

• Download binary distro: https://github.com/zendesk/maxwell/releases/download/v1.19.4/maxwell-1.19.4.tar.gz

2 解压

　　tar -zvxf maxwell-1.19.4.tar.gz

3 修改配置

　　cd maxwell-1.19.4

　　cp config.properties.example config.properties

根据自己mysql以及kafka配置　　

# tl;dr config
log_level=info

producer=kafka
kafka.bootstrap.servers=hdp1:9092,hdp2:9092,hdp3:9092,hdp4:9092

# mysql login info
host=127.0.0.1
user=maxwell
password=pwd123456

 

4 修改mysql的binlog配置  根据自己机器配置

vi /etc/my.cnf

$ vi my.cnf

[mysqld]

server_id=1

log-bin=master

binlog_format=row

 

5 创建同步用户

mysql> GRANT ALL on maxwell.* to 'maxwell'@'%' identified by 'XXXXXX';
mysql> GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE on *.* to 'maxwell'@'%';

mysql>flush privileges;

6 重启mysql查看bin log是否修改成功

service mysqld restart

show variables like '%binlog%'

 

7 测试一下maxwell 

[root@hdp1 /mnt/software/maxwell-1.19.4]#bin/maxwell --user='maxwell' --password='pwd123456' --host='127.0.0.1' --producer=stdout 

#插入数据 根据自己mysql情况测试

mysql> INSERT INTO `cmcc_offset` (`topics`, `partId`, `offsets`, `groupId`, `create_time`, `updater`, `update_time`, `is_delete`)

 成功收到数据

 

8 单独的kafka producer配置

单独copy指定版本kafka的client的jar包到lib目录下， 我自己的kafka版本是1.1.0的

 

 

 

启动kafka的producer命令，需要单独制定kafka的version

bin/maxwell --user='maxwell' --password='pwd123456' --host='127.0.0.1'  --producer=kafka --kafka_version=1.1.0 --kafka.bootstrap.servers=hdp1:9092,hdp2:9092,hdp3:9092,hdp4:9092 --kafka_topic=maxwell

 

遇到问题  Kafka的K大写了，改成小写就OK了

java.lang.RuntimeException: Unknown producer type: Kafka
at com.zendesk.maxwell.MaxwellContext.getProducer(MaxwellContext.java:372) ~[maxwell-1.19.4.jar:1.19.4]
at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:186) ~[maxwell-1.19.4.jar:1.19.4]
at com.zendesk.maxwell.Maxwell.start(Maxwell.java:156) ~[maxwell-1.19.4.jar:1.19.4]
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:246) ~[maxwell-1.19.4.jar:1.19.4]
01:08:02,127 INFO TaskManager - Stopped all tasks
java.lang.RuntimeException: Unknown producer type: Kafka
at com.zendesk.maxwell.MaxwellContext.getProducer(MaxwellContext.java:372)
at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:186)
at com.zendesk.maxwell.Maxwell.start(Maxwell.java:156)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:246)
[root@hdp1 /mnt/software/maxwell-1.19.4]#xcall.sh jps

 

其他细节详情请看官网http://maxwells-daemon.io/quickstart/