Maxwell实时采集mysql表数据到kafka
-
1、启动kafka集群和zookeeper集群
-
启动zookeeper集群
[hadoop@hadoop01 bin]zk.sh start
-
启动kafka集群
[hadoop@hadoop01 bin]kafka.sh start
-
-
2、创建topic
[hadoop@hadoop01 ~]kafka-topics.sh --create --topic maxwell --partitions 3 --replication-factor 2 --zookeeper hadoop01:2181,hadoop02:2181,hadoop03:2181
-
3、启动maxwell服务
[hadoop@hadoop03 ~]/bigdata/install/maxwell-1.21.1/bin/maxwell
-
4、进入hadoop02的mysql数据库,插入数据并进行测试
- 向mysql当中创建数据库和数据库表,向表中插入一条数据。
CREATE DATABASE `test_db`; USE `test_db`; /*Table structure for table `user` */ DROP TABLE IF EXISTS `user`; CREATE TABLE `user` ( `id` varchar(10) NOT NULL, `name` varchar(10) DEFAULT NULL, `age` int(11) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; /*Data for the table `user` */ #插入数据 insert into `user`(`id`,`name`,`age`) values ('1','xiaokai',20); #修改数据 update `user` set age= 30 where id='1'; #删除数据 delete from `user` where id='1';
- 向mysql当中创建数据库和数据库表,向表中插入一条数据。
-
5、启动kafka的自带控制台消费者
-
测试maxwell主题是否有数据进入
[hadoop@hadoop01 ~]kafka-console-consumer.sh --topic maxwell --bootstrap-server hadoop01:9092,hadoop02:9092,hadoop03:9092 --from-beginning
-
观察输出结果
{"database":"test_db","table":"user","type":"insert","ts":1621244407,"xid":985,"commit":true,"data":{"id":"1","name":"xiaokai","age":20}} {"database":"test_db","table":"user","type":"update","ts":1621244413,"xid":999,"commit":true,"data":{"id":"1","name":"xiaokai","age":30},"old":{"age":20}} {"database":"test_db","table":"user","type":"delete","ts":1621244419,"xid":1013,"commit":true,"data":{"id":"1","name":"xiaokai","age":30}}
-