Maxwell实时采集mysql表数据到kafka

• 1、启动kafka集群和zookeeper集群

  • 启动zookeeper集群

    [hadoop@hadoop01 bin]zk.sh start
    

  • 启动kafka集群

    [hadoop@hadoop01 bin]kafka.sh start
    

• 2、创建topic

  [hadoop@hadoop01 ~]kafka-topics.sh --create --topic maxwell --partitions 3 --replication-factor 2 --zookeeper hadoop01:2181,hadoop02:2181,hadoop03:2181
  

• 3、启动maxwell服务

  [hadoop@hadoop03 ~]/bigdata/install/maxwell-1.21.1/bin/maxwell
  

• 4、进入hadoop02的mysql数据库，插入数据并进行测试

  • 向mysql当中创建数据库和数据库表,向表中插入一条数据。

    CREATE DATABASE `test_db`;
    
    USE `test_db`;
    
    /*Table structure for table `user` */
    
    DROP TABLE IF EXISTS `user`;
    
    CREATE TABLE `user` (
      `id` varchar(10) NOT NULL,
      `name` varchar(10) DEFAULT NULL,
      `age` int(11) DEFAULT NULL,
      PRIMARY KEY (`id`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
    
    /*Data for the table `user` */
    #插入数据
    insert  into `user`(`id`,`name`,`age`) values  ('1','xiaokai',20);
    #修改数据
    update `user` set age= 30 where id='1';
    #删除数据
    delete from `user` where id='1';
    

• 5、启动kafka的自带控制台消费者

  • 测试maxwell主题是否有数据进入

    [hadoop@hadoop01 ~]kafka-console-consumer.sh --topic maxwell --bootstrap-server hadoop01:9092,hadoop02:9092,hadoop03:9092 --from-beginning 
    

  • 观察输出结果

    {"database":"test_db","table":"user","type":"insert","ts":1621244407,"xid":985,"commit":true,"data":{"id":"1","name":"xiaokai","age":20}}
    
    {"database":"test_db","table":"user","type":"update","ts":1621244413,"xid":999,"commit":true,"data":{"id":"1","name":"xiaokai","age":30},"old":{"age":20}}
    
    {"database":"test_db","table":"user","type":"delete","ts":1621244419,"xid":1013,"commit":true,"data":{"id":"1","name":"xiaokai","age":30}}