debezium mongodb 集成测试

debezium 是一个方便的cdc connector 可以帮助我们解决好多数据实时变更处理、数据分析、微服务的数据通信
从上次跑简单demo到现在，这个工具是有好多的变更，添加了好多方便的功能，支持了越来越多的数据库。
demo 使用了官方提供的docker-compose 文件

环境准备

  • docker-compose 文件
 
version: '2'
services:
  zookeeper:
    image: debezium/zookeeper:0.9
    ports:
     - 2181:2181
     - 2888:2888
     - 3888:3888
  kafka:
    image: debezium/kafka:0.9
    ports:
     - 9092:9092
    links:
     - zookeeper
    environment:
     - ZOOKEEPER_CONNECT=zookeeper:2181
  mongodb:
    image: debezium/example-mongodb:0.9
    hostname: mongodb
    ports:
     - 27017:27017
    environment:
     - MONGODB_USER=debezium
     - MONGODB_PASSWORD=dbz
  connect:
    image: debezium/connect:0.9
    ports:
     - 8083:8083
    links:
     - kafka
     - mongodb
    environment:
     - BOOTSTRAP_SERVERS=kafka:9092
     - GROUP_ID=1
     - CONFIG_STORAGE_TOPIC=my_connect_configs
     - OFFSET_STORAGE_TOPIC=my_connect_offsets
  • 启动
docker-compose up -d
 
  • 加载mongodb 数据
    demo 镜像内置了一个mongo 数据的脚本
 
docker-compose -f docker-compose-mongodb.yaml exec mongodb bash -c '/usr/local/bin/init-inventory.sh'
 
 

连接的用户账户密码： debezium dbz
效果

  • 启动connector
 
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8083/connectors/ -d @register-mongodb.json
 

register-mongodb.json 内容

{
    "name": "inventory-connector",
    "config": {
        "connector.class" : "io.debezium.connector.mongodb.MongoDbConnector",
        "tasks.max" : "1",
        "mongodb.hosts" : "rs0/mongodb:27017",
        "mongodb.name" : "dbserver1",
        "mongodb.user" : "debezium",
        "mongodb.password" : "dbz",
        "database.whitelist" : "inventory",
        "database.history.kafka.bootstrap.servers" : "kafka:9092"
    }
}
 
 

效果

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8083/connect
ors/ -d @register-mongodb.json
​
HTTP/1.1 201 Created
Date: Tue, 25 Dec 2018 12:15:59 GMT
Location: http://localhost:8083/connectors/inventory-connector
Content-Type: application/json
Content-Length: 379
Server: Jetty(9.4.12.v20180830)
​
{"name":"inventory-connector","config":{"connector.class":"io.debezium.connector.mongodb.MongoDbConnector","tasks.max":"1","mongodb.hosts":"rs0/mongodb:27017","mongodb.name":"dbserver1","mongodb.user":"debezium","mongodb.password":"dbz","database.whitelist":"inventory","database.history.kafka.bootstrap.servers":"kafka:9092","name":"inventory-connector"},"tasks":[],"type":null}%
 
 

测试

  • 打开消费者（kafka 命令行）
docker-compose -f docker-compose-mongodb.yaml exec kafka /kafka/bin/kafka-console-consumer.sh \
    --bootstrap-server kafka:9092 \
    --from-beginning \
    --property print.key=true \
    --topic dbserver1.inventory.customers
 
  • 修改数据&&查看效果
    修改数据：

    效果
 
{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"}],"optional":false,"name":"dbserver1.inventory.customers.Key"},"payload":{"id":"1001"}} {"schema":{"type":"struct","fields":[{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1,"field":"after"},{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1,"field":"patch"},{"type":"struct","fields":[{"type":"string","optional":true,"field":"version"},{"type":"string","optional":true,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"string","optional":false,"field":"rs"},{"type":"string","optional":false,"field":"ns"},{"type":"int32","optional":false,"field":"sec"},{"type":"int32","optional":false,"field":"ord"},{"type":"int64","optional":true,"field":"h"},{"type":"boolean","optional":true,"default":false,"field":"initsync"}],"optional":false,"name":"io.debezium.connector.mongo.Source","version":1,"field":"source"},{"type":"string","optional":true,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"}],"optional":false,"name":"dbserver1.inventory.customers.Envelope"},"payload":{"after":null,"patch":"{\"$v\" : 1,\"$set\" : {\"first_name\" : \"dalong\"}}","source":{"version":"0.9.0.Beta2","connector":"mongodb","name":"dbserver1","rs":"rs0","ns":"inventory.customers","sec":1545740280,"ord":1,"h":5866229044893041819,"initsync":false},"op":"u","ts_ms":1545740280088}}
 

说明

debezium mongodb 的cdc 是基于复制集实现的，实际上mongodb 已经支持了stream 可以进行数据的捕获处理

参考资料

https://github.com/debezium/debezium-examples/tree/master/tutorial#using-mongodb
https://debezium.io/

posted on 2018-12-25 20:26  荣锋亮  阅读(3140)  评论(0)  编辑  收藏  举报

导航