Streamset 安装与实践

注意：要先注册才能下载。

官网地址：

https://accounts.streamsets.com/install/instruction/data-collector/linux/common-tarball

1、官方步骤

1、安装jdk8

2、系统文件数设置必须要大于32768

3、注册之后下载是自动的

4、解压：tar xvzf streamsets-datacollector-common-3.22.3.tgz

5、cd streamsets-datacollector-3.22.3

6、启动：bin/streamsets dc

7、http://ip:18360

8、要用注册的账户登录

 

 2、创建自己的PipLine

 

3、添加组件

 

 

4、Mysql组件配置

 

 

 

 

 

 5、Kafka组件配置

 

 

 

 

 

 FAQ

 

1、mysql只支持mysql5.7以上

 

2、com.streamsets.pipeline.api.base.OnRecordErrorException: TEXT_GENERATOR_01 - Record 'mysql-bin.000001:3800' is missing field /text
    at com.streamsets.pipeline.stage.destination.kafka.KafkaTarget.writeOneMessagePerRecord(KafkaTarget.java:255)
    at com.streamsets.pipeline.stage.destination.kafka.KafkaTarget.write(KafkaTarget.java:84)
    at com.streamsets.pipeline.api.base.configurablestage.DTarget.write(DTarget.java:34)
    at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:303)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:244)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:311)
    at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:221)

这个错误就是kafka设置为Text导致，因为从mysql抽取的数据为json格式。如：

{"BinLogFilename":"mysql-bin.000001","Type":"UPDATE","Table":"weblog","ServerId":2,"BinLogPosition":6408,"Database":"test","OldData":{"args":"123","id":10,"method":"12df34","create_time":1625251291000,"url":"name1qw0","ip":"123"},"Data":{"args":"123","id":10,"method":"12df34","create_time":1625251291000,"url":"name3423qw0","ip":"123"},"Timestamp":1626261330000,"Offset":"mysql-bin.000001:6408"}

 

3、mysql驱动一定要正确，这个看日志，如果显示

表示你缺少对应高版本的mysql驱动包