logstash

 

官网下载： https://www.elastic.co/cn/downloads/past-releases/

 

windows环境：

bin文件夹新建run_default.bat（随意命名）写入

?

1	logstash -f mysqletc/mysql.conf

bin文件夹新建文件夹mysqletc（随意命名）

?

1 2 3

- mysql.conf （脚本将logstash输入，输出到elasticsearch） - lico_user.sql（sql语句将数据录入elasticsearch） - mysql-connector-java-5.1.47.jar（需要下载独立开发包。警告：高开发包版本可能造成时区问题）

mysql.conf配置示例：

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

input {         stdin {}         jdbc {               # mysql 数据库链接,shop为数据库名               jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/lico?serverTimezone=UTC"               # 用户名和密码               jdbc_user => "root"               jdbc_password => "123456"               # 驱动               #jdbc_driver_library => "D:/ELK/logstash-7.8.0/logstash-7.8.0/bin/mysqletc/mysql-connector-java-5.1.47.jar"               jdbc_driver_library => "D:/ELK/logstash-7.8.0/logstash-7.8.0/bin/mysqletc/mysql-connector-java-6.0.6.jar"               # 驱动类名               jdbc_driver_class => "com.mysql.jdbc.Driver"               jdbc_paging_enabled => "true"               jdbc_page_size => "50000"               # 执行的sql 文件路径+名称               statement_filepath => "D:/ELK/logstash-7.8.0/logstash-7.8.0/bin/mysqletc/lico_user.sql"               # 设置监听间隔  各字段含义（由左至右）分、时、天、月、年，全部为*默认含义为每分钟都更新               schedule => "* * * * *"               # 索引类型               type => "user"             }    }    filter {         json {             source => "message"             remove_field => ["message"]         }     }     output {         if [type]=="user"{             elasticsearch {                 hosts => ["localhost:9200"]                 index => "user"                 document_id => "%{id}"             }         }         stdout {             codec => json_lines         }     }

lico_user.sql示例

?

1	SELECT * FROM lico_user

进入bin文件执行run_default.bat

 

参考： https://blog.csdn.net/qq_15273019/article/details/89642750

 

1.启动时可能遇到java mysql的时区问题

unrecognized or represents more than one time zone. You must configure either the server or JDBC driver

解决方案：需要在Url后面添加一个参数：?serverTimezone=UTC

?

1	jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/data?serverTimezone=UTC"

 

2.[ERROR] 2020-07-12 11:16:20.630 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (ArgumentError) invalid byte sequence in US-ASCII

有很大的可能是存放logstash的路径有中文，我就是这样,修改一下路径就可以，不要出现中文

 

3.运行出现you must change the "path.data" setting.

修改启动命令：  /bin/logstash  -f xxxxx.conf  

为： /bin/logstash  -f xxxxxx.conf  --path.data=/root/