Logstash(二) 使用
基本用法
测试配置文件是否在安装路径
./filebeat -configtest -e
如下输入：
2017/06/23 10:04:00.515550 beat.go:285: INFO Home path: [/home/map/tools/filebeat-5.4.1-linux-x86_64] Config path: [/home/map/tools/filebeat-5.4.1-linux-x86_64] Data path: [/home/map/tools/filebeat-5.4.1-linux-x86_64/data] Logs path: [/home/map/tools/filebeat-5.4.1-linux-x86_64/logs]
2017/06/23 10:04:00.515612 beat.go:186: INFO Setup Beat: filebeat; Version: 5.4.1
2017/06/23 10:04:00.515722 logstash.go:90: INFO Max Retries set to: 3
2017/06/23 10:04:00.515784 metrics.go:23: INFO Metrics logging every 10s
2017/06/23 10:04:00.515871 outputs.go:108: INFO Activated logstash as output plugin.
2017/06/23 10:04:00.516005 publish.go:295: INFO Publisher name: cp01-map-2016-52.epc.baidu.com
2017/06/23 10:04:00.516164 async.go:63: INFO Flush Interval set to: 1s
2017/06/23 10:04:00.516180 async.go:64: INFO Max Bulk Size set to: 2048
Config OK
简单的例子
cd logstash-5.4.2
bin/logstash -e 'input { stdin { } } output { stdout {} }'
从文件中读取数据
- 写 taxi-pipeline.conf, 如下
input {
file {
path => "/home/map/data/taxi.txt"
}
}
filter {
}
output {
stdout {
}
}
- 测试配置文件是否OK
map@cp01-map-2016-52.epc.baidu.com ~/tools/logstash-5.4.1 16:34:42 $
bin/logstash -f taxi-pipeline.conf --config.test_and_exit
Sending Logstash's logs to /home/map/tools/logstash-5.4.1/logs which is now configured via log4j2.properties
Configuration OK
[2017-06-22T16:35:01,525][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
The --config.test_and_exit option parses your configuration file and reports any errors.
- 运行
bin/logstash -f taxi-pipeline.conf --config.reload.automatic
使用Filebeat搜集日志
配置filebeat.yml文件
启动服务
./filebeat -e -c filebeat.yml -d "publish"
修改Filebeat检测log的时间
scan_frequency: 3s
Filebeat
Command Line Options
https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html
How Filebeat Works
https://www.elastic.co/guide/en/beats/filebeat/current/how-filebeat-works.html
Filebeat consists of two main components: prospectors and harvesters. These components work together to tail files and send event data to the output that you specify.
Configuring Filebeat
https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html
Configure File Format
Configuration Options (Reference)
Logstash
Retrieve general information about the Logstash instance
curl -XGET 'localhost:9600/?pretty'
{
"host" : "cp01-map-2016-52.epc.baidu.com",
"version" : "5.4.1",
"http_address" : "127.0.0.1:9600",
"id" : "b5d32f3d-903c-42f8-a59c-0f0567db1c12",
"name" : "cp01-map-2016-52.epc.baidu.com",
"build_date" : "2017-05-29T16:40:20Z",
"build_sha" : "cf39b7a82225994a0a3e716021c66f7a45fae46c",
"build_snapshot" : false
}
Retrieve information about the node
curl -XGET 'localhost:9600/_node?pretty'
{
"host" : "cp01-map-2016-52.epc.baidu.com",
"version" : "5.4.1",
"http_address" : "127.0.0.1:9600",
"id" : "b5d32f3d-903c-42f8-a59c-0f0567db1c12",
"name" : "cp01-map-2016-52.epc.baidu.com",
"pipeline" : {
"workers" : 12,
"batch_size" : 125,
"batch_delay" : 5,
"config_reload_automatic" : true,
"config_reload_interval" : 3,
"id" : "main"
},
"os" : {
"name" : "Linux",
"arch" : "amd64",
"version" : "2.6.32_1-16-0-0_virtio",
"available_processors" : 12
},
"jvm" : {
"pid" : 6909,
"version" : "1.8.0_111",
"vm_name" : "Java HotSpot(TM) 64-Bit Server VM",
"vm_version" : "1.8.0_111",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1498462270980,
"mem" : {
"heap_init_in_bytes" : 268435456,
"heap_max_in_bytes" : 1037959168,
"non_heap_init_in_bytes" : 2555904,
"non_heap_max_in_bytes" : 0
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ]
}
}
Plugins Info API
curl -XGET 'localhost:9600/_node/plugins?pretty'
Node Stats API
curl -XGET 'localhost:9600/_node/stats/<types>'
curl -XGET 'localhost:9600/_node/stats/jvm?pretty'
curl -XGET 'localhost:9600/_node/stats/pipeline?pretty'
curl -XGET 'localhost:9600/_node/stats/os?pretty'
Hot Threads API
curl -XGET 'localhost:9600/_node/hot_threads?pretty'
