Logstash安装和设置（图文详解）（多节点的ELK集群安装在一个节点就好）
前提
Elasticsearch-2.4.3的下载（图文详解）
Elasticsearch-2.4.3的单节点安装（多种方式图文详解）
Elasticsearch-2.4.3的3节点安装（多种方式图文详解）
Logstash-2.4.1的下载（图文详解）
Logstash是一个管理日志和事件的工具。
我这里的机器集群情况分别是：
HadoopMaster（192.168.80.10）、HadoopSlave1（192.168.80.11）和HadoopSlave2（192.168.80.12）。
1、上传logstash-2.4.1.tar.gz压缩包
[hadoop@HadoopMaster app]$ ll total 16832 drwxrwxr-x. 9 hadoop hadoop 4096 Feb 22 06:05 elasticsearch-2.4.3 -rw-r--r--. 1 hadoop hadoop 908862 Jan 10 11:38 elasticsearch-head-master.zip -rw-r--r--. 1 hadoop hadoop 2228252 Jan 10 11:38 elasticsearch-kopf-master.zip drwxr-xr-x. 2 hadoop hadoop 4096 Mar 27 01:36 filebeat-1.3.1-x86_64 drwxr-xr-x. 10 hadoop hadoop 4096 Oct 31 17:15 hadoop-2.6.0 drwxr-xr-x. 15 hadoop hadoop 4096 Nov 14 2014 hadoop-2.6.0-src drwxrwxr-x. 8 hadoop hadoop 4096 Nov 2 18:20 hbase-1.2.3 drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 drwxrwxr-x. 11 hadoop hadoop 4096 Nov 4 23:24 kibana-4.6.3-linux-x86_64 -rw-r--r--. 1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz -rw-r--r--. 1 hadoop hadoop 2332033 Jan 16 17:25 shield-2.4.3.zip drwxrwxr-x. 9 hadoop hadoop 4096 Feb 25 19:18 tomcat-7.0.73 -rw-r--r--. 1 hadoop hadoop 1556618 Jan 16 17:22 watcher-2.4.3.zip drwxr-xr-x. 10 hadoop hadoop 4096 Nov 1 23:39 zookeeper-3.4.6 [hadoop@HadoopMaster app]$ rz [hadoop@HadoopMaster app]$ ll total 98864 drwxrwxr-x. 9 hadoop hadoop 4096 Feb 22 06:05 elasticsearch-2.4.3 -rw-r--r--. 1 hadoop hadoop 908862 Jan 10 11:38 elasticsearch-head-master.zip -rw-r--r--. 1 hadoop hadoop 2228252 Jan 10 11:38 elasticsearch-kopf-master.zip drwxr-xr-x. 2 hadoop hadoop 4096 Mar 27 01:36 filebeat-1.3.1-x86_64 drwxr-xr-x. 10 hadoop hadoop 4096 Oct 31 17:15 hadoop-2.6.0 drwxr-xr-x. 15 hadoop hadoop 4096 Nov 14 2014 hadoop-2.6.0-src drwxrwxr-x. 8 hadoop hadoop 4096 Nov 2 18:20 hbase-1.2.3 drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 drwxrwxr-x. 11 hadoop hadoop 4096 Nov 4 23:24 kibana-4.6.3-linux-x86_64 -rw-r--r--. 1 hadoop hadoop 83999654 Feb 25 11:06 logstash-2.4.1.tar.gz -rw-r--r--. 1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz
-rw-r--r--. 1 hadoop hadoop 2332033 Jan 16 17:25 shield-2.4.3.zip
drwxrwxr-x. 9 hadoop hadoop 4096 Feb 25 19:18 tomcat-7.0.73
-rw-r--r--. 1 hadoop hadoop 1556618 Jan 16 17:22 watcher-2.4.3.zip
drwxr-xr-x. 10 hadoop hadoop 4096 Nov 1 23:39 zookeeper-3.4.6
[hadoop@HadoopMaster app]$
2、解压
[hadoop@HadoopMaster app]$ ll total 98864 drwxrwxr-x. 9 hadoop hadoop 4096 Feb 22 06:05 elasticsearch-2.4.3 -rw-r--r--. 1 hadoop hadoop 908862 Jan 10 11:38 elasticsearch-head-master.zip -rw-r--r--. 1 hadoop hadoop 2228252 Jan 10 11:38 elasticsearch-kopf-master.zip drwxr-xr-x. 2 hadoop hadoop 4096 Mar 27 01:36 filebeat-1.3.1-x86_64 drwxr-xr-x. 10 hadoop hadoop 4096 Oct 31 17:15 hadoop-2.6.0 drwxr-xr-x. 15 hadoop hadoop 4096 Nov 14 2014 hadoop-2.6.0-src drwxrwxr-x. 8 hadoop hadoop 4096 Nov 2 18:20 hbase-1.2.3 drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 drwxrwxr-x. 11 hadoop hadoop 4096 Nov 4 23:24 kibana-4.6.3-linux-x86_64 -rw-r--r--. 1 hadoop hadoop 83999654 Feb 25 11:06 logstash-2.4.1.tar.gz -rw-r--r--. 1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz -rw-r--r--. 1 hadoop hadoop 2332033 Jan 16 17:25 shield-2.4.3.zip drwxrwxr-x. 9 hadoop hadoop 4096 Feb 25 19:18 tomcat-7.0.73 -rw-r--r--. 1 hadoop hadoop 1556618 Jan 16 17:22 watcher-2.4.3.zip drwxr-xr-x. 10 hadoop hadoop 4096 Nov 1 23:39 zookeeper-3.4.6 [hadoop@HadoopMaster app]$ tar -zxvf logstash-2.4.1.tar.gz
第三步：删除安装包，并修改所属组和用户
-rw-r--r--. 1 hadoop hadoop 2228252 Jan 10 11:38 elasticsearch-kopf-master.zip drwxr-xr-x. 2 hadoop hadoop 4096 Mar 27 01:36 filebeat-1.3.1-x86_64 drwxr-xr-x. 10 hadoop hadoop 4096 Oct 31 17:15 hadoop-2.6.0 drwxr-xr-x. 15 hadoop hadoop 4096 Nov 14 2014 hadoop-2.6.0-src drwxrwxr-x. 8 hadoop hadoop 4096 Nov 2 18:20 hbase-1.2.3 drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 drwxrwxr-x. 11 hadoop hadoop 4096 Nov 4 23:24 kibana-4.6.3-linux-x86_64 drwxrwxr-x. 5 hadoop hadoop 4096 Mar 27 03:58 logstash-2.4.1 -rw-r--r--. 1 hadoop hadoop 83999654 Feb 25 11:06 logstash-2.4.1.tar.gz -rw-r--r--. 1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz -rw-r--r--. 1 hadoop hadoop 2332033 Jan 16 17:25 shield-2.4.3.zip drwxrwxr-x. 9 hadoop hadoop 4096 Feb 25 19:18 tomcat-7.0.73 -rw-r--r--. 1 hadoop hadoop 1556618 Jan 16 17:22 watcher-2.4.3.zip drwxr-xr-x. 10 hadoop hadoop 4096 Nov 1 23:39 zookeeper-3.4.6 [hadoop@HadoopMaster app]$ rm logstash-2.4.1.tar.gz [hadoop@HadoopMaster app]$ ll total 16836 drwxrwxr-x. 9 hadoop hadoop 4096 Feb 22 06:05 elasticsearch-2.4.3 -rw-r--r--. 1 hadoop hadoop 908862 Jan 10 11:38 elasticsearch-head-master.zip -rw-r--r--. 1 hadoop hadoop 2228252 Jan 10 11:38 elasticsearch-kopf-master.zip drwxr-xr-x. 2 hadoop hadoop 4096 Mar 27 01:36 filebeat-1.3.1-x86_64 drwxr-xr-x. 10 hadoop hadoop 4096 Oct 31 17:15 hadoop-2.6.0 drwxr-xr-x. 15 hadoop hadoop 4096 Nov 14 2014 hadoop-2.6.0-src drwxrwxr-x. 8 hadoop hadoop 4096 Nov 2 18:20 hbase-1.2.3 drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 drwxrwxr-x. 11 hadoop hadoop 4096 Nov 4 23:24 kibana-4.6.3-linux-x86_64 drwxrwxr-x. 5 hadoop hadoop 4096 Mar 27 03:58 logstash-2.4.1 -rw-r--r--. 1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz -rw-r--r--. 1 hadoop hadoop 2332033 Jan 16 17:25 shield-2.4.3.zip drwxrwxr-x. 9 hadoop hadoop 4096 Feb 25 19:18 tomcat-7.0.73 -rw-r--r--. 1 hadoop hadoop 1556618 Jan 16 17:22 watcher-2.4.3.zip drwxr-xr-x. 10 hadoop hadoop 4096 Nov 1 23:39 zookeeper-3.4.6
第四步：认识目录结构
[hadoop@HadoopMaster app]$ cd logstash-2.4.1/ [hadoop@HadoopMaster logstash-2.4.1]$ pwd /home/hadoop/app/logstash-2.4.1 [hadoop@HadoopMaster logstash-2.4.1]$ ll total 160 drwxrwxr-x. 2 hadoop hadoop 4096 Mar 27 03:58 bin -rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md -rw-rw-r--. 1 hadoop hadoop 2249 Nov 14 10:04 CONTRIBUTORS -rw-rw-r--. 1 hadoop hadoop 5084 Nov 14 10:07 Gemfile -rw-rw-r--. 1 hadoop hadoop 23015 Nov 14 10:04 Gemfile.jruby-1.9.lock drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 lib -rw-rw-r--. 1 hadoop hadoop 589 Nov 14 10:04 LICENSE -rw-rw-r--. 1 hadoop hadoop 149 Nov 14 10:04 NOTICE.TXT drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 vendor [hadoop@HadoopMaster logstash-2.4.1]$
Filebeat啊，根据input来监控数据，根据output来使用数据！！！
对应于，Logstash啊，有input、filter和output。
最简单的Logstash测试（即，输入什么，直接在console打印输出）
[hadoop@HadoopMaster logstash-2.4.1]$ pwd /home/hadoop/app/logstash-2.4.1 [hadoop@HadoopMaster logstash-2.4.1]$ ll total 160 drwxrwxr-x. 2 hadoop hadoop 4096 Mar 27 03:58 bin -rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md -rw-rw-r--. 1 hadoop hadoop 2249 Nov 14 10:04 CONTRIBUTORS -rw-rw-r--. 1 hadoop hadoop 5084 Nov 14 10:07 Gemfile -rw-rw-r--. 1 hadoop hadoop 23015 Nov 14 10:04 Gemfile.jruby-1.9.lock drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 lib -rw-rw-r--. 1 hadoop hadoop 589 Nov 14 10:04 LICENSE -rw-rw-r--. 1 hadoop hadoop 149 Nov 14 10:04 NOTICE.TXT drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 vendor [hadoop@HadoopMaster logstash-2.4.1]$ bin/logstash -e 'input { stdin { } } output { stdout {} }' Settings: Default pipeline workers: 1 Pipeline main started （输入回车） 2017-03-26T21:01:02.849Z HadoopMaster （显示回车） abcd 2017-03-26T21:01:10.559Z HadoopMaster abcd
以上是最简单的Logstash测试（即，输入什么，直接在console打印输出）。
^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>"main"}
Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss. {:level=>:warn}
^CSIGINT received. Terminating immediately.. {:level=>:fatal}
[hadoop@HadoopMaster logstash-2.4.1]$
Logstash可以以指定某种格式来输入。比如如下：
[hadoop@HadoopMaster logstash-2.4.1]$ pwd /home/hadoop/app/logstash-2.4.1 [hadoop@HadoopMaster logstash-2.4.1]$ ll total 160 drwxrwxr-x. 2 hadoop hadoop 4096 Mar 27 03:58 bin -rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md -rw-rw-r--. 1 hadoop hadoop 2249 Nov 14 10:04 CONTRIBUTORS -rw-rw-r--. 1 hadoop hadoop 5084 Nov 14 10:07 Gemfile -rw-rw-r--. 1 hadoop hadoop 23015 Nov 14 10:04 Gemfile.jruby-1.9.lock drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 lib -rw-rw-r--. 1 hadoop hadoop 589 Nov 14 10:04 LICENSE -rw-rw-r--. 1 hadoop hadoop 149 Nov 14 10:04 NOTICE.TXT drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 vendor [hadoop@HadoopMaster logstash-2.4.1]$ bin/logstash -e 'input { stdin { } } output { stdout { codec => json} }' Settings: Default pipeline workers: 1 Pipeline main started {"message":"","@version":"1","@timestamp":"2017-03-26T21:13:09.879Z","host":"HadoopMaster"}hjjjk {"message":"hjjjk","@version":"1","@timestamp":"2017-03-26T21:13:23.484Z","host":"HadoopMaster"}^CSIGINT received. Shutting down the agent. {:level=>:warn} stopping pipeline {:id=>"main"} ^CSIGINT received. Terminating immediately.. {:level=>:fatal} [hadoop@HadoopMaster logstash-2.4.1]$
我们可以看到，我们输入什么内容logstash按照某种格式输出，其中-e参数参数允许Logstash直接通过命令行接受设置。这点尤其快速的帮助我们反复的测试配置是否正确而不用写配置文件。使用Ctrl + C命令可以退出之前运行的Logstash。
使用-e参数在命令行中指定配置是很常用的方式，不过如果需要配置更多设置则需要很长的内容。这种情况，我们首先创建一个简单的配置文件，并且指定logstash使用这个配置文件。例如：在logstash安装目录下创建一个“基本配置”测试文件logstash-simple.conf。
Logstash使用-f参数替换命令行中的-e参数（既可以写到配置文件里，为了方便）
[hadoop@HadoopMaster logstash-2.4.1]$ pwd /home/hadoop/app/logstash-2.4.1 [hadoop@HadoopMaster logstash-2.4.1]$ ll total 160 drwxrwxr-x. 2 hadoop hadoop 4096 Mar 27 03:58 bin -rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md -rw-rw-r--. 1 hadoop hadoop 2249 Nov 14 10:04 CONTRIBUTORS -rw-rw-r--. 1 hadoop hadoop 5084 Nov 14 10:07 Gemfile -rw-rw-r--. 1 hadoop hadoop 23015 Nov 14 10:04 Gemfile.jruby-1.9.lock drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 lib -rw-rw-r--. 1 hadoop hadoop 589 Nov 14 10:04 LICENSE -rw-rw-r--. 1 hadoop hadoop 149 Nov 14 10:04 NOTICE.TXT drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 vendor [hadoop@HadoopMaster logstash-2.4.1]$ vim logstash-simple.conf
input {
stdin { }
}
output {
stdout { }
}
[hadoop@HadoopMaster logstash-2.4.1]$ pwd /home/hadoop/app/logstash-2.4.1 [hadoop@HadoopMaster logstash-2.4.1]$ ll total 164 drwxrwxr-x. 2 hadoop hadoop 4096 Mar 27 03:58 bin -rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md -rw-rw-r--. 1 hadoop hadoop 2249 Nov 14 10:04 CONTRIBUTORS -rw-rw-r--. 1 hadoop hadoop 5084 Nov 14 10:07 Gemfile -rw-rw-r--. 1 hadoop hadoop 23015 Nov 14 10:04 Gemfile.jruby-1.9.lock drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 lib -rw-rw-r--. 1 hadoop hadoop 589 Nov 14 10:04 LICENSE -rw-rw-r--. 1 hadoop hadoop 46 Mar 27 05:30 logstash-simple.conf -rw-rw-r--. 1 hadoop hadoop 149 Nov 14 10:04 NOTICE.TXT drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 vendor [hadoop@HadoopMaster logstash-2.4.1]$ bin/logstash -f logstash-simple.conf Settings: Default pipeline workers: 1 Pipeline main started 2017-03-26T21:32:32.782Z HadoopMaster abcd 2017-03-26T21:32:36.848Z HadoopMaster abcd ^CSIGINT received. Shutting down the agent. {:level=>:warn} stopping pipeline {:id=>"main"} ^CSIGINT received. Terminating immediately.. {:level=>:fatal} [hadoop@HadoopMaster logstash-2.4.1]$
推荐用这个！！！
bin/logstash -f logstash-simple.conf --auto-reload
因为，在调试，每次都要重启。加这个，不需每次去重启Logstash，即自己会加载。
作者：大数据和人工智能躺过的坑
出处：http://www.cnblogs.com/zlslch/
本文版权归作者和博客园共有，欢迎转载，但未经作者同意必须保留此段声明，且在文章页面明显位置给出原文链接，否则保留追究法律责任的权利。
如果您认为这篇文章还不错或者有所收获，您可以通过右边的“打赏”功能 打赏我一杯咖啡【物质支持】，也可以点击右下角的【好文要顶】按钮【精神支持】，因为这两种支持都是我继续写作，分享的最大动力！
