|NO.Z.00051|——————————|^^ 部署 ^^|——|Hadoop&ElasticSearch.V03|——|ELK.v03Logstash部署.V3|

一、filter插件
### --- Filter插件

~~~     Logstash之所以强悍的主要原因是filter插件；
~~~     通过过滤器的各种组合可以得到我们想要的结构化数据。
~~~     官网地址：https://www.elastic.co/guide/en/Logstash/current/plugins-filters-grok.html
### --- grok正则表达式
~~~     grok正则表达式是Logstash非常重要的一个环节；可以通过grok非常方便的将数据拆分和索引

~~~     # 语法格式：
~~~     (?<name>pattern)
~~~     ？<name>表示要取出里面的值，pattern就是正则表达式
二、收集控制台输入数据，采集日期时间出来
### --- 开发配置文件

[root@hadoop02 ~]# vim /opt/yanqi/servers/es/Logstash/config/filter.conf
 ~~~写入配置文件参数
input {stdin{}} filter {grok
  { match => {"message" => "(?<date>\d+\.\d+)\s+"}
 }
}
output {stdout{codec => rubydebug}}
### --- 检查配置文件完整性

[root@hadoop02 ~]# /opt/yanqi/servers/es/Logstash/bin/logstash \
-f /opt/yanqi/servers/es/Logstash/config/filter.conf -t
~~~输出如下配置参数
Configuration OK
Config Validation Result: OK. Exiting Logstash
### --- 启动logstash服务

[root@hadoop02 ~]# /opt/yanqi/servers/es/Logstash/bin/logstash \
-f /opt/yanqi/servers/es/Logstash/config/filter.conf
 ~~~ 控制台输入文字
11.11 神棍节！！
  ~~~输出参数：
{
          "date" => "11.11",
       "message" => "11.11 神棍节！！",
      "@version" => "1",
    "@timestamp" => 2021-11-26T09:06:02.387Z,
          "host" => "hadoop02"
}
三、使用grok收集nginx日志数据
### --- nginx一般打印出来的日志格式如下

~~~     这种日志是非格式化的，通常，我们获取到日志后，
~~~     还要使用mapreduce 或者spark 做一下清洗操作，就是将非格式化日志编程格式化日志；
~~~     在清洗的时候，如果日志的数据量比较大，那么也是需要花费一定的时间的；
~~~     所以可以使用Logstash 的grok 功能，将nginx 的非格式化数据采集成格式化数据：
### --- 插入参数解析后的数据：详见四.6章节

36.157.150.1 - - [05/Nov/2019:12:59:28 +0800] "GET/phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1" 200 139613 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36"
~~~输出参数
{
         "time_local" => "05/Nov/2019:12:59:28 +0800",
           "@version" => "1",
               "host" => "hadoop02",
            "message" => "36.157.150.1 - - [05/Nov/2019:12:59:28 +0800] \"GET/phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1\" 200 139613 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\"",
         "rawrequest" => "GET/phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1",
         "@timestamp" => 2021-11-26T09:40:40.657Z,
           "clientip" => "36.157.150.1",
       "http_referer" => "\"-\"",
             "status" => "200",
    "body_bytes_sent" => "139613",
              "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\""
}
四、在线安装frok插件
### --- 在线安装grok插件

~~~     # 更改镜像源地址
[root@hadoop02 ~]# vim /opt/yanqi/servers/es/Logstash/Gemfile
 ~第4/5行配置如下参数
# source "https://rubygems.org"                                 # 将这个镜像源注释掉         
source "https://gems.ruby-china.com/"                           # 配置成中国的这个镜像源   
### --- 准备在线安装

~~~     # 在线安装grok插件
[root@hadoop02 ~]# cd /opt/yanqi/servers/es/Logstash/
[root@hadoop02 Logstash]# bin/logstash-plugin install logstash-filter-grok
~~~输出参数
Validating logstash-filter-grok
Installing logstash-filter-grok
Installation successful
### --- 开发Logstash的配置文件

~~~     # 定义Logstash的配置文件如下，我们从控制台输入nginx的日志数据，然后经过filter的过滤，将我们的日志文件转换成为标准的数据格式
[root@hadoop02 ~]# vim /opt/yanqi/servers/es/Logstash/config/monitor_nginx.conf
 ~~~写入配置参数
input {stdin{}}

filter {

grok {
match => {"message" => "%{IPORHOST:clientip} \- \- \[%{HTTPDATE:time_local}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?:HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS:http_referer} %{QS:agent}"
   }
   }
}
output {stdout{codec => rubydebug}}
### --- 检查配置文件完整性

[root@hadoop02 ~]# /opt/yanqi/servers/es/Logstash/bin/logstash \
-f /opt/yanqi/servers/es/Logstash/config/monitor_nginx.conf -t
 ~~~输出参数
Configuration OK
Config Validation Result: OK. Exiting Logstash
### --- 启动Logstash

~~~     # 执行以下命令启动Logstash
[root@hadoop02 ~]# /opt/yanqi/servers/es/Logstash/bin/logstash \
-f /opt/yanqi/servers/es/Logstash/config/monitor_nginx.conf
 ~~~输出参数：详情查看6数据参数输出
### --- 从控制台输入nginx日志文件数据

~~~     # 输入第一条数据
113.31.119.183 - - [05/Nov/2019:12:59:27 +0800] "GET /phpmyadmin_8c1019c9c0de7a0f/js/messages.php? lang=zh_CN&db=&collation_connection=utf8_unicode_ci&token=6a44d72481633c90bffcfd42f11e25a1 HTTP/1.1" 200 8131 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36"
~~~输出参数
{
         "time_local" => "05/Nov/2019:12:59:27 +0800",
           "@version" => "1",
               "host" => "hadoop02",
            "message" => "113.31.119.183 - - [05/Nov/2019:12:59:27 +0800] \"GET /phpmyadmin_8c1019c9c0de7a0f/js/messages.php? lang=zh_CN&db=&collation_connection=utf8_unicode_ci&token=6a44d72481633c90bffcfd42f11e25a1 HTTP/1.1\" 200 8131 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\"",
         "rawrequest" => "GET /phpmyadmin_8c1019c9c0de7a0f/js/messages.php? lang=zh_CN&db=&collation_connection=utf8_unicode_ci&token=6a44d72481633c90bffcfd42f11e25a1 HTTP/1.1",
         "@timestamp" => 2021-11-26T09:35:04.242Z,
           "clientip" => "113.31.119.183",
       "http_referer" => "\"-\"",
             "status" => "200",
    "body_bytes_sent" => "8131",
              "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\""
}
~~~     # 输入第二条数据

36.157.150.1 - - [05/Nov/2019:12:59:28 +0800] "GET /phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1" 200 139613 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36"
~~~输出参数
{
         "time_local" => "05/Nov/2019:12:59:28 +0800",
           "@version" => "1",
               "host" => "hadoop02",
            "message" => "36.157.150.1 - - [05/Nov/2019:12:59:28 +0800] \"GET /phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1\" 200 139613 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\"",
         "rawrequest" => "GET /phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1",
         "@timestamp" => 2021-11-26T09:35:28.894Z,
           "clientip" => "36.157.150.1",
       "http_referer" => "\"-\"",
             "status" => "200",
    "body_bytes_sent" => "139613",
              "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\""
}
 
 
 
 
 
 
 
 
 
Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart
                                                                                                                                                   ——W.S.Landor
 

 

posted on   yanqi_vip  阅读(46)  评论(0)  编辑  收藏  举报

相关博文：
· |NO.Z.00011|——————————|Deployment|——|Hadoop&ElasticSearch集中式日志分析系统.v11|——|Elasticsearch.v11|Logstash&Logstash部署.V3|
· |NO.Z.00050|——————————|^^ 部署 ^^|——|Hadoop&ElasticSearch.V02|——|ELK.v02|Logstash部署.V2|
· Logstash 入门实战(2)--安装及使用
· Logstash 入门实战(4)--filter plugin 介绍
· logstash
阅读排行：
· 全程不用写代码，我用AI程序员写了一个飞机大战
· MongoDB 8.0这个新功能碉堡了，比商业数据库还牛
· 记一次.NET内存居高不下排查解决与启示
· 白话解读 Dapr 1.15：你的「微服务管家」又秀新绝活了
· DeepSeek 开源周回顾「GitHub 热点速览」
< 2025年3月 >
日 一 二 三 四 五 六
23 24 25 26 27 28 1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31 1 2 3 4 5

导航

统计

点击右上角即可分享
微信分享提示