Logstash 入门实战(2)--安装及使用

本文主要介绍 Logstash 的安装及简单的使用，相关的环境及软件信息如下：CentOS 7.9、Logstash 8.2.2、Elasticsearch 8.2.2。

1、安装

根据环境下载对应的安装包：https://www.elastic.co/cn/downloads/logstash，这里选择 Linux x86_64 版本；下载完成后在服务器上解压即可：

tar zxvf logstash-8.2.2-linux-x86_64.tar.gz

2、简单使用

这里使用 Logstash 来收集 Nginx 访问日志并保存到 Elasticsearch 中。

2.1、创建索引

先创建用于存放 Nginx 访问日志的索引。

curl -X PUT -H 'Content-Type:application/json' 'http://10.49.196.11:9200/nginx-index' -d '
{
  "mappings": {
    "properties": {
      "message": {
        "type": "text"
      },
      "ip": {
        "type": "text"
      },
      "remoteUser": {
        "type": "text"
      },
      "accessTime": {
        "type": "date"
      },
      "method": {
        "type": "keyword"
      },
      "path": {
        "type": "text"
      },
      "protocal": {
        "type": "keyword"
      },
      "version": {
        "type": "keyword"
      },
      "status": {
        "type": "integer"
      },
      "bytes": {
        "type": "integer"
      },
      "referer": {
        "type": "text"
      },
      "userAgent": {
        "type": "text"
      }
    }
  }
}'

2.2、Logstash 配置输入

input {
  file {
    path => ["/home/hadoop/app/nginx-1.8.0/logs/access.log"]
    start_position => "beginning"
  }
}

这里指定了 Nginx 日志文件的路径。

2.3、Logstash 配置过滤器

我们需要对日志进行处理，提取出我们需要的字段。

filter {
  grok {
    match => { "message" => "%{IP:ip} - %{USER:remoteUser} \[%{HTTPDATE:accessTimeStr}\] \"%{WORD:method} %{URIPATHPARAM:path} %{WORD:
protocal}/%{NUMBER:version}\" %{INT:status} %{INT:bytes} \"%{DATA:referer}\" \"%{DATA:userAgent}\"" }
  }

  if [tags][0] == '_grokparsefailure' {
    drop{}
  }
  
  date {
    match => ["accessTimeStr", "dd/MMM/yyyy:HH:mm:ss Z"]
    target => "accessTime"
  }
  
  mutate {
    convert => {
      "bytes" => "integer"
      "status" => "integer"
    }
  }
  
  prune {
    blacklist_names => ["log","@version","host","@timestamp","accessTimeStr","event"]
  }
}

grok 插件通过正则表达把原始日志拆分成相应的字段；date 插件把字段转成日期格式；mutate 插件把字段转成我们需要的类型；prune 插件过滤出不需要存到 Elasticsearch 的字段。

2.4、Logstash 配置输出

配置输出到本地的 Elasticsearch。

output {
  stdout { }
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "nginx-index"
  }
}

2.5、完整配置

input {
  file {
    path => ["/home/hadoop/app/nginx-1.8.0/logs/access.log"]
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{IP:ip} - %{USER:remoteUser} \[%{HTTPDATE:accessTimeStr}\] \"%{WORD:method} %{URIPATHPARAM:path} %{WORD:
protocal}/%{NUMBER:version}\" %{INT:status} %{INT:bytes} \"%{DATA:referer}\" \"%{DATA:userAgent}\"" }
  }

  if [tags][0] == '_grokparsefailure' {
    drop{}
  }
  
  date {
    match => ["accessTimeStr", "dd/MMM/yyyy:HH:mm:ss Z"]
    target => "accessTime"
  }
  
  mutate {
    convert => {
      "bytes" => "integer"
      "status" => "integer"
    }
  }
  
  prune {
    blacklist_names => ["log","@version","host","@timestamp","accessTimeStr","event"]
  }
}


output {
  stdout { }
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "nginx-index"
  }
}

nginx.conf

2.6、运行 Logstash

bin/logstash -f nginx.conf

2.7、验证

Nginx 的访问日志信息如下：

10.49.196.1 - - [07/Sep/2022:11:04:15 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
...

Logstash 的控制台日志信息如下：

{
         "bytes" => 0,
       "referer" => "-",
      "protocal" => "HTTP",
     "userAgent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
            "ip" => "10.49.196.1",
    "accessTime" => 2022-09-07T03:04:15Z,
        "method" => "GET",
       "message" => "10.49.196.1 - - [07/Sep/2022:11:04:15 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
    "remoteUser" => "-",
        "status" => 304,
       "version" => "1.1",
          "path" => "/"
}
{
         "bytes" => 0,
       "referer" => "-",
      "protocal" => "HTTP",
     "userAgent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
            "ip" => "10.49.196.1",
    "accessTime" => 2022-09-07T03:04:16Z,
        "method" => "GET",
       "message" => "10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
    "remoteUser" => "-",
        "status" => 304,
       "version" => "1.1",
          "path" => "/"
}
{
         "bytes" => 0,
       "referer" => "-",
      "protocal" => "HTTP",
     "userAgent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
            "ip" => "10.49.196.1",
    "accessTime" => 2022-09-07T03:04:16Z,
        "method" => "GET",
       "message" => "10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
    "remoteUser" => "-",
        "status" => 304,
       "version" => "1.1",
          "path" => "/"
}
...

Elasticsearch 中查询数据：

curl -X GET -H 'Content-Type:application/json' 'http://10.49.196.11:9200/nginx-index'

结果如下：

{
  "took": 530,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "nginx-index",
        "_id": "nSjnFYMB-RPngHUTzpDo",
        "_score": 1.0,
        "_source": {
          "bytes": 0,
          "referer": "-",
          "protocal": "HTTP",
          "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
          "ip": "10.49.196.1",
          "accessTime": "2022-09-07T03:04:15Z",
          "method": "GET",
          "message": "10.49.196.1 - - [07/Sep/2022:11:04:15 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
          "remoteUser": "-",
          "status": 304,
          "version": "1.1",
          "path": "/"
        }
      },
      {
        "_index": "nginx-index",
        "_id": "nijnFYMB-RPngHUT0JCK",
        "_score": 1.0,
        "_source": {
          "bytes": 0,
          "referer": "-",
          "protocal": "HTTP",
          "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
          "ip": "10.49.196.1",
          "accessTime": "2022-09-07T03:04:16Z",
          "method": "GET",
          "message": "10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
          "remoteUser": "-",
          "status": 304,
          "version": "1.1",
          "path": "/"
        }
      },
      {
        "_index": "nginx-index",
        "_id": "nyjnFYMB-RPngHUT0JCK",
        "_score": 1.0,
        "_source": {
          "bytes": 0,
          "referer": "-",
          "protocal": "HTTP",
          "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
          "ip": "10.49.196.1",
          "accessTime": "2022-09-07T03:04:16Z",
          "method": "GET",
          "message": "10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
          "remoteUser": "-",
          "status": 304,
          "version": "1.1",
          "path": "/"
        }
      }
    ]
  }
}

posted @ 2022-09-25 09:28 且行且码阅读(372) 评论(0) 编辑收藏举报

刷新页面返回顶部