ELK日志分析

ELK日志分析系统

ELK介绍

1. 应用/需求前景

（1）业务发展越来越庞大，服务器越来越多；

（2）各种访问日志、应用日志、错误日志量越来越多，导致运维人员无法很好的去管理日志；

（3）开发人员排查问题，需要到服务器上查日志，不方便；

（4）运营人员需要一些数据，需要我们运维到服务器上分析日志

2. 为什么要用ELK

如果我们查看某些服务为什么错误，可以直接使用grep等命令进行查看，可是如果我们查看规模较大，日志较多的时候，此方法效率就小了很多。现在我们对待大规模的日志，解决思路是建立集中式日志收集系统，将所有节点上的日志统一收集，管理，访问。

一个完整的集中式日志系统，需要包含以下几个主要特点：

收集：能够采集多种来源的日志数据。
传输：能够稳定的把日志数据传输到中央系统。
存储：如何存储日志数据。
分析：可以支持 UI 分析。
警告：能够提供错误报告，监控机制。

而ELK则提供一整套的解决方案，并且都是开源软件，之间相互配合，完美衔接，高效的满足了很多场合的应用。

3. ELK 简介

ELK是3个开源软件的缩写，分别为Elasticsearch 、 Logstash和Kibana , 它们都是开源软件。不过现在还新增了一个Beats，它是一个轻量级的日志收集处理工具（Agent），Beats占用资源少，适合于在各个服务器上搜集日志后传输给Logstash，官方也推荐此工具，目前由于原本的ELK Stack成员中加入了Beats工具所以已改名为Elastic Stack。

Elasticsearch是个开源分布式搜索引擎，提供搜集、分析、存储数据3大功能。它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。

Logstash主要是用来日志的搜集、分析、过滤日志的工具，支持大量的数据获取方式。一般工作方式为c/s架构，Client端安装在需要收集日志的主机上，server端负责将收到的各节点日志进行过滤、修改等操作在一并发往Elasticsearch上去。

Kibana也是一个开源和免费的工具，Kibana可以为 Logstash和 ElasticSearch提供的日志分析友好的 Web 界面，可以帮助汇总、分析和搜索重要数据日志。

Beats在这里是一个轻量级日志采集器，其实Beats家族有6个成员，早期的ELK架构中使用Logstash收集、解析日志，但是Logstash对内存、CPU、io等资源消耗比较高。相比 Logstash，Beats所占系统的CPU和内存几乎可以忽略不计。

4. 环境配置

1.节点规划

节点IP	节点规划	主机名
192.168.200.10	Elasticsearch+Kibana（主）	elk-1
192.168.200.20	Elasticsearch+Logstash（数据）	elk-2
192.168.200.30	Elasticsearch（数据）	elk-3

2. 修改主机名和关闭防火墙

三台机器都修改主机名和防火墙
elk-1:
[root@localhost ~]# sudo hostname elk-1
[root@localhost ~]# bash
[root@elk-1 ~]# systemctl stop firewalld
[root@elk-1 ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@elk-1 ~]# setenforce 0
[root@elk-1 ~]# vi /etc/selinux/config
[root@elk-1 ~]# cat /etc/selinux/config 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

3. 配置 hosts 文件

三个节点配置相同(elk-1节点为例)

[root@elk-1 ~]# vim /etc/hosts
[root@elk-1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.200.10 elk-1
192.168.200.20 elk-2
192.168.200.30 elk-3

配置完hosts用scp命令把hosts里的配置文件复制到elk-2和elk-3下
[root@elk-1 ~]# scp /etc/hosts elk-2:/etc/hosts
The authenticity of host 'elk-2 (192.168.200.20)' can't be established.
ECDSA key fingerprint is SHA256:dxl72pjdE9IagOFsebD4+gNRSPEWnHLyDWMqpg9nNBo.
ECDSA key fingerprint is MD5:9c:7d:69:67:4a:93:93:12:4b:f0:ac:4a:1e:4f:1e:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'elk-2,192.168.200.20' (ECDSA) to the list of known hosts.
root@elk-2's password: 
hosts                                                                                     100%  222   150.6KB/s   00:00

4. 安装 JDK

部署ELK环境需要jdk1.8以上的JDK版本软件环境，我们使⽤opnejdk1.8，3节点全部安装（以 elk-1节点为例），命令如下:

[root@elk-1 ~]# yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel net tools 
[root@elk-1 ~]# java -version	#查看jdk版本
openjdk version "1.8.0_322"
OpenJDK Runtime Environment (build 1.8.0_322-b06)
OpenJDK 64-Bit Server VM (build 25.322-b06, mixed mode)

5. Elasticserach安装

下载 elasticserach-6.0.0.rpm 包

[root@elk-1 ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.0.0.rpm
--2022-04-12 15:11:29--  https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.0.0.rpm
正在解析主机 artifacts.elastic.co (artifacts.elastic.co)... 34.120.127.130, 2600:1901:0:1d7::
正在连接 artifacts.elastic.co (artifacts.elastic.co)|34.120.127.130|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：27970243 (27M) [binary/octet-stream]
正在保存至: “elasticsearch-6.0.0.rpm”

75% [=============================================================>                     ] 21,159,094  21.9KB/s 剩余 1m 46s +
100%[==================================================================================>] 27,970,243  65.4KB/s 用时 6m 49s 

2022-04-12 15:18:26 (66.8 KB/s) - 已保存 “elasticsearch-6.0.0.rpm” [27970243/27970243])

用scp命令把elasticserach-6.0.0.rpm包复制到elk-2和elk-3节点中
[root@elk-1 ~]# scp elasticsearch-6.0.0.rpm elk-2:/root/
root@elk-2's password: 
elasticsearch-6.0.0.rpm                                                                   100%   27MB  81.8MB/s   00:00    
[root@elk-1 ~]# scp elasticsearch-6.0.0.rpm elk-3:/root/
root@elk-3's password: 
elasticsearch-6.0.0.rpm                                                                   100%   27MB 108.1MB/s   00:00

安装Elasticserach

[root@elk-1 ~]# rpm -ivh elasticsearch-6.0.0.rpm 		#i 表示安装，v 表示显示安装过程，h 表示显示进度
警告：elasticsearch-6.0.0.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
准备中...                          ################################# [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
正在升级/安装...
   1:elasticsearch-0:6.0.0-1          ################################# [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

6. 配置 Elasticserach

配置elasticsearch的配置⽂件，配置⽂件：/etc/elasticsearch/elasticsearch.yml elk-1节点：增加以下红⾊字样（//为解释，这⾥⽤不到的配置⽂件被删除），注意IP

[root@elk-1 ~]# vim /etc/elasticsearch/elasticsearch.yml
[root@elk-1 ~]# cat /etc/elasticsearch/elasticsearch.yml 
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: ELK		#集群名称
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
node.name: elk-1		#节点名
node.master: true		#指定该节点是否有资格被选举成为master，默认是true，es是默认集群中的第⼀台机器为master，
						如果这台机挂了就会重新选举master
node.data: false		#指定该节点是否存储索引数据，默认为true。其他两节点为true
#node.name: node-1	
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch	#索引数据存储位置(保持默认,不要开启注释)
#
# Path to log files:
#
path.logs: /var/log/elasticsearch	#设置日志文件的存储路径，默认是es根目录下的logs文件夹
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
network.host: 192.168.200.10	#设置绑定的ip地址，可以是ipv4或ipv6的，默认为0.0.0.0。
# Set a custom port for HTTP:
#
#http.port: 9200
 http.port: 9200		#启动的es对外访问的http端口，默认9200
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
discovery.zen.ping.unicast.hosts: ["elk-1", "elk-2","elk-3"]	#设置集群中master节点的初始列表，可以通过这些节点来⾃动发现新加⼊集群的节点。
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

elk-2节点

[root@elk-2 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: ELK #设置集群的名称
node.name: elk-2	#设置节点名称
node.master: false	 
node.data: true		#指定该节点是否存储索引数据
network.host: 192.168.200.20	#设置绑定的ip地址，可以是ipv4或ipv6的，默认为0.0.0.0。
http.port: 9200		#es对外访问的http端口，默认9200
discovery.zen.ping.unicast.hosts: ["elk-1","elk-2","elk-3"]

elk-3节点

[root@elk-3 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: ELK #设置集群的名称
node.name: elk-3	#设置节点名称
node.master: false
node.data: true		#指定该节点是否存储索引数据
network.host: 192.168.200.30	#设置绑定的ip地址，可以是ipv4或ipv6的，默认为0.0.0.0。
http.port: 9200		#es对外访问的http端口，默认9200
discovery.zen.ping.unicast.hosts: ["elk-1","elk-2","elk-3"]

7. 启动服务(三个节点)

通过命令启动es服务，启动后使用ps命令查看进程是否存在或者使用netstat命令查看是否端口启动。命令如下：（3个节点命令相同）

[root@elk-1 ~]# systemctl restart elasticsearch
[root@elk-1 ~]# ps -ef |grep elasticsearch
elastic+  11880      1 79 16:21 ?        00:00:11 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
root      11956   1943  0 16:22 pts/0    00:00:00 grep --color=auto elasticsearch
[root@elk-1 ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1057/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1310/master         
tcp6       0      0 192.168.200.10:9200     :::*                    LISTEN      11880/java          
tcp6       0      0 192.168.200.10:9300     :::*                    LISTEN      11880/java          
tcp6       0      0 :::22                   :::*                    LISTEN      1057/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1310/master

8. 检测集群状态

[root@elk-1 ~]# curl '192.168.200.10:9200/_cluster/health?pretty'
{
  "cluster_name" : "ELK",
  "status" : "green",	#为green则代表健康没问题，yellow或者red 则是集群有问题
  "timed_out" : false,	#是否有超时
  "number_of_nodes" : 3,	#集群中的节点数量
  "number_of_data_nodes" : 2, #集群中的data节点的数量
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

9. 部署 Kibana

在elk-1节点下载kibana包

[root@elk-1 ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-6.0.0-x86_64.rpm
--2022-04-12 16:35:25--  https://artifacts.elastic.co/downloads/kibana/kibana-6.0.0-x86_64.rpm
正在解析主机 artifacts.elastic.co (artifacts.elastic.co)... 34.120.127.130, 2600:1901:0:1d7::
正在连接 artifacts.elastic.co (artifacts.elastic.co)|34.120.127.130|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：63979183 (61M) [binary/octet-stream]
正在保存至: “kibana-6.0.0-x86_64.rpm”

100%[==================================================================================>] 63,979,183  6.37MB/s 用时 17s    

2022-04-12 16:35:53 (3.69 MB/s) - 已保存 “kibana-6.0.0-x86_64.rpm” [63979183/63979183])

安装kibana

[root@elk-1 ~]# rpm -ivh kibana-6.0.0-x86_64.rpm 
警告：kibana-6.0.0-x86_64.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:kibana-6.0.0-1                   ################################# [100%]

配置监听

[root@elk-1 ~]# vim /etc/nginx.conf
upstream elasticsearch {
         zone elasticsearch 64K;
         server elk-1:9200;
         server elk-2:9200;
         server elk-3:9200;
}

server {
         listen 8080;
         server_name 192.168.200.10;
         location / {
         proxy_pass http://elasticsearch;
         proxy_redirect off;
         proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         }
         access_log /var/log/es_access.log;
        }

配置Kibana的配置⽂件，配置⽂件在/etc/kibana/kibana.yml，在配置⽂件增加或修改以下内容：

[root@elk-1 ~]# vim /etc/kibana/kibana.yml
server.port: 5601
server.host: 192.168.200.10
elasticsearch.url: "http://192.168.200.10:9200"

[root@elk-1 ~]# systemctl start kibana
[root@elk-1 ~]# systemctl restart kibana
[root@elk-1 ~]# ps -ef |grep kibana
kibana    12124      1 18 16:56 ?        00:00:01 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
root      12136   1943  0 16:56 pts/0    00:00:00 grep --color=auto kibana
[root@elk-1 ~]# netstat -ntpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 192.168.200.10:5601     0.0.0.0:*               LISTEN      12124/node          
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1057/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1310/master         
tcp6       0      0 192.168.200.10:9200     :::*                    LISTEN      11880/java          
tcp6       0      0 192.168.200.10:9300     :::*                    LISTEN      11880/java          
tcp6       0      0 :::22                   :::*                    LISTEN      1057/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1310/master

启动成功后网页访问，可以访问到如下界面

![批注 2022-04-12 172244](D:\桌面\批注 2022-04-12 172244.png)

10. 安装配置logstash

1.在elk-2节点安装logstash

[root@elk-2 ~]# wget  https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.rpm
--2022-04-12 17:28:14--  https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.rpm
正在解析主机 artifacts.elastic.co (artifacts.elastic.co)... 34.120.127.130, 2600:1901:0:1d7::
正在连接 artifacts.elastic.co (artifacts.elastic.co)|34.120.127.130|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：113288712 (108M) [binary/octet-stream]
正在保存至: “logstash-6.0.0.rpm”

100%[========================================================================================>] 113,288,712 49.3MB/s 用时 2.2s   

2022-04-12 17:28:17 (49.3 MB/s) - 已保存 “logstash-6.0.0.rpm” [113288712/113288712])


[root@elk-2 ~]# rpm -ivh logstash-6.0.0.rpm 	#i 安装 v 显示安装过程 h 显示进度
警告：logstash-6.0.0.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:logstash-1:6.0.0-1               ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash

增加logstash权限

[root@elk-2 ~]# chmod 644 /var/log/messages 
[root@elk-2 ~]# chown -R logstash:logstash /var/log/logstash/
[root@elk-2 ~]# chown -R logstash /var/lib/logstash/

配置logstash收集syslog日志

[root@elk-2 ~]# chmod 644 /var/log/messages 
[root@elk-2 ~]# chown -R logstash:logstash /var/log/logstash/
[root@elk-2 ~]# chown -R logstash /var/lib/logstash/
[root@elk-2 ~]# vim /etc/logstash/logstash.yml 
http.host: "192.168.200.20"
[root@elk-2 ~]# vim /etc/rsyslog.conf 
#### RULES ####
*.* @@192.168.200.20:10514

[root@elk-2 ~]# systemctl restart rsyslog
[root@elk-2 ~]# vim /etc/logstash/conf.d/syslog.conf
input {         #定义日志源
    file {
        path => "/var/log/messages"     #定义日志来源路径  目录要给644权限，不然无法读取日志
        type => "systemlog"   #定义类型
        start_position => "beginning"
        stat_interval => "3"
    }
}
output {   #定义日志输出
            elasticsearch {
            hosts => ["192.168.200.20:9200"]
            index => "system-log-%{+YYYY.MM.dd}"
        }
  }

[root@elk-2 ~]# ln -s /usr/share/logstash/bin/logstash /usr/bin/	#创建软连接，方便使用logstash命令
[root@elk-2 ~]# logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK		#为ok则没问题
[root@elk-2 ~]# systemctl restart logstash

--path.settings : 用于指定logstash的配置文件所在的目录
-f : 指定需要被检测的配置文件的路径
--config.test_and_exit : 指定检测完之后就退出，不然就会直接启动了

2. 启动服务

[root@elk-2 ~]# netstat -ntpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      952/sshd            
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1150/master         
tcp6       0      0 192.168.200.20:9600     :::*                    LISTEN      3486/java           
tcp6       0      0 192.168.200.20:9200     :::*                    LISTEN      3383/java           
tcp6       0      0 192.168.200.20:9300     :::*                    LISTEN      3383/java           
tcp6       0      0 :::22                   :::*                    LISTEN      952/sshd            
tcp6       0      0 ::1:25                  :::*                    LISTEN      1150/master         

[root@elk-2 ~]# curl '192.168.200.20:9200/_cat/indices?v'
health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana               BOee2fBHSB2E4se-M7nbhg   1   1          1            0      7.3kb          3.6kb
green  open   system-log-2022.04.13 S6wBvb8sSMGtmqU_-T58TQ   5   1     305036            0    104.5mb         52.2mb

# 有system-log则代表成功。



#获取/删除指定索引详细信息
curl -XGET/DELETE '192.168.200.10:9200/system-log-2022.04.13?pretty'	
[root@elk-1 ~]# curl -XGET '192.168.200.10:9200/system-log-2022.04.13?pretty'
{
  "system-log-2022.04.13" : {
    "aliases" : { },
    "mappings" : {
      "systemlog" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "@version" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "host" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "message" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "path" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "type" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "creation_date" : "1649818767880",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "S6wBvb8sSMGtmqU_-T58TQ",
        "version" : {
          "created" : "6000099"
        },
        "provided_name" : "system-log-2022.04.13"
      }
    }
  }
}

配置system Web监听

配置完成后，选择Discover，进入“Discover”页面后，无法查找到日志信息,这种情况一般是时间的问题，单击右上角信息切换成查看当天的日志信息即可。

10. Logstash收集Nginx日志

#elk-2上操作

[root@elk-2 ~]# wget http://nginx.org/packages/centos/7/x86_64/RPMS/nginx-1.16.1-1.el7.ngx.x86_64.rpm

--2022-04-13 12:31:18--  http://nginx.org/packages/centos/7/x86_64/RPMS/nginx-1.16.1-1.el7.ngx.x86_64.rpm
正在解析主机 nginx.org (nginx.org)... 3.125.197.172, 52.58.199.22, 2a05:d014:edb:5704::6, ...
正在连接 nginx.org (nginx.org)|3.125.197.172|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：784272 (766K) [application/x-redhat-package-manager]
正在保存至: “nginx-1.16.1-1.el7.ngx.x86_64.rpm”

100%[=================================================================>] 784,272      101KB/s 用时 8.1s   

2022-04-13 12:31:27 (94.1 KB/s) - 已保存 “nginx-1.16.1-1.el7.ngx.x86_64.rpm” [784272/784272])


[root@elk-2 ~]# rpm -ivh nginx-1.16.1-1.el7.ngx.x86_64.rpm 
警告：nginx-1.16.1-1.el7.ngx.x86_64.rpm: 头V4 RSA/SHA1 Signature, 密钥 ID 7bd9bf62: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:nginx-1:1.16.1-1.el7.ngx         ################################# [100%]
----------------------------------------------------------------------

Thanks for using nginx!

Please find the official documentation for nginx here:
* http://nginx.org/en/docs/

Please subscribe to nginx-announce mailing list to get
the most important news about nginx:
* http://nginx.org/en/support.html

Commercial subscriptions for nginx are available on:
* http://nginx.com/products/

----------------------------------------------------------------------


[root@elk-2 ~]# vim /etc/logstash/conf.d/nginx.conf 
[root@elk-2 ~]# cat /etc/logstash/conf.d/nginx.conf 
input {
  file {
    path => "/tmp/elk_access.log"
    start_position => "beginning"
    type => "nginx"
  }
}
filter {
    grok {
        match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    stdout { codec => rubydebug }
    elasticsearch {
        hosts => ["192.168.200.20:9200"]
	index => "nginx-test-%{+YYYY.MM.dd}"
  }
}

[root@elk-2 ~]# logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf --config.test_and_exit
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK
# 为ok则没有问题

[root@elk-2 ~]# vim /etc/nginx/conf.d/elk.conf 
[root@elk-2 ~]# cat /etc/nginx/conf.d/elk.conf 
server {
      listen 80;
      server_name elk.com;
      location / {
          proxy_pass      http://192.168.200.10:5601;
          proxy_set_header Host   $host;
          proxy_set_header X-Real-IP      $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      }
      access_log  /tmp/elk_access.log main2;
}

[root@elk-2 ~]# echo "192.168.200.20 elk.com" >> /etc/hosts #添加hosts配置
[root@elk-2 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.200.10 elk-1
192.168.200.20 elk-2
192.168.200.30 elk-3

192.168.200.20 elk.com

[root@elk-2 ~]# vim /etc/nginx/nginx.conf  #额外添加以下内容

  log_format main2 '$http_host $remote_addr - $remote_user
[$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$upstream_addr"
$request_time';

access_log  /var/log/nginx/access.log  main2;

[root@elk-2 ~]# systemctl start nginx
[root@elk-2 ~]# systemctl restart nginx
[root@elk-2 ~]# systemctl status nginx
● nginx.service - nginx - high performance web server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
   Active: active (running) since 三 2022-04-13 12:52:25 CST; 3s ago
     Docs: http://nginx.org/en/docs/
  Process: 9327 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=0/SUCCESS)
 Main PID: 9328 (nginx)
   CGroup: /system.slice/nginx.service
           ├─9328 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
           └─9329 nginx: worker process

4月 13 12:52:25 elk-2 systemd[1]: Starting nginx - high performance web server...
4月 13 12:52:25 elk-2 systemd[1]: Started nginx - high performance web server.

[root@elk-2 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

[root@elk-2 ~]# curl elk.com
<script>var hashRoute = '/app/kibana';
var defaultRoute = '/app/kibana';

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}</script>


[root@elk-2 ~]# netstat -ntpl	#查看是否有9200端口
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      11393/nginx: master 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      952/sshd            
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1150/master         
tcp6       0      0 192.168.200.20:9600     :::*                    LISTEN      11260/java          
tcp6       0      0 192.168.200.20:9200     :::*                    LISTEN      3383/java           
tcp6       0      0 192.168.200.20:9300     :::*                    LISTEN      3383/java           
tcp6       0      0 :::22                   :::*                    LISTEN      952/sshd            
tcp6       0      0 ::1:25                  :::*                    LISTEN      1150/master         

[root@elk-2 ~]# curl 'elk-2:9200/_cat/indices?v'
health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   nginx-test-2022.04.13 gbk8HHdJTUaRtO2Fy3Ik0w   5   1       4238            0      1.3mb        670.1kb
green  open   .kibana               BOee2fBHSB2E4se-M7nbhg   1   1          2            0       14kb            7kb
green  open   system-log-2022.04.13 S6wBvb8sSMGtmqU_-T58TQ   5   1     309284            0    208.3mb        104.1mb

web界面配置

浏览器访问192.168.200.10:5601，到Kibana上配置索引

选择nginx 如果出现无法查看到日志信息，这种情况一般是时间问题，单击右上角信息切换查看当天的日志信息，修改一下时间即可。

11. Beat 采集日志

[root@elk-3 ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-x86_64.rpm
--2022-04-13 13:29:15--  https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-x86_64.rpm
正在解析主机 artifacts.elastic.co (artifacts.elastic.co)... 34.120.127.130, 2600:1901:0:1d7::
正在连接 artifacts.elastic.co (artifacts.elastic.co)|34.120.127.130|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：11988378 (11M) [binary/octet-stream]
正在保存至: “filebeat-6.0.0-x86_64.rpm”

100%[=============================================================================================================>] 11,988,378  4.26MB/s 用时 2.7s   

2022-04-13 13:29:19 (4.26 MB/s) - 已保存 “filebeat-6.0.0-x86_64.rpm” [11988378/11988378])


[root@elk-3 ~]# rpm --install filebeat-6.0.0-x86_64.rpm 
警告：filebeat-6.0.0-x86_64.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY


[root@elk-3 ~]# vim /etc/filebeat/filebeat.yml

filebeat.prospectors:
enabled: true
paths:
    - /var/log/yum.log	#此处可自行改为想要监听的日志文件
output.elasticsearch:
  hosts: ["elk-1:9200","elk-2:9200","elk-3:9200"]
  
  
[root@elk-3 ~]# systemctl start filebeat
[root@elk-3 ~]# curl 'elk-1:9200/_cat/indices?v'
health status index                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   filebeat-6.0.0-2022.04.13 Qqp1F2hWT02ZRctFC7qnNg   3   1        101            0     73.4kb         39.6kb
green  open   .kibana                   BOee2fBHSB2E4se-M7nbhg   1   1          3            0     24.9kb         12.4kb
green  open   nginx-test-2022.04.13     gbk8HHdJTUaRtO2Fy3Ik0w   5   1       5992            0      1.9mb            1mb
green  open   system-log-2022.04.13     S6wBvb8sSMGtmqU_-T58TQ   5   1     311038            0    104.6mb         52.3mb

按照上面的步骤在浏览器中添加filebeat日志文件的监控。