FileBeat + Logstash + Elasticsearch + Grafana 日志监控

1. filebeat 配置

====================Filebeat inputs==================
- type: log

  paths:
    - /data/log/project_name/*.log
  tags: ["project_name"]

- type: log
  paths:
    - /data/log/project_name/*.log
  tags: ["project_name"]

 

[program:filebeat]
command = /opt/sites/filebeat/filebeat -e -c /opt/sites/filebeat/filebeat.yml
process_name=%(process_num)d
stopsignal=KILL
user=root
redirect_stderr=true
stdout_logfile_maxbytes=5MB
stdout_logfile_backups=20
stdout_logfile=/data/log/filebeat/filebeat.log

 

2. logstash 配置

input {
  beats {
    port => 6068
  }
}
filter {

  if "project_name" in [tags] {
    #不符合条件的删除 每条日志中必须要具有log_json
    if ([message] =~  "^(?!.*?log_json).*$") {
        drop {}
    }

    json {
        source => "message"
        remove_field => ["message"]
    }
    date {
        match => ["time", "UNIX_MS"]
        target => "@timestamp"
    }

  }
}

output { # 按日志tags的不同存入es中不同的index中
  if "project_name" in [tags] {
    elasticsearch {
      hosts => ["127.0.0.1:9200"]
      # 索引名字
      index => "project_name.log"
    }
  }

  stdout {}
}

 

 4. Grafana 安装：

wget https://dl.grafana.com/oss/release/grafana-6.7.1-1.x86_64.rpm
sudo yum install grafana-6.7.1-1.x86_64.rpm

centos6 启动命令

启动：service grafana-server start
停止：service grafana-server stop
重启：service grafana-server restart
加入开机自启动： chkconfig --add grafana-server on

centos7 启动命令

systemctl start grafana-server

 5. Elasticsearch 安装

1. 安装 ElasticSearch 7.8
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.8.0-x86_64.rpm
rpm -ivh elasticsearch-7.8.0-x86_64.rpm

2.设置开机启动
systemctl daemon-reload      # 重新加载systemd程序的配置文件
systemctl enable elasticsearch.service     # 设置开机自启动
systemctl start elasticsearch.service      # 启动服务

3. 验证：
curl -XGET http://localhost:9200

5.elasticsearch.yml 是es的配置文件
node.name: node-1
cluster.initial_master_nodes: ["node-1"]
network.host: 0.0.0.0
http.port: 9200

防火墙开放9200端口，云服务器安全组规则里开放9200端口

 

方法二：
1. 安装 ElasticSearch 
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.0.0-linux-x86_64.tar.gz 
tar -zxvf elasticsearch-8.0.0-linux-x86_64.tar.gz 进行解压缩

2.修改 config/elasticsearch.yml 配置文件

vim /etc/elasticsearch/elasticsearch.yml

cluster.name: my-application
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
cluster.initial_master_nodes: ["node-1"]

3. es5之后的都不能使用添加启动参数或者修改配置文件等方法启动了，必须要创建用户， 
创建 ElasticSearch 用户：adduser es
将对应的文件夹权限赋给该用户 chown -R es elasticsearch-8.0.0

4. 修改配置文件
vi /etc/security/limits.conf
在文末添加：

  es soft nofile 65536
  es hard nofile 65536
  es soft nproc 4096
  es hard nproc 4096

  vi /etc/sysctl.conf

  在文末添加：

  vm.max_map_count=655360

  然后执行：sysctl -p

  5.配置java环境

  vi /etc/profile

  在文末添加：

  export JAVA_HOME=/opt/sites/elasticsearch-8.0.0/jdk
  export PATH=$PATH:${JAVA_HOME}/bin

  然后执行：source /etc/profile

6. supervisorct 配置
[program:elasticsearch]
command=/opt/sites/elasticsearch-8.0.0/bin/elasticsearch
numprocs=1
directory=/opt/sites/elasticsearch-8.0.0
stopsignal=INT
user=elasticsearch
redirect_stderr=true
stdout_logfile_maxbytes=100MB
stdout_logfile_backups=20
stdout_logfile=/data/log/elasticsearch/super.log

 

 

6. kibana 安装

1. kibana 下载
https://www.elastic.co/cn/downloads/past-releases/kibana-7-8-0
kibana版本和es版本尽量一致
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.8.0-x86_64.rpm

2.解压
tar -zxvf kibana-7.8.0-linux-x86_64.tar.gz
rpm -ivh kibana-7.8.0-x86_64.rpm

3. 导入密钥：
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

4. 设置开机重启：

   systemctl enable kibana.service

  5. 启动 

   systemctl start kibana.service

6.修改配置文件
vim config/kibana.yml
server.port: 9201
server.host: "0.0.0.0" #对外暴露服务的地址
elasticsearch.url: "http://127.0.0.1:9200" #配置Elasticsearch

7.启动
./bin/kibana

8.通过浏览器进行访问
http://0.0.0.0:9201/app/kibana