before#

centos7.9 + chrome + elasticsearch-6.8.15 + kibana-6.8.15 + filebeat-6.8.15 + nginx

本篇通过kibana演示各种图表制作，比如统计指定时间段内网站访问量前十的IP、URL等等，通过这些图表，可以帮助我们分析网站的使用情况。且这些图表是实时动态更新的。

首先，保证es和kibana以及elasticsearch-head都已经配置好了。

elasticsearch安装：https://www.cnblogs.com/Neeo/articles/10368280.html#rpm
elasticsearch-head：https://www.cnblogs.com/Neeo/articles/9146695.html#chrome
kibana安装：https://www.cnblogs.com/Neeo/articles/10623632.html#rpm
filebeat安装：https://www.cnblogs.com/Neeo/articles/9017936.html

开始吧！

必要的准备#

具体步骤

nginx日志准备
首先感谢路飞学城提供的真实的nginx访问日志。
日志内容记录的是从2021/4/20~2021/4/23号，这三天的部分日志，大约5万条。且我对日志数据做了清洗，然后处理成json格式的日志，方便kibana分析日志信息。唯一不好之处，就是数据是死的，展示不出动态的效果。
链接：https://pan.baidu.com/s/12z6qvhdJKzw_YgPTZEa1uA 提取码：rwwf
日志内容如下：

{"remote_addr": "111.18.75.210", "time_local": "20/Apr/2021:10:35:35 +0800", "request": "GET /_nuxt/6288107c3da5b062eedb.js HTTP/1.1", "status": "200", "bytes": "1038", "referer": "https://www.luffycity.com/study/degree", "agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.72", "equipment": "Safari/537.36"}
{"remote_addr": "1.198.17.134", "time_local": "21/Apr/2021:16:00:43 +0800", "request": "GET /_nuxt/dd9ba08d41a9b6adc600.js HTTP/1.1", "status": "200", "bytes": "10700", "referer": "https://www.luffycity.com/study/degree", "agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114", "equipment": "Safari/537.36"}
{"remote_addr": "111.225.149.16", "time_local": "22/Apr/2021:04:30:42 +0800", "request": "GET /python-book/assets/MySQL-7-11.png HTTP/1.1", "status": "301", "bytes": "178", "referer": "-", "agent": "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider;", "equipment": "https://zhanzhang.toutiao.com/)"}
{"remote_addr": "223.64.64.147", "time_local": "23/Apr/2021:14:19:30 +0800", "request": "GET /_nuxt/2e522c735bc48a7b89e4.js HTTP/1.1", "status": "200", "bytes": "5405", "referer": "https://www.luffycity.com/study/degree", "agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36", "equipment": "Edg/90.0.818.42"}

对日志内容有了大概的了解，我们开始配置吧。
也没啥配置的，就是把日志文件上传到了服务器：

[root@cs tmp]# ls ng*
nginx_json.log
[root@cs tmp]# pwd
/tmp
[root@cs tmp]# wc -l nginx_json.log 
49879 nginx_json.log

OK了。

filebeat配置
filebeat配置也相当简单：

[root@cs tmp]# vim /etc/filebeat/filebeat.yml
[root@cs tmp]# cat /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/nginx_json.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["icon"]

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

setup.kibana:

output.elasticsearch:
  hosts: ["10.0.0.200:9200"]
  indices:
    - index: "nginx-icon-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "icon"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

完事之后，重启filebeat：

[root@cs tmp]# systemctl restart filebeat
[root@cs tmp]# tail -f /var/log/filebeat/filebeat
2021-04-27T16:10:16.860+0800	INFO	input/input.go:114	Starting input of type: log; ID: 10100389906076279661 
2021-04-27T16:10:16.861+0800	INFO	log/input.go:148	Configured paths: [/tmp/nginx_json.log]
2021-04-27T16:10:16.861+0800	INFO	input/input.go:114	Starting input of type: log; ID: 12788937719430288701 
2021-04-27T16:10:16.862+0800	INFO	log/input.go:148	Configured paths: [/var/log/elasticsearch/elasticsearch.log]
2021-04-27T16:10:16.862+0800	INFO	input/input.go:114	Starting input of type: log; ID: 335975315166146607 
2021-04-27T16:10:16.862+0800	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 4
2021-04-27T16:10:16.869+0800	INFO	cfgfile/reload.go:150	Config reloader started
2021-04-27T16:10:16.869+0800	INFO	log/harvester.go:255	Harvester started for file: /tmp/nginx_json.log
2021-04-27T16:10:16.869+0800	INFO	cfgfile/reload.go:205	Loading of config files completed.
2021-04-27T16:10:19.852+0800	INFO	add_cloud_metadata/add_cloud_metadata.go:340	add_cloud_metadata: hosting provider type not detected.

ok，filebeat也没问题。

kibana准备
filebeat正常运行，说明它读到日志内容并且发送到了es种，我们在kibana中添加索引模式(es中已经存在了索引，这一步是为了在kibana中使用)。

下面这一步，一定要等kibana自己完成，不要着急手动去点点。

完事：

然后在discover栏，就可以看到了。