ELK-Stack 最后一次全篇文档
简介：
ELK-Stack 日志收集系统。最后一次全篇记录的笔记，之后关于 ELK 的笔记都将是片段型、针对性的。
环境介绍：
ELK-Stack：192.168.1.25 ( Redis、LogStash、Kibana )
ES-1: 192.168.1.26 ( ElasticSearch )
ES-2: 192.168.1.27 ( ElasticSearch )
Node: *.*.*.* ( LogStash )
工作流程：
1、Node:LogStash 收集日志发往 --> ELK-Stack:Redis
2、ELK-Stack:LogStash 将 Redis 消息队列中的数据取出发往 --> ElasticSearch
3、ELK-Stack:Kibana 去 ElasticSearch 读取数据并展示
# 接下来我们将按 ELK 工作流程来搭建并配置环境
一、Node:LogStash 客户端收集日志
shell > yum -y install java shell > cd /usr/local/src shell > wget https://download.elastic.co/logstash/logstash/logstash-2.4.0.tar.gz shell > tar zxf logstash-2.4.0.tar.gz shell > mv logstash-2.4.0 ../; cd ~ shell > vim /usr/local/logstash-2.4.0/logstash.conf # Logstash input { file { type => "nginx_access" path => ["/tmp/access.log"] start_position => "beginning" } } output { stdout { codec => rubydebug } }
# input 插件定义从哪里读取日志，output 插件定义将日志输出到哪里。
shell > /usr/local/logstash-2.4.0/bin/logstash -f /usr/local/logstash-2.4.0/logstash.conf -t # 测试配置文件正确 Configuration OK shell > /usr/local/logstash-2.4.0/bin/logstash -f /usr/local/logstash-2.4.0/logstash.conf & # 启动
# 进入测试阶段，导入一段日志文件，查看是否有输出
{ "message" => "125.118.134.85 - - [18/Sep/2016:14:06:58 +0800] \"POST / HTTP/1.0\" 200 8317 \"-\" \"-\" \"ptv.giv.tv\" \"-\" cid=3&tvtoken=78%3A0a%3Ac7%3A01%3A7b%3Ae1&year=0&tag=0&version=1.0&area=0&method=bftv.video.list&extend=isVip&sn=600000MW600T165J1764_5188&page=4&pageSize=20&apptoken=282340ce12c5e10fa84171660a2054f8&time=1474178818312&channel=0&hot=1&plateForm=bftv_android \"125.118.134.85\"", "@version" => "1", "@timestamp" => "2016-09-18T06:07:30.425Z", "path" => "/tmp/access.log", "host" => "king", "type" => "nginx_access" } { "message" => "59.172.199.235 - - [18/Sep/2016:14:06:58 +0800] \"POST / HTTP/1.0\" 200 822 \"-\" \"-\" \"ptv.giv.tv\" \"-\" tvtoken=78%3A0a%3Ac7%3A04%3Aa9%3Ac0&version=1.0&method=bftv.tv.topicpiclist&sn=600000MW301D167B3808_214E&apptoken=282340ce12c5e10fa84171660a2054f8&time=1474178817053&plateForm=bftv_android \"59.172.199.235\"", "@version" => "1", "@timestamp" => "2016-09-18T06:07:30.425Z", "path" => "/tmp/access.log", "host" => "king", "type" => "nginx_access" }
# 屏幕立即出现数据，说明客户端 LogStash 配置无误。
二、ELK-Stack:Redis 客户端将日志写入 Redis
shell > yum -y install gcc shell > cd /usr/local/src shell > wget http://download.redis.io/releases/redis-3.2.3.tar.gz shell > tar zxf redis-3.2.3.tar.gz shell > cd redis-3.2.3; make; make install shell > mkdir /usr/local/redis-3.2.3; cp redis.conf ../../redis-3.2.3; cd ~ shell > sed -i 's/bind 127.0.0.1/bind 127.0.0.1 192.168.1.25/' /usr/local/redis-3.2.3/redis.conf shell > sed -i '/daemonize/s/no/yes/' /usr/local/redis-3.2.3/redis.conf shell > sed -i 's#dir ./#dir /data/redis_data#' /usr/local/redis-3.2.3/redis.conf shell > redis-server /usr/local/redis-3.2.3/redis.conf shell > redis-cli ping PONG shell > redis-cli -h 192.168.1.25 ping PONG
# 进入测试阶段，修改客户端 LogStash 配置文件，使其收集到的日志输出到该 Redis 中。
shell > vim /usr/local/logstash-2.4.0/logstash.conf # Logstash input { file { type => "nginx_access" path => ["/tmp/access.log"] start_position => "beginning" } } output { # stdout { # codec => rubydebug # } redis { host => "192.168.1.25" port => 6379 data_type => "list" key => "logstash-list" } }
# 回到 ELK-Stack:Redis 上查看数据，结果为空... 问题不大，因为客户端 LogStash 上次读取的日志文件是输出到屏幕，而不是 Redis
shell > redis-cli 127.0.0.1:6379> keys * (empty list or set)
# 重新导入一段日志测试
shell > redis-cli 127.0.0.1:6379> keys * 1) "logstash-list"
# 再次查看，redis 中已经有数据了。可以证明，工作流程中第一步正常。
shell > lrange logstash-list 0 0 1) "{\"message\":\"58.39.181.33 - - [18/Sep/2016:14:06:45 +0800] \\\"POST / HTTP/1.0\\\" 200 352 \\\"-\\\" \\\"-\\\" \\\"ptv.giv.tv\\\" \\\"-\\\" tvtoken=78%3A0a%3Ac7%3A01%3A49%3A97&requestplatform=4&version=1.0&method=core.pic.list&sn=600000MW300D165J0484_8B59&apptoken=282340ce12c5e10fa84171660a2054f8&type=10&time=1474178780284&plateForm=bftv_android \\\"58.39.181.33\\\"\",\"@version\":\"1\",\"@timestamp\":\"2016-09-18T06:37:46.961Z\",\"path\":\"/tmp/access.log\",\"host\":\"king\",\"type\":\"nginx_access\"}"
# key:logstash-list 中第一个值得信息如上，正式日志信息。
# 查看该 key 的所有值：lrange logstash-list 0 -1 ，发现第一个到最后一个总共有 110 条，然后 wc -l < /tmp/access.log 发现也是 110 条，证明数据无误。
三、ELK-Stash:LogStash 服务端 LogStash 从 Redis 读取数据并存入 ElasticSearch
shell > vim /usr/local/logstash-2.4.0/logstash.conf # Logstash.conf input { redis { host => "127.0.0.1" port => 6379 data_type => "list" key => "logstash-list" } } output { stdout { codec => rubydebug } } shell > /usr/local/logstash-2.4.0/bin/logstash -f /usr/local/logstash-2.4.0/logstash.conf -t Configuration OK shell > /usr/local/logstash-2.4.0/bin/logstash -f /usr/local/logstash-2.4.0/logstash.conf # 调试
# 再次追加日志，查看有无输出
"message" => "125.118.134.85 - - [18/Sep/2016:14:06:58 +0800] \"POST / HTTP/1.0\" 200 8317 \"-\" \"-\" \"ptv.giv.tv\" \"-\" cid=3&tvtoken=78%3A0a%3Ac7%3A01%3A7b%3Ae1&year=0&tag=0&version=1.0&area=ideo.list&extend=isVip&sn=600000MW600T165J1764_5188&page=4&pageSize=20&apptoken=282340ce12c5e10fa84171660a2054f8&time=1474178818312&channel=0&hot=1&plateForm=bftv_android \"125.", "@version" => "1", "@timestamp" => "2016-09-18T08:36:11.621Z", "path" => "/tmp/access.log", "host" => "king", "type" => "nginx_access" } { "message" => "59.172.199.235 - - [18/Sep/2016:14:06:58 +0800] \"POST / HTTP/1.0\" 200 822 \"-\" \"-\" \"ptv.gv.tv\" \"-\" tvtoken=78%3A0a%3Ac7%3A04%3Aa9%3Ac0&version=1.0&method=bftv.tv.topicpicli01D167B3808_214E&apptoken=282340ce12c5e10fa84171660a2054f8&time=1474178817053&plateForm=bftv_android \"59.172.199.235\"", "@version" => "1", "@timestamp" => "2016-09-18T08:36:11.622Z", "path" => "/tmp/access.log", "host" => "king", "type" => "nginx_access" }
# 至此，说明 ELK-Stack:LogStash 可以正常从 ELK-Stack:redis 读取数据，下面将读取到的数据写入到 ElasticSearch 。
# 注意：这只是个示例，将来还要将日志条目拆分，然后写入到 Kibana 才能更好的分析日志 ( filter )
# http://kibana.logstash.es/content/
shell > yum -y install java shell > cd /usr/local/src shell > wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.0/elasticsearch-2.4.0.tar.gz shell > tar zxf elasticsearch-2.4.0.tar.gz shell > mv elasticsearch-2.4.0 ../; cd ~ shell > useradd elast shell > chown -R elast.elast /usr/local/elasticsearch-2.4.0 shell > vim /usr/local/elasticsearch-2.4.0/config/elasticsearch.yml # 编辑配置文件 cluster.name: my-elk # 集群名称 node.name: node-1 # 节点名称 path.data: /data/esearch/data # 数据存放路径 path.logs: /data/esearch/logs # 日志 network.host: 192.168.1.26 # 监听地址 http.port: 9200 # 监听端口 discovery.zen.ping.unicast.hosts: ["192.168.1.26", "192.168.1.27"] # 指定谁可以成为 Master ，如不指定无法加入到集群中 shell > mkdir -p /data/esearch/{data,logs} shell > chown -R elast.elast /data/esearch shell > su - elast -c "/usr/local/elasticsearch-2.4.0/bin/elasticsearch -d" shell > echo 'su - elast -c "/usr/local/elasticsearch-2.4.0/bin/elasticsearch -d"' >> /etc/rc.local
# 简单测试 ElasticSearch
shell > curl -X GET 192.168.1.26:9200/_cat/nodes?v host ip heap.percent ram.percent load node.role master name 192.168.1.26 192.168.1.26 7 43 0.10 d * node-1
# 发现当前集群只有一个节点，名称为 node-1 ，角色为 master 。
# 好了，按照这个步骤开始安装下一台节点服务器。
# 注意：集群名称必须一致，节点名称必须不一致，防火墙开启 TCP 9200/9300，9300 用于节点间通信。
# node-2 安装略
shell > curl -X GET 192.168.1.27:9200/_cat/nodes?v host ip heap.percent ram.percent load node.role master name 192.168.1.27 192.168.1.27 6 42 0.23 d m node-2 192.168.1.26 192.168.1.26 7 43 0.04 d * node-1
# 搭建好 node-2，再次查看节点信息，显示集群中有两个节点，node-1 角色为 master，node-2 角色 m 代表可以成为 master。
curl -X GET 192.168.1.26:9200/_cat/nodes?v host ip heap.percent ram.percent load node.role master name 192.168.1.26 192.168.1.26 7 43 0.32 d * node-1 192.168.1.27 192.168.1.27 6 43 0.06 d m node-2
# 再次从 node-1 上查看节点信息，发现一致。
# 至此，ElasticSearch 集群安装完毕，默认 master 为 node-1，下面开始配置 LogStash 将数据存入 ElasticSearch 集群。
shell > vim /usr/local/logstash-2.4.0/logstash.conf output { # stdout { # codec => rubydebug # } elasticsearch { hosts => ["192.168.1.26:9200"] index => "logstash-%{type}-%{+YYYY.MM.dd}" document_type => "%{type}" workers => 1 flush_size => 20000 idle_flush_time => 10 template_overwrite => true } }
# 记得重启服务
# 再次追加日志，然后查看 ElasticSearch 索引信息
shell > curl -X GET 192.168.1.26:9200/_cat/indices green open logstash-nginx_access-2016.09.18 5 1 79 0 65.3kb 39.8kb
# 查看 ElasticSearch 索引情况
shell > curl -X GET 192.168.1.26:9200/_cat/shards?v index shard prirep state docs store ip node logstash-nginx_access-2016.09.18 1 r STARTED 13 12.8kb 192.168.1.26 node-1 logstash-nginx_access-2016.09.18 1 p STARTED 13 12.8kb 192.168.1.27 node-2 logstash-nginx_access-2016.09.18 2 p STARTED 19 14.3kb 192.168.1.26 node-1 logstash-nginx_access-2016.09.18 2 r STARTED 19 14.3kb 192.168.1.27 node-2 logstash-nginx_access-2016.09.18 3 r STARTED 16 13.9kb 192.168.1.26 node-1 logstash-nginx_access-2016.09.18 3 p STARTED 16 13.9kb 192.168.1.27 node-2 logstash-nginx_access-2016.09.18 4 p STARTED 15 12.5kb 192.168.1.26 node-1 logstash-nginx_access-2016.09.18 4 r STARTED 15 12.5kb 192.168.1.27 node-2 logstash-nginx_access-2016.09.18 0 p STARTED 16 12.6kb 192.168.1.26 node-1 logstash-nginx_access-2016.09.18 0 r STARTED 16 12.6kb 192.168.1.27 node-2
# 至此，ELK:LogStash 读取 Redis 信息写入 ElasticSearch 完成。
四、ELK:Kibana 去读取 ElasticSearch 数据并展示
shell > cd /usr/local/src shell > wget https://download.elastic.co/kibana/kibana/kibana-4.6.1-linux-x86_64.tar.gz shell > tar zxf kibana-4.6.1-linux-x86_64.tar.gz shell > mv kibana-4.6.1-linux-x86_64 ../; cd shell > vim /usr/local/kibana-4.6.1-linux-x86_64/config/kibana.yml elasticsearch.url: "http://192.168.1.26:9200" shell > /usr/local/kibana-4.6.1-linux-x86_64/bin/kibana > /data/logs/kibana.log &
# 防火墙开放 TCP 5601 端口。访问 http://192.168.1.25:5601
# 剩下的工作就是根据业务，来抓取、拆分日志，并制作图标等。
五、留一份配置文件
1、客户端 LogStash
# Logstash input { file { type => "nginx_access" path => ["/tmp/access.log"] start_position => "beginning" } } output { # stdout { # codec => rubydebug # } redis { host => "192.168.1.25" port => 6379 data_type => "list" key => "logstash-list" } }
2、服务端 LogStash
# Logstash.conf input { redis { host => "127.0.0.1" port => 6379 data_type => "list" key => "logstash-list" } } output { # stdout { # codec => rubydebug # } elasticsearch { hosts => ["192.168.1.26:9200"] index => "logstash-%{type}-%{+YYYY.MM.dd}" document_type => "%{type}" workers => 1 flush_size => 20000 idle_flush_time => 10 template_overwrite => true } }
3、ElasticSearch
shell > grep -vP '^#|^$' /usr/local/elasticsearch-2.4.0/config/elasticsearch.yml cluster.name: my-elk node.name: node-1 path.data: /data/esearch/data path.logs: /data/esearch/logs network.host: 192.168.1.26 http.port: 9200 discovery.zen.ping.unicast.hosts: ["192.168.1.26","192.168.1.27"]
# End
