ELK5.0部署
开源实时日志分析系统ELK 5.0 部署
A.环境准备
A.1、操作系统为：centos7.2 1台或2台 内存单台8G,Elasticsearch 5.2.0,Kibana 5.2.0, Logstash 5.2.0, Beats 5.2.0(含filebeat,metricbeat)
A.2、安全配置
A.2.1、SELINUX is disabled
A.2.2、Iptables or firewalld is stop
A.3、JAVA安装（默认亦可）
B.Elasticsearch安装
B.1、配置yum源并安装
#rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# cat > /etc/yum.repos.d/elasticsearch.repo <<EOF
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/
yumgpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
# yum install elasticsearch
# systemctl start elasticsearch
# systemctl status elasticsearch
# /usr/share/elasticsearch/bin/elasticsearch -V ##查看elasticsearch版本
B.2、如遇到提供内存等问题，可以尝试以下解决方法：
修改/etc/elasticsearch下的jvm.options：
# vi /etc/elasticsearch/jvm.options
-Xms4g ##启用如下两项
-Xmx4g
##-Xms2g ##关闭如下两项
##-Xmx2g
B.3、验证
B.3.1、ElasticSearch默认的对外服务的HTTP端口是9200，节点间交互的TCP端口是9300。
# ss -tlnp |grep -E '9200|9300'
B.3.2、测试服务
# curl -X GET http://localhost:9200
or
# curl -i -X GET 'localhost:9200/'
C、Logstash安装
C.1、配置yum源并安装
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# cat > /etc/yum.repos.d/logstash.repo <<EOF[logstash-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
# yum install logstash
# systemctl start logstash
# systemctl status logstash
# /usr/share/logstash/bin/logstash -V ##查看logstash版本
D、Kibana安装
D.1、配置yum源并安装
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# cat > /etc/yum.repos.d/kibana.repo <<EOF[kibana-5.x]
name=Kibana repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1autorefresh=1
type=rpm-md
EOF
# yum install kibana
# systemctl start kibana
# systemctl status kibana
# /usr/share/kibana/bin/kibana -V ##查看kibana版本
D.2、修改配置文件（/etc/kibana/kibana.yml）
#cat /etc/kibana/kibana.yml | grep -v "#"
server.port: 5601
server.host: "192.168.1.115" //部署kibana服务所有的节点地址，方便外网访问
elasticsearch.url: "http://192.168.1.115:9200" //elasticsearch所在的节点服务地址
D.3、服务验证
D.3.1、检查端口
# ss -tlnp|grep 5601 ##Kibana默认进程名：node ，端口5601
D.3.2、检查UI页面
firefox:http://localhost:5601 //如果配置文件中已修改server.host地址，此处为其地址。
E、Beats部署
E.1、FileBeat安装
#curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.0.1-x86_64.rpm
# rpm -ivh filebeat-5.0.1-x86_64.rpm
# systemctl start filebeat
# systemctl status filebeat
# filebeat.sh -version ##查看版本
E.1.1、FileBeat配置
# cd /etc/filebeat
# cp filebeat.yml filebeat.yml.bak
# vi /etc/filebeat/filebeat.yml ##配置filebeat
#============= Filebeat prospectors ===============
filebeat.prospectors:
- input_type: log
paths: - /var/log/*.log
#==================== Outputs =====================
#------------- Elasticsearch output ---------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
#---------------- Logstash output -----------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5043"] ##只需配置该处，其他默认即可
# filebeat.sh -configtest -e ##验证配置文件
E.1.2、Logstash配置(添加额外配置文件，以达到收集日志的目的)
# cat /etc/logstash/conf.d/logstash.conf
input {
beats {
port => "5043" }
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip { source => "clientip"
}
}
output {
elasticsearch { hosts => [ "localhost:9200" ]
}
}
# systemctl restart logstash ##重启logstash
# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit ##验证配置文件
# ss -tlnp|grep -E '5043|9600'
E.1.3、Kibana-UI页面设置
浏览器输入http://localhost:5601，配置filebeat的索引（只需输入filebeat-*即可）。如下图所示：
输入filebeat-*后稍等片刻，kibana会自动识别，OK后下面的按钮会由灰色变为可操控的按钮"Create",如上图所示。点击该按钮后，最后就会呈现如下图所示:
浏览器输入http://localhost:5601，点击左边栏的”Management”===>然后点击“index Patterns”===>点击“Add New”===>
===>点击“Crete”按钮创建logstash索引，创建完成后即会展现如下图所示:
E.2、Packetbeat安装
E.2.1、下载并安装
# yum install libpcap
# curl -L -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-5.0.1-x86_64.rpm
# rpm -ivh packetbeat-5.0.1-x86_64.rpm
# cat /etc/packetbeat/packetbeat.yml
==================== Network device ===================
packetbeat.interfaces.device: any ##捕获所有消息发送或接收的网络接口======================== Flows ========================
packetbeat.flows:
timeout: 30s
period: 10s
================== Transaction protocols ==============
##如下是packetbeat默认支持的主要协议及端口
packetbeat.protocols.icmp:
enabled: true
packetbeat.protocols.amqp:
ports: [5672]
packetbeat.protocols.cassandra:
ports: [9042]
packetbeat.protocols.dns:
ports: [53]
include_authorities: true
include_additionals: true
packetbeat.protocols.http: ports: [80, 8080, 8000, 5000, 8002]
packetbeat.protocols.memcache:
ports: [11211]
packetbeat.protocols.mysql:
ports: [3306]
packetbeat.protocols.pgsql:
ports: [5432]
packetbeat.protocols.redis:
ports: [6379]
packetbeat.protocols.thrift:
ports: [9090]
packetbeat.protocols.mongodb:
ports: [27017]
packetbeat.protocols.nfs:
ports: [2049]
========================= General ========================
========================== Outputs =========================
------------------- Elasticsearch output ------------------
output.elasticsearch:
hosts: ["localhost:9200"]
--------------------- Logstash output ---------------------
output.logstash:
hosts: ["localhost:5043"] ##只需配置该处，其他默认即可============================= Logging =====================
E.2.2、验证配置并启动packetbeat
# packetbeat.sh -version ##查看packetbeat版本packetbeat version 5.0.1 (amd64), libbeat 5.0.1
# packetbeat.sh -configtest -e ##测试配置文件
......Config OK
# systemctl start packetbeat
# systemctl status packetbeat
E.2.3、配置Kibana(新建packetbeat索引)
浏览器打开http://localhost:5601，新建索引页面输入“packetbeat-*”，之后kibana会自动更新，在“Time-field name”下面的三个选项中选择“@timestamp”，最后点击“Create”创建即可。
E.3、Metricbeat安装
E.3.1、下载并安装
# curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-5.0.1-x86_64.rpm
# rpm -ivh metricbeat-5.0.1-x86_64.rpm
# cat /etc/metricbeat/metricbeat.yml
================= Modules configuration =================
metricbeat.modules:
---------------------- System Module ----------------------
module: system
metricsets:
- cpu
- load
- filesystem
- fsstat -
memory
- network
- process
enabled: true
period: 10s
processes: ['.*']
========================= General ========================
======================== Outputs =======================
------------------- Elasticsearch output ----------------
output.elasticsearch:
hosts: ["localhost:9200"]
--------------------- Logstash output -------------------
output.logstash:
hosts: ["localhost:5043"] ##只需配置该项，其他默认即可======================= Logging =========================
E.3.2、验证并启动metricbeat
# metricbeat.sh -version ##查看版本信息metricbeat version 5.0.1 (amd64), libbeat 5.0.1
# metricbeat.sh -configtest -e ##验证配置文件
# systemctl start metricbeat
# systemctl status metricbeat
E.3.3、配置kibana(新建metricbeat索引)
在http://localhost:5601下新建索引页面输入“metricbeat-*”，之后kibana会自动更新，在“Time-field name”下面的选项中选择“@timestamp”，最后点击“Create”创建即可。
F、问题集
F.1、查询集群健康度
# curl 'localhost:9200/_cat/health?v'
F.2、查看节点列表
# curl 'localhost:9200/_cat/nodes?v'
F.3、列出所有索引
# curl 'localhost:9200/_cat/indices?v'
F.4、查询filebeat,packetbeat,metricbeat
# curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'
# curl -XGET 'http://localhost:9200/packetbeat-*/_search?pretty'
# curl -XGET 'http://localhost:9200/metricbeat-*/_search?pretty'
F.5、涉及到证书配置，可以使用以下方法
#cd /etc/pki/tls
#openssl req -subj '/CN=node.abclocal.com/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
//CN名一定是服务器主机名，在其它被监控节点上也会用到。
一般都在logstash配置文件及Beats中使用到证书。
当然ELK不仅于此，有更多的功能，需要开发。
