ELK
ELK介绍
http://blog.51cto.com/zero01/2082794
https://www.cnblogs.com/aresxin/p/8035137.html
需求背景：
业务发展越来越庞大，服务器越来越多，各种访问日志、应用日志、错误日志量越来越多，开发人员排查问题，需要到服务器上查日志，不方便，运营人员需要一些数据，需要我们运维到服务器上分析日志。
ELK安装准备工作
1.准备三台机器，三台机器都安装es，主节点131，数据节点132,133，es主节点上安装kibana，132上安装logstash数据收集
2.三台机器都配置下host
[root@centos-01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.242.131 centos-01
192.168.242.132 centos-02
192.168.242.133 centos-03
[root@centos-01 ~]#
3.三台机器都安装jdk
[root@centos-01 ~]# yum install -y java-1.8.0-openjdk
安装es
http://www.mamicode.com/info-detail-2207957.html
1.三台机器都导入key
[root@centos-01 ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch [root@centos-01 ~]#
2.三台机器都写一个repo文件
[root@centos-01 ~]# cat !$ cat /etc/yum.repos.d/elastic.repo [elasticsearch-6.x] name=Elasticsearch repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md [root@centos-01 ~]#
3.检查下yum源是否可以找的elastic
[root@centos-01 ~]# yum list|grep elastic apm-server.i686 6.3.0-1 elasticsearch-6.x apm-server.x86_64 6.3.0-1 elasticsearch-6.x auditbeat.i686 6.3.0-1 elasticsearch-6.x auditbeat.x86_64 6.3.0-1 elasticsearch-6.x elasticsearch.noarch 6.3.0-1 elasticsearch-6.x filebeat.i686 6.3.0-1 elasticsearch-6.x filebeat.x86_64 6.3.0-1 elasticsearch-6.x heartbeat-elastic.i686 6.3.0-1 elasticsearch-6.x heartbeat-elastic.x86_64 6.3.0-1 elasticsearch-6.x kibana.x86_64 6.3.0-1 elasticsearch-6.x kibana-oss.x86_64 6.3.0-1 elasticsearch-6.x logstash.noarch 1:6.3.0-1 elasticsearch-6.x metricbeat.i686 6.3.0-1 elasticsearch-6.x metricbeat.x86_64 6.3.0-1 elasticsearch-6.x packetbeat.i686 6.3.0-1 elasticsearch-6.x packetbeat.x86_64 6.3.0-1 elasticsearch-6.x pcp-pmda-elasticsearch.x86_64 3.12.2-5.el7 base rsyslog-elasticsearch.x86_64 8.24.0-16.el7_5.4 updates [root@centos-01 ~]#
4.三台都安装elasticsearch(下载rpm包会慢，可以提前下载好再用rpm -ivh安装)
[root@centos-01 ~]# yum install -y elasticsearch
1.查看elasticsearch安装了哪些文件
[root@centos-01 ~]# rpm -ql elasticsearch
2.es有两个配置文件
/etc/elasticsearch/elasticsearch.yml （配置集群的时候设置主节点父节点等配置） /etc/sysconfig/elasticsearch (和服务本身相关的配置)
3.编辑31服务器elasticsearch.yml
[root@centos-01 ~]# vim /etc/elasticsearch/elasticsearch.yml
修改
cluster.name: centos （簇名称） node.name: centos-01 （节点名称）
network.host: 192.168.242.131 （定义绑定的ip，需要在哪个ip上监听端口，为了安全我们这里设置成本机ip，可以写成0.0.0.0代表所有ip）
http.port: 9200
在node配置块儿添加
node.master: true （是主节点）
node.data: false （不是数据节点）
discovery.zen.ping.unicast.hosts: ["centos-01", "centos-02", "centos-03"] （定义集群里面都有哪些角色，也可以写IP）
4.将配置好的文件拷贝到另外两个节点上去
[root@centos-01 ~]# scp /etc/elasticsearch/elasticsearch.yml centos-02:/tmp/ The authenticity of host 'centos-02 (192.168.242.132)' can't be established. ECDSA key fingerprint is 15:ce:01:bd:cf:ed:1c:10:b5:e7:d3:8f:ca:66:a1:19. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'centos-02,192.168.242.132' (ECDSA) to the list of known hosts. root@centos-02's password: elasticsearch.yml 100% 3016 3.0KB/s 00:00 [root@centos-01 ~]# scp /etc/elasticsearch/elasticsearch.yml centos-03:/tmp/ The authenticity of host 'centos-03 (192.168.242.133)' can't be established. ECDSA key fingerprint is 0e:f3:c9:fe:6e:3a:e9:09:ed:b3:47:72:9d:c7:e0:ae. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'centos-03,192.168.242.133' (ECDSA) to the list of known hosts. root@centos-03's password: elasticsearch.yml 100% 3016 3.0KB/s 00:00 [root@centos-01 ~]#
5.到02和03服务器上修改下配置文件
[root@centos-02 ~]# cp /tmp/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml [root@centos-02 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name不变
node.name改变成centos-02
node.master: false （改成false）
node.data: true （改成true）
network.host: 192.168.242.132 （改成132）
[root@centos-03 ~]# cp /tmp/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml
[root@centos-03 ~]# vi /etc/elasticsearch/elasticsearch.yml
node.name: centos-03 node.master: false node.data: true network.host: 192.168.242.133
6.这样我们就可以启动三台机器的es服务了（我们这里省略xpack安装，xpack是收费的）
7.先启动主节点es，然后再启动其他两个节点
[root@centos-01 ~]# systemctl start elasticsearch.service [root@centos-01 ~]# [root@centos-02 ~]# systemctl start elasticsearch.service [root@centos-02 ~]# [root@centos-03 ~]# systemctl start elasticsearch.service [root@centos-03 ~]#
8.查看是否启动成功，结果没有启动成功，查看下日志情况，先看var/log/elasticsearch下面的，如果没有再看/var/log/messages
[root@centos-01 ~]# ps aux|grep elastic root 6640 0.0 0.0 112668 976 pts/0 R+ 15:50 0:00 grep --color=auto elastic [root@centos-01 ~]#
[root@centos-01 ~]# vi /var/log/elasticsearch/
[root@centos-01 ~]# less /var/log/messages
9.原来是配置选项冒号后一定要加空格，重启master(01)服务，成功
[root@centos-01 local]# systemctl start elasticsearch.service [root@centos-01 local]# ps aux |grep elasticsearch elastic+ 8379 3.4 70.0 3280924 701048 ? Ssl 17:03 0:51 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:
+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.
recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.Tm3xQEXo -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/
lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/
var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribut
ion.flavor=default -Des.distribution.type=rpm -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet elastic+ 8507 0.0 0.0 72076 860 ? Sl 17:04 0:00 /usr/share/elasticsearch/modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller root 9200 0.0 0.0 112668 980 pts/0 R+ 17:29 0:00 grep --color=auto elasticsearch [root@centos-01 local]#
curl查看es
1.查看端口，9200、9300端口也起来了
[root@centos-01 local]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 2729/mysqld
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1849/nginx: master
tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 2865/dnsmasq
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1172/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1173/cupsd
tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 1849/nginx: master
tcp6 0 0 192.168.242.131:9200 :::* LISTEN 8379/java
tcp6 0 0 192.168.242.131:9300 :::* LISTEN 8379/java
tcp6 0 0 :::22 :::* LISTEN 1172/sshd
tcp6 0 0 ::1:631 :::* LISTEN 1173/cupsd
[root@centos-01 local]#
2.启动02、03服务器
[root@centos-02 ~]# systemctl start elasticsearch.service
[root@centos-02 ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1437/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2350/master
tcp6 0 0 192.168.242.132:9200 :::* LISTEN 2873/java
tcp6 0 0 192.168.242.132:9300 :::* LISTEN 2873/java
tcp6 0 0 :::22 :::* LISTEN 1437/sshd
tcp6 0 0 ::1:25 :::* LISTEN 2350/master
[root@centos-02 ~]#
[root@centos-03 ~]# systemctl start elasticsearch
[root@centos-03 ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1435/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2570/master
tcp6 0 0 192.168.242.133:9200 :::* LISTEN 2926/java
tcp6 0 0 192.168.242.133:9300 :::* LISTEN 2926/java
tcp6 0 0 :::22 :::* LISTEN 1435/sshd
tcp6 0 0 ::1:25 :::* LISTEN 2570/master
[root@centos-03 ~]#
3.三台机器都关掉防火墙
[root@centos-01 local]# systemctl stop firewalld [root@centos-01 local]# iptables -nvL Chain INPUT (policy ACCEPT 20558 packets, 2482K bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 27470 packets, 11M bytes) pkts bytes target prot opt in out source destination [root@centos-01 local]#
[root@centos-01 local]# systemctl stop iptables
4.检测，发现只有一个节点，原来是服务器的senlinux没有关闭，把三台机器的senlinux都关闭再重新启动三台机器的elasticsearch服务，成功。
[root@centos-01 local]# curl '192.168.242.131:9200/_cluster/health?pretty'
{
"cluster_name" : "centos",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 0,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
[root@centos-01 local]#
[root@centos-01 local]# curl '192.168.242.131:9200/_cluster/health?pretty'
{
"cluster_name" : "centos",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 2,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
[root@centos-01 local]#
5.查看集群详细信息，端口9200是通信的，9300是数据传输用到的
[root@centos-01 local]# curl '192.168.242.131:9200/_cluster/state?pretty'|less
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{
"cluster_name" : "centos",
"compressed_size_in_bytes" : 9376,
"version" : 7,
{
"cluster_name" : "centos",
"compressed_size_in_bytes" : 9376,
"version" : 7,
"state_uuid" : "AYXexjAZRhC53TySE-_1ug",
"master_node" : "LuQzu2TISnmztU5O2R5DWw",
"blocks" : { },
"nodes" : {
"rDHWQTRwQRunAWdlalN4Nw" : {
"name" : "centos-02",
"ephemeral_id" : "ERGYa3b9SzewG4Dr4VOAew",
"transport_address" : "192.168.242.132:9300",
"attributes" : {
"ml.machine_memory" : "1025363968",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"ml.enabled" : "true"
}
},
"FNnHSA2rT2m6s7Ez8tYyLg" : {
{
"cluster_name" : "centos",
"compressed_size_in_bytes" : 9376,
"version" : 7,
"state_uuid" : "AYXexjAZRhC53TySE-_1ug",
"master_node" : "LuQzu2TISnmztU5O2R5DWw",
"blocks" : { },
"nodes" : {
"rDHWQTRwQRunAWdlalN4Nw" : {
"name" : "centos-02",
"ephemeral_id" : "ERGYa3b9SzewG4Dr4VOAew",
"transport_address" : "192.168.242.132:9300",
"attributes" : {
"ml.machine_memory" : "1025363968",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"ml.enabled" : "true"
}
},
"FNnHSA2rT2m6s7Ez8tYyLg" : {
"name" : "centos-03",
"ephemeral_id" : "2OUZMwW-TOWMpOBZUY6Erw",
"transport_address" : "192.168.242.133:9300",
"attributes" : {
"ml.machine_memory" : "1025363968",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"ml.enabled" : "true"
}
},
"LuQzu2TISnmztU5O2R5DWw" : {
"name" : "centos-01",
"ephemeral_id" : "jSwXaS5dR0SMy1FI9uPhrQ",
"transport_address" : "192.168.242.131:9300",
"attributes" : {
"ml.machine_memory" : "1025363968",
"xpack.installed" : "true",
安装kibana
kibana是用nodejs开发的
kibana是用来展示图像的
1.可以用yum直接安装kibana，如果太慢了可以预先下载rpm包用rmp -ivh安装包
[root@centos-01 local]# yum install -y kibana
2.编辑kibana配置文件
[root@centos-01 local]# vim /etc/kibana/kibana.yml
server.port: 5601 （端口） server.host: "192.168.242.131" （为了安装设置成只监听内网不让外网访问，如果还想让公网访问可以用nginx做一个代理，加一个安全认证） elasticsearch.url: "http://192.168.242.131:9200" （设置elastic服务器的ip，因为kibana需要与elastic通信）
3.启动kibana
[root@centos-01 local]# systemctl start kibana [root@centos-01 local]#
4.查看进程和端口
[root@centos-01 local]# ps aux|grep kibana kibana 28148 46.1 19.0 1143756 190284 ? Rsl 10:38 0:15 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml root 28199 0.0 0.0 112668 976 pts/0 R+ 10:38 0:00 grep --color=auto kibana [root@centos-01 local]# [root@centos-01 local]# netstat -lntp|grep node tcp 0 0 192.168.242.131:5601 0.0.0.0:* LISTEN 28148/node [root@centos-01 local]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 2729/mysqld tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1849/nginx: master tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 2865/dnsmasq tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1172/sshd tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1173/cupsd tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 1849/nginx: master tcp 0 0 192.168.242.131:5601 0.0.0.0:* LISTEN 28148/node tcp6 0 0 192.168.242.131:9200 :::* LISTEN 26052/java tcp6 0 0 192.168.242.131:9300 :::* LISTEN 26052/java tcp6 0 0 :::22 :::* LISTEN 1172/sshd tcp6 0 0 ::1:631 :::* LISTEN 1173/cupsd [root@centos-01 local]#
5.浏览器访问kibana
http://192.168.242.131:5601
6.kibana默认日志在/var/log/message下
安装logstash
1.lagstash不支持java9
2.我们在132服务器上yum安装logstash，可以下载rpm包安装（浏览器提前下载好rpm包，然后rz命令上传到服务器，rpm -ivh 安装即可）
[root@centos-02 ~]# yum install -y logstash
3.安装完之后配置logstash，我们这里收集系统日志，input就是我们的进入源，output就是我们的输出源
[root@centos-02 ~]# vim /etc/logstash/conf.d/syslog.conf
[root@centos-02 ~]# cat /etc/logstash/conf.d/syslog.conf
input {
syslog {
type => "system-syslog"
port => 10514
}
}
output {
stdout {
codec => rubydebug
}
}
[root@centos-02 ~]#
4.检查配置是否有错，path.settings指定配置文件所在的目录，-f指定我配置的logstash相关的配置文件
[root@centos-02 ~]# cd /usr/share/logstash/bin/ [root@centos-02 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf /conf.d/syslog.conf --config.test_and_exit
5.编辑rsyslog.conf文件在RULES下加一行，将日志输出到10514端口
[root@centos-02 bin]# vim /etc/rsyslog.conf #### RULES #### *.* @@192.168.242.132:10514
6.启动logstash
[root@centos-02 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf
7.再打开一个132服务器终端，重启rsyslog服务，第一个打开的终端有日志输出了
[root@centos-02 ~]# systemctl restart rsyslog [root@centos-02 ~]#
{
"severity_label" => "Informational",
"facility" => 5,
"severity" => 6,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:41:06",
"facility_label" => "syslogd",
"logsource" => "centos-02",
"@version" => "1",
"program" => "rsyslogd",
"priority" => 46,
"message" => "[origin software=\"rsyslogd\" swVersion=\"7.4.7\" x-pid=\"4749\" x-info=\"http://www.rsyslog.com\"] start\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:41:06.000Z
}
{
"severity_label" => "Informational",
"facility" => 3,
"severity" => 6,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:41:06",
"facility_label" => "system",
"logsource" => "centos-02",
"@version" => "1",
"program" => "systemd",
"priority" => 30,
"message" => "Stopping System Logging Service...\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:41:06.000Z
}
{
"severity_label" => "Informational",
"facility" => 3,
"severity" => 6,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:41:06",
"facility_label" => "system",
"logsource" => "centos-02",
"@version" => "1",
"program" => "systemd",
"priority" => 30,
"message" => "Starting System Logging Service...\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:41:06.000Z
}
{
"severity_label" => "Informational",
"facility" => 3,
"severity" => 6,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:41:06",
"facility_label" => "system",
"logsource" => "centos-02",
"@version" => "1",
"program" => "systemd",
"priority" => 30,
"message" => "Started System Logging Service.\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:41:06.000Z
}
{
"severity_label" => "Notice",
"facility" => 10,
"severity" => 5,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:41:06",
"facility_label" => "security/authorization",
"logsource" => "centos-02",
"@version" => "1",
"program" => "polkitd",
"pid" => "1232",
"priority" => 85,
"message" => "Unregistered Authentication Agent for unix-process:4742:8497324 (system bus name :1.71, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale zh_CN.UTF-8)
(disconnected from bus)\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:41:06.000Z
}
8.查看10514端口是否启动
[root@centos-02 ~]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1437/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2350/master tcp6 0 0 192.168.242.132:9200 :::* LISTEN 4254/java tcp6 0 0 :::10514 :::* LISTEN 4678/java tcp6 0 0 192.168.242.132:9300 :::* LISTEN 4254/java tcp6 0 0 :::22 :::* LISTEN 1437/sshd tcp6 0 0 ::1:25 :::* LISTEN 2350/master tcp6 0 0 127.0.0.1:9600 :::* LISTEN 4678/java [root@centos-02 ~]#
9.再用03服务器登录下02服务器看看有日志输出么？输出了
[root@centos-03 ~]# ssh centos-02
"severity_label" => "Notice",
"facility" => 10,
"severity" => 5,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:53:12",
"facility_label" => "security/authorization",
"logsource" => "centos-02",
"@version" => "1",
"program" => "unix_chkpwd",
"pid" => "4760",
"priority" => 85,
"message" => "password check failed for user (root)\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:53:12.000Z
}
{
"severity_label" => "Notice",
"facility" => 10,
"severity" => 5,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:53:12",
"facility_label" => "security/authorization",
"logsource" => "centos-02",
"@version" => "1",
"program" => "sshd",
"pid" => "4758",
"priority" => 85,
"message" => "pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=centos-03 user=root\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:53:12.000Z
}
{
"severity_label" => "Informational",
"facility" => 10,
"severity" => 6,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:53:12",
"facility_label" => "security/authorization",
"logsource" => "centos-02",
"@version" => "1",
"program" => "sshd",
"pid" => "4758",
"priority" => 86,
"message" => "pam_succeed_if(sshd:auth): requirement \"uid >= 1000\" not met by user \"root\"\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:53:12.000Z
}
{
"severity_label" => "Informational",
"facility" => 10,
"severity" => 6,
"type" => "system-syslog",
"timestamp" => "Jul 6 20:53:14",
"facility_label" => "security/authorization",
"logsource" => "centos-02",
"@version" => "1",
"program" => "sshd",
"pid" => "4758",
"priority" => 86,
"message" => "Failed password for root from 192.168.242.133 port 60658 ssh2\n",
"host" => "192.168.242.132",
"@timestamp" => 2018-07-06T12:53:14.000Z
}
10.日志配置成功
配置logstash
怎么把日志输出到es里面去呢，下面来讲解
1.首先把第一个终端退出ctrl+c，编辑配置文件，output改成elasticsearch
[root@centos-02 bin]# vim /etc/logstash/conf.d/syslog.conf
[root@centos-02 bin]# cat /etc/logstash/conf.d/syslog.conf
input {
syslog {
type => "system-syslog"
port => 10514
}
}
output {
elasticsearch {
hosts => ["192.168.242.131:9200"]
index => "system-syslog-%{+YYYY.MM}"
}
}
[root@centos-02 bin]#
2.检查下配置文件语法
[root@centos-02 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
[2018-07-06T21:06:23,517][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2018-07-06T21:06:29,416][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@centos-02 bin]#
3.以服务的形式启动logstash。
[root@centos-02 bin]# systemctl start logstash [root@centos-02 bin]# ps aux|grep logstash
4.查看端口，有10514和logstash的监听端口9600说明启动成功了，logstash日志文件如下
[root@centos-02 bin]# ls /var/log/logstash/logstash-plain.log /var/log/logstash/logstash-plain.log [root@centos-02 bin]#
[root@centos-02 bin]# vim /etc/logstash/logstash.yml path.config: /etc/logstash/conf.d/*.conf
[root@centos-02 bin]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1437/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2350/master tcp6 0 0 192.168.242.132:9200 :::* LISTEN 4254/java tcp6 0 0 192.168.242.132:9300 :::* LISTEN 4254/java tcp6 0 0 :::22 :::* LISTEN 1437/sshd tcp6 0 0 ::1:25 :::* LISTEN 2350/master [root@centos-02 bin]#
5.一直没有9600端口，可能是权限问题，我们修改下日志权限
[root@centos-02 bin]# ll /var/log/logstash/logstash-plain.log -rw-r--r--. 1 root root 2928 7月 6 21:06 /var/log/logstash/logstash-plain.log [root@centos-02 bin]# chown logstash /var/log/logstash/logstash-plain.log [root@centos-02 bin]# systemctl restart logstash [root@centos-02 bin]# tail /var/log/logstash/logstash-plain.log
6.查看日志提示没有写的权限，这是因为我们之前是用root身份启动的logstash，所以logstash对应的/var/lib/logstash下的文件都是root权限了
[root@centos-02 bin]# tail /var/log/logstash/logstash-plain.log
[2018-07-06T20:41:06,868][INFO ][logstash.inputs.syslog ] new connection {:client=>"192.168.242.132:55245"}
[2018-07-06T20:57:05,143][WARN ][logstash.runner ] SIGINT received. Shutting down.
[2018-07-06T20:57:05,692][INFO ][logstash.inputs.syslog ] connection error: stream closed
[2018-07-06T20:57:05,888][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x1868527b run>"}
[2018-07-06T21:06:23,517][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-07-06T21:06:29,416][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[2018-07-06T21:39:23,273][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not
writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:448:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:230:in `validate_value'",
"/usr/share/logstash/logstash-core/lib/logstash/settings.rb:141:in `block in validate_all'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.
rb:140:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:279:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:
in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:238:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/sha
re/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
[2018-07-06T21:39:23,325][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-07-06T21:40:28,999][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not
writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:448:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:230:in `validate_value'",
"/usr/share/logstash/logstash-core/lib/logstash/settings.rb:141:in `block in validate_all'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.
rb:140:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:279:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:
in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:238:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/sh
are/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
[2018-07-06T21:40:29,047][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[root@centos-02 bin]#
7.我们修改下权限
[root@centos-02 bin]# chown -R logstash /var/lib/logstash [root@centos-02 bin]#
[root@centos-02 bin]# systemctl restart logstash [root@centos-02 bin]#
8.9600端口出现了，成功
[root@centos-02 bin]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1437/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2350/master tcp6 0 0 192.168.242.132:9200 :::* LISTEN 4254/java tcp6 0 0 :::10514 :::* LISTEN 5828/java tcp6 0 0 192.168.242.132:9300 :::* LISTEN 4254/java tcp6 0 0 :::22 :::* LISTEN 1437/sshd tcp6 0 0 ::1:25 :::* LISTEN 2350/master tcp6 0 0 127.0.0.1:9600 :::* LISTEN 5828/java [root@centos-02 bin]#
kibana上查看日志
1.在es上看看有没有日志，获取索引（如果获取不到配置host），有索引说明logstash和es通信正常了
[root@centos-02 bin]# vim /etc/logstash/logstash.yml
http.host: "192.168.242.132"
root@centos-01 local]# curl '192.168.242.131:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open system-syslog-2018.07 Nh8lRsu3T56U1BIE-_mLnw 5 1 1 0 24.9kb 12.4kb green open .kibana n2dgGyqCRs-9KTIs1A-uAw 1 1 1 0 8kb 4kb [root@centos-01 local]#
2.获取索引的详细信息
[root@centos-01 local]# curl '192.168.242.131:9200/system-syslog-2018.07?pretty'
{
"system-syslog-2018.07" : {
"aliases" : { },
"mappings" : {
"doc" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"facility" : {
"type" : "long"
},
"facility_label" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"host" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"logsource" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"pid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"priority" : {
"type" : "long"
},
"program" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"severity" : {
"type" : "long"
},
"severity_label" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"timestamp" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
},
"settings" : {
"index" : {
"creation_date" : "1530856398824",
"number_of_shards" : "5",
"number_of_replicas" : "1",
"uuid" : "Nh8lRsu3T56U1BIE-_mLnw",
"version" : {
"created" : "6030099"
},
"provided_name" : "system-syslog-2018.07"
}
}
}
}
[root@centos-01 local]#
配置kibana
1.创建索引，将我们刚看到的索引，粘贴到kibana中（如下图），这个地方支持正则可以写成system-syslog-*
[root@centos-01 local]# curl '192.168.242.131:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open system-syslog-2018.07 Nh8lRsu3T56U1BIE-_mLnw 5 1 1 0 24.9kb 12.4kb
green open .kibana n2dgGyqCRs-9KTIs1A-uAw 1 1 1 0 8kb 4kb
[root@centos-01 local]#
2.成功创建，这个过程就是讲es里面的索引搞到kibana里面来
3.点击discover
收集nginx日志
1.在132服务器，也就是logstash所在的服务器上添加一个nginx配置文件
[root@centos-02 bin]# vim /etc/logstash/conf.d/nginx.conf^C
[root@centos-02 bin]# cat /etc/logstash/conf.d/nginx.conf
input {
file { # 指定一个文件作为logstash的输入源
path => "/tmp/elk_access.log" # 指定文件的路径
start_position => "beginning" # 指定何时开始收集
type => "nginx" # 定义日志类型，可自定义
}
}
filter { # 配置过滤器
grok {
match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER
:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"} # 定义日志的输出格式
}
geoip {
source => "clientip"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["192.168.242.132:9200"]
index => "nginx-test-%{+YYYY.MM.dd}"
}
}
[root@centos-02 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCTh
reads=N
^[[ASending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
[2018-07-06T23:12:09,630][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2018-07-06T23:12:19,638][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@centos-02 bin]#
1.安装nginx
https://my.oschina.net/andyfeng/blog/1634805
[root@centos-02 yum.repos.d]# rpm -ivh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm 获取http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm 警告：/var/tmp/rpm-tmp.RGilFh: 头V4 RSA/SHA1 Signature, 密钥 ID 7bd9bf62: NOKEY 准备中... ################################# [100%] 正在升级/安装... 1:nginx-release-centos-7-0.el7.ngx ################################# [100%] [root@centos-02 yum.repos.d]# yum install -y nginx
2.启动nginx
[root@centos-02 yum.repos.d]# systemctl start nginx [root@centos-02 yum.repos.d]# ps aux|grep nginx root 6091 1.0 0.0 48524 968 ? Ss 23:33 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf nginx 6092 0.3 0.1 48928 1920 ? S 23:33 0:00 nginx: worker process root 6094 0.0 0.0 112660 964 pts/0 R+ 23:33 0:00 grep --color=auto nginx [root@centos-02 yum.repos.d]#
[root@centos-02 yum.repos.d]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 6091/nginx: master
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1437/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2350/master
tcp6 0 0 192.168.242.132:9200 :::* LISTEN 4254/java
tcp6 0 0 :::10514 :::* LISTEN 5828/java
tcp6 0 0 192.168.242.132:9300 :::* LISTEN 4254/java
tcp6 0 0 :::22 :::* LISTEN 1437/sshd
tcp6 0 0 ::1:25 :::* LISTEN 2350/master
tcp6 0 0 127.0.0.1:9600 :::* LISTEN 5828/java
[root@centos-02 yum.repos.d]#
3.建立虚拟主机并配置代理ip为kibana所在的服务器
[root@centos-02 conf.d]# vim elk.conf^C
[root@centos-02 conf.d]# cat elk.conf
server {
listen 80;
server_name elk.test.com;
location / {
proxy_pass http://192.168.242.131:5601;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
access_log /tmp/elk_access.log main2;
}
[root@centos-02 conf.d]#
4.编辑nginx的配置文件，在http选项中添加main2日志配置
[root@centos-02 nginx]# vim nginx.conf
log_format main2 '$http_host $remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$upstream_addr" $request_time';
5.检查配置文件是否有错误
[root@centos-02 usr]# sbin/nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@centos-02 usr]#
6.重新加载配置文件
[root@centos-02 usr]# /usr/sbin/nginx -s reload [root@centos-02 usr]#
[root@centos-02 conf.d]# mv default.conf default.conf.bak [root@centos-02 conf.d]# ls default.conf.bak elk.conf [root@centos-02 conf.d]#
7.将detault.conf重命名再重新加载配置文件
[root@centos-02 conf.d]# /usr/sbin/nginx -s reload [root@centos-02 conf.d]#
http://elk.test.com/app/kibana
8.检查是否生成了，已经生成了，并且有日志内容
[root@centos-02 conf.d]# ls /tmp/elk_access.log /tmp/elk_access.log [root@centos-02 conf.d]#
1.再次测试logstash配置文件是否正确
[root@centos-02 bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCTh
reads=N Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties [2018-07-07T00:37:57,356][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [2018-07-07T00:38:03,846][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@centos-02 bin]#
2.重启logstash
[root@centos-02 bin]# systemctl restart logstash [root@centos-02 bin]# ps aux|grep logstash logstash 6250 89.5 31.3 3180544 313724 ? SNsl 00:39 0:31 /bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOcc
upancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /u
sr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-19.0.
jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/jackson
-databind-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.8.jar:/usr/share/logstash/logstash-core/
lib/jars/jruby-complete-9.1.13.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/share/logstash/logstash-
core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/
logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/
lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.reso
urces-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/
logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/
lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.o
sgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings
/etc/logstash root 6276 0.0 0.0 112660 964 pts/0 S+ 00:40 0:00 grep --color=auto logstash [root@centos-02 bin]# netstat -lntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 6091/nginx: master tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1437/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2350/master tcp6 0 0 192.168.242.132:9200 :::* LISTEN 4254/java tcp6 0 0 :::10514 :::* LISTEN 6250/java tcp6 0 0 192.168.242.132:9300 :::* LISTEN 4254/java tcp6 0 0 :::22 :::* LISTEN 1437/sshd tcp6 0 0 ::1:25 :::* LISTEN 2350/master tcp6 0 0 127.0.0.1:9600 :::* LISTEN 6250/java [root@centos-02 bin]#
3.再次查看有没有生成nginx-test
[root@centos-01 local]# curl '192.168.242.131:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open system-syslog-2018.07 Nh8lRsu3T56U1BIE-_mLnw 5 1 113 0 914.8kb 484kb green open .kibana n2dgGyqCRs-9KTIs1A-uAw 1 1 2 0 22kb 11kb green open nginx-test-2018.07.06 -Nkch6RJTt2bFVyl-2XpHA 5 1 56 0 227.3kb 121.1kb [root@centos-01 local]#
4.配置kibana的索引，点击index patterns
5.点击create index pattern
6.创建新索引
7.点击discover查看nginx-test
使用beats采集日志
轻量型数据采集器
1.下载filebeat
[root@centos-03 src]# pwd /usr/local/src [root@centos-03 src]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.1-x86_64.rpm
2.安装filebeat
[root@centos-03 src]# rpm -ivh filebeat-6.3.1-x86_64.rpm
3.编辑配置文件
# Change to true to enable this input configuration.
enabled: true （改为true）
paths:
- /var/log/messages (修改paths)
#- c:\programdata\elasticsearch\logs\*
# Configure what output to use when sending the data collected by the beat.
output.console: （新增）
enable: true （新增）
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch: （注释掉）
# Array of hosts to connect to.
# hosts: ["localhost:9200"] （注释掉）
4.启动filebeat
[root@centos-03 src]# /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml
5.用02服务器登录03服务器看看是否有日志生成，成功生成
[root@centos-02 ~]# ssh centos-03
root@centos-03's password:
Last login: Mon Jul 9 19:29:07 2018 from centos-02
[root@centos-03 ~]#
","offset":1680,"message":"Jul 9 19:29:40 centos-03 systemd-logind: New session 6 of user root.","input":{"type":"log"}}
{"@timestamp":"2018-07-09T11:29:42.186Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.3.1"},"message":"Jul 9 19:29:40 centos-03 systemd: Starting Session 6 of user root.","prospect
or":{"type":"log"},"input":{"type":"log"},"beat":{"name":"centos-03","hostname":"centos-03","version":"6.3.1"},"host":{"name":"centos-03"},"source":"/var/log/messages","offset":1750}
6.filebeat已服务形式启动
1.编辑配置文件，我们将paths指定的日志文件路径修改日志路径
[root@centos-03 src]# ls /var/log/elasticsearch/centos.log /var/log/elasticsearch/centos.log [root@centos-03 src]#
[root@centos-03 src]# vim /etc/filebeat/filebeat.yml
paths:
- /var/log/elasticsearch/centos.log
# Configure what output to use when sending the data collected by the beat. #output.console: （注释掉） # enable: true （注释掉） #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: （打开） # Array of hosts to connect to. hosts: ["192.168.242.131:9200"] （打开）
2.启动filebeat
[root@centos-03 src]# systemctl start filebeat [root@centos-03 src]# ps aux|grep filebeat root 4999 0.2 1.2 360060 12772 ? Ssl 19:49 0:00 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat root 5008 0.0 0.0 112660 964 pts/0 R+ 19:50 0:00 grep --color=auto filebeat [root@centos-03 src]#
3.查看es服务器是否生成了新的索引
[root@centos-01 ~]# curl '192.168.242.131:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open system-syslog-2018.07 Nh8lRsu3T56U1BIE-_mLnw 5 1 34663 0 8.7mb 4.3mb green open .kibana n2dgGyqCRs-9KTIs1A-uAw 1 1 3 0 35.9kb 17.9kb green open nginx-test-2018.07.06 -Nkch6RJTt2bFVyl-2XpHA 5 1 34856 0 8.7mb 4.3mb green open filebeat-6.3.1-2018.07.09 CbdrEwpqTEK1WJdXGZg92g 3 1 412 0 281kb 151.4kb [root@centos-01 ~]#
4.在kibana上建立filebeat日志配置（和之前做过的配置日志一样）
