logstash收集TCP与UDP日志
通过 logstash 的 tcp/udp 插件收集日志，通常用于在向 elasticsearch 日志补录丢
失的部分日志，可以将丢失的日志写到一个文件，然后通过 TCP 日志收集方式直
接发送给 logstash 然后再写入到 elasticsearch 服务器。
https://www.elastic.co/guide/en/logstash/5.6/input-plugins.html
准备条件：
环境：jdk，安装好 logstash
安装nc
[root@es-web2 ~]# apt install nc
安装jdk
[root@es-web2 ~]# apt install openjdk-8-jdk -y
dpkg安装
[root@es-web2 src]# dpkg -i logstash-7.12.1-amd64.deb
配置个文件，先进行收集测试
[root@es-web2 ]# vim /etc/logstash/conf.d/tcp-log-es.conf
input{
tcp{
port => 8899
type => "tcplog"
mode => "server"
}
}
output{
stdout{
codec => rubydebug
}
}
验证
[root@es-web2 ]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcp-log-es.conf
其他服务器安装 “瑞士军刀” nc
[root@es-web2 ]# apt install nc
测试
[root@es-web2 ]# echo "nc test" | nc 172.31.2.107 8899
检查端口
[root@es-web2 ]# ss -tnl | grep 8899
LISTEN 0 128 *:8899 *:*
测试接收文件
[root@es-web2 ]# nc 172.31.2.107 8899 < /etc/passwd
将输出改为 elasticsearch
root@long:~# vim /etc/logstash/conf.d/tcp-log-es.conf
input{
tcp{
port => 8899
type => "tcplog"
mode => "server"
}
}
output{
elasticsearch{
hosts => ["172.31.2.101:9200"]
index => "long-tcplog-%{+YYYY.MM.dd}"
}
}
重启
root@long:~# systemctl restart logstash
再用nc 传数据
root@long:~# echo "nc test1" | nc 172.31.2.108 8899
root@long:~# echo "伪设备1" > /dev/tcp/172.31.2.108/8899
查看es
添加到kibana
略
logstash收集UDP日志
准备一台CentOS代替交换机
安装rsyslog和haproxy
[root@localhost ~]# yum install rsyslog
[root@localhost ~]# yum install haproxy -y
rsyslog配置
[ root@localhost ~]# vim /etc/rsyslog.conf
$ModLoad imudp
$UDPServerRun 514
# 最后一行添加
local2.* @@remote-host:514
haproxy配置
[ root@localhost ~]# vim /etc/haproxy/haproxy.cfg
listen web-port
bind 0.0.0.0:80
server 172.31.2.108 172.31.2.108:80 check inter 3s fall 3 rise 5
重启
[ root@localhost ~]# systemctl restart haproxy
测试网页，可以访问即可
配置rsyslog写入日志
[ root@localhost ~]# vim /etc/rsyslog.conf
#local2.* @@remote-host:514
local2.* /var/log/haproxy.log
重启
[ root@localhost ~]# systemctl restart rsyslog
改haproxy配置
[ root@localhost ~]# vim /etc/haproxy/haproxy.cfg
listen web-port
bind 0.0.0.0:80
log global
mode http
server 172.31.2.108 172.31.2.108:80 check inter 3s fall 3 rise 5
重启
[ root@localhost ~]# systemctl restart haproxy
查看日志写入
[ root@localhost ~]# tail -f /var/log/haproxy.log
配置测试logstash输出到终端显示
input{
syslog{
host => "172.31.0.18"
port => "6514"
type => "ststem-rsyslog"
}
}
output {
stdout {}
}
停止
root@long:~# systemctl stop logstash
启动
root@long:~# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/rsys-log-es.conf
然后在centos-18服务器改rsyslog配置
[ root@localhost ~]# systemctl restart rsyslog
local2.* @@172.31.0.18:6514
重启
[ root@localhost ~]# systemctl restart rsyslog
启动
root@long:~# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/rsys-log-es.conf
然后刷新页面看看机台有没有获取到信息，获取到信息说明是成功的
在haproxy配置添加
[ root@localhost ~]# vim /etc/haproxy/haproxy.cfg
listen web1-port
bind 172.31.2.108:5601
log global
mode tcp
server 172.31.2.101 172.31.2.101:5601 check inter 3s fall 3 rise 5
server 172.31.2.102 172.31.2.102:5601 check inter 3s fall 3 rise 5
重启
[ root@localhost ~]# systemctl restart haproxy
检查端口
[ root@localhost ~]# ss -tnl
9200
在上面的基础上修改配置（Ubuntu的rsyslog日志配置有问题，所有这里使用的CentOS系统）
[root@es-web2 ~]# vim /etc/logstash/conf.d/tcp-log-es.conf
input{
tcp{
port => 8899
type => "tcplog"
mode => "server"
}
syslog {
type => "ststem-rsyslog"
port => "6514"
}
}
#output{
# stdout{
# codec => rubydebug
# }
#}
output{
if [type] == "tcplog" {
elasticsearch {
hosts => ["172.31.2.101:9200"]
index => "long-tcplog-%{+YYYY.MM.dd}"
}}
if [type] == "ststem-rsyslog" {
elasticsearch {
hosts => ["172.31.2.101:9200"]
index => "long-rsyslog-%{+YYYY.MM.dd}"
}}
}
启动
root@long:~# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/rsys-log-es.conf
访问几下网页，出现下面的即可
然后添加到 kibana
略
如果时间没有同步，执行下面命令即可
[root@localhost ~]# ntpdate time1.aliyun.com
[root@localhost ~]# hwclock -w
重启
[ root@localhost ~]# systemctl restart rsyslog
时区不对，执行如下命令即可（CentOS7）
[root@localhost ~]# cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
