收集日志之二:logstash的TCP/UDP
收集日志的几种方式: 1.logstash (消耗内存多。功能性好) 2.logstash的TCP/UDP 监听端口,在”其他“服务器安装 nc 命令 3.通过 rsyslog 收集日志,要logstash接收再转发到ES : 4.filebeat 收集日志:写入es redis logstash kafka (消耗内存少,不使用java,不支持多输出 ,不支持IF的type判断,filebeat服务器的配置文件中先定义fields:) 实验说明 192.168.80.100 localhost7A.localdomain node1 head cerebro kibana 192.168.80.110 localhost7B.localdomain node2 192.168.80.120 localhost7C.localdomain node3 192.168.80.130 localhost7D.localdomain logstash 192.168.80.140 localhost7D.localdomain nc 使用 logstash TCP/UDP 收集日志 通过 logstash 的 tcp/udp(开启监听端口)插件收集日志,通常用于在向 elasticsearch 日志补录丢失的部分日志, 可以将丢失的日志写到一个文件,然后通过 TCP 日志收集方式直接发送给 logstash 然后再写入到 elasticsearch 服务器。 [root@localhost7D conf.d]# cat logtcp.conf input { tcp { port => 9889 host => "0.0.0.0" #监听本机地址 type => "tcplog" mode => "server" } } output { stdout { codec => rubydebug } } 验证端口启动成功: # /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcp.conf 客户端的测试和验证 1.在其他服务器安装 nc (netcat)命令 # yum instll nc –y 2.1 通过管理传递 # echo "nc test" | nc 192.168.80.130 9889 2.2 通过 nc 命令发送一个文件: # nc 192.168.80.130 9889 < /etc/hosts 2.3 通过伪设备的方式发送消息: 在类 Unix 操作系统中,块设备有硬盘、内存的硬件,但是还有设备节点并不一定要对应物理设备,我们把没有这种对应关系的设备是伪设备,比如/dev/null, /dev/zero,/dev/random 以及/dev/tcp 和/dev/upd 等,Linux 操作系统使用这些伪设备提供了多种不通的功能,tcp 通信只是 dev 下面众多伪设备当中的一种设备。 # echo "伪设备" > /dev/tcp/192.168.80.130/9889 4.验证 logstash 是否接收到数据: [INFO ] 2021-12-08 10:17:53.038 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated { "@version" => "1", "message" => "nc test ", "@timestamp" => 2021-12-08T02:18:09.337Z, "port" => 48935, "type" => "tcplog", "host" => "192.168.80.140" } { "@version" => "1", "message" => "127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4", "@timestamp" => 2021-12-08T02:18:40.035Z, "port" => 48936, "type" => "tcplog", "host" => "192.168.80.140" } { "@version" => "1", "message" => "伪设备", "@timestamp" => 2021-12-08T02:19:44.715Z, "port" => 48937, "type" => "tcplog", "host" => "192.168.80.140" } 4.输出改为 elasticsearch: [root@localhost7D conf.d]# cat logtcp.conf input { tcp { port => "8888" #监听端口 host => "0.0.0.0" #监听地址 type => "tcplog" mode => "server" } } output { if [type] == "tcplog" { elasticsearch { hosts => ["192.168.80.100:9200"] index => "linux_tcp_140_%{+YYYY.MM.dd}" } } } # systemctl restart logstas 通过 nc 命令或伪设备输入日志: [root@linux-host1 ~]# echo "伪设备 1" > /dev/tcp/192.168.80.130/9889 [root@linux-host1 ~]# echo "伪设备 2" > /dev/tcp/192.168.80.130/9889