ELK之使用filebeat收集系统数据及其他程序并生成可视化图表
当您要面对成百上千、甚至成千上万的服务器、虚拟机和容器生成的日志时,请告别 SSH 吧。Filebeat 将为您提供一种轻量型方法,用于转发和汇总日志与文件,让简单的事情不再繁杂。
1,安装filebeat
1 | rpm - ivh / nas / nas / softs / elk / 6.5 . 4 / filebeat - 6.5 . 4 - x86_64.rpm |
查看模块
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | [root@prd - elk - kafka - 01 ~] # filebeat modules list Enabled: kafka system Disabled: apache2 auditd elasticsearch haproxy icinga iis kibana logstash mongodb mysql nginx osquery postgresql redis suricata traefik |
如需要启用某个模块使用命令
1 | filebeat modules enable system |
修改配置文件把filebeat输出至elasticsearch和kibana
1 | / etc / filebeat / filebeat.yml |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | filebeat.config.modules: path: ${path.config} / modules.d / * .yml reload .enabled: false setup.template.settings: index.number_of_shards: 3 output.elasticsearch: hosts: [ "172.16.90.24:9200" ] # username: "admin" # password: "admin" setup.kibana: host: "172.16.90.24:5601" processors: - add_host_metadata: - add_cloud_metadata: # - drop_fields: # fields: ["beat", "input", "source", "offset", "prospector","host"] |
模块路径为/etc/filebeat/modules.d/
2,设置系统模块system
启用系统模块
1 | / etc / filebeat / filebeat.yml |
编辑系统模块配置文件,这里使用默认
1 | / etc / filebeat / modules.d / system.yml |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | - module: system # Syslog syslog: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. #var.paths: # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1. #var.convert_timezone: false # Authorization logs auth: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. #var.paths: # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1. #var.convert_timezone: false |
系统模块每一台主机都需要安装,启用后查看kibana
3,设置nginx模块
启动nginx模块
1 | filebeat modules enable nginx |
修改配置文档
1 | / etc / filebeat / modules.d / nginx.yml |
输入access日志及error日志路径
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | - module: nginx # Access logs access: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: - "/opt/log/wwwaccess.log" - "/opt/log/workaccess.log" - "/opt/log/datavaccess.log" # Error logs error: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: - "/opt/log/wwwerror.log" - "/opt/log/workerror.log" - "/opt/log/dataverror.log" |
kibana查看
要显示客户端城市及浏览器信息需要安装elasticsearch插件
1 2 | bin / elasticsearch - plugin install ingest - geoip bin / elasticsearch - plugin install ingest - user - agent |
4,设置redis模块
启用redis模块
1 | filebeat modules enable redis |
修改redis配置文件
1 | / etc / filebeat / modules.d / redis.yml |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | - module: redis # Main logs log: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: [ "/nas/nas/logs/redis/redis-server.log" ] # Slow logs, retrieved via the Redis API (SLOWLOG) slowlog: enabled: true # The Redis hosts to connect to. var.hosts: [ "localhost:6379" ] # Optional, the password to use when connecting to Redis. var.password: "password" |
kibana页面查看
5,设置kafka模块
启用kafka模块
1 | filebeat modules enable kafka |
修改配置文件
1 | / etc / filebeat / modules.d / kafka.yml |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | - module: kafka # All logs log: enabled: true # Set custom paths for Kafka. If left empty, # Filebeat will look under /opt. #var.kafka_home: # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: - "/usr/local/kafka/logs/controller.log*" - "/usr/local/kafka/logs/server.log*" - "/usr/local/kafka/logs/state-change.log*" - "/usr/local/kafka/logs/kafka-*.log*" # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1. #var.convert_timezone: false |
kibana查看
6,设置MySQL模块
开启mysql模块
1 | filebeat modules enable mysql |
修改配置文件
1 | / etc / filebeat / modules.d / mysql.yml |
输入日志及慢查询日志文件路径
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | - module: mysql # Error logs error: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: [ "/var/log/mysql3306.log" ] # Slow logs slowlog: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: [ "/nas/nas/logs/mysql/slow3306.log" ] |
kibana查看
更多模块查看官方文档 https://www.elastic.co/guide/en/beats/filebeat/6.5/filebeat-modules.html
【推荐】国内首个AI IDE,深度理解中文开发场景,立即下载体验Trae
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· SQL Server 2025 AI相关能力初探
· Linux系列:如何用 C#调用 C方法造成内存泄露
· AI与.NET技术实操系列(二):开始使用ML.NET
· 记一次.NET内存居高不下排查解决与启示
· 探究高空视频全景AR技术的实现原理
· 阿里最新开源QwQ-32B,效果媲美deepseek-r1满血版,部署成本又又又降低了!
· AI编程工具终极对决:字节Trae VS Cursor,谁才是开发者新宠?
· 开源Multi-agent AI智能体框架aevatar.ai,欢迎大家贡献代码
· Manus重磅发布:全球首款通用AI代理技术深度解析与实战指南
· 被坑几百块钱后,我竟然真的恢复了删除的微信聊天记录!
2018-03-30 Zabbix添加web页面监控告警
2018-03-30 SaltStack数据系统-Pillar
2018-03-30 SaltStack数据系统-Grains