|NO.Z.00290|——————————|CloudNative|——|KuberNetes&运维.V12|——|EFLK架构.v07|验证日志输出采集是否正常|
一、验证日志输出采集是否正常
### --- 进入Filebeat-pod查看是否正常启动
~~~ 查看Filebeat容器
[root@k8s-master01 filebeat]# kubectl get po -n public-service -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
app-d585bb7cf-d5gpg 2/2 Running 0 7m55s 172.27.14.206 k8s-node02 <none> <none>### --- 进入Filebeat容器
~~~ 说明容器启动正常
[root@k8s-master01 filebeat]# kubectl exec -ti app-d585bb7cf-d5gpg -n public-service -- bash
[root@app-d585bb7cf-d5gpg filebeat]# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.2 0.9 310220 49460 ? Ssl 15:29 0:00 filebeat -e
root 14 0.3 0.0 11828 3048 pts/0 Ss 15:33 0:00 bash
root 28 0.0 0.0 51752 3456 pts/0 R+ 15:34 0:00 ps aux 二、通过Filebeat模拟写入日志，查看日志记录过程
### --- 打开kafka的日志记录
[root@k8s-master01 filebeat]# kubectl logs -f kafka-0 -n public-service
[2021-05-24 08:02:41,968] INFO [GroupCoordinator 0]: Stabilized group logstash generation 1 (__consumer_offsets-49) with 1 members (kafka.coordinator.group.GroupCoordinator)
[2021-05-24 08:02:42,020] INFO [GroupCoordinator 0]: Assignment received from leader for group logstash for generation 1. The group has 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)### --- 打开logstash
~~~ 注：状态显示已经连接上kafka
[root@k8s-master01 filebeat]# kubectl logs -f logstash-deployment-78cd865bcd-nbs58 -n public-service
[INFO ] 2021-05-24 08:02:42.539 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-kafka-9.1.0/lib/logstash/inputs/kafka.rb:244] ConsumerCoordinator - [Consumer clientId=logstash-0, groupId=logstash] Found no committed offset for partition test-filebeat-0
[INFO ] 2021-05-24 08:02:42.613 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-kafka-9.1.0/lib/logstash/inputs/kafka.rb:244] SubscriptionState - [Consumer clientId=logstash-0, groupId=logstash] Resetting offset for partition test-filebeat-0 to offset 0.### --- 打开filebeat-app日志记录
[root@k8s-master01 filebeat]# kubectl logs -f app-d585bb7cf-rlfkw -n public-service filebeat
2021-05-24T16:11:28.891+0800 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":570,"time":{"ms":10}},"total":{"ticks":1430,"time":{"ms":47},"value":1430},"user":{"ticks":860,"time":{"ms":37}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"636fcd01-3b01-4337-bc8c-cdaca461f344","uptime":{"ms":751607}},"memstats":{"gc_next":7173840,"memory_alloc":3805008,"memory_total":19027144},"runtime":{"goroutines":20}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.02,"15":1.3,"5":1.02,"norm":{"1":0.51,"15":0.65,"5":0.51}}}}}}### --- 再开启一个窗口，模拟写入文件，查看日志记录过程
~~~ 注：touch一个文件
~~~ 注：挂载了一个共享目录/home/tomcat/target
[root@k8s-master01 filebeat]# kubectl exec -ti app-d585bb7cf-rlfkw -n public-service -c app -- sh
/ # cd /home/tomcat/target/
/home/tomcat/target # touch app.log ~~~ # 进入/home/tomcat/target后创建文件，查看/data/log/app下是否会监听到
~~~ 注：在容器中touch完文件之后，Filebeat会打印一条日志；这个就是创建文件记录的日志记录
~~~ 可以正常监听到数据
root@k8s-master01 filebeat]# kubectl logs -f app-d585bb7cf-rlfkw -n public-service filebeat
2021-05-24T16:29:19.333+0800 INFO log/harvester.go:251 Harvester started for file: /data/log/app/app.log
[root@k8s-master01 filebeat]# kubectl exec -ti app-d585bb7cf-rlfkw -n public-service -c filebeat -- sh
sh-4.2# more /data/log/app/app.log
123三、通过app-filebeat写入文件内容，验证是否采集日志记录
### --- 通过app-filebeat写入文件内容
[root@k8s-master01 filebeat]# kubectl exec -ti app-d585bb7cf-rlfkw -n public-service -c app -- sh
/ # cd /home/tomcat/target/
/home/tomcat/target # touch app.log
/home/tomcat/target # echo 123 >> app.log // 写入123到app.log文件中### --- 查看Filebeat收集日志过程
~~~ 注：Filebeat是没有打印出来详情，只是把数据给传输过去
[root@k8s-master01 filebeat]# kubectl logs -f app-d585bb7cf-rlfkw -n public-service filebeat
2021-05-24T16:33:45.394+0800 INFO pipeline/output.go:95 Connecting to kafka(kafka:9092)
2021-05-24T16:33:45.394+0800 INFO kafka/log.go:53 kafka message: Initializing new client
2021-05-24T16:33:45.395+0800 INFO kafka/log.go:53 kafka message: Successfully initialized new client
2021-05-24T16:33:45.395+0800 INFO pipeline/output.go:105 Connection to kafka(kafka:9092) established
2021-05-24T16:33:45.397+0800 INFO kafka/log.go:53 client/metadata fetching metadata for [test-filebeat] from broker kafka:9092
2021-05-24T16:33:45.487+0800 INFO kafka/log.go:53 Connected to broker at kafka:9092 (unregistered)
2021-05-24T16:33:45.495+0800 INFO kafka/log.go:53 client/brokers registered new broker #0 at kafka-0.kafka-headless.public-service.svc.cluster.local:9092
2021-05-24T16:33:45.498+0800 INFO kafka/log.go:53 producer/broker/0 starting up
2021-05-24T16:33:45.498+0800 INFO kafka/log.go:53 producer/broker/0 state change to [open] on test-filebeat/0
2021-05-24T16:33:45.596+0800 INFO kafka/log.go:53 Connected to broker at kafka-0.kafka-headless.public-service.svc.cluster.local:9092 (registered as #0)### --- 查看logstash的日志收集变化过程
~~~ 注：logstash是有记录日志详情的
~~~ logstash会把这个日志推送到es上面
[root@k8s-master01 filebeat]# kubectl logs -f logstash-deployment-78cd865bcd-nbs58 -n public-service
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"tags" => [
[0] "test-filebeat"
],
"message" => "123",
"host" => {
"name" => "app-d585bb7cf-rlfkw"
},
"@timestamp" => 2021-05-24T08:33:44.391Z,
"log" => {
"offset" => 0,
"file" => {
"path" => "/data/log/app/app.log"
}
},
"@version" => "1",
"ecs" => {
"version" => "1.1.0"
},
"fields" => {
"pod_deploy_name" => "app",
"pod_namespace" => "public-service",
"pod_ip" => "172.27.14.207",
"pod_name" => "app-d585bb7cf-rlfkw"
},
"agent" => {
"id" => "8800213a-8bf3-4baf-9585-16510c642069",
"version" => "7.4.2",
"type" => "filebeat",
"ephemeral_id" => "636fcd01-3b01-4337-bc8c-cdaca461f344",
"hostname" => "app-d585bb7cf-rlfkw"
}
}
[WARN ] 2021-05-24 08:33:46.483 [[main]>worker0] elasticsearch - Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch-logging:9200/][Manticore::SocketException] Connection reset {:url=>http://elasticsearch-logging:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch-logging:9200/][Manticore::SocketException] Connection reset", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[ERROR] 2021-05-24 08:33:46.484 [[main]>worker0] elasticsearch - Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://elasticsearch-logging:9200/][Manticore::SocketException] Connection reset", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[ERROR] 2021-05-24 08:33:48.491 [[main]>worker0] elasticsearch - Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
[WARN ] 2021-05-24 08:33:50.727 [Ruby-0-Thread-4: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:232] elasticsearch - Restored connection to ES instance {:url=>"http://elasticsearch-logging:9200/"}四、多写入文件内容，查看es是否会采集到数据
### --- 通过Filebeat写入文件内容
[root@k8s-master01 filebeat]# kubectl exec -ti app-d585bb7cf-rlfkw -n public-service -c app -- sh
/ # cd /home/tomcat/target/
/home/tomcat/target # for i in 'seq 1 6000'; do echo $i >> app.log ; sleep 2;done### --- 查看Filebeat收集到的日志记录
[root@k8s-master01 filebeat]# kubectl logs -f app-d585bb7cf-rlfkw -n public-service filebeat
2021-05-24T16:56:09.810+0800 INFO log/harvester.go:251 Harvester started for file: /data/log/app/app.log
2021-05-24T16:56:10.822+0800 INFO kafka/log.go:53 producer/broker/0 state change to [closing] because write tcp 172.27.14.207:55108->172.25.244.203:9092: write: broken pipe
2021-05-24T16:56:10.823+0800 INFO kafka/log.go:53 Closed connection to broker kafka-0.kafka-headless.public-service.svc.cluster.local:9092
2021-05-24T16:56:10.823+0800 INFO kafka/log.go:53 producer/leader/test-filebeat/0 state change to [retrying-1]
2021-05-24T16:56:10.824+0800 INFO kafka/log.go:53 producer/leader/test-filebeat/0 abandoning broker 0
2021-05-24T16:56:10.824+0800 INFO kafka/log.go:53 producer/broker/0 shut down
2021-05-24T16:56:10.928+0800 INFO kafka/log.go:53 client/metadata fetching metadata for [test-filebeat] from broker kafka:9092
2021-05-24T16:56:10.955+0800 INFO kafka/log.go:53 producer/broker/0 starting up
2021-05-24T16:56:10.956+0800 INFO kafka/log.go:53 producer/broker/0 state change to [open] on test-filebeat/0
2021-05-24T16:56:10.957+0800 INFO kafka/log.go:53 producer/leader/test-filebeat/0 selected broker 0
2021-05-24T16:56:10.958+0800 INFO kafka/log.go:53 producer/leader/test-filebeat/0 state change to [flushing-1]
2021-05-24T16:56:10.958+0800 INFO kafka/log.go:53 producer/leader/test-filebeat/0 state change to [normal]
2021-05-24T16:56:10.961+0800 INFO kafka/log.go:53 Connected to broker at kafka-0.kafka-headless.public-service.svc.cluster.local:9092 (registered as #0)### --- 查看logstash采集到的数据详情
[root@k8s-master01 filebeat]# kubectl logs -f logstash-deployment-78cd865bcd-nbs58 -n public-service
{
"tags" => [
[0] "test-filebeat"
],
"host" => {
"name" => "app-d585bb7cf-rlfkw"
},
"message" => "seq 1 6000",
"@timestamp" => 2021-05-24T08:56:09.811Z,
"log" => {
"offset" => 4,
"file" => {
"path" => "/data/log/app/app.log"
}
},
"@version" => "1",
"ecs" => {
"version" => "1.1.0"
},
"fields" => {
"pod_deploy_name" => "app",
"pod_namespace" => "public-service",
"pod_ip" => "172.27.14.207",
"pod_name" => "app-d585bb7cf-rlfkw"
},
"agent" => {
"id" => "8800213a-8bf3-4baf-9585-16510c642069",
"version" => "7.4.2",
"type" => "filebeat",
"ephemeral_id" => "636fcd01-3b01-4337-bc8c-cdaca461f344",
"hostname" => "app-d585bb7cf-rlfkw"
}
} Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart
——W.S.Landor
【推荐】国内首个AI IDE，深度理解中文开发场景，立即下载体验Trae
【推荐】编程新体验，更懂你的AI，立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包，你的智能百科全书，全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell：AI 加持，快人一步
· 全程不用写代码，我用AI程序员写了一个飞机大战
· MongoDB 8.0这个新功能碉堡了，比商业数据库还牛
· 记一次.NET内存居高不下排查解决与启示
· DeepSeek 开源周回顾「GitHub 热点速览」
· 白话解读 Dapr 1.15：你的「微服务管家」又秀新绝活了