4.安装fluentd用于收集集群内部应用日志
作者
微信:tangy8080
电子邮箱:914661180@qq.com
更新时间：2019-06-13 11:02:14 星期四
欢迎您订阅和分享我的订阅号,订阅号内会不定期分享一些我自己学习过程中的编写的文章
如您在阅读过程中发现文章错误,可添加我的微信 tangy8080 进行反馈.感谢您的支持。
文章主题
- 在大多数情况下,我们需要集中管理应用的日志.但是我们又不能强制要求开发者直接对日志进行统一输出
对开发者来说这可能是侵入式的,为了统一输出日志,可能导致业务收到影响.
在这种情况下我们可以自己采集日志,采集工具比较多,这里我们选择fluentd,用于收集集群内的日志 - 集群外部日志的收集将在下一节涉及
- 日志收集原则:在任意时刻(由Es高性能索引速度支持)可以实时的查看到某一节点(由physics.hostname支持)的某一服务(由tag支持)的日志
前置条件
- 已经完成了本章节的第一，第二节
安装
开始安装fluentd-elasticsearch
由于我们对fluentd-elasticsearch的定制比较多,所以我们选择使用源码来安装fluentd-elasticsearch
#克隆源码库
cd /usr/local/src/
git clone https://github.com/kiwigrid/helm-charts
cd helm-charts/charts/fluentd-elasticsearch/
#为所有输出添加hostname,physics.hostname,tag字段,方便查找日志
vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/templates/configmaps.yaml
#在containers.input.conf节点下添加如下配置
<filter **>
@id filter_hostname
@type record_transformer
<record>
hostname "#{Socket.gethostname}"
physics.hostname "#{ENV['K8S_NODE_NAME']}"
tag ${tag}
</record>
</filter>
- 在filter中添加了hostname,该值为k8s中的podName
- physics.hostname标识节点的名称,该值为k8s中节点的名称,在daemonset.yaml中定义默认定义了K8S_NODE_NAME名称
#执行安装
cd /usr/local/src/helm-charts/charts/fluentd-elasticsearch
helm install --name fluentd-elasticsearch --set elasticsearch.host=elasticsearch-client,hostLogDir.dockerContainers=/data/k8s/docker/data/containers .
- hostLogDir.dockerContainers 用于收集docker产生的日志,请按实际情况更改
- stable/fluentd-elasticsearch is deprecated as we move to our own repo (https://kiwigrid.github.io) which will be puplished on hub.helm.sh soon. The chart source can be found here: https://github.com/kiwigrid/helm-charts/tree/master/charts/fluentd-elasticsearch
[按需]卸载fluentd-elasticsearch
helm del --purge fluentd-elasticsearch
采集思路解析
- 我们在每个节点上运行一个fluentd代理（采用DaemonSet）,用于采集部署在k8s容器环境中程序和部署在物理主机中程序的日志。
- 对于容器内部应用.它的生命周期是不固定的,所以我们可以把日志放在宿主机上,（也可以使用sidecar容器，https://kubernetes.io/docs/concepts/cluster-administration/logging/）,出于性能原因,这里会选择只运行一个fluentd代理)
- pos_file需要放在何处? 需要放在宿主机上的一个稳定位置（通常和设定和path一个目录）.它记录了采集进度,假如容器销毁重启后,它依然可以从上次进度的位置开始采集
- 日志轮转问题, 如果应用本身不支持日志轮转.我们需要考虑日志轮转问题.避免日志越来越大
采集实例配置
实例一:运行在物理机器中的Nginx日志
配置挂载目录
vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/values.yaml
#nginx日志不在/var/log目录下,在文件中最后添加Nginx日志文件挂载
extraVolumes:
- name: nginxlog
hostPath:
path: /usr/local/nginx/logs
extraVolumeMounts:
- name: nginxlog
mountPath: /var/log/nginx
readOnly: true
配置采集数据源
vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/templates/configmaps.yaml
# service.honeysuckle-log-consumer
<source>
@id honeysuckle-log-consumer.log
@type tail
<parse>
@type regexp
expression /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<thread>.*) (?<level>[a-zA-Z]+) (?<logger>.*) (?<property>\[.*\]) - (?<msg>[\s\S]*)$/
time_key time
</parse>
path /var/log/businesslogs/honeysuckle-log-consumer/Log.txt
pos_file /var/log/businesslogs/honeysuckle-log-consumer/Log.txt.pos
tag service.honeysuckle-log-consumer
</source>
# service.honeysuckle-configmanager-service
<source>
@id honeysuckle-configmanager-service.log
@type tail
<parse>
@type regexp
expression /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<thread>.*) (?<level>[a-zA-Z]+) (?<logger>.*) (?<property>\[.*\]) - (?<msg>[\s\S]*)$/
time_key time
</parse>
path /var/log/businesslogs/honeysuckle-configmanager-service/Log.txt
pos_file /var/log/businesslogs/honeysuckle-configmanager-service/Log.txt.pos
tag service.honeysuckle-configmanager-service
</source>
# Nginx Access Log Source
<source>
@id nginx.accesslog
@type tail
path /var/log/nginx/access.log
pos_file /var/log/access.log.pos
format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"(?:\s+(?<http_x_forwarded_for>[^ ]+))?)?$/
time_format %d/%b/%Y:%H:%M:%S %z
tag nginx
</source>
- nginx默认的日志格式format可以在这里找到
https://docs.fluentd.org/parser/nginx
实例二:运行在容器中的Helloword程序
该项目用于模拟一个服务,它每隔3秒向 /app/App_Data/Logs/Log.txt 写入一条日志
该项目采用Log4net写入日志,应用本身支持日志轮转,最新的日志都在Log.txt中
log4net.config的配置如下:
<?xml version="1.0" encoding="utf-8" ?>
<log4net>
<!--文本文件appender-->
<appender name="RollingFileAppender" type="log4net.Appender.RollingFileAppender" >
<file value="App_Data/Logs/Log.txt" />
<appendToFile value="true" />
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="1024KB" />
<staticLogFileName value="true" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date [%thread] %-5level %logger [%property{NDC}] - %message%newline" />
</layout>
</appender>
<root>
<appender-ref ref="RollingFileAppender" />
<level value="ALL" />
</root>
</log4net>
日志输出的样例类似于:
2019-06-17 07:29:52,038 [5] INFO honeysuckle.log.consumer.HoneysuckleWebModule [(null)] - electronicinvoice_log_queue.BasicConsume 已创建.
2019-06-17 07:31:51,510 [WorkPool-Session#1:Connection(21956434-0138-45f3-be65-848ca544cad3,amqp://honeysuckle.site:5672)] ERROR honeysuckle.log.consumer.HoneysuckleWebModule [(null)] - Error in EventingBasicConsumer.Received
Elasticsearch.Net.UnexpectedElasticsearchClientException: The operation was canceled. ---> System.OperationCanceledException: The operation was canceled.
at System.Net.Http.HttpClient.HandleFinishSendAsyncError(Exception e, CancellationTokenSource cts)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
--- End of inner exception stack trace ---
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
at Nest.LowLevelDispatch.BulkDispatch[TResponse](IRequest`1 p, SerializableData`1 body)
at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[TRequest,TQueryString,TResponse](TRequest request, Func`3 responseGenerator, Func`3 dispatch)
at honeysuckle.log.consumer.HoneysuckleWebModule.<>c__DisplayClass10_0.<CreateNewChannel>b__0(Object sender, BasicDeliverEventArgs e) in /src/src/honeysuckle.log.consumer/HoneysuckleWebModule.cs:line 141
基于日志输出格式,我们采用正则表达式插件进行日志解析（表达式在下面的source中可以看到）
可以使用fluentular工具进行表达式的正确性测试
该项目托管在:
http://admin@gitblit.honeysuckle.site/r/public/helloworld.git
如有测试需要.欢迎clone
配置采集数据源
vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/templates/configmaps.yaml
# service.helloworld.log Log Source
<source>
@id helloworld.log
@type tail
<parse>
@type regexp
expression /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<thread>\[\d+\]) (?<level>[a-zA-Z]+) (?<logger>.*) (?<property>\[.*\]) - (?<msg>.*)$/
time_key time
</parse>
path /var/log/businesslogs/helloworld/Log.txt
pos_file /var/log/businesslogs/helloworld/Log.txt.pos
tag service.helloworld
</source>
查看fluentd是否采集到了数据
待所有组件全部runing之后,可以通过curl查看fluentd是否采集到了数据
curl 'http://10.254.193.78:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open logstash-2019.06.11 _v7q6b4DSt6mLJ_GLDyFUQ 5 1 4112 0 4.6mb 2.2mb
yellow open logstash-2019.06.12 mz3sF15LTbqQwZiXMubdgg 5 1 29394 0 27.9mb 15.2mb
green open logstash-2019.06.13 rtjvfGVWSM-5rLvq72jxlA 5 1 971 0 2.4mb 1.1mb
在Kibana中检查日志收集情况
-
检查点一:日志是否根据tag正确分类
-
检查点二：日志否则附加了hostname和physics.hostname属性
处理升级问题
随着业务的需要,我们后期可能会加入其他的服务.这个时候我们可以在配置文件（configmaps.yaml）中添加好采集规则之后,进行升级
cd /usr/local/src/helm-charts/charts/fluentd-elasticsearch
helm upgrade fluentd-elasticsearch .
一些问题的处理方式记录
- failed to read data from plugin storage file path="/var/log/kernel.pos/worker0/storage.json" error_class=Fluent::ConfigError error="Invalid contents (not object) in plugin storage file: '/var/log/kernel.pos/worker0/storage.json
rm /var/log/kernel.pos/worker0/storage.json
引用链接
https://github.com/helm/charts/tree/master/stable/kibana#configuration
请尽量按照自己期望的生活
email:18980489167@189.cn
