ELK系列~Nxlog日志收集加转发(解决log4日志换行导致json转换失败问题)
本文章将会继承上一篇文章,主要讲通过工具来进行日志的收集与发送,《ELK系列~NLog.Targets.Fluentd到达如何通过tcp发到fluentd》
Nxlog是一个日志收集工具,它将系统日志,或者指定的日志文件,统配符文件找到,然后加工,最后发送到目标位置。而目标位置有很多种,如文件系统,fluentd系统等,下面我们介绍一个使用场景,也是经常涉及到的场景。
- log4产生日期,日期文件名,统一后缀,按日志级别命名
- nxlog工具,配置,启动,发送数据
- fluentd配置,接受数据,打印数据
一 log4产生日期,日期文件名,统一后缀,按日志级别命名
<?xml version="1.0"?> <configuration> <configSections> <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler,log4net" /> </configSections> <system.web> <compilation debug="true" targetFramework="4.0" /> </system.web> <log4net> <logger name="ErrorLog"> <level value="ERROR"></level> <appender-ref ref="ErrorLog"></appender-ref> </logger> <appender name="ErrorLog" type="log4net.Appender.RollingFileAppender"> <param name="File" value="logs\\" /> <param name="AppendToFile" value="true" /> <param name="StaticLogFileName" value="false" /> <rollingStyle value="Date" /> <datePattern value="yyyyMMdd"Error.log"" /> <layout type="log4net.Layout.PatternLayout"> </layout> </appender> <logger name="InfoLog"> <level value="INFO"></level> <appender-ref ref="InfoLog"></appender-ref> </logger> <appender name="InfoLog" type="log4net.Appender.RollingFileAppender"> <param name="File" value="logs\\" /> <param name="AppendToFile" value="true" /> <param name="StaticLogFileName" value="false" /> <rollingStyle value="Date" /> <datePattern value="yyyyMMdd"Info.log"" /> <layout type="log4net.Layout.PatternLayout"> </layout> </appender> <logger name="WarnLog"> <level value="Warn"></level> <appender-ref ref="WarnLog"></appender-ref> </logger> <appender name="WarnLog" type="log4net.Appender.RollingFileAppender"> <param name="File" value="logs\\" /> <param name="AppendToFile" value="true" /> <param name="StaticLogFileName" value="false" /> <rollingStyle value="Date" /> <datePattern value="yyyyMMdd"Warn.log"" /> <layout type="log4net.Layout.PatternLayout"> </layout> </appender> <logger name="FatalLog"> <level value="Fatal"></level> <appender-ref ref="FatalLog"></appender-ref> </logger> <appender name="FatalLog" type="log4net.Appender.RollingFileAppender"> <param name="File" value="logs\\" /> <param name="AppendToFile" value="true" /> <param name="StaticLogFileName" value="false" /> <rollingStyle value="Date" /> <datePattern value="yyyyMMdd"Fatal.log"" /> <layout type="log4net.Layout.PatternLayout"> </layout> </appender> </log4net> </configuration>
产生日志的方法如下
log4net.Config.XmlConfigurator.Configure(new FileInfo("log4.config")); for (int i = 0; i < 10; i++) { LogHelper.Info("test"); } Console.WriteLine("end");
上面代码将会产生日志文件,命名比较固定,方法日志的收集工作。
二 nxlog工具,配置,启动,发送数据
安装:https://docs.fluentd.org/v0.12/articles/windows
配置:去掉了log4产生日志每行后面的\r标记,如果不进行处理,在json转换时会有问题
## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/docs/ ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension json> Module xm_json </Extension> <Extension _syslog> Module xm_syslog </Extension> <Input in> Module im_file File "c:\dotnet\20*.log" Exec $raw_event = replace($raw_event, "\r\n", " "); Exec $raw_event = replace($raw_event, "\r", " "); Exec $raw_event = replace($raw_event, "\n", " "); Exec $raw_event = replace($raw_event, "0x0A", " "); Exec $raw_event = replace($raw_event, "0x0DA", " "); Exec $raw_event = replace($raw_event, "0x0D", " "); </Input> <Output out> Module om_tcp Host 192.168.200.214 Port 24224 Exec $raw_event =$raw_event + "\n"; </Output> <Route 1> Path in => out </Route>
启动:nxlog -f -c conf/nxlog.conf
查看:打开fluentd端,查看它的日志,发现我们的日志已经过来了
这个东西,我在google上找了好久,最后总算是功夫不负有心人,让我找对了!当然也证明了最初的猜想是正确的,即在output中对字符进行过滤!
三 fluentd配置,接受数据,打印数据
1 运行脚本,升级需要的文件夹和文件
mkdir /scripts/fluentd -p cd /scripts/fluentd cat > Dockerfile << 'EOF' FROM fluent/fluentd:v0.12-onbuild ENV TZ=Asia/Shanghai RUN echo "http://mirrors.aliyun.com/alpine/v3.5/main" >/etc/apk/repositories \ && echo "http://mirrors.aliyun.com/alpine/v3.5/community" >>/etc/apk/repositories \ && apk add --update tzdata \ && apk add curl \ && ln -snf /usr/share/zoneinfo/$TZ /etc/localtime \ && echo $TZ > /etc/timezone RUN apk add --virtual .build-deps \ sudo build-base ruby-dev \ && sudo gem sources --clear-all \ && apk del .build-deps \ && rm -rf /var/cache/apk/* \
/home/fluent/.gem/ruby/2.3.0/cache/*.gem EOF mkdir plugins -p mkdir -p /srv/volume/fluentd/ cat > /scripts/fluentd/fluent.conf << 'EOF' <source> @type forward port 24224 bind 0.0.0.0 </source> <match **> @type stdout
</match> EOF
2 生成镜像
docker build --no-cache --pull -t pilipa/tools/fluentd ./
2 docker上直接运行
docker run --privileged=true -v /scripts/fluentd/fluent.conf:/fluentd/etc/fluent.conf pilipa/tools/fluentd
启动后出现了配置和日志相关的输出信息
日志收集总算可以告一段落了!
感谢各位的阅读!1024节日快乐!