利用Filebeat+ELK 来收集.Net Core 微服务日志 (一)

对于所有的服务的日志，应该有相同的格式，收集到一起，称为日志中心，方便发现错误的时候，在统一的一个地方可以debug

整个项目结构图

 

集成后效果图：

 

 

 

拿出错日志为例：

一：建立你的service （我使用 WEB API为例）

1.引用nlog 负责 日志记录 并配置 nlog.config  配置文件如下： 

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true"
      internalLogLevel="Warn"
      internalLogFile="internal-nlog.txt">
  <extensions>
    <!--enable NLog.Web for ASP.NET Core-->
    <add assembly="NLog.Web.AspNetCore"/>
  </extensions>
  <!-- define various log targets -->
  <!--定义日志文件 docker  目录-->
  <variable name="logDirectory" value="/data/logs"/>
  <!--定义本机日志文件  目录-->
  <!--<variable name="logDirectory" value="${basedir}/logs"/>-->
  <!--定义本机你的服务名-->
  <variable name="ServceName" value="API001"/>
  <targets async="true">
    
    <!-- 本地文件日志target --> 
    <!-- 约定以 err 为文件后缀的日志文件记录了程序输出的警告或者错误。 -->
    <target name="ERROR_LOG_FILE"
            xsi:type="File"
            layout="[${longdate}]#${ServceName}#${message}#"
            encoding="utf-8"
            archiveNumbering="Date"
            archiveEvery="Day"
            archiveDateFormat="yyyy-MM-dd"
            fileName="${logDirectory}/${ServceName}/${shortdate}/${shortdate}.err" />

    <!--grok 规则-->
    <!--\[%{DATA:time}\]#%{DATA:ser_name}\#%{DATA:req_url}\#%{DATA:req_method}\#%{DATA:rep_param}\#\s*(?<err_msg>([\s\S]*))-->
    <!--空白-->
    <target xsi:type="Null" name="blackhole" />
  </targets>
  <!--日志级别 Trace -》Debug-》 Info -》Warn-》 Error-》 Fatal-->
  <!--日志规则-->
  <rules>
    <!-- 记录所有日志级别不低于 Warn 的日志到日志文件 -->
    <logger name="*" minlevel="Error" writeTo="ERROR_LOG_FILE" /> 
    <!--自定义日志，排除Microsoft日志-->
    <logger name="Microsoft.*" minlevel="Trace" writeTo="blackhole" final="true" />
  </rules>
</nlog>

 

2.编写全局异常中间件：

   /// <summary>
    /// 全局异常处理中间件
    /// </summary>
    public class GlobalErrorHandlingMiddleware
    {
        private readonly RequestDelegate next;
        private readonly ILogger<GlobalErrorHandlingMiddleware> _logger;

        public GlobalErrorHandlingMiddleware(RequestDelegate next, ILogger<GlobalErrorHandlingMiddleware> logger)
        {
            this.next = next;
            this._logger = logger;
        }

        public async Task Invoke(HttpContext context)
        {
            try
            {
                await next(context);
            }
            catch (Exception ex)
            { 

                var Request = context.Request;
                ///访问路径
                string visit_url = Request.Path;
                ///URL 请求方法
                string method = Request.Method.ToUpper();
                ///URL 请求的参数
                string url_paramters = string.Empty;

                if (method == "GET") url_paramters = Request.QueryString.Value;


                if (method == "POST")
                {
                    foreach (var item in Request.Form)
                        url_paramters = url_paramters + item.Key + "=" + item.Value + "&";
                }

                ///错识信息
                string err_msg = ex.StackTrace;

                ///日志格式内容
                var logs_msg = $"{visit_url}#{method}#{url_paramters}#{err_msg}";

                _logger.LogError(logs_msg);

                var statusCode = context.Response.StatusCode;

                var msg = $"错误代码：{statusCode}，提示信息：{ex.Message}";

                await HandleExceptionAsync(context, msg);
            }
        }

        private static Task HandleExceptionAsync(HttpContext context, string msg)
        {
            var data = new Result { Title = "异常中间件返回", Msg = msg };
            var result = JsonConvert.SerializeObject(data);
            context.Response.ContentType = "application/json;charset=utf-8";
            return context.Response.WriteAsync(result);
        }
    }

 

3.在API Startup.cs 文件配置nlog

        public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }

            //添加NLog
            loggerFactory.AddNLog();

            env.ConfigureNLog(Path.Combine("configs", "nlog.config"));

            //异常处理中间件
            app.UseMiddleware<GlobalErrorHandlingMiddleware>();         

            app.UseMvc();
        }

 

到这里。API 出错就会捕获至日志里，日志格式如下：

[2018-06-26 15:07:22.1562]#API001#/api/get#GET#?id=ad#   at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
   at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)
   at API001.Controllers.ValuesController.test(String id) in D:\code\study_demo\API001\Controllers\ValuesController.cs:line 26
   at lambda_method(Closure , Object , Object[] )
   at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionMethodAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextActionFilterAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at API001.GlobalErrorHandlingMiddleware.<Invoke>d__3.MoveNext() in D:\code\study_demo\API001\GlobalErrorHandlingMiddleware.cs:line 28#

 

到这里你的service 负责日志创造的部份已完成

注意：使用docker 运行容器的同志，要将 服务的日志目录挂载 至 filebeat 目录。这样方便统一监控目录

 ///将日志挂载到filebeat 目录
sudo docker run --name api001 -e "ASPNETCORE_URLS=http://+:5004" --restart=always -d -p 5004:5004 -v /home/aimei/apps/api001:/publish  -v /home/aimei/filebeat/logs:/data/logs -v /etc/localtime:/etc/localtime -w /publish --network=default 730e3899d926 dotnet API001.dll