logstash使用grok过滤数据

有一段线上日志:

2020-07-14 11:37:04.556 INFO  [com.lyf.action.PlayAction:124] - [ 播放日志 add ] userid: 0 vid: 8079245, vtime: -1
┏━━━━━ Debug [native.update d_stcs_month_page set vcount = ifnul...] ━━━
┣ SQL:  update d_stcs_month_page set vcount = ifnull(vcount,0) + 1 where deptid = ? and vyear = ? and vmonth = ?
┣ 参数:         [0, 2020, 7]
┣ 位置:         com.chaoxing.video.mobile.front.service.stcs.PageStcsThreadTask.run(PageStcsThreadTask.java:76)
┣ 时间:         1ms
┣ 更新:         [1]
┗━━━━━ Debug [native.update d_stcs_month_page set vcount = ifnul...] ━━━

想把log打印的内容部分通过grok读取出来(前提:使用了多行合并),也就是[ 播放日志 add ] userid: 0 vid: 8079245, vtime: -1这一段,下面是过滤规则:

filter {
  if [type] == "debug" {
    grok {
      match => ["message", "%{TIMESTAMP_ISO8601:logdate} %{LOGLEVEL:level} (?<class>.*) - (?<log>[^┏]*)(?<sql>.*)"]
    }
    date {
      match => ["logdate", "yyyy-MM-dd HH:mm:ss.SSS"]
    }
  }
}

grokdebug分析结果如下:

{
  "logdate": [
    [
      "2020-07-14 11:37:04.556"
    ]
  ],
  "YEAR": [
    [
      "2020"
    ]
  ],
  "MONTHNUM": [
    [
      "07"
    ]
  ],
  "MONTHDAY": [
    [
      "14"
    ]
  ],
  "HOUR": [
    [
      "11",
      null
    ]
  ],
  "MINUTE": [
    [
      "37",
      null
    ]
  ],
  "SECOND": [
    [
      "04.556"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "level": [
    [
      "INFO"
    ]
  ],
  "class": [
    [
      " [com.lyf.action.PlayAction:124]"
    ]
  ],
  "log": [
    [
      "[ 播放日志 add ] userid: 0 vid: 8079245, vtime: -1\n"
    ]
  ],
  "sql": [
    [
      "┏━━━━━ Debug [native.update d_stcs_month_page set vcount = ifnul...] ━━━"
    ]
  ]
}
posted @ 2020-07-14 14:30  林宇风  阅读(936)  评论(0编辑  收藏  举报