Loki Promtail 使用 multiline 对Java 堆栈日志进行多行处理的示例

promtail 使用multiline处理 java 多行日志文本的示例

Environment:

promtail_version: v2.7.1

origin Java log content：

log file content stdout-a.log:

2022-Dec-27 08:30:02.501 ERROR [com.alibaba.nacos.naming.beat.sender] [com.alibaba.nacos.client.naming] [CLIENT-BEAT] failed to send beat: {"cluster":"DEFAULT","ip":"10.200.216.51","metadata":{"preserved.register.source":"SPRING_CLOUD"},"period":5000,"port":8268,"scheduled":false,"serviceName":"DEFAULT_GROUP@@open-business-api","stopped":false,"weight":1.0}
java.lang.IllegalStateException: failed to req API:/nacos/v1/ns/instance/beat after all servers([192.168.1.5:8848]) tried: failed to req API:192.168.1.5:8848/nacos/v1/ns/instance/beat. code:500 msg: java.net.SocketTimeoutException: connect timed out
  at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:464)
  at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:386)
  at com.alibaba.nacos.client.naming.net.NamingProxy.sendBeat(NamingProxy.java:309)
  at com.alibaba.nacos.client.naming.beat.BeatReactor.run(BeatReactor.java:92)
  at java.util.concurrent.Executors.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ScheduledThreadPoolExecutor.access01(ScheduledThreadPoolExecutor.java:180)
  at java.util.concurrent.ScheduledThreadPoolExecutor.run(ScheduledThreadPoolExecutor.java:293)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
2022-Dec-27 08:33:05.212 INFO [aaaa] [bbb .....] .......

promtail configure file：

Promtail 配置文件内容 promtail-local-config.yaml：

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
  pipeline_stages:
  - match:
      selector: '{job="varlogs"}'
      stages:
      - multiline:
          firstline: '^\d{4}-\w{2,3}-\d{2} \d{1,2}:\d{2}:\d{2}.\d{3}'
          max_wait_time: 3s
      - regex:
          expression: '(?P<time>^[\w-]+\s[\d:.]+) (?P<LEVEL>[[:upper:]]{4,6}) (?P<msg>(.|\n)*$)'
      - labels:
          LEVEL:
          time:
      - timestamp:
          source: time
          format: '2006-Jan-02 15:04:05.999'
          location: "UTC"
      - output:
          source: msg

其中 pipeline 部分配置值解释如下：

run promtail local test：

本地调试测试：

cat stdout-a.log | ./promtail-linux-amd64 -stdin -inspect --dry-run -log.level=debug --config.file promtail-local-config.yaml

output:

Clients configured:
----------------------
url: http://localhost:3100/loki/api/v1/push
batchwait: 1s
batchsize: 1048576
follow_redirects: false
enable_http2: false
backoff_config:
  min_period: 500ms
  max_period: 5m0s
  max_retries: 10
timeout: 10s
tenant_id: ""
stream_lag_labels: ""

[inspect: regex stage]:
{stages.Entry}.Extracted["LEVEL"]:
        +: ERROR
{stages.Entry}.Extracted["msg"]:
        +: [com.alibaba.nacos.naming.beat.sender] [com.alibaba.nacos.client.naming] [CLIENT-BEAT] failed to send beat: {"cluster":"DEFAULT","ip":"10.200.216.51","metadata":{"preserved.register.source":"SPRING_CLOUD"},"period":5000,"port":8268,"scheduled":false,"serviceName":"DEFAULT_GROUP@@operate-business-api","stopped":false,"weight":1.0}
java.lang.IllegalStateException: failed to req API:/nacos/v1/ns/instance/beat after all servers([192.168.10.5:8848]) tried: failed to req API:192.168.10.5:8848/nacos/v1/ns/instance/beat. code:500 msg: java.net.SocketTimeoutException: connect timed out
  at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:464)
  at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:386)
  at com.alibaba.nacos.client.naming.net.NamingProxy.sendBeat(NamingProxy.java:309)
  at com.alibaba.nacos.client.naming.beat.BeatReactor.run(BeatReactor.java:92)
  at java.util.concurrent.Executors.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ScheduledThreadPoolExecutor.access01(ScheduledThreadPoolExecutor.java:180)
  at java.util.concurrent.ScheduledThreadPoolExecutor.run(ScheduledThreadPoolExecutor.java:293)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
{stages.Entry}.Extracted["time"]:
        +: 2022-Dec-27 08:30:02.501
[inspect: regex stage]:
{stages.Entry}.Extracted["LEVEL"]:
        +: INFO
{stages.Entry}.Extracted["msg"]:
        +: [aaaa sd*f*as*dfas .....] .......
{stages.Entry}.Extracted["time"]:
        +: 2022-Dec-27 08:33:05.212
[inspect: labels stage]:
{stages.Entry}.Entry.Labels:
        -: {__path__="/var/log/*log", job="varlogs"}
        +: {LEVEL="ERROR", __path__="/var/log/*log", job="varlogs", time="2022-Dec-27 08:30:02.501"}
[inspect: labels stage]:
{stages.Entry}.Entry.Labels:
        -: {__path__="/var/log/*log", job="varlogs"}
        +: {LEVEL="INFO", __path__="/var/log/*log", job="varlogs", time="2022-Dec-27 08:33:05.212"}
[inspect: timestamp stage]:
{stages.Entry}.Entry.Entry.Timestamp:
        -: 2022-12-27 21:37:55.565994731 +0800 CST
        +: 2022-12-27 08:30:02.501 +0000 UTC
[inspect: timestamp stage]:
{stages.Entry}.Entry.Entry.Timestamp:
        -: 2022-12-27 21:37:55.568107676 +0800 CST
        +: 2022-12-27 08:33:05.212 +0000 UTC
[inspect: output stage]:
{stages.Entry}.Entry.Entry.Line:
        -: 2022-Dec-27 08:30:02.501 ERROR [com.alibaba.nacos.naming.beat.sender] [com.alibaba.nacos.client.naming] [CLIENT-BEAT] failed to send beat: {"cluster":"DEFAULT","ip":"10.200.216.51","metadata":{"preserved.register.source":"SPRING_CLOUD"},"period":5000,"port":8268,"scheduled":false,"serviceName":"DEFAULT_GROUP@@operate-business-api","stopped":false,"weight":1.0}
java.lang.IllegalStateException: failed to req API:/nacos/v1/ns/instance/beat after all servers([192.168.10.5:8848]) tried: failed to req API:192.168.10.5:8848/nacos/v1/ns/instance/beat. code:500 msg: java.net.SocketTimeoutException: connect timed out
  at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:464)
  at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:386)
  at com.alibaba.nacos.client.naming.net.NamingProxy.sendBeat(NamingProxy.java:309)
  at com.alibaba.nacos.client.naming.beat.BeatReactor.run(BeatReactor.java:92)
  at java.util.concurrent.Executors.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ScheduledThreadPoolExecutor.access01(ScheduledThreadPoolExecutor.java:180)
  at java.util.concurrent.ScheduledThreadPoolExecutor.run(ScheduledThreadPoolExecutor.java:293)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
        +: [com.alibaba.nacos.naming.beat.sender] [com.alibaba.nacos.client.naming] [CLIENT-BEAT] failed to send beat: {"cluster":"DEFAULT","ip":"10.200.216.51","metadata":{"preserved.register.source":"SPRING_CLOUD"},"period":5000,"port":8268,"scheduled":false,"serviceName":"DEFAULT_GROUP@@operate-business-api","stopped":false,"weight":1.0}
java.lang.IllegalStateException: failed to req API:/nacos/v1/ns/instance/beat after all servers([192.168.10.5:8848]) tried: failed to req API:192.168.10.5:8848/nacos/v1/ns/instance/beat. code:500 msg: java.net.SocketTimeoutException: connect timed out
  at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:464)
  at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:386)
  at com.alibaba.nacos.client.naming.net.NamingProxy.sendBeat(NamingProxy.java:309)
  at com.alibaba.nacos.client.naming.beat.BeatReactor.run(BeatReactor.java:92)
  at java.util.concurrent.Executors.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ScheduledThreadPoolExecutor.access01(ScheduledThreadPoolExecutor.java:180)
  at java.util.concurrent.ScheduledThreadPoolExecutor.run(ScheduledThreadPoolExecutor.java:293)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
[inspect: output stage]:
{stages.Entry}.Entry.Entry.Line:
        -: 2022-Dec-27 08:33:05.212 INFO [aaaa sd*f*as*dfas .....] .......
        +: [aaaa sd*f*as*dfas .....] .......
2022-12-27T08:30:02.501+0000    {LEVEL="ERROR", __path__="/var/log/*log", job="varlogs", time="2022-Dec-27 08:30:02.501"}       [com.alibaba.nacos.naming.beat.sender] [com.alibaba.nacos.client.naming] [CLIENT-BEAT] failed to send beat: {"cluster":"DEFAULT","ip":"10.200.216.51","metadata":{"preserved.register.source":"SPRING_CLOUD"},"period":5000,"port":8268,"scheduled":false,"serviceName":"DEFAULT_GROUP@@operate-business-api","stopped":false,"weight":1.0}
java.lang.IllegalStateException: failed to req API:/nacos/v1/ns/instance/beat after all servers([192.168.10.5:8848]) tried: failed to req API:192.168.10.5:8848/nacos/v1/ns/instance/beat. code:500 msg: java.net.SocketTimeoutException: connect timed out
  at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:464)
  at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:386)
  at com.alibaba.nacos.client.naming.net.NamingProxy.sendBeat(NamingProxy.java:309)
  at com.alibaba.nacos.client.naming.beat.BeatReactor.run(BeatReactor.java:92)
  at java.util.concurrent.Executors.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ScheduledThreadPoolExecutor.access01(ScheduledThreadPoolExecutor.java:180)
  at java.util.concurrent.ScheduledThreadPoolExecutor.run(ScheduledThreadPoolExecutor.java:293)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
2022-12-27T08:33:05.212+0000    {LEVEL="INFO", __path__="/var/log/*log", job="varlogs", time="2022-Dec-27 08:33:05.212"}        [aaaa sd*f*as*dfas .....] .......