Loki Promtail 使用 multiline 对Java 堆栈日志进行多行处理的示例
promtail 使用multiline处理 java 多行日志文本的示例
Environment:
promtail_version: v2.7.1
origin Java log content：
log file content stdout-a.log:
2022-Dec-27 08:30:02.501 ERROR [com.alibaba.nacos.naming.beat.sender] [com.alibaba.nacos.client.naming] [CLIENT-BEAT] failed to send beat: {"cluster":"DEFAULT","ip":"10.200.216.51","metadata":{"preserved.register.source":"SPRING_CLOUD"},"period":5000,"port":8268,"scheduled":false,"serviceName":"DEFAULT_GROUP@@open-business-api","stopped":false,"weight":1.0}
java.lang.IllegalStateException: failed to req API:/nacos/v1/ns/instance/beat after all servers([192.168.1.5:8848]) tried: failed to req API:192.168.1.5:8848/nacos/v1/ns/instance/beat. code:500 msg: java.net.SocketTimeoutException: connect timed out
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:464)
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:386)
at com.alibaba.nacos.client.naming.net.NamingProxy.sendBeat(NamingProxy.java:309)
at com.alibaba.nacos.client.naming.beat.BeatReactor.run(BeatReactor.java:92)
at java.util.concurrent.Executors.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor.access01(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-Dec-27 08:33:05.212 INFO [aaaa] [bbb .....] .......
promtail configure file：
Promtail 配置文件内容 promtail-local-config.yaml：
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
pipeline_stages:
- match:
selector: '{job="varlogs"}'
stages:
- multiline:
firstline: '^\d{4}-\w{2,3}-\d{2} \d{1,2}:\d{2}:\d{2}.\d{3}'
max_wait_time: 3s
- regex:
expression: '(?P<time>^[\w-]+\s[\d:.]+) (?P<LEVEL>[[:upper:]]{4,6}) (?P<msg>(.|\n)*$)'
- labels:
LEVEL:
time:
- timestamp:
source: time
format: '2006-Jan-02 15:04:05.999'
location: "UTC"
- output:
source: msg
其中 pipeline 部分配置值解释如下：
run promtail local test：
本地调试测试：
cat stdout-a.log | ./promtail-linux-amd64 -stdin -inspect --dry-run -log.level=debug --config.file promtail-local-config.yaml
output:
Clients configured:
----------------------
url: http://localhost:3100/loki/api/v1/push
batchwait: 1s
batchsize: 1048576
follow_redirects: false
enable_http2: false
backoff_config:
min_period: 500ms
max_period: 5m0s
max_retries: 10
timeout: 10s
tenant_id: ""
stream_lag_labels: ""
[inspect: regex stage]:
{stages.Entry}.Extracted["LEVEL"]:
+: ERROR
{stages.Entry}.Extracted["msg"]:
+: [com.alibaba.nacos.naming.beat.sender] [com.alibaba.nacos.client.naming] [CLIENT-BEAT] failed to send beat: {"cluster":"DEFAULT","ip":"10.200.216.51","metadata":{"preserved.register.source":"SPRING_CLOUD"},"period":5000,"port":8268,"scheduled":false,"serviceName":"DEFAULT_GROUP@@operate-business-api","stopped":false,"weight":1.0}
java.lang.IllegalStateException: failed to req API:/nacos/v1/ns/instance/beat after all servers([192.168.10.5:8848]) tried: failed to req API:192.168.10.5:8848/nacos/v1/ns/instance/beat. code:500 msg: java.net.SocketTimeoutException: connect timed out
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:464)
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:386)
at com.alibaba.nacos.client.naming.net.NamingProxy.sendBeat(NamingProxy.java:309)
at com.alibaba.nacos.client.naming.beat.BeatReactor.run(BeatReactor.java:92)
at java.util.concurrent.Executors.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor.access01(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{stages.Entry}.Extracted["time"]:
+: 2022-Dec-27 08:30:02.501
[inspect: regex stage]:
{stages.Entry}.Extracted["LEVEL"]:
+: INFO
{stages.Entry}.Extracted["msg"]:
+: [aaaa sd*f*as*dfas .....] .......
{stages.Entry}.Extracted["time"]:
+: 2022-Dec-27 08:33:05.212
[inspect: labels stage]:
{stages.Entry}.Entry.Labels:
-: {__path__="/var/log/*log", job="varlogs"}
+: {LEVEL="ERROR", __path__="/var/log/*log", job="varlogs", time="2022-Dec-27 08:30:02.501"}
[inspect: labels stage]:
{stages.Entry}.Entry.Labels:
-: {__path__="/var/log/*log", job="varlogs"}
+: {LEVEL="INFO", __path__="/var/log/*log", job="varlogs", time="2022-Dec-27 08:33:05.212"}
[inspect: timestamp stage]:
{stages.Entry}.Entry.Entry.Timestamp:
-: 2022-12-27 21:37:55.565994731 +0800 CST
+: 2022-12-27 08:30:02.501 +0000 UTC
[inspect: timestamp stage]:
{stages.Entry}.Entry.Entry.Timestamp:
-: 2022-12-27 21:37:55.568107676 +0800 CST
+: 2022-12-27 08:33:05.212 +0000 UTC
[inspect: output stage]:
{stages.Entry}.Entry.Entry.Line:
-: 2022-Dec-27 08:30:02.501 ERROR [com.alibaba.nacos.naming.beat.sender] [com.alibaba.nacos.client.naming] [CLIENT-BEAT] failed to send beat: {"cluster":"DEFAULT","ip":"10.200.216.51","metadata":{"preserved.register.source":"SPRING_CLOUD"},"period":5000,"port":8268,"scheduled":false,"serviceName":"DEFAULT_GROUP@@operate-business-api","stopped":false,"weight":1.0}
java.lang.IllegalStateException: failed to req API:/nacos/v1/ns/instance/beat after all servers([192.168.10.5:8848]) tried: failed to req API:192.168.10.5:8848/nacos/v1/ns/instance/beat. code:500 msg: java.net.SocketTimeoutException: connect timed out
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:464)
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:386)
at com.alibaba.nacos.client.naming.net.NamingProxy.sendBeat(NamingProxy.java:309)
at com.alibaba.nacos.client.naming.beat.BeatReactor.run(BeatReactor.java:92)
at java.util.concurrent.Executors.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor.access01(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
+: [com.alibaba.nacos.naming.beat.sender] [com.alibaba.nacos.client.naming] [CLIENT-BEAT] failed to send beat: {"cluster":"DEFAULT","ip":"10.200.216.51","metadata":{"preserved.register.source":"SPRING_CLOUD"},"period":5000,"port":8268,"scheduled":false,"serviceName":"DEFAULT_GROUP@@operate-business-api","stopped":false,"weight":1.0}
java.lang.IllegalStateException: failed to req API:/nacos/v1/ns/instance/beat after all servers([192.168.10.5:8848]) tried: failed to req API:192.168.10.5:8848/nacos/v1/ns/instance/beat. code:500 msg: java.net.SocketTimeoutException: connect timed out
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:464)
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:386)
at com.alibaba.nacos.client.naming.net.NamingProxy.sendBeat(NamingProxy.java:309)
at com.alibaba.nacos.client.naming.beat.BeatReactor.run(BeatReactor.java:92)
at java.util.concurrent.Executors.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor.access01(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[inspect: output stage]:
{stages.Entry}.Entry.Entry.Line:
-: 2022-Dec-27 08:33:05.212 INFO [aaaa sd*f*as*dfas .....] .......
+: [aaaa sd*f*as*dfas .....] .......
2022-12-27T08:30:02.501+0000 {LEVEL="ERROR", __path__="/var/log/*log", job="varlogs", time="2022-Dec-27 08:30:02.501"} [com.alibaba.nacos.naming.beat.sender] [com.alibaba.nacos.client.naming] [CLIENT-BEAT] failed to send beat: {"cluster":"DEFAULT","ip":"10.200.216.51","metadata":{"preserved.register.source":"SPRING_CLOUD"},"period":5000,"port":8268,"scheduled":false,"serviceName":"DEFAULT_GROUP@@operate-business-api","stopped":false,"weight":1.0}
java.lang.IllegalStateException: failed to req API:/nacos/v1/ns/instance/beat after all servers([192.168.10.5:8848]) tried: failed to req API:192.168.10.5:8848/nacos/v1/ns/instance/beat. code:500 msg: java.net.SocketTimeoutException: connect timed out
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:464)
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:386)
at com.alibaba.nacos.client.naming.net.NamingProxy.sendBeat(NamingProxy.java:309)
at com.alibaba.nacos.client.naming.beat.BeatReactor.run(BeatReactor.java:92)
at java.util.concurrent.Executors.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor.access01(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-12-27T08:33:05.212+0000 {LEVEL="INFO", __path__="/var/log/*log", job="varlogs", time="2022-Dec-27 08:33:05.212"} [aaaa sd*f*as*dfas .....] .......
explain output：
PS:
Online regex test: https://regex101.com/
