grok 生产场景示例

/var/log/syslog

日志信息：

Oct 18 22:53:08 C1-M620-16 systemd[31255]: Listening on GnuPG network certificate management daemon.

grok表达式：

^%{SYSLOGBASE} %{GREEDYDATA:log_message}

解析结果：

{
  "pid": "31255",
  "program": "systemd",
  "logsource": "C1-M620-16",
  "log_message": "Listening on GnuPG network certificate management daemon.",
  "timestamp": "Oct 18 22:53:08"
}

/var/log/ceph/ceph.log

日志信息：

2019-10-22 11:40:04.675969 mgr.C2-M620-15 client.1354132 10.60.11.31:0/794903793 1242832 : cluster [DBG] pgmap v1242788: 1056 pgs: 1056 active+clean; 439GiB data, 868GiB used, 14.4TiB / 15.3TiB avail; 0B/s rd, 891KiB/s wr, 51op/s

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:logsource} %{NOTSPACE:client} %{NOTSPACE:client_ip} %{NOTSPACE} : %{NOTSPACE:program} \[%{NOTSPACE:log_level}\] %{GREEDYDATA:log_message}

解析结果：

{
  "program": "cluster",
  "log_level": "DBG",
  "client": "client.1354132",
  "client_ip": "10.60.11.31:0/794903793",
  "log_message": "pgmap v1242788: 1056 pgs: 1056 active+clean; 439GiB data, 868GiB used, 14.4TiB / 15.3TiB avail; 0B/s rd, 891KiB/s wr, 51op/s",
  "logsource": "mgr.C2-M620-15",
  "timestamp": "2019-10-22 11:40:04.675969"
}

/var/log/ceph/ceph.audit.log

日志信息：

2019-10-22 15:35:48.378098 mon.C1-M620-16 mon.0 10.60.11.16:6789/0 48846 : audit [DBG] from='client.? 10.60.11.16:0/4272646193' entity='client.admin' cmd=[{,",p,r,e,f,i,x,",:,",o,s,d, ,p,o,o,l, ,g,e,t,-,q,u,o,t,a,",,, ,",p,o,o,l,",:, ,",v,o,l,u,m,e,s,",,, ,",f,o,r,m,a,t,",:,",j,s,o,n,",}]: dispatch

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:logsource} %{NOTSPACE:client} %{NOTSPACE:client_ip} %{NOTSPACE} : %{NOTSPACE:program} \[%{NOTSPACE:log_level}\] %{GREEDYDATA:log_message}

解析结果：

{
  "log_level": "DBG",
  "client": "mon.0",
  "client_ip": "10.60.11.16:6789/0",
  "log_message": "from='client.? 10.60.11.16:0/4272646193' entity='client.admin' cmd=[{,\",p,r,e,f,i,x,\",:,\",o,s,d, ,p,o,o,l, ,g,e,t,-,q,u,o,t,a,\",,, ,\",p,o,o,l,\",:, ,\",v,o,l,u,m,e,s,\",,, ,\",f,o,r,m,a,t,\",:,\",j,s,o,n,\",}]: dispatch",
  "program": "audit",
  "logsource": "mon.C1-M620-16",
  "timestamp": "2019-10-22 15:35:48.378098"
}

/var/log/ceph/ceph-mds.*.log
/var/log/ceph/ceph-osd.*.log
/var/log/ceph/ceph-mon.*.log
/var/log/ceph/ceph-mgr.*.log

日志信息：

2019-10-22 15:41:06.569773 7f3b5d52b700  1 mgr send_beacon standby

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE}%{SPACE}+%{GREEDYDATA:log_message}

/var/log/haproxy.log

日志信息：

Oct 22 11:22:54 C1-M620-16 haproxy[13816]: 10.60.13.14:42686 [22/Oct/2019:11:22:54.028] os_placement_api os_placement_api/controller2 0/0/75 679 -- 293/28/0/1/0 0/0

grok表达式：

^%{HAPROXYTCP}

解析结果：

{
  "server_name": "controller2",
  "srvconn": "1",
  "actconn": "293",
  "time_queue": "0",
  "pid": "13816",
  "haproxy_second": "54",
  "program": "haproxy",
  "haproxy_year": "2019",
  "haproxy_time": "11:22:54",
  "client_port": "42686",
  "syslog_timestamp": "Oct 22 11:22:54",
  "backend_name": "os_placement_api",
  "beconn": "0",
  "client_ip": "10.60.13.14",
  "haproxy_milliseconds": "028",
  "haproxy_monthday": "22",
  "termination_state": "--",
  "feconn": "28",
  "srv_queue": "0",
  "syslog_server": "C1-M620-16",
  "bytes_read": "679",
  "haproxy_minute": "22",
  "haproxy_hour": "11",
  "retries": "0",
  "backend_queue": "0",
  "accept_date": "22/Oct/2019:11:22:54.028",
  "frontend_name": "os_placement_api",
  "time_duration": "75",
  "time_backend_connect": "0",
  "haproxy_month": "Oct"
}

/var/log/pacemaker.log

日志信息：

Oct 22 15:38:48 [1560] C1-M620-16       crmd:     info: do_state_transition:    State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response

grok表达式：

^%{SYSLOGTIMESTAMP:timestamp} \[%{NUMBER:pid}\] %{NOTSPACE:logsource}%{SPACE}+%{NOTSPACE:program}:%{SPACE}+%{NOTSPACE:log_level}: %{NOTSPACE:log_type}%{SPACE}+%{GREEDYDATA:log_message}

解析结果：

{
  "log_type": "do_state_transition:",
  "log_level": "info",
  "pid": "1560",
  "log_message": "State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response",
  "program": "crmd",
  "logsource": "C1-M620-16",
  "timestamp": "Oct 22 15:38:48"
}

/var/log/pcsd/pcsd.log

日志信息：

I, [2019-10-24T11:05:36.544037 #888]  INFO -- : Return Value: 0

grok表达式：

^%{NOTSPACE} \[%{NOTSPACE:timestamp} %{NOTSPACE}\]%{SPACE}+%{NOTSPACE:log_level} -- : %{GREEDYDATA:log_message}

日志信息：

10.60.13.32 - - [24/Oct/2019:11:05:36 +0800] "GET /remote/get_configs?cluster_name=openstack-controller-cluster HTTP/1.1" 200 1570 0.0534

grok表达式：

^%{IP:client_ip} - - \[(?<timestamp>.*)\] \"%{WORD:verb} %{NOTSPACE:request} HTTP\/%{NOTSPACE:httpversion}\" %{NUMBER:response} %{NUMBER:bytes} %{BASE16FLOAT:time}

日志信息：

10.60.13.32 - - [24/Oct/2019:11:05:36 CST] "GET /remote/get_configs?cluster_name=openstack-controller-cluster HTTP/1.1" 200 1570

grok表达式：

^%{IP:client_ip} - - \[(?<timestamp>.*)\] \"%{WORD:verb} %{NOTSPACE:request} HTTP\/%{NOTSPACE:httpversion}\" %{NUMBER:response} %{NUMBER:bytes}

-> /remote/get_configs?cluster_name=openstack-controller-cluster

grok表达式：

^- %{GREEDYDATA:log_message}

/var/log/rabbitmq/rabbit@*.log
/var/log/rabbitmq/rabbit@*-sasl.log

日志信息：

=WARNING REPORT==== 20-Oct-2019::10:25:25 ===
closing AMQP connection <0.6930.374> (10.60.13.32:45624 -> 10.60.13.16:5672 - nova-novncproxy:5107:77722cd1-81e7-479d-8960-242efe26b963, vhost: '/', user: 'admin'):
client unexpectedly closed TCP connection

grok表达式：

(?m)^=%{NOTSPACE:log_level} %{NOTSPACE}==== %{NOTSPACE:timestamp} ===%{GREEDYDATA:log_message}

/var/log/nova/nova-api.log

日志信息：

2019-10-23 06:34:47.001 4938 INFO nova.osapi_compute.wsgi.server [-] 10.60.13.16 "OPTIONS / HTTP/1.0" status: 200 len: 499 time: 0.0042779

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[-\] %{IP:client_ip} \"%{WORD:verb} %{NOTSPACE:request} HTTP\/%{NOTSPACE:httpversion}\" status: %{NUMBER:response} len: %{NUMBER:bytes} time: %{GREEDYDATA:time}

解析结果：

{
  "request": "/",
  "log_level": "INFO",
  "verb": "OPTIONS",
  "pid": "4938",
  "program": "nova.osapi_compute.wsgi.server",
  "response": "200",
  "bytes": "499",
  "client_ip": "10.60.13.16",
  "httpversion": "1.0",
  "time": "0.0042779",
  "timestamp": "2019-10-23 06:34:47.001"
}

日志信息：

2019-10-23 06:34:48.373 4924 INFO nova.osapi_compute.wsgi.server [req-33b64ce5-c3c6-487a-9be6-7ab861d881c2 02f572b883df493b9eed0b0d95562647 ea7764de8f334b8cb79e54e13eae434c - default default] 10.60.13.16 "GET /v2.1 HTTP/1.1" status: 302 len: 290 time: 0.2573490

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] %{IP:client_ip} \"%{WORD:verb} %{NOTSPACE:request} HTTP\/%{NOTSPACE:httpversion}\" status: %{NUMBER:response} len: %{NUMBER:bytes} time: %{GREEDYDATA:time}

解析结果：

{
  "request": "/v2.1",
  "log_level": "INFO",
  "verb": "GET",
  "pid": "4924",
  "program": "nova.osapi_compute.wsgi.server",
  "user_id": "02f572b883df493b9eed0b0d95562647",
  "project_id": "ea7764de8f334b8cb79e54e13eae434c",
  "user_domain": "default",
  "response": "302",
  "bytes": "290",
  "client_ip": "10.60.13.16",
  "httpversion": "1.1",
  "time": "0.2573490",
  "project_domain": "default",
  "timestamp": "2019-10-23 06:34:48.373",
  "request_id": "33b64ce5-c3c6-487a-9be6-7ab861d881c2"
}

日志信息：

2019-10-23 06:34:48.527 4924 ERROR oslo_db.sqlalchemy.engines [req-587a1d65-ce8e-41a1-b228-bbd5d7c69b5b 02f572b883df493b9eed0b0d95562647 ea7764de8f334b8cb79e54e13eae434c - default default] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1']

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] %{GREEDYDATA:log_message}

/var/log/nova/nova-conductor.log
/var/log/nova/nova-consoleauth.log
/var/log/nova/nova-novncproxy.log
/var/log/nova/nova-scheduler.log

日志信息：

2019-10-22 15:47:50.897 5023 INFO nova.scheduler.host_manager [req-8610b1c1-bd5d-4e1c-a79e-d0174f466867 - - - - -] Successfully synced instances from host 'C2-M620-2'.

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} .*\] %{GREEDYDATA:log_message}

解析结果：

{
  "log_level": "INFO",
  "pid": "5023",
  "log_message": "Successfully synced instances from host 'C2-M620-2'.",
  "program": "nova.scheduler.host_manager",
  "timestamp": "2019-10-22 15:47:50.897",
  "request_id": "8610b1c1-bd5d-4e1c-a79e-d0174f466867"
}

/var/log/glance/glance-api.log

日志信息：

2019-10-22 15:50:06.469 14058 INFO eventlet.wsgi.server [-] 10.60.13.16 - - [22/Oct/2019 15:50:06] "OPTIONS / HTTP/1.0" 200 94 0.001604

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[-\] %{IP:client_ip} - - \[(?<request_time>.*)\] \"%{WORD:verb} %{NOTSPACE:request} HTTP\/%{NOTSPACE:httpversion}\" %{NUMBER:response} %{NUMBER:bytes} %{GREEDYDATA:time}

解析结果：

{
  "request": "/",
  "log_level": "INFO",
  "verb": "OPTIONS",
  "pid": "14058",
  "program": "eventlet.wsgi.server",
  "request_time": "22/Oct/2019 15:50:06",
  "response": "200",
  "bytes": "94",
  "client_ip": "10.60.13.16",
  "httpversion": "1.0",
  "time": "0.001604",
  "timestamp": "2019-10-22 15:50:06.469"
}

日志信息：

2019-10-24 14:40:47.433 14065 INFO eventlet.wsgi.server [req-cc080dbc-bf43-4d1b-889c-9935e9c07e32 02f572b883df493b9eed0b0d95562647 ea7764de8f334b8cb79e54e13eae434c - default default] 10.60.13.16 - - [24/Oct/2019 14:40:47] "GET /v2/images?marker=eef3b511-aeea-4d8d-afec-e3eb0e03836f HTTP/1.1" 200 243 0.027979

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] %{IP:client_ip} - - \[(?<request_time>.*)\] \"%{WORD:verb} %{NOTSPACE:request} HTTP\/%{NOTSPACE:httpversion}\" %{NUMBER:response} %{NUMBER:bytes} %{GREEDYDATA:time}

解析结果：

{
  "request": "/v2/images?marker=eef3b511-aeea-4d8d-afec-e3eb0e03836f",
  "log_level": "INFO",
  "verb": "GET",
  "pid": "14065",
  "program": "eventlet.wsgi.server",
  "request_time": "24/Oct/2019 14:40:47",
  "user_id": "02f572b883df493b9eed0b0d95562647",
  "project_id": "ea7764de8f334b8cb79e54e13eae434c",
  "user_domain": "default",
  "response": "200",
  "bytes": "243",
  "client_ip": "10.60.13.16",
  "httpversion": "1.1",
  "time": "0.027979",
  "project_domain": "default",
  "request_id": "cc080dbc-bf43-4d1b-889c-9935e9c07e32",
  "timestamp": "2019-10-24 14:40:47.433"
}

日志信息：

2019-10-24 14:40:47.364 14065 ERROR oslo_db.sqlalchemy.engines [req-cd6f35d6-51a9-4c29-8b11-45389caf0b56 02f572b883df493b9eed0b0d95562647 ea7764de8f334b8cb79e54e13eae434c - default default] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1']
2019-10-24 14:40:47.364 14065 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] %{GREEDYDATA:log_message}

/var/log/cinder/cinder-scheduler.log

日志信息：

2019-10-21 05:21:59.672 8751 INFO cinder.message.api [req-df902f61-7f39-4f91-b957-8dfe740b2ae4 - - - - -] Deleted 0 expired messages.

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} .*\] %{GREEDYDATA:log_message}

解析结果：

{
  "log_level": "INFO",
  "pid": "8751",
  "log_message": "Deleted 0 expired messages.",
  "program": "cinder.message.api",
  "timestamp": "2019-10-21 05:21:59.672",
  "request_id": "df902f61-7f39-4f91-b957-8dfe740b2ae4"
}

/var/log/keystone/keystone-wsgi-public.log

日志信息：

2019-10-22 15:52:00.608 29526 INFO keystone.common.wsgi [req-a1932d49-8308-4ceb-a6c1-a1810d46fba1 32187e16f6224683a3275d9b1709cfde da0710ae6e814683ab922c60cd96e5d7 - default default] GET http://10.60.13.100:5000/v3/auth/tokens

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] %{WORD:verb} %{GREEDYDATA:request}

解析结果：

{
  "request": "http://10.60.13.100:5000/v3/auth/tokens",
  "log_level": "INFO",
  "verb": "GET",
  "pid": "29526",
  "program": "keystone.common.wsgi",
  "user_id": "32187e16f6224683a3275d9b1709cfde",
  "project_id": "da0710ae6e814683ab922c60cd96e5d7",
  "user_domain": "default",
  "project_domain": "default",
  "timestamp": "2019-10-22 15:52:00.608",
  "request_id": "a1932d49-8308-4ceb-a6c1-a1810d46fba1"
}

日志信息：

2019-10-22 15:52:00.973 29524 INFO keystone.common.wsgi [req-2a166e00-f527-4e5d-b721-8dbf2d63dfad - - - - -] GET http://10.60.13.100:5000/v3/

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} .*\] %{WORD:verb} %{GREEDYDATA:request}

解析结果：

{
  "request": "http://10.60.13.100:5000/v3/",
  "log_level": "INFO",
  "verb": "GET",
  "pid": "29524",
  "program": "keystone.common.wsgi",
  "timestamp": "2019-10-22 15:52:00.973",
  "request_id": "2a166e00-f527-4e5d-b721-8dbf2d63dfad"
}

/var/log/keystone/keystone-manage.log

日志信息：

2019-08-21 14:51:48.823 16802 INFO keystone.cmd.cli [req-9c47b587-5a9f-4b0d-97b8-53a004601571 - - - - -] Skipping public endpoint as already created

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} .*\] %{GREEDYDATA:log_message}

解析结果：

{
  "log_level": "INFO",
  "pid": "16802",
  "log_message": "Skipping public endpoint as already created",
  "program": "keystone.cmd.cli",
  "timestamp": "2019-08-21 14:51:48.823",
  "request_id": "9c47b587-5a9f-4b0d-97b8-53a004601571"
}

/var/log/neutron/neutron-server.log

日志信息：

2019-10-26 14:55:34.142 6680 INFO neutron.wsgi [req-c9700810-47de-444c-b9e8-e25a4464224b 9bb22287438a411787a4afe09cc9d925 da0710ae6e814683ab922c60cd96e5d7 - default default] 10.60.13.16 "GET /v2.0/networks?id=4f1ff150-f358-4311-ab64-19521df964fd HTTP/1.1" status: 200  len: 884 time: 0.1054780

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] %{IP:client_ip} \"%{WORD:verb} %{NOTSPACE:request} HTTP\/%{NOTSPACE:httpversion}\" status: %{NUMBER:response}%{SPACE}+len: %{BASE16FLOAT:bytes} time: %{GREEDYDATA:time}

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] %{GREEDYDATA:log_message}

/var/log/apache2/access.log

日志信息：

10.60.13.16 - - [22/Oct/2019:11:43:19 +0800] "OPTIONS / HTTP/1.0" 200 181 "-" "-"

grok表达式：

^%{COMBINEDAPACHELOG}  匹配组合日志

解析结果：

{
  "request": "/",
  "agent": "\"-\"",  <--
  "auth": "-",
  "ident": "-",
  "verb": "OPTIONS",
  "referrer": "\"-\"",  <--
  "response": "200",
  "bytes": "181",
  "clientip": "10.60.13.16",
  "httpversion": "1.0",
  "timestamp": "22/Oct/2019:11:43:19 +0800"
}

/var/log/apache2/cinder.log

日志信息：

10.60.13.16 - - [22/Oct/2019:12:00:01 +0800] "GET /v3/ea7764de8f334b8cb79e54e13eae434c/os-services HTTP/1.1" 200 459 "-" "python-cinderclient" 288383(us)

grok表达式：

^%{COMBINEDAPACHELOG} %{GREEDYDATA:time}

解析结果：

{
  "request": "/v3/ea7764de8f334b8cb79e54e13eae434c/os-services",
  "agent": "\"python-cinderclient\"",
  "auth": "-",
  "ident": "-",
  "verb": "GET",
  "referrer": "\"-\"",
  "response": "200",
  "bytes": "459",
  "clientip": "10.60.13.16",
  "httpversion": "1.1",
  "time": "288383(us)",
  "timestamp": "22/Oct/2019:12:00:01 +0800"
}

/var/log/apache2/keystone_access.log

日志信息：

10.60.13.16 - - [22/Oct/2019:12:05:16 +0800] "POST /v3/auth/tokens HTTP/1.1" 201 5104 "-" "neutron/12.0.6 keystonemiddleware.auth_token/4.21.0 keystoneauth1/3.17.0 python-requests/2.22.0 CPython/2.7.15+"

grok表达式：

^%{COMBINEDAPACHELOG}

解析结果：

{
  "request": "/v3/auth/tokens",
  "agent": "\"neutron/12.0.6 keystonemiddleware.auth_token/4.21.0 keystoneauth1/3.17.0 python-requests/2.22.0 CPython/2.7.15+\"",
  "auth": "-",
  "ident": "-",
  "verb": "POST",
  "referrer": "\"-\"",
  "response": "201",
  "bytes": "5104",
  "clientip": "10.60.13.16",
  "httpversion": "1.1",
  "timestamp": "22/Oct/2019:12:05:16 +0800"
}

/var/log/apache2/nova_placement_access.log

grok表达式：

^%{COMBINEDAPACHELOG}

/var/log/apache2/error.log

日志信息：

[Tue Oct 22 06:25:02.868623 2019] [mpm_event:notice] [pid 13824:tid 139685582539712] AH00493: SIGUSR1 received.  Doing graceful restart

grok表达式：

^%{HTTPD24_ERRORLOG}

解析结果：

{
  "module": "mpm_event",
  "loglevel": "notice",
  "pid": "13824",
  "message": "SIGUSR1 received.  Doing graceful restart",
  "tid": "139685582539712",
  "errorcode": "AH00493",
  "timestamp": "Tue Oct 22 06:25:02.868623 2019"
}

/var/log/apache2/keystone.log

日志信息：

2019-09-18 09:27:24.473403 Truncated or oversized response headers received from daemon process 'keystone-public': /usr/bin/keystone-wsgi-public

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:message}

解析结果：

{
  "message": "Truncated or oversized response headers received from daemon process 'keystone-public': /usr/bin/keystone-wsgi-public",
  "timestamp": "2019-09-18 09:27:24.473403"
}

/var/log/apache2/nova_placement_error.log

日志信息：

2019-10-22 12:15:32.388235 2019-10-22 12:15:32.387 29533 INFO nova.api.openstack.placement.requestlog [req-efdc4393-7544-4e6d-b71b-fe3ab4d5663c 74a6ed6f0e9740bcbdaf924109361a4e da0710ae6e814683ab922c60cd96e5d7 - default default] 10.60.13.16 "GET /resource_providers/56a3c5a5-ed9e-44a5-b0c3-85763c35e6e6/inventories" status: 200 len: 406 microversion: 1.0

grok表达式：

^%{TIMESTAMP_ISO8601} %{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[%{WORD}-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] %{IPV4:client_ip} \"%{WORD:verb} %{NOTSPACE:request}\" status\: %{NUMBER:response} len\: %{NUMBER:bytes} microversion\: %{NUMBER:httpversion}

解析结果：

{
  "client_ip": "10.60.13.16",
  "program": "nova.api.openstack.placement.requestlog",
  "log_level": "INFO",
  "pid": "29533",
  "verb": "GET",
  "httpversion": "1.0",
  "user_id": "74a6ed6f0e9740bcbdaf924109361a4e",
  "project_id": "da0710ae6e814683ab922c60cd96e5d7",
  "user_domain": "default",
  "timestamp": "2019-10-22 12:15:32.387",
  "response": "200",
  "bytes": "406",
  "project_domain": "default",
  "request_id": "efdc4393-7544-4e6d-b71b-fe3ab4d5663c",
  "request": "/resource_providers/56a3c5a5-ed9e-44a5-b0c3-85763c35e6e6/inventories"
}

/var/log/apache2/cinder_error.log

日志信息：

2019-10-22 15:00:07.090544 2019-10-22 15:00:07.090 29519 INFO cinder.api.openstack.wsgi [req-2ed8b30f-2ed1-4846-8e22-efa5ea1b1bf6 02f572b883df493b9eed0b0d95562647 ea7764de8f334b8cb79e54e13eae434c - default default] OPTIONS http://controller-1:8776/

grok表达式：

^%{TIMESTAMP_ISO8601} %{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] %{WORD:verb} (?<reguest>http://.*)

日志信息：

2019-10-24 10:53:02.143423 2019-10-24 10:53:02.142 21967 ERROR oslo_db.sqlalchemy.engines [req-225a3778-1739-4c79-afa8-6ac31c644bab 02f572b883df493b9eed0b0d95562647 ea7764de8f334b8cb79e54e13eae434c - default default] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1']

grok表达式：

^%{TIMESTAMP_ISO8601} %{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] %{GREEDYDATA:log_message}

日志信息：

2019-10-26 14:02:27.822646 2019-10-26 14:02:27.822 13036 INFO cinder.api.openstack.wsgi [req-b64dcd78-253a-41b6-ace3-698facfc1730 02f572b883df493b9eed0b0d95562647 ea7764de8f334b8cb79e54e13eae434c - default default] http://controller-1:8776/ returned with HTTP 300

grok表达式：

^%{TIMESTAMP_ISO8601} %{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] (?<reguest>http://.*) returned with HTTP %{NUMBER:response}

/var/log/nova/nova-compute.log

日志信息：

2019-10-22 15:18:55.381 27144 INFO nova.compute.resource_tracker [req-c8cf8279-77e0-449e-9290-7992a66c659d - - - - -] Final resource view: name=C1-M620-14 phys_ram=64368MB used_ram=512MB phys_disk=15640GB used_disk=0GB total_vcpus=24 used_vcpus=0 pci_stats=[]

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] %{GREEDYDATA:log_message}

/var/log/neutron/neutron-dhcp-agent.log

日志信息：

2019-10-15 17:27:18.249 1471 INFO neutron.agent.dhcp.agent [req-7294f983-b35e-454f-bd42-795392790f34 ab4cf2b0fb154e938e02d94be61b1ef8 c9dbfb9910cd4421a58a65b2ad408b6e - - -] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[], binding:host_id=, binding:profile=, binding:vif_details=, binding:vif_type=unbound, binding:vnic_type=normal, created_at=2019-10-15T09:27:17Z, description=, device_id=c58f288f-ead8-4d76-942f-55717ca4a9b5, device_owner=, extra_dhcp_opts=[], fixed_ips=[{u'subnet_id': u'0a708e34-aa55-45dc-9f5b-25fe2886465f', u'ip_address': u'10.60.101.30'}], id=4c868d82-af12-4e99-b99a-2c86e5e0b1bd, mac_address=fa:16:3e:75:9a:4c, name=, network_id=4f1ff150-f358-4311-ab64-19521df964fd, port_security_enabled=True, project_id=c9dbfb9910cd4421a58a65b2ad408b6e, revision_number=6, security_groups=[u'6b31721f-7c47-405b-a610-23f85b35637e'], status=DOWN, tags=[], tenant_id=c9dbfb9910cd4421a58a65b2ad408b6e, updated_at=2019-10-15T09:27:18Z

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] %{GREEDYDATA:log_message}

/var/log/neutron/neutron-metadata-agent.log

日志信息：

2019-10-20 23:29:59.768 1518 INFO eventlet.wsgi.server [-] 10.60.101.22,<local> "GET /metadata/instance?api-version=2017-04-02 HTTP/1.1" status: 404 len: 247 time: 0.0736740

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[-\] %{IP:client_ip}%{NOTSPACE} \"%{WORD:verb} %{NOTSPACE:request} HTTP\/%{NOTSPACE:httpversion}\" status: %{NUMBER:response} len: %{NUMBER:bytes} time: %{GREEDYDATA:time}

/var/log/neutron/neutron-openvswitch-agent.log

日志信息：

2019-10-19 11:47:41.324 27638 INFO neutron.agent.securitygroups_rpc [req-b9a61e39-9d7a-4844-9254-4f1f7f72f801 ab4cf2b0fb154e938e02d94be61b1ef8 7a79302e363b4e16ad72b32c63287876 - - -] Security group rule updated ['cc834728-9a0a-4e9c-a125-a3b97875b5fd']

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] %{GREEDYDATA:log_message}

/var/log/neutron/neutron-ovs-cleanup.log

日志信息：

2019-09-23 09:33:53.437 1279 INFO neutron.cmd.ovs_cleanup [-] OVS cleanup completed successfully

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[-\] %{GREEDYDATA:log_message}

/var/log/openvswitch/ovsdb-server.log

日志信息：

2019-10-21T22:25:01.402Z|00068|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log

grok表达式：

^%{NOTSPACE:timestamp}\|%{NOTSPACE}\|%{NOTSPACE}\|%{NOTSPACE:log_level}\|%{GREEDYDATA:log_message}

/var/log/openvswitch/ovs-vswitchd.log

日志信息：

2019-10-21T22:25:01.399Z|00115|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log

/var/log/cinder/cinder-volume.log

日志信息：

2019-10-12 16:06:59.172 12191 INFO cinder.volume.manager [req-3c50956d-8575-49a4-9582-98f3b4824a17 ab4cf2b0fb154e938e02d94be61b1ef8 c9dbfb9910cd4421a58a65b2ad408b6e - default default] attachment_update completed successfully.

grok表达式：

^%{TIMESTAMP_ISO8601:timestamp} %{NUMBER:pid} (?<log_level>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) %{NOTSPACE:program} \[req-%{NOTSPACE:request_id} %{NOTSPACE:user_id} %{NOTSPACE:project_id} - %{NOTSPACE:user_domain} %{NOTSPACE:project_domain}\] %{GREEDYDATA:log_message}

posted @ 2021-08-20 09:58 Varden 阅读(120) 评论(0) 编辑收藏举报

刷新页面返回顶部

Varden

grok 生产场景示例

公告