nginx日志分析工具

安装goaccess 

$ wget -O - https://deb.goaccess.io/gnugpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/goaccess.gpg >/dev/null
$ echo "deb [signed-by=/usr/share/keyrings/goaccess.gpg] https://deb.goaccess.io/ $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/goaccess.list
$ sudo apt-get update
$ sudo apt-get install goaccess

goaccess使用帮助

GoAccess - 1.5.5

Usage: goaccess [filename] [ options ... ] [-c][-M][-H][-S][-q][-d][...]
The following options can also be supplied to the command:

LOG & DATE FORMAT OPTIONS

  --date-format=<dateformat>      - Specify log date format. e.g., %d/%b/%Y
  --log-format=<logformat>        - Specify log format. Inner quotes need escaping, or use single quotes.
  --time-format=<timeformat>      - Specify log time format. e.g., %H:%M:%S


USER INTERFACE OPTIONS

  -c --config-dialog              - Prompt log/date/time configuration window.
  -i --hl-header                  - Color highlight active panel.
  -m --with-mouse                 - Enable mouse support on main dashboard.
  --color=<fg:bg[attrs, PANEL]>   - Specify custom colors. See manpage for more details.
  --color-scheme=<1|2|3>          - Schemes: 1 => Grey, 2 => Green, 3 => Monokai.
  --html-custom-css=<path.css>    - Specify a custom CSS file in the HTML report.
  --html-custom-js=<path.js>      - Specify a custom JS file in the HTML report.
  --html-prefs=<json_obj>         - Set default HTML report preferences.
  --html-report-title=<title>     - Set HTML report page title and header.
  --html-refresh=<secs>           - Refresh HTML report every X seconds (>=1 or <=60).
  --json-pretty-print             - Format JSON output w/ tabs & newlines.
  --max-items                     - Maximum number of items to show per panel. See man page for limits.
  --no-color                      - Disable colored output.
  --no-column-names               - Don't write column names in term output.
  --no-csv-summary                - Disable summary metrics on the CSV output.
  --no-html-last-updated          - Hide HTML last updated field.
  --no-parsing-spinner            - Disable progress metrics and parsing spinner.
  --no-progress                   - Disable progress metrics.
  --no-tab-scroll                 - Disable scrolling through panels on TAB.

SERVER OPTIONS

  --addr=<addr>                   - Specify IP address to bind server to.
  --unix-socket=<addr>            - Specify UNIX-domain socket address to bind server to.
  --daemonize                     - Run as daemon (if --real-time-html enabled).
  --fifo-in=<path>                - Path to read named pipe (FIFO).
  --fifo-out=<path>               - Path to write named pipe (FIFO).
  --origin=<addr>                 - Ensure clients send this origin header upon the WebSocket handshake.
  --pid-file=<path>               - Write PID to a file when --daemonize is used.
  --port=<port>                   - Specify the port to use.
  --real-time-html                - Enable real-time HTML output.
  --ssl-cert=<cert.crt>           - Path to TLS/SSL certificate.
  --ssl-key=<priv.key>            - Path to TLS/SSL private key.
  --user-name=<username>          - Run as the specified user.
  --ws-url=<url>                  - URL to which the WebSocket server responds.

FILE OPTIONS

  -                               - The log file to parse is read from stdin.
  -f --log-file=<filename>        - Path to input log file.
  -l --debug-file=<filename>      - Send all debug messages to the specified file.
  -p --config-file=<filename>     - Custom configuration file.
  -S --log-size=<number>          - Specify the log size, useful when piping in logs.
  --invalid-requests=<filename>   - Log invalid requests to the specified file.
  --no-global-config              - Don't load global configuration file.
  --unknowns-log=<filename>       - Log unknown browsers and OSs to the specified file.

PARSE OPTIONS

  -a --agent-list                 - Enable a list of user-agents by host.
  -b --browsers-file=<path>       - Use additional custom list of browsers.
  -d --with-output-resolver       - Enable IP resolver on HTML|JSON output.
  -e --exclude-ip=<IP>            - Exclude one or multiple IPv4/6. Allows IP ranges
                                    e.g. 192.168.0.1-192.168.0.10
  -H --http-protocol=<yes|no>     - Set/unset HTTP request protocol if found.
  -M --http-method=<yes|no>       - Set/unset HTTP request method if found.
  -o --output=file.html|json|csv  - Output either an HTML, JSON or a CSV file.
  -q --no-query-string            - Strip request's query string. This can decrease memory consumption.
  -r --no-term-resolver           - Disable IP resolver on terminal output.
  --444-as-404                    - Treat non-standard status code 444 as 404.
  --4xx-to-unique-count           - Add 4xx client errors to the unique visitors count.
  --all-static-files              - Include static files with a query string.
  --anonymize-ip                  - Anonymize IP addresses before outputting to report.
  --crawlers-only                 - Parse and display only crawlers.
  --date-spec=<date|hr>           - Date specificity. Possible values: `date` (default), or `hr`.
  --double-decode                 - Decode double-encoded values.
  --enable-panel=<PANEL>          - Enable parsing/displaying the given panel.
  --hide-referrer=<NEEDLE>        - Hide a referrer but still count it. Wild cards are allowed.
                                    i.e., *.bing.com
  --hour-spec=<hr|min>            - Hour specificity. Possible values: `hr` (default),
                                    or `min` (tenth of a min).
  --ignore-crawlers               - Ignore crawlers.
  --ignore-panel=<PANEL>          - Ignore parsing/displaying the given panel.
  --ignore-referrer=<NEEDLE>      - Ignore a referrer from being counted. Wild cards are allowed.
                                    i.e., *.bing.com
  --ignore-statics=<req|panel>    - Ignore static requests.
                                    req => Ignore from valid requests.
                                    panel => Ignore from valid requests and panels.
  --ignore-status=<CODE>          - Ignore parsing the given status code.
  --keep-last=<NDAYS>             - Keep the last NDAYS in storage.
  --no-ip-validation              - Disable client IPv4/6  validation.
  --no-strict-status              - Disable HTTP status code validation.
  --num-tests=<number>            - Number of lines to test. >= 0 (10 default)
  --persist                       - Persist data to disk on exit to the given --db-path or to /tmp.
  --process-and-exit              - Parse log and exit without outputting data.
  --real-os                       - Display real OS names. e.g, Windows XP, Snow Leopard.
  --restore                       - Restore data from disk from the given --db-path or from /tmp.
  --sort-panel=PANEL,METRIC,ORDER - Sort panel on initial load. e.g., --sort-panel=VISITORS,BY_HITS,ASC.
                                    See manpage for a list of panels/fields.
  --static-file=<extension>       - Add static file extension. e.g.: .mp3. Extensions are case sensitive.

GEOIP OPTIONS

  --geoip-database=<path>         - Specify path to GeoIP database file.
                                    i.e., GeoLiteCity.dat, GeoIPv6.dat ...

OTHER OPTIONS

  -h --help                       - This help.
  -s --storage                    - Display current storage method. e.g., Hash.
  -V --version                    - Display version information and exit.
  --dcf                           - Display the path of the default config file when `-p` is not used.

Examples can be found by running `man goaccess`.

For more details visit: https://goaccess.io/
GoAccess Copyright (C) 2009-2020 by Gerardo Orellana

添加中文支持

apt -y install language-pack-zh-hans
export LANG=zh_CN.UTF-8

查看nginx记录格式

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" $request_time $upstream_response_time $request_method $host $uri "$request_body" '
                      '"$http_user_agent" "$http_x_forwarded_for" $upstream_cache_status ';

添加goaccess配置

和nginx日志记录格式对应

time-format %T
date-format %d/%b/%Y
log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" %T %^ %m %v %^ "%r_body" "%u" "%^"

生成报表

goaccess -f /usr/local/nginx/logs/access.log -p ./nginlog.conf -o report.html