可视化日志--goaccess

可视化日志--goaccess

http://www.goaccess.io/

http://goaccess.io/manual.php
http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
http://dev.maxmind.com/geoip/legacy/downloadable/

http://www.cnblogs.com/yjf512/p/3640346.html


YUM安装

rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

yum -y install goaccess


源码安装
一.安装编译依赖库

yum -y install ncurses-devel glib2-devel tokyocabinet-devel

wget ftp://rpmfind.net/linux/epel/6/x86_64/geoipupdate-2.2.1-2.el6.x86_64.rpm

wget ftp://rpmfind.net/linux/centos/5.11/extras/x86_64/RPMS/GeoIP-data-20090201-1.el5.centos.x86_64.rpm

wget ftp://rpmfind.net/linux/epel/6/x86_64/GeoIP-1.6.5-1.el6.x86_64.rpm

wget ftp://rpmfind.net/linux/epel/6/x86_64/GeoIP-devel-1.6.5-1.el6.x86_64.rpm

rpm -ivh GeoIP-* geoipupdate-2.2.1-2.el6.x86_64.rpm

二.安装goaccess

wget http://tar.goaccess.io/goaccess-0.9.4.tar.gz

tar -xvf goaccess-0.9.4.tar.gz -C /usr/local/src/

cd /usr/local/src/goaccess-0.9.4
./configure --enable-geoip --enable-utf8 && make -j4 && make install

三.配置IP地址库

wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz

mkdir -p /usr/share/GeoIP

gunzip GeoLiteCity.dat.gz

mv GeoLiteCity.dat /usr/share/GeoIP/GeoIP.dat

四.使用范例

日志格式样例:

10.157.20.131 - - #2016-04-11 23:59:58# "GET /webapp/wcs/stores/servlet/SearchComponentAutoSuggestView?langId=-7&storeId=10001&catalogId=10052&term=c ha&catalogIdX=10052 HTTP/1.1" 200 "RESPONSE_TIME: 47169" 20"http://www.sephora.cn/brand/dior-184/" "URL_PATH: /webapp/wcs/stores/servlet/SearchComponentAutoSuggestView" "Mozilla/5.0 (iPad; CPU OS 9_3_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) BaiduHD/4.7.1.0 Mobile/10A406 Safari/8536.25" "CLIENT: 183.251.70.65, 218.67.60.13, 10.157.20.147, 10.157.20.147"

对应的日志格式:

cat >/root/.goaccessrc <<HERE

date-format %Y-%m-%d

time-format %H:%M:%S

log-format %h %^#%d %t# "%r" %s "RESPONSE_TIME: %D" %b "%R" "%^" "%u"

HERE

注意:log-format最重要，如果没有准确对应到日志的各个栏位，会分析错位至分析不准， 如上，己对各位栏位作了对应

不要加引号,否则通不过,之前加了引号死活没反应

(log-format '%h %^#%d %t# "%r" %s "RESPONSE_TIME: %D" %b "%R" "%^" "%u"')

常用命令:

goaccess -m -d -f /var/log/httpd/foo.com_access.log -a -p /root/.goaccessrc --geoip-database=/usr/share/GeoIP/GeoLiteCity.dat > /tmp/test.html

GoAccess使用介绍

启动参数介绍：

• -b 流量消耗统计，如果想要提高解析速度，不要开启这个选项。缺省值为不开启此选项
• -f 日志文件路径。
• -s HTTP 状态码报告，为了能够更好的解析日志，选项被默认关闭。
• -e 指定IP地址的访问统计。
• -a 通过代理的请求统计。

菜单介绍

• F1 帮助菜单
• F5 刷新
• q 退出当前口窗、菜单、或是当前查看的选项
• o 打开当前的选项、菜单
• c 改变窗口配色（目前只有两种 默认和绿色）
• SHIFT + TAB 从当前选定模块向后切换
• RIGHT 打开当前选中模块，查看详细信息
• s 通过日期排序，只会在访问请求模块起作用
• S 通过点击次数排序，只会在访问请求模块起作用
• / 查看详细信息的窗口进行搜索
• n 通过/进行查找后，查找下个匹配的内容的位置，如果没有则在窗口底部显示“search hit BOTTOM”
• t 在查看详细信息窗口，移动指针到最顶部
• b 在查看详细信息窗口，移动指针到最底部

goaccess时间和日志格式支持的参数：

CUSTOM LOG/DATE FORMAT

       GoAccess can parse virtually any web log format.

       Predefined options include, Common Log Format (CLF), Combined Log Format  (XLF/ELF),  including  virtual  host,

       Amazon CloudFront (Download Distribution), Google Cloud Storage and W3C format (IIS).

       GoAccess allows any custom format string as well.

       There  are two ways to configure the log format.  The easiest is to run GoAccess with -c to prompt a configuration window. Otherwise, it can be configured under ~/.goaccessrc or the %sysconfdir%.

       time_format

              The time_format variable followed by a space, specifies the log format time containing  any  combination

              of  regular  characters  and  special format specifiers.  They all begin with a percentage (%) sign. See‘man strftime‘.  %T or %H:%M:%S.

              Note: If a timestamp is given in microseconds, %f must be used as time_format

       date_format

              The date_format variable followed by a space, specifies the log format date containing  any  combination

              of regular characters and special format specifiers. They all begin with a percentage (%) sign. See ‘man strftime‘. e.g., %Y-%m-%d.

              Note: If a timestamp is given in microseconds, %f must be used as date_format

       log_format

              The log_format variable followed by a space or \t , specifies the log format string.

       %x     A date and time field matching the time_format and date_format variables. This is used when a  timestamp

              is given instead of the date and time being in two separated variables.

       %t     time field matching the time_format variable.

       %d     date field matching the date_format variable.

       %h     host (the client IP address, either IPv4 or IPv6)

       %r     The  request  line  from  the  client.  This  requires specific delimiters around the request (as single

              quotes, double quotes, or anything else) to be parsable. If not, we have to use a combination of special

              format specifiers as %m %U %H.

       %q     The query string.

       %m     The request method.

       %U     The URL path requested.

              Note:  If  the  query  string  is in %U, there is no need to use %q.  However, if the URL path, does not

              include any query string, you may use %q and the query string will be appended to the request.

       %H     The request protocol.

       %s     The status code that the server sends back to the client.

       %b     The size of the object returned to the client.

       %R     The "Referrer" HTTP request header.

       %u     The user-agent HTTP request header.

       %D     The time taken to serve the request, in microseconds as a decimal number.

       %T     The time taken to serve the request, in seconds with milliseconds resolution.

       %L     The time taken to serve the request, in milliseconds as a decimal number.

              Note: If multiple time served specifiers are used at the same time, the first option  specified  in  the

              format string will take priority over the other specifiers.

       %^     Ignore this field.

       GoAccess requires the following fields:

              %h a valid IPv4/6

              %d a valid date

              %r the request