Nginx配置GeoIP库或者直接通过修改Logstash将日志写入ES

GItHub：https://github.com/TravelEngineers/ngx_http_geoip2_module

一、DB文件下载

先注册用户：https://dev.maxmind.com/geoip/geoip2/geolite2/

手动进入下载页面：https://www.maxmind.com/en/accounts/455551/geoip/downloads   # 455551为用户ID

或

生成一个用户自己的 License Key

 

然后配置到自己的系统里

# vim /etc/GeoIP.conf 

# geoipupdate

执行 geoipupdate 此时会自动下载或更新 /usr/share/GeoIP 文件夹内的mmdb文件

 

二、Nginx配置引用GeoIP变量

首先安装ngx_http_geoip2_module依赖项

wget https://github.com/maxmind/libmaxminddb/releases/download/1.3.2/libmaxminddb-1.3.2.tar.gz
tar -xvf libmaxminddb-1.3.2.tar.gz
cd libmaxminddb-1.3.2
./configure
make && make config
echo "/usr/local/lib" >> /etc/ld.so.conf
ldconfig

然后重新编译安装nginx，支持GeoIP模块

./configure --prefix=/usr/local/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-http_stub_status_module --with-http_realip_module --with-http_v2_module --add-module=/root/ngx_http_geoip2_module

 

nginx.conf:

    log_format main escape=json '{"remote_addr":"$remote_addr",'
        '"timestamp":"$time_local",'
        '"country":"$geoip2_data_country_name",'
        '"city":"$geoip2_city_name",'
        '"method":"$request_method",'
        '"request":"$uri",'
        '"requestParam":"$query_string",'
        '"status":"$status",'
        '"referrer":"$http_referer",'
        '"agent":"$http_user_agent",'
        '"elapsed":"$request_time",'
        '"serverelapsed":"$upstream_response_time"}';

    geoip2 /usr/share/GeoIP/GeoLite2-Country.mmdb {
        auto_reload 5m;
        $geoip2_metadata_country_build metadata build_epoch;
        $geoip2_data_country_code default=US country iso_code;
        $geoip2_data_country_name country names en;
    }

    geoip2 /usr/share/GeoIP/GeoLite2-City.mmdb {
         $geoip2_city_name default=BeiJing city names en;
         $geoip2_continent_code continent code;
    }

 

---

 

通过修改Logstash使用geoip插件定制日志内容传入ES，并通过Kibana的Worldmap Panel Plugin展示在Grafana

项目地址：

https://github.com/grafana/worldmap-panel

https://grafana.com/grafana/plugins/grafana-worldmap-panel

1、插件安装：grafana-cli plugins install grafana-worldmap-panel

2、Nginx配置文件日志格式编辑指定

    log_format main escape=json
        '{"client_ip":"$remote_addr",'
        '"timestamp":"$time_local",'
        '"method":"$request_method",'
        '"request":"$uri",'
        '"requestParam":"$query_string",'
        '"status":"$status",'
        '"referrer":"$http_referer",'
        '"agent":"$http_user_agent",'
        '"elapsed":"$request_time",'
        '"serverelapsed":"$upstream_response_time"'
    '}';

3、Logstash配置文件

input {
    file {
      path => "/usr/local/nginx/logs/adg-access.log"
      codec => json
      type => "adg-nginx"
    }
}

filter {
    geoip {
      source => "client_ip"
      target => "geoip"
      database =>"/usr/share/GeoIP/GeoLite2-City.mmdb"
      remove_field => [ "[geoip][timezone]","[geoip][country_code3]","[geoip][region_code]","[geoip][city_name]","[geoip][ip]","[geoip][region_name]","[geoip][continent_code]","[geoip][longitude]","[geoip][latitude]" ]
    }
    mutate {
      convert => [ "elapsed", "float" ]
      convert => [ "serverelapsed", "float" ]
      convert => [ "status", "integer" ]
      #convert => [ "[geoip][coordinates]", "float" ]
      remove_field => [ "messages","timestamp","@version","host","path","agent" ] #agent此处未进行采集，如果需要的话使用ua插件，请参考下面的链接
    }
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
}

output {
    if [type] == "adg-nginx" {
      elasticsearch {
      hosts => ["192.168.0.7:9200", "192.168.0.8:9200", "192.168.0.9:9200"]
      index => "nginx-adg-kr.%{+YYYY-MM-dd}"
      }
    }
}

 输出日志格式如下：（此处主要获取了经纬度、country_code、country_name，未使用省会或城市名称）

 

4、配置Grafana

Worldmap Panel并不支持通过经纬度数据对 e.g. (latitude, longtitude)在地图上定位与可视化， 其支持的数据格式有且仅有两种：Country/State Code 或 geohash

以下从官方文档中摘出的这句话很好地的解释了这两种数据类型：

There are currently two ways to connect data with points on a map. Either by matching a tag or series name to a country code/state code (e.g. SE for Sweden, TX for Texas) or by using geohashes to map against geographic coordinates。  

 

 

5、完成后，效果如下图：

 

 

 

参考资料：https://www.ucloud.cn/yun/90596.html

               https://grafana.com/grafana/dashboards/11190