使用openresty+empty_gif 进行请求数据分析
empty_gif 是一个很不错的nginx 模块，可以方便的生成1*1 像素的图片（很适合数据分析）
以下是一个基于empty_gif模块以及openresty 的access_by_lua_block 阶段进行简单的demo 演示
具体的处理可以自己扩展下
环境准备
- nginx.conf
worker_processes 1;
user root;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
gzip on;
real_ip_header X-Forwarded-For;
log_format jsonlog escape=json '{ "timestamp": "$time_iso8601", '
'"remote_addr": "$remote_addr", '
'"body_bytes_sent": $body_bytes_sent, '
'"request_time": $request_time, '
'"response_status": $status, '
'"request": "$request", '
'"request_method": "$request_method", '
'"host": "$host",'
'"content":"$content",'
'"upstream_cache_status": "$upstream_cache_status",'
'"upstream_addr": "$upstream_addr",'
'"http_x_forwarded_for": "$http_x_forwarded_for",'
'"http_referrer": "$http_referer", '
'"http_user_agent": "$http_user_agent" }';
server {
listen 80;
charset utf-8;
default_type text/html;
location / {
default_type text/html;
index index.html;
}
location /sys/ {
set $content "";
if_modified_since off;
etag off;
add_header Cache-Control no-store;
add_header Cache-Control must-revalidate;
access_log /usr/local/openresty/nginx/logs/access_test.log jsonlog;
access_by_lua_block {
local args = ngx.req.get_uri_args()
local info = {
id = args["id"] or "",
psnid = args["psnid"] or ""
}
local res = ngx.location.capture('/logs?id='..info.id..'&psnid='..info.psnid)
ngx.var.content = res.body
}
empty_gif;
}
location /logs {
content_by_lua_block {
local args = ngx.req.get_uri_args()
ngx.say(string.format("result is: %s,%s",args["id"],args["psnid"]))
}
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
- 简单说明
因为默认的图片是会cache 的，所以添加了以下请求头确认每次都会发起请求
核心参数
add_header Cache-Control no-store;
add_header Cache-Control must-revalidate;
为了方便，日志存储使用了本地文件，实际如果需要可以结合数据库处理，日志格式基于json
同时对于请求的数据基于openresty 的location.capture 请求处理，日志中添加了nginx 变量，方便
存储结果
- docker-compose 文件
version: "3"
services:
api:
image: openresty/openresty:alpine
volumes:
- "./nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf"
- "./index.html:/usr/local/openresty/nginx/html/index.html"
- "./access_test.log:/usr/local/openresty/nginx/logs/access_test.log"
ports:
- "80:80"
- index.html
引用一像素分析的页面(基于img src)
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>demo</title>
</head>
<body>
<img src="./sys/333333?id=33333&psnid=222" />
</body>
</html>
运行&&效果
- 运行
docker-compose up -d
- 效果
说明
以上只是一个简单的处理，实际使用中我们可以基于此模型添加比较强大的数据分析&&处理,比如streamsets，webhook。。。。
参考资料
https://nginx.org/en/docs/http/ngx_http_empty_gif_module.html
https://github.com/rongfengliang/empty_gif_openresty_learning
