nginx499 body_bytes_sent 为0 的情况

发现线上有大量499的请求，然后看见有的499的请求body_bytes_sent 为0，所以就此情况查看一番。

复现场景：192.168.60.128 上面请求192.168.60.150的机器，150上的应用接口执行时间30秒，128上面的超时时间2秒，此时nginx返回了499，body_bytes_sent为0。

1、128请求模拟

>>> requests.get('http://192.168.226.150/', timeout=2)

2、150 nginx配置

worker_processes  1;
events {
    worker_connections  1024;
}
http {
    lua_package_path "/usr/local/openresty/lualib/myapp/lua/?.lua;;";
    init_by_lua_block {
       require "hello"
    }

    log_format main '{"@timestamp":"$time_iso8601",'
                   '"host":"$server_addr",'
                   '"remote_addr":"$remote_addr",'
                   '"body_bytes_sent":"$body_bytes_sent",'
                   '"request_time":"$request_time",'
                   '"upstreamtime":"$upstream_response_time",'
                   '"upstream_addr":"$upstream_addr",'
                   '"http_host":"$host",'
                   '"url":"$uri",'
                   '"request":"$request",'
                   '"request_for_keyword":"$request",'
                   #'"request_body":"$request_body",'
                   '"domain":"$host",'
                   '"http_x_forwarded_for":"$http_x_forwarded_for",'
                   '"user-agent":"$http_user_agent",'
                   '"http_referer":"$http_referer",'
                   '"status":"$status"}';

    access_log  /usr/local/openresty/nginx/logs/access.log  main;


    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       80;
        server_name  localhost;
        location / {
        #content_by_lua_block {
            #    local hello = require "hello"
            #    hello.greet("a Lua module")
            #}
            #root index.html;
            proxy_pass http://127.0.0.1:8123;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

3、150接口

from flask import Flask
import time

app = Flask(__name__) 
@app.route('/') 
def hello():
    for i in range(30):
        time.sleep(1)
    return 'Hello World'

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=8123,debug=True)

4、请求之后日志情况

{"@timestamp":"2022-06-17T15:49:42+08:00","host":"192.168.226.150","remote_addr":"192.168.226.128","body_bytes_sent":"0","request_time":"2.007","upstreamtime":"2.007","upstream_addr":"127.0.0.1:8123","http_host":"192.168.226.150","url":"/","request":"GET / HTTP/1.1","request_for_keyword":"GET / HTTP/1.1","domain":"192.168.226.150","http_x_forwarded_for":"-","user-agent":"python-requests/2.24.0","http_referer":"-","status":"499"}

 

5、抓包对比

 

 上面是请求超时返回499的情况，客户端未等服务端返回主动断开连接，下面是请求超时返回200的情况。上图为啥不是四次挥手？网上各种解释乱七八糟的，说是第二次和第三次都放在了第二次。

 

6、nginx 源码，这个需要细品。

static ngx_int_t
ngx_http_variable_body_bytes_sent(ngx_http_request_t *r,
    ngx_http_variable_value_t *v, uintptr_t data)
{
    off_t    sent;
    u_char  *p;

    sent = r->connection->sent - r->header_size;

    if (sent < 0) {
        sent = 0;
    }

    p = ngx_pnalloc(r->pool, NGX_OFF_T_LEN);
    if (p == NULL) {
        return NGX_ERROR;
    }

    v->len = ngx_sprintf(p, "%O", sent) - p;
    v->valid = 1;
    v->no_cacheable = 0;
    v->not_found = 0;
    v->data = p;

    return NGX_OK;
}