Tengine 安装 nginx-module-vts 模块

Tengine 安装 nginx-module-vts 模块

使用 Tengine 作为反向代理服务器，发现 Tengine 日志中接口请求过慢，需要绘制 grafana 展示详细的接口情况

解决思路

• 第一版解决思路，修改 Tengine 中 access log 日志格式，转行为 json 写入 elk 中，grafana 从 elk 中获取日志信息绘图

  • access log 日志信息不全，grafana 绘图有些数据无法获取
  • access log 中 request_time 字段格式不对，无法进行聚合以及求 AVG，计算接口平均耗时等
  • 其他的一些问题，比如 我们 elk 中有 Tengine 和 k8s nginx 日志，获取数据从所以所有的 日志中获取，而我们只希望从 Tengine 日志获取
  • 基于上述的原因放弃了第一版解决思路

• access log 日志格式

# nginx.conf
    log_format  main_new escape=json  '{ "@timestamp": "$time_iso8601", '
            '"remote_addr": "$remote_addr",'
            '"remote_port": "$remote_port",'
            '"remote_user": "$remote_user",'
            '"scheme": "$scheme",'
            '"server_protocol": "$server_protocol",'
            '"http_host": "$http_host",'
            '"request_uri": "$request_uri",'
            '"request_method": "$request_method",'
            '"request_time": "$request_time",'
            '"request_length": "$request_length",'
            '"response_status": "$status",'
            '"http_cookie": "$http_cookie",'
            '"body_bytes_sent": "$body_bytes_sent",'
            '"http_referer": "$http_referer",'
            '"http_user_agent": "$http_user_agent",'
            '"http_x_forwarded_for": "$http_x_forwarded_for",'
            '"upstream_addr": "$upstream_addr",'
            '"upstream_status": "$upstream_status",'
            '"upstream_connect_time": "$upstream_connect_time",'
            '"upstream_response_time": "$upstream_response_time"}';

# default.conf
    access_log  /var/log/nginx/prod-access.log  main_new;

• 第二版解决思路，使用 nginx-module-vts 模块暴露接口代理情况，并且此模块自带 exporter 对接 prometheus

调整 Tengine 安装，编译进 nginx-module-vts 模块

Tengine 安装文档

修改 Dockerfile 重新打镜像

• 修改项

	    --add-module=modules/ngx_http_upstream_session_sticky_module \
        --add-module=modules/nginx-module-vts-0.2.2 \ # 新增

        && curl -L "https://github.com/openresty/headers-more-nginx-module/archive/v0.33.tar.gz" -o more.tar.gz \
        && curl -L "https://github.com/vozlt/nginx-module-vts/archive/refs/tags/v0.2.2.tar.gz" -o more2.tar.gz \ # 新增
        && tar -zxC /usr/src/tengine-$TENGINE_VERSION/modules -f more.tar.gz \
        && tar -zxC /usr/src/tengine-$TENGINE_VERSION/modules -f more2.tar.gz \ # 新增
	    && rm  more.tar.gz \
        && rm  more2.tar.gz \ # 新增

• 完整的 Dockerfile 文件

FROM alpine:3.17


ENV TENGINE_VERSION 2.4.0

# nginx: https://git.io/vSIyj

RUN rm -rf /var/cache/apk/* && \
    rm -rf /tmp/*

ENV CONFIG "\
        --prefix=/etc/nginx \
        --sbin-path=/usr/sbin/nginx \
        --modules-path=/usr/lib/nginx/modules \
        --conf-path=/etc/nginx/nginx.conf \
        --error-log-path=/var/log/nginx/error.log \
        --http-log-path=/var/log/nginx/access.log \
        --pid-path=/var/run/nginx.pid \
        --lock-path=/var/run/nginx.lock \
        --http-client-body-temp-path=/var/cache/nginx/client_temp \
        --http-proxy-temp-path=/var/cache/nginx/proxy_temp \
        --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
        --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
        --http-scgi-temp-path=/var/cache/nginx/scgi_temp \
        --user=nginx \
        --group=nginx \
        --with-http_ssl_module \
        --with-http_realip_module \
        --with-http_addition_module \
        --with-http_sub_module \
        --with-http_dav_module \
        --with-http_flv_module \
        --with-http_mp4_module \
        --with-http_gunzip_module \
        --with-http_gzip_static_module \
        --with-http_random_index_module \
        --with-http_secure_link_module \
        --with-http_stub_status_module \
        --with-http_auth_request_module \
        --with-http_xslt_module=dynamic \
        --with-http_image_filter_module=dynamic \
        --with-http_geoip_module=dynamic \
        --with-threads \
        --with-stream \
        --with-stream_ssl_module \
        --with-stream_ssl_preread_module \
        --with-stream_realip_module \
        --with-stream_geoip_module=dynamic \
        --with-http_slice_module \
        --with-mail \
        --with-mail_ssl_module \
        --with-compat \
        --with-file-aio \
        --with-http_v2_module \
        --add-module=modules/ngx_http_upstream_check_module \
        --add-module=modules/headers-more-nginx-module-0.33 \
	--add-module=modules/ngx_http_upstream_session_sticky_module \
        --add-module=modules/nginx-module-vts-0.2.2 \
        "
RUN     addgroup -S nginx \
        && adduser -D -S -h /var/cache/nginx -s /sbin/nologin -G nginx nginx \
        && adduser -u 82 -D -S -G www-data www-data \
        && apk add --no-cache --virtual .build-deps \
                gcc \
                libc-dev \
                make \
                openssl-dev \
                pcre-dev \
                zlib-dev \
                linux-headers \
                curl \
                libxslt-dev \
                gd-dev \
                geoip-dev \
        && curl -L "https://github.com/alibaba/tengine/archive/$TENGINE_VERSION.tar.gz" -o tengine.tar.gz \
        && mkdir -p /usr/src \
        && tar -zxC /usr/src -f tengine.tar.gz \
        && rm tengine.tar.gz \
        && cd /usr/src/tengine-$TENGINE_VERSION \
        && curl -L "https://github.com/openresty/headers-more-nginx-module/archive/v0.33.tar.gz" -o more.tar.gz \
        && curl -L "https://github.com/vozlt/nginx-module-vts/archive/refs/tags/v0.2.2.tar.gz" -o more2.tar.gz \
        && tar -zxC /usr/src/tengine-$TENGINE_VERSION/modules -f more.tar.gz \
        && tar -zxC /usr/src/tengine-$TENGINE_VERSION/modules -f more2.tar.gz \
	&& rm  more.tar.gz \
        && rm  more2.tar.gz \
	&& ls -l /usr/src/tengine-$TENGINE_VERSION/modules \
	&& ./configure $CONFIG --with-debug \
        && make -j$(getconf _NPROCESSORS_ONLN) \
        && mv objs/nginx objs/nginx-debug \
        && mv objs/ngx_http_xslt_filter_module.so objs/ngx_http_xslt_filter_module-debug.so \
        && mv objs/ngx_http_image_filter_module.so objs/ngx_http_image_filter_module-debug.so \
        && mv objs/ngx_http_geoip_module.so objs/ngx_http_geoip_module-debug.so \
        && mv objs/ngx_stream_geoip_module.so objs/ngx_stream_geoip_module-debug.so \
        && ./configure $CONFIG \
        && make -j$(getconf _NPROCESSORS_ONLN) \
        && make install \
        && rm -rf /etc/nginx/html/ \
        && mkdir /etc/nginx/conf.d/ \
        && mkdir -p /usr/share/nginx/html/ \
        && install -m644 html/index.html /usr/share/nginx/html/ \
        && install -m644 html/50x.html /usr/share/nginx/html/ \
        && install -m755 objs/nginx-debug /usr/sbin/nginx-debug \
        && install -m755 objs/ngx_http_xslt_filter_module-debug.so /usr/lib/nginx/modules/ngx_http_xslt_filter_module-debug.so \
        && install -m755 objs/ngx_http_image_filter_module-debug.so /usr/lib/nginx/modules/ngx_http_image_filter_module-debug.so \
        && install -m755 objs/ngx_http_geoip_module-debug.so /usr/lib/nginx/modules/ngx_http_geoip_module-debug.so \
        && install -m755 objs/ngx_stream_geoip_module-debug.so /usr/lib/nginx/modules/ngx_stream_geoip_module-debug.so \
        && ln -s ../../usr/lib/nginx/modules /etc/nginx/modules \
        && strip /usr/sbin/nginx* \
        && strip /usr/lib/nginx/modules/*.so \
        && rm -rf /usr/src/tengine-$NGINX_VERSION \
        \
        # Bring in gettext so we can get `envsubst`, then throw
        # the rest away. To do this, we need to install `gettext`
        # then move `envsubst` out of the way so `gettext` can
        # be deleted completely, then move `envsubst` back.
        && apk add --no-cache --virtual .gettext gettext \
        && mv /usr/bin/envsubst /tmp/ \
        \
        && runDeps="$( \
                scanelf --needed --nobanner --format '%n#p' /usr/sbin/nginx /usr/lib/nginx/modules/*.so /tmp/envsubst \
                        | tr ',' '\n' \
                        | sort -u \
                        | awk 'system("[ -e /usr/local/lib/" $1 " ]") == 0 { next } { print "so:" $1 }' \
        )" \
        && apk add --no-cache --virtual .nginx-rundeps $runDeps \
        && apk del .build-deps \
        && apk del .gettext \
        && mv /tmp/envsubst /usr/local/bin/ \
        \
        # Bring in tzdata so users could set the timezones through the environment
        # variables
        && apk add --no-cache tzdata \
        # Bring in curl and ca-certificates to make registering on DNS SD easier
        && apk add --no-cache curl ca-certificates \
        # forward request and error logs to docker log collector
        && ln -sf /dev/stdout /var/log/nginx/access.log \
        && ln -sf /dev/stderr /var/log/nginx/error.log

COPY nginx.conf /etc/nginx/nginx.conf

EXPOSE 80 443

STOPSIGNAL SIGTERM

CMD ["nginx", "-g", "daemon off;"]

• 制作 Docker 镜像

cd tengine

docker build -t harbor.evescn.com/devops/tengine:2.4.0-nginx-module-vts .

docker push harbor.evescn.com/devops/tengine:2.4.0-nginx-module-vts

• nginx.conf 文件

user  nginx;
# This number should be, at maximum, the number of CPU cores on your system. 
worker_processes auto;

error_log  /var/log/nginx/error.log error;
pid        /var/run/nginx.pid;


events {
    # The effective method, used on Linux 2.6+, optmized to serve many clients with each thread.
    use epoll;
    # Determines how many clients will be served by each worker process.
    worker_connections 4000;
    # Accept as many connections as possible, after nginx gets notification about a new connection.
    multi_accept on;
}


http {
    #################
    #     开启vts   #
    #################
    vhost_traffic_status_zone;
    map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;}

    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    #long time
    check_shm_size 5M;
    # Allow the server to close the connection after a client stops responding. 
    reset_timedout_connection on;
    client_header_timeout 15;
    # Send the client a "request timed out" if the body is not loaded by this time.
    client_body_timeout 10;
    # If the client stops reading data, free up the stale client connection after this much time.
    send_timeout 15;
    # Timeout for keep-alive connections. Server will close connections after this time.
    keepalive_timeout 30;
    # Number of requests a client can make over the keep-alive connection.
    keepalive_requests 30;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    #################
    #   新增日志格式  #
    #################
    log_format  main_new escape=json  '{ "@timestamp": "$time_iso8601", '
            '"remote_addr": "$remote_addr",'
            '"remote_port": "$remote_port",'
            '"remote_user": "$remote_user",'
            '"scheme": "$scheme",'
            '"server_protocol": "$server_protocol",'
            '"http_host": "$http_host",'
            '"request_uri": "$request_uri",'
            '"request_method": "$request_method",'
            '"request_time": "$request_time",'
            '"request_length": "$request_length",'
            '"response_status": "$status",'
            '"http_cookie": "$http_cookie",'
            '"body_bytes_sent": "$body_bytes_sent",'
            '"http_referer": "$http_referer",'
            '"http_user_agent": "$http_user_agent",'
            '"http_x_forwarded_for": "$http_x_forwarded_for",'
            '"upstream_addr": "$upstream_addr",'
            '"upstream_status": "$upstream_status",'
            '"upstream_connect_time": "$upstream_connect_time",'
            '"upstream_response_time": "$upstream_response_time"}';

    client_body_buffer_size 128k;
    client_max_body_size 40m;
    proxy_read_timeout 180s;
 
    # Compression.
    gzip on;
    gzip_min_length 10240;
    gzip_proxied expired no-cache no-store private auth;
    gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml;
    gzip_disable "msie6";

    # Sendfile copies data between one FD and other from within the kernel.
    sendfile on; 
    # Don't buffer data-sends (disable Nagle algorithm).
    tcp_nodelay on; 
    # Causes nginx to attempt to send its HTTP response head in one packet,  instead of using partial frames.
    tcp_nopush on;


    # Hide web server information
    server_tokens off;
    server_info off;
    server_tag off;

    # redirect server error pages to the static page
    error_page 404             /404.html;
    error_page 500 502 503 504 /50x.html;

    include /etc/nginx/*.d/*.conf;

    #######################
    #  暴露 vts 端口和路径  #
    #######################
    server {
        listen       9112;

        location /vts {
            vhost_traffic_status_display;
            vhost_traffic_status_display_format html;
        }
    }
}

• 启动服务

# cat docker-compose.yml 
version: '3'

services:
  rabbitmq:
    image: harbor.evescn.com/devops/tengine:2.4.0-nginx-module-vts
    hostname: tengine
    container_name: tengine
    ports:
      - "80:80"
      - "443:443"
      # 新增监控页面接口
      - "9112:9112"
    volumes:
      # 配置文件目录
      - /data/tengine/conf.d/:/etc/nginx/conf.d/
      # ssl 证书文件
      - /data/tengine/certs/:/etc/nginx/certs/
      # 日志文件
      - /data/tengine/logs/:/var/log/nginx/
    restart: always

# docker-compose up -d

# docker-compose ls
NAME                STATUS              CONFIG FILES
tengine             running(1)          /data/docker-compose/tengine/docker-compose.yml

访问监控页面测试

• web 页面

http://IP:9112/vts/

• exporter 页面

http://IP:9112/vts/format/prometheus

Grafana 绘图