网关系统 apisix/kong 日志收集方案实施对比

        include_req_body = {type = "boolean", default = false},
        include_req_body_expr = {
            type = "array",
            minItems = 1,
            items = {
                type = "array",
                items = {
                    type = "string"
                }
            }
        }

并未用kong2.0和apisix做基础性能对比，但就日志request_body收集功能上,apisix和kong 基础免费版，apisix强于kong

因此网关系统开始部分从kong迁移转至apisix

以下是部分测试验证的细节,启动log-kafka插件收集

{
    "broker_list": {
        "192.168.11.22": 9092,
        "192.168.11.23": 9092,
        "192.168.11.24": 9092
    },
    "kafka_topic": "test_apisix",
    "key": "key1",
    "batch_max_size": 1,
    "name": "kafka logger",
    "include_req_body":true,
    "meta_format":"default",
    "log_format":""
}

meta_format支持两种格式，为收集解析方便使用"default"

样例日志
{
    "start_time": 1638185753246,
    "client_ip": "172.17.0.1",
    "latency": 0,
    "service_id": "127.0.0.1",
    "server": {
        "hostname": "fe053d1baf4a",
        "version": "2.9"
    },
    "response": {
        "status": 404,
        "headers": {
            "server": "APISIX/2.9",
            "connection": "close",
            "transfer-encoding": "chunked",
            "content-type": "text/plain; charset=utf-8"
        },
        "size": 223
    },
    "request": {
        "headers": {
            "host": "127.0.0.1:8280",
            "user-agent": "curl/7.29.0",
            "accept": "*/*"
        },
        "method": "GET",
        "size": 83,
        "url": "http://127.0.0.1:9080/hello",
        "querystring": {},
        "uri": "/hello"
    }
}

自定义格式

curl http://127.0.0.1:9080/apisix/admin/plugin_metadata/kafka-logger -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "log_format": {"host": "$host", "@timestamp": "$time_iso8601",    "client_ip":"$remote_addr","request_id":"$request_id","request_length":"$request_length","request_time":"$request_time","header_test_1":"$http_nothas","header_test_1":"$http_x-api-key","body":"$request_body"}
}'

apisix从测试和查看源码上看，也有些待优化的点

1 "default"格式完全默认无法自定义，如果需要nginx原生的一些信息，则需要使用log_format

defautl信息为目前为硬编码，如果需要一些别的信息，需要直接修改代码
https://github.com/apache/apisix/blob/2.10.1.1/apisix/utils/log-util.lua

default格式实现，和上面的demo对应
local log =  {
        request = {
            url = url,
            uri = var.request_uri,
            method = ngx.req.get_method(),
            headers = ngx.req.get_headers(),
            querystring = ngx.req.get_uri_args(),
            size = var.request_length
        },
        response = {
            status = ngx.status,
            headers = ngx.resp.get_headers(),
            size = var.bytes_sent
        },
        server = {
            hostname = core.utils.gethostname(),
            version = core.version.VERSION
        },
        upstream = var.upstream_addr,
        service_id = service_id,
        route_id = route_id,
        consumer = consumer,
        client_ip = core.request.get_remote_client_ip(ngx.ctx.api_ctx),
        start_time = ngx.req.start_time() * 1000,
        latency = (ngx_now() - ngx.req.start_time()) * 1000
    }

#默认的default格式
_M.get_full_log = get_full_log
#自定义全局格式
_M.get_custom_format_log = get_custom_format_log

2 log_format目前是全局生效的，即全局只能有一种log_format格式，若想对不同的路由配置不同的规则，则作不到

3 可以通过修改 https://github.com/apache/apisix/blob/2.10.1.1/apisix/utils/log-util.lua default格式的log内容，很简单

apisix满足网关的性能和日志收集需求

END

posted @ 2021-12-10 21:57 cclient 阅读(2480) 评论(0) 编辑收藏举报

刷新页面返回顶部

吾生也有涯，而知也无涯

心有阳光，正视黑暗

网关系统 apisix/kong 日志收集方案实施对比

END

公告