ELK使用过程中遇到的一些问题

filebeat

skipped as it is a symlink

背景：log组件有变动，应用发布到k8s上后，filebeat不收集了。

排查过程

$ k -n kube-system logs -f filebeat-cx6jp
...
Configured paths: [/data/test/log/*.log]
...

配置确实加载了，但是没有启动Harvester，也就是没有类似以下的记录

Harvester started for file: /data/sync/test/abc.log

但是不知道为什么没有。

filebeat.yml中加入以下内容开启debug。(官方文档)

logging.level: debug
logging.to_stderr: true

重启后发现以下内容：

DEBUG	[input]	log/input.go:290   File /data/sync/test/abc.log skipped as it is a symlink.

原来是因为/data/sync/test/abc.log是个软连，默认是不收集的。

官网一番查找，加入了symlinks: true

- type: log
     symlinks: true
     paths:
       - "/data/test/log/*.log"

解决！

The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true.

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."
            }
        ],
        "type": "illegal_argument_exception",
        "reason": "The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."
    },
    "status": 400
}

我用的是第二种，添加了include_type_name

PUT _template/loges-pro-traefik?include_type_name