Elastic Search + Logstash + Kibana 初学者日志
因为我用的是windows操作系统，所以下面的介绍都基于windows环境。功能方面应该和Linux是一样的。
=======================================================================================
Getting Started with ELK for Apache Logs (这个例子展示了怎样用logstash把log发送到Elastic Search, 然后用Kibana做图形化展示）
https://github.com/aagea/elk-example我修改了上面的例子，用stdin作为logstash的输入（输入了hello world)，用Elastic Search作为输出。
那么为了验证，可以用下面这个命令搜索Elastic Search的内容
当然也可以直接在Kibana的Discovery界面搜索。具体方法如下。这篇文章写的很好，尤其是里面的图画得很清晰，缺点是软件版本有点老了。
https://my.oschina.net/itblog/blog/547250/
下面介绍一下ELK各个组件安装后的目录结构和help命令返回的结果。这个有助于了解组件的功能覆盖和基本使用方法。
Elastic Search 6.6
目录
bin -- 这里存放Elastic Search的执行程序. 这里我用elasticsearch -help /all > rex_output.txt命令得到所有的可选项
config -- 这里存放配置文件，主要是一些yml文件。 比如elasticsearch.yml (内容如下)
cluster.name=my-application
node.name=node-1
path.data=/path/to/data
path.logs=/path/to/logs
network.host=127.0.0.1
http.port=9200 - log4j2.properties
- role_mapping.yml
- roles.yml
- users
- users_roles
data -- 这里面好像是跟cluster有关的文件，比如默认的是cluster node-0, 所以这里面就有一个名字是“0”的文件夹
lib -- 就是elasticsearch依赖的一些JAR包
logs -- 日志文件
modules -- 这里面是ElasticSearch用到的主要模块，包括X-Pack (lets you set up watches (or rules) to detect and alert on changes in your ElasticSearch data) ，也是一些jar包。
plugins -- 插件 sudo bin/elasticsearch-plugin install [plugin_name]
Help命令返回的内容
Option Description
------ -----------
-E <KeyValuePair> Configure a setting
-V, --version Prints elasticsearch version information and exits
-d, --daemonize Starts Elasticsearch in the background
-h, --help show help
-p, --pidfile <Path> Creates a pid file in the specified path on start
-q, --quiet Turns off standard output/error streams logging in console
-s, --silent show minimal output
-v, --verbose show verbose output
------ -----------
-E <KeyValuePair> Configure a setting
-V, --version Prints elasticsearch version information and exits
-d, --daemonize Starts Elasticsearch in the background
-h, --help show help
-p, --pidfile <Path> Creates a pid file in the specified path on start
-q, --quiet Turns off standard output/error streams logging in console
-s, --silent show minimal output
-v, --verbose show verbose output
ElasticSearch的官网上有一些例子（https://github.com/elastic/examples），还没来得及细看，下一篇文章再细说吧。
Logstash 6.6
目录 （这里有详细介绍 https://www.elastic.co/guide/en/logstash/current/dir-layout.html）
bin -- 可执行程序
config -- 配置文件。可以参考logstash-sample.conf创建你自己的conf文件
data -- 这里面的文件有些看不懂。比如一个uuid文件，一个.lock文件，几个queue文件夹。好像是跟持久化有关的内容。
lib -- 这里面有bootstrap, pluginmaster, secretstore, systeminstall几个文件夹。也都是ruby程序。
logstash-core -- 这里面有一些ruby程序和JAR包，应该是logstash的核心代码。
logstash-core-plugin-api -- 插件。也是ruby程序。
modules -- 好像是用来存放打包后的ruby module的。ps: Ruby通常用rubygem来打包
tools -- 这里有一个ingest-converter文件夹，里面是一个JAR文件。
vendor -- 这里有bundle和jruby文件夹
x-pack -- 这里面有很多文件夹。可能和ElasticSearch一样，作用是lets you set up watches (or rules) to detect and alert on changes in your Logstash data
Help命令返回的内容 (初学者常用的命令用粗体表示)
Usage:
bin/logstash [OPTIONS]
Options:
-n, --node.name NAME Specify the name of this logstash instance, if no value is given
it will default to the current hostname.
(default: 这里指的是主机名)
-f, --path.config 指定配置文件 比如这样：
bin/logstash [OPTIONS]
Options:
-n, --node.name NAME Specify the name of this logstash instance, if no value is given
it will default to the current hostname.
(default: 这里指的是主机名)
-f, --path.config 指定配置文件 比如这样：
config/log4j_to_es.conf CONFIG_PATH Load the logstash config from a specific file
or directory. If a directory is given, all
files in that directory will be concatenated
in lexicographical order and then parsed as a
single config file. You can also specify
wildcards (globs) and any matched files will
be loaded in the order described above.
-e, --config.string 从命令行指定配置 比如：logstash -e “input { stdin { } } output { stdout {} }
or directory. If a directory is given, all
files in that directory will be concatenated
in lexicographical order and then parsed as a
single config file. You can also specify
wildcards (globs) and any matched files will
be loaded in the order described above.
-e, --config.string 从命令行指定配置 比如：logstash -e “input { stdin { } } output { stdout {} }
”CONFIG_STRING Use the given string as the configuration
data. Same syntax as the config file. If no
input is specified, then the following is
used as the default input:
"input { stdin { type => stdin } }"
and if no output is specified, then the
following is used as the default output:
"output { stdout { codec => rubydebug } }"
If you wish to use both defaults, please use
the empty string for the '-e' flag.
(default: nil)
--field-reference-parser MODE Use the given MODE when parsing field
references.
The field reference parser is used to expand
field references in your pipeline configs,
and will be becoming more strict to better
handle illegal and ambbiguous inputs in a
future release of Logstash.
Available MODEs are:
- `LEGACY`: parse with the legacy parser,
which is known to handle ambiguous- and
illegal-syntax in surprising ways;
warnings will not be emitted.
- `COMPAT`: warn once for each distinct
ambiguous- or illegal-syntax input, but
continue to expand field references with
the legacy parser.
- `STRICT`: parse in a strict manner; when
given ambiguous- or illegal-syntax input,
raises a runtime exception that should
be handled by the calling plugin.
The MODE can also be set with
`config.field_reference.parser`
(default: "COMPAT")
--modules MODULES Load Logstash modules.
Modules can be defined using multiple instances
'--modules module1 --modules module2',
or comma-separated syntax
'--modules=module1,module2'
Cannot be used in conjunction with '-e' or '-f'
Use of '--modules' will override modules declared
in the 'logstash.yml' file.
-M, --modules.variable MODULES_VARIABLE Load variables for module template.
Multiple instances of '-M' or
'--modules.variable' are supported.
Ignored if '--modules' flag is not used.
Should be in the format of
'-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"'
as in
'-M "example.var.filter.mutate.fieldname=fieldvalue"'
--setup Load index template into Elasticsearch, and saved searches,
index-pattern, visualizations, and dashboards into Kibana when
running modules.
(default: false)
--cloud.id CLOUD_ID Sets the elasticsearch and kibana host settings for
module connections in Elastic Cloud.
Your Elastic Cloud User interface or the Cloud support
team should provide this.
Add an optional label prefix '<label>:' to help you
identify multiple cloud.ids.
e.g. 'staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy'
--cloud.auth CLOUD_AUTH Sets the elasticsearch and kibana username and password
for module connections in Elastic Cloud
e.g. 'username:<password>'
--pipeline.id ID Sets the ID of the pipeline.
(default: "main")
-w, --pipeline.workers COUNT Sets the number of pipeline workers to run.
(default: 4)
--java-execution Use Java execution engine.
(default: false)
-b, --pipeline.batch.size SIZE Size of batches the pipeline is to work in.
(default: 125)
-u, --pipeline.batch.delay DELAY_IN_MS When creating pipeline batches, how long to wait while polling
for the next event.
(default: 50)
--pipeline.unsafe_shutdown Force logstash to exit during shutdown even
if there are still inflight events in memory.
By default, logstash will refuse to quit until all
received events have been pushed to the outputs.
(default: false)
--path.data PATH This should point to a writable directory. Logstash
will use this directory whenever it needs to store
data. Plugins will also have access to this path.
(default: "C:/Users/IBM_ADMIN/Downloads/logstash-6.6.0/data")
-p, --path.plugins PATH A path of where to find plugins. This flag
can be given multiple times to include
multiple paths. Plugins are expected to be
in a specific directory hierarchy:
'PATH/logstash/TYPE/NAME.rb' where TYPE is
'inputs' 'filters', 'outputs' or 'codecs'
and NAME is the name of the plugin.
(default: [])
-l, --path.logs PATH Write logstash internal logs to the given
file. Without this flag, logstash will emit
logs to standard output.
(default: "C:/Users/IBM_ADMIN/Downloads/logstash-6.6.0/logs")
--log.level LEVEL Set the log level for logstash. Possible values are:
- fatal
- error
- warn
- info
- debug
- trace
(default: "info")
--config.debug Print the compiled config ruby code out as a debug log (you must also have --log.level=debug enabled).
WARNING: This will include any 'password' options passed to plugin configs as plaintext, and may result
in plaintext passwords appearing in your logs!
(default: false)
-i, --interactive SHELL Drop to shell instead of running as normal.
Valid shells are "irb" and "pry"
-V, --version Emit the version of logstash and its friends,
then exit.
-t, --config.test_and_exit Check configuration for valid syntax and then exit.
(default: false)
-r, --config.reload.automatic Monitor configuration changes and reload
whenever it is changed.
NOTE: use SIGHUP to manually reload the config
(default: false)
--config.reload.interval RELOAD_INTERVAL How frequently to poll the configuration location
for changes, in seconds.
(default: 3000000000)
--http.host HTTP_HOST Web API binding host (default: "127.0.0.1")
--http.port HTTP_PORT Web API http port (default: 9600..9700)
--log.format FORMAT Specify if Logstash should write its own logs in JSON form (one
event per line) or in plain text (using Ruby's Object#inspect)
(default: "plain")
--path.settings SETTINGS_DIR Directory containing logstash.yml file. This can also be
set through the LS_SETTINGS_DIR environment variable.
(default: config目录的地址)
--verbose Set the log level to info.
DEPRECATED: use --log.level=info instead.
--debug Set the log level to debug.
DEPRECATED: use --log.level=debug instead.
--quiet Set the log level to info.
DEPRECATED: use --log.level=info instead.
-h, --help print help
data. Same syntax as the config file. If no
input is specified, then the following is
used as the default input:
"input { stdin { type => stdin } }"
and if no output is specified, then the
following is used as the default output:
"output { stdout { codec => rubydebug } }"
If you wish to use both defaults, please use
the empty string for the '-e' flag.
(default: nil)
--field-reference-parser MODE Use the given MODE when parsing field
references.
The field reference parser is used to expand
field references in your pipeline configs,
and will be becoming more strict to better
handle illegal and ambbiguous inputs in a
future release of Logstash.
Available MODEs are:
- `LEGACY`: parse with the legacy parser,
which is known to handle ambiguous- and
illegal-syntax in surprising ways;
warnings will not be emitted.
- `COMPAT`: warn once for each distinct
ambiguous- or illegal-syntax input, but
continue to expand field references with
the legacy parser.
- `STRICT`: parse in a strict manner; when
given ambiguous- or illegal-syntax input,
raises a runtime exception that should
be handled by the calling plugin.
The MODE can also be set with
`config.field_reference.parser`
(default: "COMPAT")
--modules MODULES Load Logstash modules.
Modules can be defined using multiple instances
'--modules module1 --modules module2',
or comma-separated syntax
'--modules=module1,module2'
Cannot be used in conjunction with '-e' or '-f'
Use of '--modules' will override modules declared
in the 'logstash.yml' file.
-M, --modules.variable MODULES_VARIABLE Load variables for module template.
Multiple instances of '-M' or
'--modules.variable' are supported.
Ignored if '--modules' flag is not used.
Should be in the format of
'-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"'
as in
'-M "example.var.filter.mutate.fieldname=fieldvalue"'
--setup Load index template into Elasticsearch, and saved searches,
index-pattern, visualizations, and dashboards into Kibana when
running modules.
(default: false)
--cloud.id CLOUD_ID Sets the elasticsearch and kibana host settings for
module connections in Elastic Cloud.
Your Elastic Cloud User interface or the Cloud support
team should provide this.
Add an optional label prefix '<label>:' to help you
identify multiple cloud.ids.
e.g. 'staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy'
--cloud.auth CLOUD_AUTH Sets the elasticsearch and kibana username and password
for module connections in Elastic Cloud
e.g. 'username:<password>'
--pipeline.id ID Sets the ID of the pipeline.
(default: "main")
-w, --pipeline.workers COUNT Sets the number of pipeline workers to run.
(default: 4)
--java-execution Use Java execution engine.
(default: false)
-b, --pipeline.batch.size SIZE Size of batches the pipeline is to work in.
(default: 125)
-u, --pipeline.batch.delay DELAY_IN_MS When creating pipeline batches, how long to wait while polling
for the next event.
(default: 50)
--pipeline.unsafe_shutdown Force logstash to exit during shutdown even
if there are still inflight events in memory.
By default, logstash will refuse to quit until all
received events have been pushed to the outputs.
(default: false)
--path.data PATH This should point to a writable directory. Logstash
will use this directory whenever it needs to store
data. Plugins will also have access to this path.
(default: "C:/Users/IBM_ADMIN/Downloads/logstash-6.6.0/data")
-p, --path.plugins PATH A path of where to find plugins. This flag
can be given multiple times to include
multiple paths. Plugins are expected to be
in a specific directory hierarchy:
'PATH/logstash/TYPE/NAME.rb' where TYPE is
'inputs' 'filters', 'outputs' or 'codecs'
and NAME is the name of the plugin.
(default: [])
-l, --path.logs PATH Write logstash internal logs to the given
file. Without this flag, logstash will emit
logs to standard output.
(default: "C:/Users/IBM_ADMIN/Downloads/logstash-6.6.0/logs")
--log.level LEVEL Set the log level for logstash. Possible values are:
- fatal
- error
- warn
- info
- debug
- trace
(default: "info")
--config.debug Print the compiled config ruby code out as a debug log (you must also have --log.level=debug enabled).
WARNING: This will include any 'password' options passed to plugin configs as plaintext, and may result
in plaintext passwords appearing in your logs!
(default: false)
-i, --interactive SHELL Drop to shell instead of running as normal.
Valid shells are "irb" and "pry"
-V, --version Emit the version of logstash and its friends,
then exit.
-t, --config.test_and_exit Check configuration for valid syntax and then exit.
(default: false)
-r, --config.reload.automatic Monitor configuration changes and reload
whenever it is changed.
NOTE: use SIGHUP to manually reload the config
(default: false)
--config.reload.interval RELOAD_INTERVAL How frequently to poll the configuration location
for changes, in seconds.
(default: 3000000000)
--http.host HTTP_HOST Web API binding host (default: "127.0.0.1")
--http.port HTTP_PORT Web API http port (default: 9600..9700)
--log.format FORMAT Specify if Logstash should write its own logs in JSON form (one
event per line) or in plain text (using Ruby's Object#inspect)
(default: "plain")
--path.settings SETTINGS_DIR Directory containing logstash.yml file. This can also be
set through the LS_SETTINGS_DIR environment variable.
(default: config目录的地址)
--verbose Set the log level to info.
DEPRECATED: use --log.level=info instead.
--debug Set the log level to debug.
DEPRECATED: use --log.level=debug instead.
--quiet Set the log level to info.
DEPRECATED: use --log.level=info instead.
-h, --help print help
以我初学者的眼光看，Logstash是ELK里面需要配置最多的组件。通过配置, Logstash可以搜集不同类型的日志供我们分析。下面是网上的一些例子：
Logstash收集RESTful API日志
Logstash收集log4j日志
Kibana 6.6
目录
bin -- 这个目录存放Kibana主程序
config -- 这个目录存放kibana.yml配置文件
data -- 好像是存放主程序生成的数据？
dlls (这应该只有windows才有) -- windows需要的动态链接库文件
node -- 这里面存放一个node.exe程序
node_modules -- 这里存放node插件
optimize -- 这里存放了一些JS和CSS文件，可能是Kibana界面所使用的
plugins -- 插件
src
-- 这里存放的是Kibana的源程序，比如core文件夹，里面的README描述是这样的Core is a set of systems
(frontend, backend etc.) that Kibana and its plugins are built on top
of. 通过修改源程序可以对Kibana进行定制。
target -- 这里面好像是一些.ts.map文件，应该是src编译之后产生的
webpackShims -- 这里面是一些JS文件，好像是一些common module, 比如ui-bootstrap.js, angular.js, elasticsearch.js
Help命令返回的内容
Usage: bin/kibana [command=serve] [options]
Kibana is an open source (Apache Licensed), browser based analytics and search dashboard for Elasticsearch.
Commands:
serve [options] Run the kibana server
help <command> Get the help for a specific command
"serve" Options:
-h, --help output usage information
-e, --elasticsearch <uri1,uri2> Elasticsearch instances
-c, --config <path> Path to the config file, can be changed with the CONFIG_PATH environment variable as well. Use multiple --config args to include multiple config files.
-p, --port <port> The port to bind to
-q, --quiet Prevent all logging except errors
-Q, --silent Prevent all logging
--verbose Turns on verbose logging
-H, --host <host> The host to bind to
-l, --log-file <path> The file to log to
--plugin-dir <path> A path to scan for plugins, this can be specified multiple times to specify multiple directories
--plugin-path <path> A path to a plugin which should be included by the server, this can be specified multiple times to specify multiple paths
--plugins <path> an alias for --plugin-dir
--optimize Optimize and then stop the server
Kibana is an open source (Apache Licensed), browser based analytics and search dashboard for Elasticsearch.
Commands:
serve [options] Run the kibana server
help <command> Get the help for a specific command
"serve" Options:
-h, --help output usage information
-e, --elasticsearch <uri1,uri2> Elasticsearch instances
-c, --config <path> Path to the config file, can be changed with the CONFIG_PATH environment variable as well. Use multiple --config args to include multiple config files.
-p, --port <port> The port to bind to
-q, --quiet Prevent all logging except errors
-Q, --silent Prevent all logging
--verbose Turns on verbose logging
-H, --host <host> The host to bind to
-l, --log-file <path> The file to log to
--plugin-dir <path> A path to scan for plugins, this can be specified multiple times to specify multiple directories
--plugin-path <path> A path to a plugin which should be included by the server, this can be specified multiple times to specify multiple paths
--plugins <path> an alias for --plugin-dir
--optimize Optimize and then stop the server
在实际使用中，我们可能需要自定义Kibana界面。 我在网上找到两篇文章，先收藏着吧。
Kibana界面风格重制
Kibana国际化
https://www.jianshu.com/p/57e0475fda90
