CentOS7.4部署ELK日志分析系统

官网地址:https://www.elastic.co/cn/，官网权威指南:https://www.elastic.co/guide/cn/elasticsearch/guide/current/index.html, 安装指南：https://www.elastic.co/guide/en/elasticsearch/reference/5.x/rpm.html.  ELK是Elasticsearch、Logstash、Kibana的简称，这三者是核心套件，但并非全部。Elasticsearch是实时全文搜索和分析引擎，提供搜集、分析、存储数据三大功能；是一套开放REST和JAVA API等结构提供高效搜索功能，可扩展的分布式系统。它构建于Apache Lucene搜索引擎库之上。Logstash是一个用来搜集、分析、过滤日志的工具。它支持几乎任何类型的日志，包括系统日志、错误日志和自定义应用程序日志。它可以从许多来源接收日志，这些来源包括syslog、消息传递（例如RabbitMQ）和JMX，它能够以多种方式输出数据，包括电子邮件、websockets和Elasticsearch。

一、java环境，安装jdk 1.8及以上的版本

[root@elk-node1 ~]# yum install -y java-1.8.0 

[root@elk-node1 ~]# java -version

openjdk version "1.8.0_121"

OpenJDK Runtime Environment (build 1.8.0_121-b13)

OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

   二、elasticsearch安装

1、下载Elasticsearch最新版本

解压至指定目录(安装目录)/usr/local/下，并将其重命名为elasticsearch (完整的目录应该为/usr/local/elasticsearch)

# tar xzf elasticsearch-6.4.2.tar.gz  -C /usr/local

# cd /usr/local/

# mv elasticsearch-6.4.2/ elasticsearch

2、创建一个用于运行elasticsearch的普通用户，随后修改elasticsearch家目录的所属权限为该用户；创建elasticsearch数据存储目录/data/elasticsearch

# groupadd elasticsearch

# useradd -g elasticsearch elasticsearch -m

useradd  elasticsearch(用户名) -g elasticsearch(组名) -p elasticsearch(密码)

# useradd elasticsearch -g elasticsearch -p elasticsearch

更改Elasticsearch文件夹以及内部文件的所属用户以及组为elasticsearch

# chown -R elasticsearch:elasticsearch elasticsearch

# mkdir /data/elasticsearch

# chown -R elasticsearch. /data/elasticsearch

3、配置环境变量

# vim /etc/profile

ulimit -u 4096

# source /etc/profile

4、修改elasticsearch.yml配置文件

# vim config/elasticsearch.yml

cluster.name: my-application     #ELK集群名称

path.data: /data/elasticsearch    #elasticsearch 数据存储目录

path.logs: /usr/local/elasticsearch/logs       #elasticsearch 日志存储路径

network.host: 10.66.1.23       #elasticsearch 监听地址，默认为localhost

http.port: 9200          #elasticsearch 监听端口，默认问9200

5、修改相关的内核参数

[root@localhostlocal]# vim /etc/security/limits.conf

添加以下内容

* soft nproc 65536

* hard nproc 65536

* soft nofile 65536

* hard nofile 65536

#修改vm.max_map_count=262144

# echo "vm.max_map_count=262144" >> /etc/sysctl.conf

6、以下由elasticsearch用户操作，以elasticsearch用户登录服务器，运行elasticsearch (注意：要切换到普通用户运行)

# su - elasticsearch

运行elasticsearch

$ ./elasticsearch/bin//elasticsearch或

$cd /usr/local/elasticsearch/

$ ./bin/elasticsearch

一般情况我们要求elasticsearch在后台运行，使用命令如下：

$ ./bin/elasticsearch -d

7、检查elasticsearch状态，如下则表示正常运行

# curl http://10.66.1.23:9200

{

  "name" : "dlOHzTB",

  "cluster_name" : "elasticsearch",

  "cluster_uuid" : "WuNxFom3QUWZLqC61-FSCw",

  "version" : {

    "number" : "6.4.2",

    "build_flavor" : "default",

    "build_type" : "tar",

    "build_hash" : "04711c2",

    "build_date" : "2018-09-26T13:34:09.098244Z",

    "build_snapshot" : false,

    "lucene_version" : "7.4.0",

    "minimum_wire_compatibility_version" : "5.6.0",

    "minimum_index_compatibility_version" : "5.0.0"

  },

  "tagline" : "You Know, for Search"

}

跳转到Elasticsearch的config配置文件下，使用vim打开elasticsearch.yml，找到里面的"network.host",将其改为你的IP,保存。

#cd elasticsearch/config/

#vim elasticsearch.yml

重启ElasticSearch，然后使用http://10.66.1.123:9200/访问，访问结果所示：，则说明ElasticSearch安装成功。如果是使用命令./bin/elasticsearch来启动的Elasticsearch，如果想要停止Elasticsearch的执行，则直接按住键盘Ctrl+C则会停止，停止之后你在浏览器中再次测试发现已不能操作。

http://10.66.1.23:9200/

name"dlOHzTB"

cluster_name"elasticsearch"

cluster_uuid"WuNxFom3QUWZLqC61-FSCw"

version

number"6.4.2"

build_flavor"default"

build_type"tar"

build_hash"04711c2"

build_date"2018-09-26T13:34:09.098244Z"

build_snapshotfalse

lucene_version"7.4.0"

minimum_wire_compatibility_version"5.6.0"

minimum_index_compatibility_version"5.0.0"

tagline"You Know, for Search"

三、Logstash安装

1、下载logstash软件包

# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.2.tar.gz

2、解压至指定安装目录

# tar -zxf logstash-6.4.2.tar.gz -C /usr/local

# cd /usr/local/

# mv logstash-5.5.2 logstash

3、运行logstash

# cd logstash/

# ./bin/logstash -e 'input { stdin { } } output { stdout {} }'

输入”hello world!  ”，验证是否正常输出

[root@elk-server logstash]# ./bin/logstash -e 'input { stdin { } } output { stdout {} }'ERROR StatusLogger No log4j2 configurationfilefound. Using default configuration: logging only errors to the console.Sending Logstash's logs to /usr/local/logstash/logswhichis now configured via log4j2.properties[2017-08-28T15:11:33,267][INFO][logstash.setting.writabledirectory]Creating directory{:setting=>"path.queue", :path=>"/usr/local/logstash/data/queue"}[2017-08-28T15:11:33,273][INFO][logstash.setting.writabledirectory]Creating directory{:setting=>"path.dead_letter_queue", :path=>"/usr/local/logstash/data/dead_letter_queue"}[2017-08-28T15:11:33,300][INFO][logstash.agent]No persistent UUIDfilefound. Generating new UUID{:uuid=>"2fb479ab-0ca5-4979-89b1-4246df9a7472", :path=>"/usr/local/logstash/data/uuid"}[2017-08-28T15:11:33,438][INFO][logstash.pipeline]Starting pipeline{"id"=>"main","pipeline.workers"=>8,"pipeline.batch.size"=>125,"pipeline.batch.delay"=>5,"pipeline.max_inflight"=>1000}[2017-08-28T15:11:33,455][INFO][logstash.pipeline]Pipeline main startedThe stdin plugin is now waitingforinput:[2017-08-28T15:11:33,497][INFO][logstash.agent]Successfully started Logstash API endpoint{:port=>9600}hello world!2017-08-28T07:11:42.724Z elk-server.huangming.org hello world!

四、Kibana安装

1、下载kibana

# wget https://artifacts.elastic.co/downloads/kibana/kibana-5.5.2-linux-x86_64.tar.gz

2、解压至安装目录

# tar -zxf kibana-6.4.2-linux-x86_64.tar.gz -C /usr/local

# cd /usr/local/

# mv kibana-6.4.2-linux-x86_64/ kibana

3、修改配置

# cd kibana/

# vim config/kibana.yml

server.port: 5601   # 监听端口

server.host:"10.66.1.23"    # 指定后端服务器

elasticsearch.url:"http://10.66.1.23:9200"      # 指定elasticsearch实例地址

4、运行kibana

# ./bin/kibana &

# ./bin/kibana &[1]3219

[root@Anwar01 kibana]# ./bin/kibana

  log  [06:48:27.127] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:27.222] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:27.227] [info][status][plugin:xpack_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:27.234] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:27.240] [info][status][plugin:ml@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:27.329] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:27.332] [info][status][plugin:watcher@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:27.352] [info][status][plugin:license_management@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:27.355] [info][status][plugin:index_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:28.747] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:28.751] [info][status][plugin:graph@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:28.843] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:28.849] [warning][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml

  log  [06:48:28.855] [warning][security] Session cookies will be transmitted over insecure connections. This is not recommended.

  log  [06:48:28.871] [info][status][plugin:security@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:28.913] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:28.924] [info][status][plugin:dashboard_mode@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:28.928] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:28.965] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:29.028] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:29.031] [info][status][plugin:console_extensions@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:29.037] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:29.042] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:45.846] [warning][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml

  log  [06:48:45.850] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:45.868] [error][status][plugin:xpack_main@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.869] [error][status][plugin:searchprofiler@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.870] [error][status][plugin:ml@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.871] [error][status][plugin:tilemap@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.871] [error][status][plugin:watcher@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.872] [error][status][plugin:index_management@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.873] [error][status][plugin:graph@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.874] [error][status][plugin:security@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.875] [error][status][plugin:grokdebugger@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.875] [error][status][plugin:logstash@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.876] [error][status][plugin:reporting@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.877] [error][status][plugin:elasticsearch@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:47.418] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active

  log  [06:48:47.423] [info][status][plugin:xpack_main@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.424] [info][status][plugin:searchprofiler@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.425] [info][status][plugin:ml@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.425] [info][status][plugin:tilemap@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.426] [info][status][plugin:watcher@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.427] [info][status][plugin:index_management@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.428] [info][status][plugin:graph@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.429] [info][status][plugin:grokdebugger@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.430] [info][status][plugin:logstash@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.430] [info][status][plugin:reporting@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.431] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection

  log  [06:48:47.436] [info][status][plugin:security@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.564] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active

  log  [06:48:50.527] [info][status][plugin:elasticsearch@6.4.2] Status changed from red to green - Ready

  log  [06:49:07.536] [info][listening][server][http] Server running at http://10.66.1.23:5601

5、验证kibana

在客户端浏览器打开http://10.66.1.23:5601

在该页面提示我们需要创建一个index

首先创建一个kinana默认的index(名称为.kibana),如果输入的index名不存在，则无法创建

 
 
 
 

查看运行状态及已安装的插件

 

至此ELK已经搭建完成了，下面来创建一个收集message系统日志的实例

posted on 2018-10-25 17:11  Anwar￠  阅读(384)  评论(0)  编辑  收藏  举报

Powered by: 博客园 Copyright © 2024 Anwar￠
Powered by .NET 9.0 on Kubernetes