ELK6.3版本安装部署
一、Elasticsearch 安装
1、部署系统以及环境准备 cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core) uname -r 3.10.0-693.el7.x86_64 #Firewalld and selinux systemctl stop firewalld systemctl disable firewalld sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config setenforce 0 sed -i 's/localhost.localdomain/ELK.localdomain/' /etc/hostname hostnamectl set-hostname ELK.localdomain 安装jdk8以上版本 mkdir /application/ tar xf jdk-8u151-linux-x64.tar.gz -C /application/ ln -s /application/jdk1.8.0_151 /application/jdk sed -i.ori '$a export JAVA_HOME=/application/jdk\nexport PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH\nexport CLASSPATH=.$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar' /etc/profile source /etc/profile java -version java version "1.8.0_151" Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode) 2、部署elasticsearch6.3.2 采用压缩包解压缩安装方式启动服务 创建另外独立账户专供es使用 cd /usr/local wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.tar.gz tar zxvf elasticsearch-6.3.2.tar.gz -C /application/ groupadd ela 创建ela组 useradd -g ela ela 创建ela用户，并且加入ela组 passwd ela 为ela用户设定登录密码 ln -s /application/elasticsearch-6.3.2/ /application/elasticsearch chown -R ela.ela /application/elasticsearch 配置文件 grep -n '^[a-Z]' /application/elasticsearch/config/elasticsearch.yml 17:cluster.name: elk 23:node.name: node-1 33:path.data: /home/elkdata 37:path.logs: /var/log/elasticsearch 42:bootstrap.memory_lock: true 54:network.host: 0.0.0.0 58:http.port: 9200 88:http.cors.enabled: true 89:http.cors.allow-origin: "*" mkdir /home/elkdata/ -p chown -R ela.ela /home/elkdata/ mkdir /var/log/elasticsearch -p chown -R ela.ela /var/log/elasticsearch/ vim /etc/security/limits.conf #增加以下行 * soft nofile 65536 * hard nofile 131072 ela soft memlock unlimited ela hard memlock unlimited 让设置生效重启系统或者另外开一个终端 vim /etc/sysctl.conf #增加一行 vm.max_map_count = 655360 sysctl -p 检查文件权限是否是ela用户 ls -l /application/elasticsearch-6.3.2/ ls -l /application/elasticsearch/ ls -l /application/elasticsearch/config/jvm.options 切换到ela用户，-d后台运行 [ela@elk elasticsearch-6.3.2]$ ./bin/elasticsearch -d [ela@elk elasticsearch-6.3.2]$ netstat -ntpl|grep 9200 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp6 0 0 :::9200 :::* LISTEN 147438/java
二、Kibana安装
#下载 cd /usr/local/src wget https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gz #解压 tar xzvf kibana-6.3.2-linux-x86_64.tar.gz ln -s /application/kibana-6.3.2-linux-x86_64 /application/kibana vim /application/kibana/config/kibana.yml #端口 server.port: 5601 #服务器IP server.host: "10.10.114.4" #elasticsearch服务器 elasticsearch.url: "http://10.10.114.4:9200" 启动 nohup bin/kibana & 查看端口占用命令 netstat -apn |grep 5601 #杀掉进程 kill -9 进程号
浏览器访问：http://ip:5601
kibana-6.3.0版本以后新增index索引不支持常用的正则匹配了，只能使用【*】
三、Logstash安装
另外一台10.10.114.2安装Logstash mkdir /application/ tar xf jdk-8u151-linux-x64.tar.gz -C /application/ ln -s /application/jdk1.8.0_151 /application/jdk sed -i.ori '$a export JAVA_HOME=/application/jdk\nexport PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH\nexport CLASSPATH=.$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar' /etc/profile source /etc/profile tar xf logstash-6.3.2.tar.gz -C /application/ ln -s /application/logstash-6.3.2 /application/logstash #收集IIS日志为例 cd /application/logstash/ mkdir conf && cd conf cat IIS.conf input { beats { port => "5044" codec => json } } filter { if [message] =~ "^#" { drop {} } grok { match =>{"message" => "%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:s_ip} %{WORD:request_method} %{NOTSPACE:uripath} %{NOTSPACE:uri-query} %{NUMBER:port} - %{IPORHOST:c_ip} %{NOTSPACE:agent} %{NOTSPACE:referer} %{NUMBER:status} %{NUMBER:sc_bytes} %{NUMBER:cs_bytes} %{NUMBER:time}" } } date { match => ["timestamp","YYYY-MM-dd HH:mm:ss"] } mutate { remove_field => ["message","beat","_id","host","@version","_score","tags"] } } output { elasticsearch { hosts => ["10.10.114.4:9200"] index => "logstash-iis-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } } 首先来测试配置文件是否可用,使用-f标志指定配置文件。 /application/logstash/bin/logstash -f conf/IIS.conf --config.test_and_exit --config.test_and_exit,会测试你配置文件的正确性,并给出错误信息. 如果可用,会输出 OK ,之后可以通过下面的命令来启动logstash /application/logstash/bin/logstash -f conf/IIS.conf & 查看端口 netstat -ntpl|grep 5044
客户端安装Filebeat
windows下直接解压缩后filebeat-6.3.2-windows-x86_64
修改filebeat.yml文件
filebeat.inputs: - type: log encoding: GBK #编码格式 enabled: true paths: - D:\ApplicationLogs\*\*\* #收集日志的路径 output.logstash: hosts: ["10.10.114.2:5044"] #直接写到logstash
可以在cmd下测试执行、可以查看到传输的日志文件
/路径/filebeat.exe -c filebeat.yml -e
head插件安装
Elasticsearch6.x版本不能使用命令直接安装head插件 修改配置文件/etc/elasticsearch/elasticsearch.yml增加参数 # 增加参数，使head插件可以访问es http.cors.enabled: true http.cors.allow-origin: "*" 下载head插件 cd /usr/local/src wget https://github.com/mobz/elasticsearch-head/archive/master.zip unzip master.zip mv elasticsearch-head-master/ /application/ 安装node wget https://npm.taobao.org/mirrors/node/latest-v4.x/node-v4.4.7-linux-x64.tar.gz tar -zxvf node-v4.4.7-linux-x64.tar.gz 修改环境变量/etc/profile添加 export NODE_HOME=/application/node-v4.4.7-linux-x64 export PATH=$PATH:$NODE_HOME/bin export NODE_PATH=$NODE_HOME/lib/node_modules 设置生效 source /etc/profile 安装grunt cd /application/elasticsearch-head-master npm install -g grunt-cli 检查是否安装成功 [root@elk elasticsearch]# grunt -version grunt-cli v1.3.1
修改head插件源码/application/elasticsearch-head-master/Gruntfile.js
hostname是新增的，不要忘记原有的true后面加,符号
修改连接地址/application/elasticsearch-head-master/_site/app.js
下载运行head必要的文件（放置在文件夹/tmp下）
cd /tmp wget https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2 yum -y install bzip2
运行head
cd /application/elasticsearch-head-master
npm install
后台启动
grunt server &
web页面验证
可以查看到node1节点
最简单的做法就是直接在谷歌浏览器添加应用程序
