从零开始搭建系统2.2——ELK安装及配置

 

ELK 最新版本对JDK的最低要求是1.8，安装java_1.8版本

一.Elasticsearch 

1.创建目录

2.下载安装包

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.tar.gz

3.解压安装包

tar -zxvf elasticsearch-6.2.3.tar.gz

4.新建用户

groupadd elk#创建组

useradd elk -g elk  #创建用户并将用户添加到组

 #修改文件所有者

chown -R elk /usr/elk/elasticsearch/elasticsearch-6.2.3/

 5.启动服务

切换用户至elk,启动服务

6.设置开机启动

vim /etc/init.d/elasticsearch

#!/bin/sh
# chkconfig: 2345 90 10
# description: elasticsearch service

su - elk<<!
cd  /usr/elk/elasticsearch/elasticsearch-6.2.3/
./bin/elasticsearch -d &
exit

!

保存

添加自启动权限

chmod a+x /etc/init.d/elasticsearch

使用

service elasticsearch status/start/stop
查看nexus服务状态、启动服务、停止服务等

使用
chkconfig elasticsearch on/off

设置nexus服务开机自启动或者开机不启动

设置开机启动的时候出现下面提示会

解决方案:

vim elasticsearch

在开始加上下面两行内容

# chkconfig: 2345 90 10
# description: elasticsearch service

然后重新设置

7.验证是否启动成功

如果有信息返回则说明启动成功

8.elasticsearch默认端口为9200，开启端口

添加

firewall-cmd --zone=public --add-port=9200/tcp --permanent    （--permanent永久生效，没有此参数重启后失效）

重新载入

firewall-cmd --reload

查看

firewall-cmd --zone= public --query-port=9200/tcp

查看所有打开的端口

firewall-cmd --zone=public --list-ports

9.修改elasticsearch配置，通过浏览器可以访问

如果浏览器中访问http://localhost:9200/没有返回预期的结果，就需要修改Elasticsearch的配置，使其支持外网访问。

 

 

为了能够通过浏览器访问elasticsearch，我们需要配置elasticsearch的配置文件，主要是修改 

network.host和http.port

vim config/elasticsearch.yml

#修改配置文件内容如下：
network.host: 0.0.0.0

注意：

:后面需要一个空格，格式不正确会报错误

然后保存，重新启动服务

重启服务，发现报错

ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

 

修改/etc/security/limits.conf配置文件：

1 2 3 4 5 6 7

[root@yqtrack-elk03 /]# vim /etc/security/limits.conf   添加如下配置项：   * - nproc 65535 * - nofile 409600 elastic - memlock unlimited

　修改/etc/security/limits.d/90-nproc.conf配置文件：

1 2 3 4 5 6

[root@yqtrack-elk03 /]# vim /etc/security/limits.d/90-nproc.conf   修改如下配置项目：   * soft nproc unlimited root soft nproc unlimited

　修改完成后，重新登录elk账户，查看设置是否生效。

1 2 3 4

[elk@yqtrack-elk03 /]$ ulimit -n 409600 [elk@yqtrack-elk03 /]$ ulimit -u 65535

 

以root账户更改/etc/sysctl.conf文件，添加如下内容

vm.max_map_count=655360

以root账户执行下面命令

[***@elk01 ~]$ sudo sysctl -p
vm.max_map_count = 655360

 

 

 

二.Logstash

1.创建目录

 

2.下载安装包

wget -c https://artifacts.elastic.co/downloads/logstash/logstash-6.2.3.tar.gz

3.解压安装包

tar -xzvf logstash-6.2.3.tar.gz

 4.创建配置文件

vim logstash.conf
在配置文件logstash.conf添加如下内容：

input {
log4j {
host => "192.168.0.200"
port => 4560
}
}

#过滤器
filter{

#删除无用字段
mutate {
remove_field => "file"
remove_field => "method"
remove_field => "@version"
remove_field => "thread"
remove_field => "class"
remove_field => "timestamp"
remove_field => "@timestamp"
}

#自定义字段
json
{
source=>"message"
}

}

output {
stdout
{
codec => rubydebug
}
elasticsearch{
hosts => ["127.0.0.1:9200"]
index => "app_%{appId}"
document_type => "appLog"
}
}

5.启动

 #修改文件所有者

chown -R elk /usr/elk/logstash/logstash-6.2.3

./bin/logstash ./config/logstash.conf

启动的时候如果出现下面错误，则说明缺少LOG4J插件,需要进行安装

可以通过命令查看已安装插件

安装插件 

./logstash-plugin install logstash-input-log4j

 

 6.设置开机启动

vim /etc/init.d/logstash

#!/bin/bash

# chkconfig: 2345 50 50
# description: logstash
PATH=/sbin:/usr/sbin:/bin:/usr/bin
export PATH

if [ `id -u` -ne 0 ]; then
echo "You need root privileges to run this script"
exit 1
fi

name=logstash
pidfile="/var/run/$name.pid"

LS_USER=elk
LS_GROUP=elk
LS_HOME=/opt/soft/elk/logstash/logstash-6.2.3
LS_HEAP_SIZE="500m"
LS_LOG_DIR="${LS_HOME}/logs"
LS_LOG_FILE="${LS_LOG_DIR}/$name.log"
LS_CONF_DIR=$LS_HOME/config/logstash.conf
LS_OPEN_FILES=16384
LS_NICE=19
LS_OPTS=""

[ -r /etc/default/$name ] && . /etc/default/$name
[ -r /etc/sysconfig/$name ] && . /etc/sysconfig/$name

program=${LS_HOME}/bin/logstash
args=" -f ${LS_CONF_DIR}"
start() {
LS_JAVA_OPTS="${LS_JAVA_OPTS} -Djava.io.tmpdir=${LS_HOME}"
HOME=${LS_HOME}
export PATH HOME LS_HEAP_SIZE LS_JAVA_OPTS LS_USE_GC_LOGGING

# chown doesn't grab the suplimental groups when setting the user:group - so we have to do it for it.
# Boy, I hope we're root here.
SGROUPS=$(id -Gn "$LS_USER" | tr " " "," | sed 's/,$//'; echo '')

if [ ! -z $SGROUPS ]
then
EXTRA_GROUPS="--groups $SGROUPS"
fi

# set ulimit as (root, presumably) first, before we drop privileges
ulimit -n ${LS_OPEN_FILES}

# Run the program!
nice -n ${LS_NICE} chroot --userspec $LS_USER:$LS_GROUP $EXTRA_GROUPS / sh -c "
cd $LS_HOME
ulimit -n ${LS_OPEN_FILES}
exec \"$program\" $args
" > "${LS_LOG_DIR}/$name.stdout" 2> "${LS_LOG_DIR}/$name.err" &

# Generate the pidfile from here. If we instead made the forked process
# generate it there will be a race condition between the pidfile writing
# and a process possibly asking for status.
echo $! > $pidfile
echo "$name started."
return 0
}

stop() {
# Try a few times to kill TERM the program
if status ; then
pid=`cat "$pidfile"`
echo "Killing $name (pid $pid) with SIGTERM"
kill -TERM $pid
# Wait for it to exit.
for i in 1 2 3 4 5 ; do
echo "Waiting $name (pid $pid) to die..."
status || break
sleep 1
done
if status ; then
if [ "$KILL_ON_STOP_TIMEOUT" -eq 1 ] ; then
echo "Timeout reached. Killing $name (pid $pid) with SIGKILL. This may result in data loss."
kill -KILL $pid
echo "$name killed with SIGKILL."
else
echo "$name stop failed; still running."
fi
else
echo "$name stopped."
fi
fi
}

status() {
if [ -f "$pidfile" ] ; then
pid=`cat "$pidfile"`
if kill -0 $pid > /dev/null 2> /dev/null ; then
# process by this pid is running.
# It may not be our pid, but that's what you get with just pidfiles.
# TODO(sissel): Check if this process seems to be the same as the one we
# expect. It'd be nice to use flock here, but flock uses fork, not exec,
# so it makes it quite awkward to use in this case.
return 0
else
return 2 # program is dead but pid file exists
fi
else
return 3 # program is not running
fi
}

force_stop() {
if status ; then
stop
status && kill -KILL `cat "$pidfile"`
fi
}

case "$1" in
start)
status
code=$?
if [ $code -eq 0 ]; then
echo "$name is already running"
else
start
code=$?
fi
exit $code
;;
stop) stop ;;
force-stop) force_stop ;;
status)
status
code=$?
if [ $code -eq 0 ] ; then
echo "$name is running"
else
echo "$name is not running"
fi
exit $code
;;
restart)

stop && start
;;
*)
echo "Usage: $SCRIPTNAME {start|stop|force-stop|status|restart}" >&2
exit 3
;;
esac

exit $?

保存

添加自启动权限

chmod a+x /etc/init.d/logstash

使用

service logstash status/start/stop
查看nexus服务状态、启动服务、停止服务等

使用
chkconfig logstash on/off

三.Kibana

1.创建目录

 

2.下载安装包

wget -c https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-linux-x86_64.tar.gz

3.解压安装包

tar -xzvf kibana-6.2.3-linux-x86_64.tar.gz

4.启动服务

5.开启端口

添加

firewall-cmd --zone=public --add-port=5601/tcp --permanent    （--permanent永久生效，没有此参数重启后失效）

重新载入

firewall-cmd --reload

查看

firewall-cmd --zone= public --query-port=5601/tcp

查看所有打开的端口

firewall-cmd --zone=public --list-ports

6.设置外网可访问

修改config/kibaba.yml下的server.host为0.0.0.0

7.设置开机启动

vim /etc/init.d/kibana

#!/bin/bash

# chkconfig: 2345 98 02
# description: kibana

KIBANA_HOME=/usr/elk/kibana/kibana-6.2.3-linux-x86_64

case $1 in
start) $KIBANA_HOME/bin/kibana &;;
*) echo "require start";;
esac

保存

添加自启动权限

chmod a+x /etc/init.d/kibana

使用

service kibana status/start/stop
查看nexus服务状态、启动服务、停止服务等

使用
chkconfig kibana on/off

8.验证安装结果