ELK 部署

文章转载：

http://www.open-open.com/doc/view/df156a76a824402482d1d72cd3b61e38

http://www.open-open.com/lib/view/open1441088477784.html

http://www.07net02.com/?p=563

https://www.ttlsa.com/elk/elk-topbeat-deployment-guide/

http://www.ibm.com/developerworks/cn/data/library/ba/ba-bluemix-elklog/index.html

http://www.cnblogs.com/delgyd/p/elk.html

ELK 日志平台搭建

一.平台了解

1.1 ElasticSearch是一个基于Lucene构建的开源，分布式，RESTful搜索引擎。设计用于云计算中，能够达到实时搜索，稳定，可靠，快速，安装使用方便。支持通过HTTP使用JSON进行数据索引。

1.2 logstash是一个应用程序日志、事件的传输、处理、管理和搜索的平台。你可以用它来统一对应用程序日志进行收集管理，提供 Web 接口用于查询和统计。其实logstash是可以被别的替换，比如常见的fluented

1.3 Kibana是一个为 Logstash 和 ElasticSearch 提供的日志分析的 Web 接口。可使用它对日志进行高效的搜索、可视化、分析等各种操作。

Redis是一个高性能的内存key-value数据库,非必需安装,可以防止数据丢失.

二.架构拓扑

三 安装部署

2.1 客户端：nginx+logstash 192.168.127.141

服务端：logstash+redis+elasticSearch+kibana 192.168.127.140 #本例子将三个组件同时安装在一台服务器。

 

服务器端安装

JDK 安装

yum install java-1.8.0-openjdk

redis 部署

#tar –xvzf redis-3.0.4.tar.gz

#cd redis-3.0.4

#make

#make install

#cd utils

#./install_server.sh

这样redis就安装好了
#which redis-server在/usr/local/bin/redis-server

配置文件放在/etc/redis/6379.conf文件

#redis-cli

>set aa aavalue

>get aa 即可测试。
#cd

elasticsearch 部署

#tar –xvzf elasticsearch-1.7.1.tar.gz

#cp –a elasticsearch-1.7.1 /usr/local

#cd /usr/local

#ln –s elasticsearch-1.7.1 elasticsearch

#cd

这里我们可以下载一个elasticsearch的启动脚本来启动elasticsearch

git   clone  git@github.com:smail-bao/elk.git

然后把service目录挪到elasticsearch/bin目录下

然后执行/elasticsearch/bin/service/elasticsearch install

就可以把elasticsearch 就开机自动启动，且我们可以使用/etc/init.d/elasticsearch   了

部署中心logstash

#tar –xvzf logstash-1.5.4.tar.gz

#cp –a logstash-1.5.4 /usr/local

#cd /usr/local

#ln –s logstash-1.5.4 logstash

#cd /usr/local/logstash

#cd conf && vim logstash_server.conf

input {
        redis {
                host => "127.0.0.1"           
                port => 6379                  
                type => "redis-input"             
            data_type => "list"                
                key => "logstash:redis"
        }
}
output {
        stdout {}
        elasticsearch {
                host => "192.168.220.116"               
                cluster => "elasticsearch"
                codec => "json"
                protocol => "http"                   
        }
}

配置文件的意思就是，输入来自redis，使用redis的list类型存储数据，key为"logstash:redis"，输出到elasticsearch,cluster的名称为elasticsearch,这里的cluster 就是elasticsearch.yml里面的cluster.name,集群的名称

kibana部署

#tar –xvzf kibana-4.1.2-linux-x64.tar.gz

#cp –a kibana-4.1.2-linux-x64 /usr/local

#cd /usr/local

#ln –s kibana-4.1.2-linux-x64 kibana

我们再下载一个kibana的脚本文件，注意这里要修改自己相应的软件安装路劲

该脚本来自：http://www.open-open.com/lib/view/open1441088477784.html

git  clone   git@github.com:smail-bao/elk.git
#创建kibana服务
vi /etc/rc.d/init.d/kibana

#!/bin/bash
### BEGIN INIT INFO
# Provides:          kibana
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Runs kibana daemon
# Description: Runs the kibana daemon as a non-root user
### END INIT INFO

# Process name
NAME=kibana
DESC="Kibana4"
PROG="/etc/init.d/kibana"

# Configure location of Kibana bin
KIBANA_BIN=/usr/local/kibana/bin

# PID Info
PID_FOLDER=/var/run/kibana/
PID_FILE=/var/run/kibana/$NAME.pid
LOCK_FILE=/var/lock/subsys/$NAME
PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN
DAEMON=$KIBANA_BIN/$NAME

# Configure User to run daemon process
DAEMON_USER=root
# Configure logging location
KIBANA_LOG=/var/log/kibana.log

# Begin Script
RETVAL=0

if [ `id -u` -ne 0 ]; then
        echo "You need root privileges to run this script"
        exit 1
fi

# Function library
. /etc/init.d/functions
 
start() {
        echo -n "Starting $DESC : "

pid=`pidofproc -p $PID_FILE kibana`
        if [ -n "$pid" ] ; then
                echo "Already running."
                exit 0
        else
        # Start Daemon
if [ ! -d "$PID_FOLDER" ] ; then
                        mkdir $PID_FOLDER
                fi
daemon --user=$DAEMON_USER --pidfile=$PID_FILE $DAEMON 1>"$KIBANA_LOG" 2>&1 &
                sleep 2
                pidofproc node > $PID_FILE
                RETVAL=$?
                [[ $? -eq 0 ]] && success || failure
echo
                [ $RETVAL = 0 ] && touch $LOCK_FILE
                return $RETVAL
        fi
}

reload()
{
    echo "Reload command is not implemented for this service."
    return $RETVAL
}

stop() {
        echo -n "Stopping $DESC : "
        killproc -p $PID_FILE $DAEMON
        RETVAL=$?
echo
        [ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE
}
 
case "$1" in
  start)
        start
;;
  stop)
        stop
        ;;
  status)
        status -p $PID_FILE $DAEMON
        RETVAL=$?
        ;;
  restart)
        stop
        start
        ;;
  reload)
reload
;;
  *)
# Invalid Arguments, print the following message.
        echo "Usage: $0 {start|stop|status|restart}" >&2
exit 2
        ;;
esac

#修改启动权限
chmod +x /etc/rc.d/init.d/kibana

#启动kibana服务
service kibana start
service kibana status

程序安装全部完成。

由于elasticsearch和logstash是安装在一台机器上所以elasticsearch默认配置即可。
/usr/local/elasticsearch/bin/elasticsearch –d （以deamon方式启动elasticsearch）

访问192.168.127.141:9200即可看到

配置logstash程序
# cd /usr/local/logstash下，mkdir etc logs两个文件夹，etc用于存放配置文件，logs用于存放日志文件
在该etc目录下建立一个central.conf配置文件：

# cat /usr/local/logstash/etc/central.conf

input {

redis {

host => "127.0.0.1"

port => 6379

type => "redis_input"

data_type => "list"

key => "logstash:redis"

}

}

 

output {

stdout {}

elasticsearch {

cluster => "elasticsearch"

codec => "json"

protocol => "http"

}

}

 

该文件说明是以redis为输入，输出到elasticsearch程序，格式为json协议为http.

启动logstash程序如下：

#/usr/local/logstash/bin/logstash -f /usr/local/logstash/etc/central.conf -l /usr/local/logstash/logs/stdout.log

Sending logstash logs to /usr/local/logstash/logs/stdout.log.

 

Kibana程序现在是4版本了，自带web，端口为5601:

#cd /usr/local/kibana/bin/

#./kibana程序即可。

[root@localhost bin]# ./kibana

{"name":"Kibana","hostname":"localhost.localdomain","pid":1645,"level":30,"msg":"Found kibana index","time":"2016-01-28T05:00:17.462Z","v":0}

{"name":"Kibana","hostname":"localhost.localdomain","pid":1645,"level":30,"msg":"Listening on 0.0.0.0:5601","time":"2016-01-28T05:00:18.205Z","v":0}

现在可以打开 kibana的页面了。

192.168.127.140:5601

 

四 客户端安装

现在要在客户端（192.168.127.141）收集日志

#tar –xvzf logstash-1.5.4.tar.gz

#cp –a logstash-1.5.4 /usr/local

#cd /usr/local

#ln –s logstash-1.5.4 logstash

#cd /usr/local/logstash目录
同样需要#mkdir etc logs存放配置文件和日志文件。
# cat etc/logstash_agent.conf

input {

file {

type => "nginx_access"

path => ["/usr/local/nginx/logs/access.log"]

}

}

output {

redis {

host => "192.168.127.140"

data_type => "list"

key => "logstash:redis"

}

}

启动该logstash程序即可。
#/usr/local/logstash/bin/logstash –f /usr/local/logstash/etc/logstash_agent.conf

最后打开kibana查看采集到的nginx访问日志