elk 分布式部署

这个logstash 读取日志 是增量的 还是怎么读的?

定时每秒读增量


机器配置；

elasticsearch-192.168.32.80


elasticsearch-192.168.32.81


elasticsearch-192.168.32.82


redis-192.168.32.67


logstash-192.168.32.76



日志需要传送到logstash 对应的服务器

nginx 配置:

http {
    include       mime.types;
    default_type  application/octet-stream;
    log_format logstash '$http_host $server_addr $remote_addr [$time_local] "$request" '
                    '$request_body $status $body_bytes_sent "$http_referer" "$http_user_agent" '
                    '$request_time $upstream_response_time';

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /usr/local/nginx/logs/access.log  logstash;



logstash 配置:





/*** 写入redis
[elk@zjtest7-frontend config]$ cat logstash_agent.conf 
input {
        file {
                type => "nginx_access"
                path => ["/usr/local/nginx/logs/access.log"]
        }
}
filter {
    grok {
        match => {
            "message" => "%{IPORHOST:clientip} \[%{HTTPDATE:time}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:http_status_code} %{NUMBER:bytes} \"(?

<http_referer>\S+)\" \"(?<http_user_agent>\S+)\" \"(?<http_x_forwarded_for>\S+)\""
        }
    }   
}
output {
        redis {
                host => "192.168.32.67"
                data_type => "list"
                key => "logstash:redis"
                port=>"6379"
                password => "1234567"
        }
}


/***从redis读取,发送到elasticsearch

[elk@zjtest7-frontend config]$ cat logstash_indexer.conf 
input {
        redis {
                host => "192.168.32.67"
                data_type => "list"
                key => "logstash:redis"
                type => "redis-input"
                password => "1234567"
                port =>"6379"
        }
}
output {
        elasticsearch {
                hosts => "192.168.32.80:9200"
                index => "logstash-nginx-%{+YYYY.MM.dd}"
        }
		stdout {
			codec => rubydebug
		}
}




写入到redis的数据:

127.0.0.1:6379> keys *
1) "\xac\xed\x00\x05t\x00\x18contract_rebuild_qty:423"
2) "logstash:redis"
3) "\xac\xed\x00\x05t\x00Dapp_permission_cache:com.zjzc.common.vo.permission.AppPermissionBean"
4) "\xac\xed\x00\x05t\x00\x18contract_rebuild_qty:427"
5) "\xac\xed\x00\x05t\x00\x18contract_rebuild_qty:422"
6) "\xac\xed\x00\x05t\x00!message_left:20160630:18158464881"
7) "\xac\xed\x00\x05t\x00\x18contract_rebuild_qty:417"
127.0.0.1:6379> LLEN "logstash:redis"
(integer) 167