ELK7.6+Filebeat集群部署
| 软件名称 | 版本 | 操作系统 | 内核版本 |
|---|---|---|---|
| Elasticsearch | 7.6.2 | CentOS 7.5.1804 | 3.10.0-862.el7 |
| Logstach | 7.6.2 | CentOS 7.5.1804 | 3.10.0-862.el7 |
| Kibana | 7.6.2 | CentOS 7.5.1804 | 3.10.0-862.el7 |
| Filebeat | 7.6.2 | CentOS 7.5.1804 | 3.10.0-862.el7 |
| JDK | 11.0.7 | CentOS 7.5.1804 | 3.10.0-862.el7 |
| kafka/zookeeper | 2.12-2.3.1 | CentOS 7.5.1804 | 3.10.0-862.el7 |
二、安装JVM(所有服务器上)
tar xf jdk-11.0.7_linux-x64_bin.tar.gz -C /usr/local/ vim /etc/profile.d/java.sh #设置环境变量，如果服务器上当前已经存在JVM环境变量请删除 export JAVA_HOME=/usr/local/jdk-11.0.7/ export PATH=$PATH:$JAVA_HOME/bin export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar source /etc/profile.d/java.sh java -version #检查 java version "11.0.7" 2020-04-14 LTS Java(TM) SE Runtime Environment 18.9 (build 11.0.7+8-LTS) Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.7+8-LTS, mixed mode)
三、ES集群安装配置(ES集群上)
tar xf elasticsearch-7.6.2-linux-x86_64.tar.gz -C /usr/local/ cd /usr/local/ ln -sv elasticsearch-7.6.2/ elasticsearch cd elasticsearch/config/ grep "^[a-Z]" /usr/local/elasticsearch/config/elasticsearch.yml #修改ES配置如下 cluster.name: pwb-elk-cluster #集群名称，所有机器相同 node.name: node-2 #当前服务器的node名称，集群中保持唯一 path.data: /Data/es/data path.logs: /Data/es/log bootstrap.memory_lock: true network.host: 172.16.150.158 #当前主机IP地址 http.port: 9200 discovery.seed_hosts: ["172.16.150.157", "172.16.150.158","172.16.150.159"] #集群主机IP cluster.initial_master_nodes: ["172.16.150.157", "172.16.150.158","172.16.150.159"] #集群中首次启动时可被选举为master的节点 discovery.zen.minimum_master_nodes: 2 #最少有两个节点存活才可以选举master gateway.recover_after_nodes: 2 #最少两个节点存活在开始数据存活
network.host: #本机IP地址
node.name: #分配的节点名称
mkdir -pv /Data/es/ useradd elastic chown -R elastic:elastic /Data/es/ chown -R elastic:elastic /usr/local/elasticsearch-7.6.2/
tail /etc/security/limits.conf #新增或修改以下选项 * soft nofile 65536 * hard nofile 131072 * soft nproc 2048 * hard nproc 4096 * soft memlock unlimited * hard memlock unlimited echo "vm.max_map_count=262144 " >> /etc/sysctl.conf sysctl -p reboot
su - elastic cd /usr/local/elasticsearch nohup ./bin/elasticsearch > /tmp/elastic.log & tailf /tmp/elastic.log
master node changed {previous [], current [{node-2}{TA9XcpyMS8yH1YIkq7fN-Q}{FPgTcZnNRgSiKnHfrjsd-A}{172.16.150.158}{172.16.150.158:9300}
netstat -tnlp|grep -E "9200|9300"curl http://172.16.150.159:9200/ #任意节点IP地址{ "name" : "node-3", "cluster_name" : "pwb-elk-cluster", "cluster_uuid" : "mSE1bV1UTh-p1VSPLLQLLQ", "version" : { "number" : "7.6.2", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f", "build_date" : "2020-03-26T06:34:37.794943Z", "build_snapshot" : false, "lucene_version" : "8.4.0", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search"}
四、安装Kibana
1.安装配置Kibana(kibana服务器上)
tar xf kibana-7.6.2-linux-x86_64.tar.gz -C /usr/local/ cd /usr/local/ ln -sv kibana-7.6.2-linux-x86_64/ kibana cd kibana/config grep "^[a-Z]" /usr/local/kibana/config/kibana.yml server.port: 5601 #服务器端口，默认5601 必须 server.host: "172.16.150.159" #主机IP地址 必须 elasticsearch.hosts: ["http://172.16.150.157:9200"] #ES地址 必须 i18n.locale: "zh-CN" #7版本支持中文，按需配置
nohup ./kibana --allow-root > /tmp/kibana.log & tailf /tmp/kibana.log #确保出现一下信息 "tags":["listening","info"],"pid":13922,"message":"Server running at http://172.16.150.159:5601"}
web界面打开http://172.16.150.159:5601连接
五、安装ZK/kafka(zk/kafka集群)
tar xf kafka_2.12-2.3.1.tgz -C /usr/local/ cd /usr/local/ ln -sv kafka_2.12-2.3.1 kafka cd kafka/config/ grep "^[a-Z]" /usr/local/kafka/config/zookeeper.properties dataDir=/Data/zookeeper clientPort=2181 maxClientCnxns=0 tickTime=2000 initLimit=20 syncLimit=10 server.1=172.16.150.164:2888:3888 server.2=172.16.150.165:2888:3888 server.3=172.16.150.166:2888:3888 mkdir -pv /Data/zookeeper #创建日志及快照目录 echo "1" > /Data/zookeeper/myid #创建myid文件
grep "^[a-Z]" /usr/local/kafka/config/server.properties broker.id=1 listeners=PLAINTEXT://172.16.150.164:9092 #服务器IP地址和端口 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/Data/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=172.16.150.164:2181,172.16.150.166:2181,172.16.150.166:2181 #zookeeper服务器IP和端口 zookeeper.connection.timeout.ms=20000 group.initial.rebalance.delay.ms=0
其他节点配置相同，除以下几点：
（1）zookeeper的配置 echo "x" > /Data/zookeeper/myid #唯一 （2）kafka的配置 broker.id=1 #唯一 host.name=本机IP
nohup /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties & netstat -nlpt | grep -E "2181|2888|3888" #哪台是leader，那么他就拥有2888端口
vim /etc.hosts #编辑hosts文件，添加127.0.0.1 对当前主机名称的解析 /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &
/usr/local/kafka/bin/kafka-topics.sh --create --zookeeper 172.16.150.164:2181 --replication-factor 2 --partitions 1 --topic summer #创建一个测试 topic /usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 172.16.150.164:2181 #查看创建的topic /usr/local/kafka/bin/kafka-topics.sh --describe --zookeeper 172.16.150.164:2181 --topic summer #查看topic的详情 /bin/bash /usr/local/kafka/bin/kafka-console-producer.sh --broker-list 172.16.150.164:9092 --topic summer #模拟生产者往 summertopic发送消息 #另起一个页面 /usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server 172.16.150.165:9092 --topic summer --from-beginning #另起一个页面，查看是否可以读取summertopic消息
六、安装配置filebeat(日志客户端)
tar xf filebeat-7.6.2-linux-x86_64.tar.gz -C /usr/local/ cd /usr/local/filebeat-7.6.2-linux-x86_64/ vim filebeat.yml 15 filebeat.inputs: 16 17 # Each - is an input. Most options can be set at the input level, so 18 # you can use different inputs for various configurations. 19 # Below are the input specific configurations. 20 21 - type: log #日志类型 22 23 # Change to true to enable this input configuration. 24 enabled: true 25 json.keys_under_root: true #可以让字段位于根节点 26 json.overwrite_keys: true #对于同名的key，覆盖原有key值 27 fields_under_root: true #可以让字段位于根节点 28 29 # Paths that should be crawled and fetched. Glob based paths. 30 paths: 31 - /opt/logs/nginx/access.log #日志文件路径 32 # document_type: dev-nginx-access 33 fields: 34 type: log 35 log_topic: dev-nginx-access #指定日志topic名称 96 name: dev-nginx-150-153 229 output.kafka: 230 # Boolean flag to enable or disable the output module. 231 enabled: true 232 233 # The list of Kafka broker addresses from which to fetch the cluster metadata. 234 # The cluster metadata contain the actual Kafka brokers events are published 235 # to. 236 hosts: ["172.16.150.164:9092","172.16.150.165:9092","172.16.150.166:9092"] #kafka集群地址 237 238 # The Kafka topic used for produced events. The setting can be a format string 239 # using any event field. To set the topic from document type use `%{[type]}`. 240 topic: '%{[log_topic]}' #fileds.log_topic 定义的值
Nginx安装步骤略vim nginx.conf #修改Nginx配置文件添加以下内容 log_format json '{"@timestamp":"$time_iso8601",' '"@version":"1",' '"client_ip":"$remote_addr",' '"status":"$status",' '"host":"$server_addr",' '"url":"$request_uri",' '"domain":"$host",' '"size":"$body_bytes_sent",' '"responsetime":"$request_time",' '"referer":"$scheme://$server_addr$request_uri",' '"user_agent":"$http_user_agent"' '}'; access_log /opt/logs/nginx/access.log json; /usr/local/nginx/sbin/nginx -t -c /usr/local/nginx/conf/nginx.conf/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf #启动Nginx
nohup /usr/local/filebeat/filebeat -e -c /usr/local/filebeat/filebeat.yml > /tmp/filebeat.log &
vim /etc/systemd/system/filebeat.service [Unit] Description=filebeat server daemon Documentation=/usrl/local/filebeat/filebeat -help Wants=network-online.target After=network-online.target [Service] User=root Group=root ExecStart=/usr/local/filebeat/filebeat -c /usr/local/filebeat/filebeat.yml --path.logs /usr/local/filebeat/logs Restart=always [Install] WantedBy=multi-user.target systemctl restart filebeat.service
/usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 172.16.150.164:2181
tar xf logstash-7.6.2.tar.gz -C /usr/local/ cd /usr/local/logstash-7.6.2/config/ vim messages.conf input { kafka { bootstrap_servers => "172.16.150.164:9092,172.16.150.165:9092,172.16.150.166:9092" #kafka集群地址 topics => "dev-nginx-access" #接受topic的名称 codec => "json" #解析格式 consumer_threads => 5 #最大线程 decorate_events => true #将当前topic、offset、group、partition等信息也带到message中 } } output { elasticsearch { hosts => ["172.16.150.157:9200","172.16.150.158:9200"] #ES集群信息 index => "dev-nginx-access-%{+YYYY-MM-dd}" #索引格式建议按天切割 } } ../bin/logstash -f messages.conf -t --verbose nohup /usr/local/logstash-7.6.2/bin/logstash -f messages.conf > /tmp/logstch.log &
验证：
"一劳永逸" 的话，有是有的，而 "一劳永逸" 的事却极少
