- 1. 安装Elasticsearch
1.1 安装及配置Elasticsearch
1.1.1 安装
官方文档:https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
cd /usr/local #进入local目录 mkdir elasticsearch #创建elasticsearch文件夹 cd elasticsearch #进入elasticsearch文件夹 wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.0.rpm #开始下载 rpm -ivh elasticsearch-6.6.0.rpm #开始安装
使用Docker安装ElasticSearch
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html #获取镜像 docker pull docker.elastic.co/elasticsearch/elasticsearch:6.7.0 #运行容器 docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.7.0
1.1.2 配置
whereis elasticsearch #查找安装目录 vi /etc/elasticsearch/elasticsearch.yml #编辑配置文件
主要配置Network.host(本机ip)和http.port(默认9200)(目前单节点模式,其他参数请参考官方文档)
1.1.3 启动服务
firewall-cmd --add-port=9200/tcp --permanent #开启端口9200 firewall-cmd --reload #重新加载配置 systemctl enable elasticsearch #设置服务开机启动 systemctl start elasticsearch #启动服务
在浏览器打开http://192.168.30.128:9200,如下图所示表示启动成功了
1.1.4 卸载Elasticsearch
systemctl stop elasticsearch #停止elasticsearch rpm -e elasticsearch #卸载elasticsearch ps aux | grep elasticsearch #查看gitlab进程
1.1.5 相关问题
l 安装完毕后,elasticSearch无法启动
解决方案:修改jvm配置,减少elasticSearch占用内存
vi /etc/elasticsearch/jvm.options #修改为:-Xms256m -Xmx256m
#保存后,重启elasticsearch服务
l elasticSearch启动报错:elasticsearch: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
解决方案:修改jvm配置(/etc/elasticsearch/jvm.options)
vi /etc/elasticsearch/jvm.options #文件中加入:-XX:-AssumeMP
l 其他
1.2 安装及配置logstash
1.2.1 安装
官方文档:https://www.elastic.co/guide/en/logstash/current/index.html
cd /usr/local/elasticsearch #进入elasticsearch目录 wget https://artifacts.elastic.co/downloads/logstash/logstash-6.6.0.rpm rpm -ivh logstash-6.6.0.rpm #安装rpm包
使用Docker安装logstash
https://www.elastic.co/guide/en/logstash/current/docker.html
docker pull docker.elastic.co/logstash/logstash:6.7.0
1.2.2 配置
vi /etc/logstash/conf.d/nlog.conf #新增配置信息
input:采用TCP监控本机8001端口的消息
filter:使用grok 插件,自定义消息格式,推荐使用grokdebug在线进行调试
output:使用elasticsearch作为数据存储
1.2.3 启动服务
firewall-cmd --add-port=8001/tcp --permanent #开启端口8001 firewall-cmd --reload #重载配置 systemctl enable logstash #设置开机启动 systemctl start logstash #启动logstash
1.2.4 相关问题
l 设置开机启动时报:Failed to start logstash.service: Unit not found
vi /etc/logstash/startup.options #修改启动配置,如下图
cd /usr/share/logstash/bin /usr/share/logstash/bin/system-install /etc/logstash/startup.options
https://www.centos.org/forums/viewtopic.php?t=67591
https://discuss.elastic.co/t/logstash-service-unit-not-found-centos-7/138446
https://stackoverflow.com/questions/41986441/unable-to-start-logstash-service-on-centos7
l 其他
1.3 安装及配置Kibana
1.3.1 安装Kibana
cd /usr/local/elasticsearch #进入elasticsearch目录 wget https://artifacts.elastic.co/downloads/kibana/kibana-6.6.0-x86_64.rpm rpm -ivh kibana-6.6.0-x86_64.rpm #安装Kibana
1.3.2 配置Kibana
cd /etc/kibana #进入安装目录 vi kibana.yml #编辑配置文件
设置端口号:5601,Host地址:"192.168.30.128" ,elasticsearch服务地址为:http://192.168.30.128:9200
1.3.3 启动服务
firewall-cmd --add-port=5601/tcp --permanent #开启端口5601 firewall-cmd --reload #重新加载配置 systemctl enable kibana #设置服务开机启动 systemctl start kibana #启动服务
在浏览器打开http://192.168.30.128:5601,将进入到Kibana管理界面
1.4 Nginx使用logstash输出日志
1.4.1 Nginx设置日志格式
vi /usr/local/nginx/conf/nginx.conf
#在http模块中添加
log_format json '{"@timestamp":"$time_iso8601",' '"@version":"1",' '"client":"$remote_addr",' '"url":"$uri",' '"status":"$status",' '"domain":"$host",' '"host":"$server_addr",' '"size":$body_bytes_sent,' '"responsetime":$request_time,' '"referer": "$http_referer",' '"ua": "$http_user_agent"' '}';
#在server模块中添加
access_log /var/log/nginx/access.log json;
修改后的Nignx.conf文件
1.4.2 使用logstash
1.4.2.1 配置logstash
vi /etc/logstash/conf.d/nginx_log.conf
配置内容如下:
input { file { path => "/var/log/elasticsearch/elasticsearch.log" type => "es-error" start_position => "beginning" codec => multiline { pattern => "^\[" negate => true what => "previous" } } file { path => "/usr/local/nginx/logs/access.log" codec => json start_position => "beginning" type => "nginx-log" } } output { if [type] == "es-error"{ elasticsearch { hosts => ["192.168.56.99:9200"] index => "es-error-%{+YYYY.MM.dd}" } } if [type] == "nginx-log"{ elasticsearch { hosts => ["192.168.56.99:9200"] index => "nginx-log-%{+YYYY.MM.dd}" } } }
1.4.2.2 检测logstash配置是否正确
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx_log.conf --configtest
1.4.2.3 使用kibana查看日志
l 使用浏览器访问kibana:http://192.168.56.99:5601
l 选择management/Index Patterns创建nginx索引
l 单击“Discover”查看日志