ELK01-Elasticsearch
1.Elastic search（ES）介绍
到底什么是Elasticsearch呢？官方网站给的解释如下：
What is Elasticsearch?
You know, for search (and analysis)
Elasticsearch is the distributed search and analytics engine at the heart of the Elastic Stack. Logstash and Beats facilitate collecting, aggregating, and enriching your data and storing it in Elasticsearch. Kibana enables you to interactively explore, visualize, and share insights into your data and manage and monitor the stack. Elasticsearch is where the indexing, search, and analysis magic happens.
Elasticsearch provides near real-time search and analytics for all types of data. Whether you have structured or unstructured text, numerical data, or geospatial data, Elasticsearch can efficiently store and index it in a way that supports fast searches. You can go far beyond simple data retrieval and aggregate information to discover trends and patterns in your data. And as your data and query volume grows, the distributed nature of Elasticsearch enables your deployment to grow seamlessly right along with it.
百度百科对于Elastic search的定义和解释：Elasticsearch 是一个分布式、高扩展、高实时的搜索与数据分析引擎。它能很方便的使大量数据具有搜索、分析和探索的能力。充分利用Elasticsearch的水平伸缩性，能使数据在生产环境变得更有价值。Elasticsearch 的实现原理主要分为以下几个步骤，首先用户将数据提交到Elasticsearch 数据库中，再通过分词控制器去将对应的语句分词，将其权重和分词结果一并存入数据，当用户搜索数据时候，再根据权重将结果排名，打分，再将返回结果呈现给用户。
2.Elasticsearch（ES）有关概念理解
Elasticsearch功能：
a.分布式搜索和分析引擎;
b.全文检索，结构化检索，数据分析;
c.对海量数据进行实时处理。
Elasticsearch常见使用场景：
a.搜索，高亮显示
b.商城搜索
c.日志收集分析展示
Elasticsearch和mysql数据库名词术语概念一一对应：
elasticsearch 数据库
Document 行
Type 表
Index 库
filed 字段
3.Elasticsearch（ES）安装和配置
Elasticsearch 安装部署方式：
a.docker方式；
b.tar包安装;
c.ansible;
d.rpm/deb。
本人以rpm安装方式安装elasticsearch进行研究！
RPM安装Elasticsearch主要步骤：
1）安装jdk
yum search java | grep -i --color JDK
yum install -y java-1.8.0-openjdk.x86_64
2）下载安装elasticsearch
mkdir -p /data/es_soft/
cd /data/es_soft/
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.0.rpm
rpm -ivh elasticsearch-6.6.0.rpm
3）配置启动
[root@localhost es_soft]# systemctl daemon-reload
[root@localhost es_soft]# systemctl enable elasticsearch.service
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
[root@localhost es_soft]# systemctl start elasticsearch.service
[root@localhost es_soft]# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since 二 2022-01-11 11:03:20 CST; 36s ago
Docs: http://www.elastic.co
Main PID: 25175 (java)
CGroup: /system.slice/elasticsearch.service
├─25175 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch...
└─25251 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
1 月 11 11:03:20 localhost.localdomain systemd[1]: Started Elasticsearch.
4）检查elasticsearch服务是否启动成功
[root@localhost es_soft]# ps -ef|grep elastic
elastic+ 25175 1 6 11:03 ? 00:00:17 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-3742633244208252722 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
elastic+ 25251 25175 0 11:03 ? 00:00:00 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
root 30177 1528 0 11:07 pts/0 00:00:00 grep --color=auto elastic
[root@localhost es_soft]# lsof -i:9200
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 25175 elasticsearch 204u IPv6 124374 0t0 TCP localhost:wap-wsp (LISTEN)
java 25175 elasticsearch 205u IPv6 124375 0t0 TCP localhost:wap-wsp (LISTEN)
[root@localhost es_soft]# netstat -lntup|grep 9200
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 25175/java
tcp6 0 0 ::1:9200 :::* LISTEN 25175/java
[root@localhost es_soft]# curl 127.0.0.1:9200
{
"name" : "Plo4-LY",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "qpS8OUS5ThSzrzPNZ4dVXQ",
"version" : {
"number" : "6.6.0",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "a9861f4",
"build_date" : "2019-01-24T11:27:09.439740Z",
"build_snapshot" : false,
"lucene_version" : "7.6.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
注意：curl命令出现如上信息，说明elasticsearch启动成功！
elasticsearch输出日志如下：
[root@localhost es_soft]# tail -f /var/log/elasticsearch/elasticsearch.log
[2022-01-11T11:03:31,931][INFO ][o.e.g.GatewayService ] [Plo4-LY] recovered [0] indices into cluster_state
[2022-01-11T11:03:32,111][INFO ][o.e.c.m.MetaDataIndexTemplateService] [Plo4-LY] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2022-01-11T11:03:32,175][INFO ][o.e.c.m.MetaDataIndexTemplateService] [Plo4-LY] adding template [.watches] for index patterns [.watches*]
[2022-01-11T11:03:32,208][INFO ][o.e.c.m.MetaDataIndexTemplateService] [Plo4-LY] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[2022-01-11T11:03:32,230][INFO ][o.e.c.m.MetaDataIndexTemplateService] [Plo4-LY] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
[2022-01-11T11:03:32,297][INFO ][o.e.c.m.MetaDataIndexTemplateService] [Plo4-LY] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
[2022-01-11T11:03:32,326][INFO ][o.e.c.m.MetaDataIndexTemplateService] [Plo4-LY] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
[2022-01-11T11:03:32,351][INFO ][o.e.c.m.MetaDataIndexTemplateService] [Plo4-LY] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
[2022-01-11T11:03:32,374][INFO ][o.e.c.m.MetaDataIndexTemplateService] [Plo4-LY] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
[2022-01-11T11:03:32,464][INFO ][o.e.l.LicenseService ] [Plo4-LY] license [cf912932-b640-402a-b7fa-d80ad6aeac0d] mode [basic] - valid
5）查看elasticsearch有哪些配置文件：
[root@localhost ~]# rpm -qc elasticsearch
/etc/elasticsearch/elasticsearch.yml ES主配置文件
/etc/elasticsearch/jvm.options JVM虚拟机配置
/etc/elasticsearch/log4j2.properties
/etc/elasticsearch/role_mapping.yml
/etc/elasticsearch/roles.yml
/etc/elasticsearch/users
/etc/elasticsearch/users_roles
/etc/init.d/elasticsearch init的启动文件
/etc/sysconfig/elasticsearch 环境变量相关参数
/usr/lib/sysctl.d/elasticsearch.conf JVM相关配置
/usr/lib/systemd/system/elasticsearch.service systemc启动配置
4.ES-head插件
1）elasticsearch 支持的常用查询指令
[root@localhost ~]# curl 10.96.211.105:9200/_cat
=^.^=
/_cat/allocation
/_cat/shards
/_cat/shards/{index}
/_cat/master
/_cat/nodes
/_cat/tasks
/_cat/indices
/_cat/indices/{index}
/_cat/segments
/_cat/segments/{index}
/_cat/count
/_cat/count/{index}
/_cat/recovery
/_cat/recovery/{index}
/_cat/health
/_cat/pending_tasks
/_cat/aliases
/_cat/aliases/{alias}
/_cat/thread_pool
/_cat/thread_pool/{thread_pools}
/_cat/plugins
/_cat/fielddata
/_cat/fielddata/{fields}
/_cat/nodeattrs
/_cat/repositories
/_cat/snapshots/{repository}
/_cat/templates
[root@localhost ~]# curl 10.96.211.105:9200/_cat/nodes
10.96.211.105 9 80 2 0.09 0.07 0.08 mdi * node-1
2) elasticsearch交互方式
elasticsearch有三种交互方式如下：
a.curl命令
b.es-head插件
c.kibana
3) es-head插件安装
此插件安装赞略，不是研究重点，知道即可。
5.ES集群
ES集群就是多台主机在同一个组里。
注意：默认所有节点都是工作节点，主节点既负责调度，又是工作节点！
1）ES配置部署集群
ES配置部署集群，我以3台主机为例进行研究。
---------------------------------------------------------------------------------------------------------------------
集群中第一台主机主配置文件内容如下：
[root@db01 elasticsearch]# grep "^[a-z]" elasticsearch.yml
cluster.name: Linux #集群名称，同一个集群内所有节点集群名称要一模一样
node.name: node-1 #节点名称，同一个集群内所有节点的节点名称不能重复
path.data: /data/elasticsearch #数据目录
path.logs: /var/log/elasticsearch #日志目录
bootstrap.memory_lock: true #内存锁定
network.host: 10.96.211.209,127.0.0.1 #绑定监听地址
http.port: 9200 #默认端口号
discovery.zen.ping.unicast.hosts: ["10.96.211.209", "10.96.211.110"] #集群发现节点配置
discovery.zen.minimum_master_nodes: 2 #选项相关参数,有公式 master/2 +1
新增节点配置步骤:集群中第二台主机主配置文件内容如下：
1.安装软件
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.0.rpm
rpm -ivh elasticsearch-6.6.0.rpm
2.修改配置文件
[root@db02 elasticsearch]# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: Linux
node.name: node-2
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 10.96.211.110,127.0.0.1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.96.211.209", "10.96.211.110"]
discovery.zen.minimum_master_nodes: 2
3.修改内存锁定
[root@db02 ~]# systemctl edit elasticsearch
[Service]
LimitMEMLOCK=infinity
4.创建数据目录并授权
mkidr /data/elasticsearch
chown =R elasticsearch:elasticsearch /data/elasticsearch
5.重启服务
systemctl daemon-reload
systemctl start elasticsearch
6.查看日志和端口
tail -f /var/log/elasticsearch/Linux.log
netstat -lntup:grep 9200
检查有关ES集群状态的一些操作指令:
curl -XGET 'http://localhost:9200/_nodes/procese?human&pretty'
curl -XGET 'http://localhost:9200/_nodes/_all/info/jvm,process?human&pretty'
curl -XGET 'http://localhost:9200/_cat/nodes?human&pretty'
curl -XGET 'http://localhost:9200/_cluster/health?pretty'
curl -XPUT 'localhost:9200/_cat/indices?pretty'
新增节点配置步骤:集群中第三台主机主配置文件内容如下：
1.安装软件
[root@localhost ~]# yum search java | grep -i --color JDK
[root@localhost ~]# yum install -y java-1.8.0-openjdk.x86_64
[root@localhost ~]# mkdir -p /data/es_soft/
[root@localhost ~]# cd /data/es_soft/
[root@localhost es_soft]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.0.rpm
rpm -ivh elasticsearch-6.6.0.rpm
2.修改配置文件
[root@db02 elasticsearch]# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: Linux
node.name: node-3
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 10.96.211.111,127.0.0.1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.96.211.209", "10.96.211.111"]
discovery.zen.minimum_master_nodes: 2
[root@localhost es_soft]# grep "^-Xm" /etc/elasticsearch/jvm.options
-Xms2g
-Xmx2g
3.修改内存锁定
[root@db02 ~]# systemctl edit elasticsearch
[Service]
LimitMEMLOCK=infinity
4.创建数据目录并授权
mkdir /data/elasticsearch
chown -R elasticsearch:elasticsearch /data/elasticsearch
5.重启服务
systemctl daemon-reload
systemctl start elasticsearch
6.查看日志和端口
tail -f /var/log/elasticsearch/Linux.log
netstat -lntup:grep 9200
检查有关ES集群状态的一些操作指令:
curl -XGET 'http://localhost:9200/_nodes/procese?human&pretty'
curl -XGET 'http://localhost:9200/_nodes/_all/info/jvm,process?human&pretty'
curl -XGET 'http://localhost:9200/_cat/nodes?human&pretty'
curl -XGET 'http://localhost:9200/_cluster/health?pretty'
curl -XPUT 'localhost:9200/_cat/indices?pretty'
---------------------------------------------------------------------------------------------------------------------
2）ES配置部署集群常见问题和处理
a.防火墙没有关闭，导致配置集群失败。处理：关闭防火墙或者开启有关端口策略。
b.没有配置内存锁定。处理：配置内存锁定：即[root@db02 ~]# systemctl edit elasticsearch
[Service]
LimitMEMLOCK=infinity
6.ES集群故障和解决
ES故障1：
2个节点，master设置为2的时候，一台出现故障会导致ES集群不可用，具体解决方法如下：
解决方式：把还存活的节点的配置文件集群选举相关的选项注释掉或者改成1,然后重启elasticsearch服务就可以解决。即
文件： /etc/elasticsearch/elasticsearch.yml
修改选项：discovery.zen.minimum_master_nodes: 1或者#discovery.zen.minimum_master_nodes
重启elasticsearch服务：systemctl restart elasticsearch
注意:
两个节点数据不一致会导致查询结果不一致；
找出不一致的数据,清空一个节点,以另一个节点的数据为准；
然后手动插入修改后的数据。
7.ES集群分片与副本
8.ES相关工具以及ES有关review
1)ES有关review
默认设置
es默认设置：
5分片
1副本 在其他机器上备份
应用场景
应用场景：
a.搜索，高亮显示
b.商城搜索
c.日志收集分析展示
集群状态颜色
集群状态颜色：
绿色:所有条件都满足，数据完整，副本满足
黄色:数据完整，副本不满足
红色:有索引里的数据出现不完整了
紫色:有分片正在同步中
默认自己就是一个集群，默认的集群名称为:elasticsearch
集群安装需要注意
安装注意的内容：
a.锁定内存要修改配置
b.JVM虚拟机最大最小内存设置为一样
c.最大内存不要超过30G
d.更改数据目录需要授权用户给elasticsearch
e.es启动比较慢
数据操作
数据操作：
增删改查
a.插入数据不需要提前创建好数据库
b.index -- 库
type -- 表
filter -- 字段
c.默认随机生成_ID -- 唯一键ID
交互方式
交互方式：
a.curl命令
b.es-head插件
c.kibana
2)ES有关工具
9.ES集群监控
监控ES指标
1）监控集群状态
2）监控集群节点数
对于ES集群监控来说，以上二者缺一不可，集群状态正常不代表集群节点都是OK！
监控ES集群报警条件：集群状态不是绿色（OK）或者集群节点不等于所有节点数就触发报警！！！
查询ES集群节点数：
[root@localhost elasticsearch]# curl -s -XGET 'http://localhost:9200/_cat/nodes?human&pretty' |wc -l
3
