Elasticsearch循序渐进

前言

Elasticsearch是一个基于Lucene的搜索服务器。它提供了一个分布式多用户能力的全文搜索引擎，基于RESTful web接口。Elasticsearch是用Java语言开发的，并作为Apache许可条款下的开放源码发布，是一种流行的企业级搜索引擎。Elasticsearch用于云计算中，能够达到实时搜索，稳定，可靠，快速，安装使用方便

规划

1、主机信息

节点名称	节点IP	软件版本	OS版本
server50	192.168.4.50	elasticsearch-8.5.3、cerebro-0.9.4	coentos 4.19
server51	192.168.4.51	elasticsearch-8.5.3	coentos 4.19
server52	192.168.4.52	elasticsearch-8.5.3	coentos 4.19

2、软件版本

elasticsearch：8.5.3
cerebro：v0.9.4

地址：https://artifacts.elastic.co/downloads
地址：https://github.com/lmenezes/cerebro/releases

部署

1、系统初始化

1、修改/etc/security/limits.conf

vi /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65535
* soft nproc 65535
* hard nproc 65535
* soft memlock unlimited
* hard memlock unlimited

2、修改/etc/systemd/system.conf

vi /etc/systemd/system.conf
DefaultLimitNOFILE=65535
DefaultLimitNPROC=65535

3、修改/etc/sysctl.conf

vi /etc/sysctl.conf 
vm.max_map_count=655360

sysctl -p

4、关闭防火墙和selinux

systemctl stop firewalld
setenforce 0
sed -i 's/=enforcing/=disabled/g' /etc/selinux/config

2、下载安装

useradd  elasticsearch
echo "111111" | passwd elasticsearch --stdin
mkdir /data/app/
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.5.3-linux-x86_64.tar.gz
tar -xvf elasticsearch-8.5.3-linux-x86_64.tar.gz -C /data/app/
chown -R elasticsearch:elasticsearch /data/app/elasticsearch-8.5.3

3、修改配置文件

1、主机50

cd /data/app/elasticsearch-8.5.3/config
cp elasticsearch.yml  elasticsearch.yml.bak
vim elasticsearch.yml
cluster.name: cluster_prd_es
node.name: server50
bootstrap.memory_lock: true
network.host: 192.168.4.50
http.port: 9200
discovery.seed_hosts: ["192.168.4.50:9300", "192.168.4.51:9300", "192.168.4.52"]
cluster.initial_master_nodes: ["server50"]

2、主机51

cd /data/app/elasticsearch-8.5.3/config
cp elasticsearch.yml  elasticsearch.yml.bak
vim elasticsearch.ymlvim elasticsearch.yml
cluster.name: cluster_prd_es
node.name: server51
bootstrap.memory_lock: true
network.host: 192.168.4.51
http.port: 9200
discovery.seed_hosts: ["192.168.4.50:9300", "192.168.4.51:9300", "192.168.4.52"]
cluster.initial_master_nodes: ["server50"]

3、主机52

cd /data/app/elasticsearch-8.5.3/config
cp elasticsearch.yml  elasticsearch.yml.bak
vim elasticsearch.yml
cluster.name: cluster_prd_es
node.name: server52
bootstrap.memory_lock: true
network.host: 192.168.4.52
http.port: 9200
discovery.seed_hosts: ["192.168.4.50:9300", "192.168.4.51:9300", "192.168.4.52"]
cluster.initial_master_nodes: ["server50"]

4、修改jvm

vi jvm.options
-Xms512m
-Xmx512m

4、启动&查看集群状态

su - elasticsearch
cd /data/app/elasticsearch-8.5.3
./bin/elasticsearch -d

查看集群状态


```[root@server50 ~]# curl -XGET "http://192.168.4.50:9200/_cluster/health?pretty"
{
  "cluster_name" : "cluster_prd_es",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

监控

1、部署

tar -xvf cerebro-0.9.4.tgz -C /data/app/
cd /data/app/cerebro-0.9.4
nohup bin/cerebro 2>&1 >/dev/null &

注意：默认cerebro监听的端口是9000，如果出现端口冲突，需要修改cerebro监控的端口。
在启动cerebro的时候可以通过http.port参数指定端口号，如下命令：

bin/cerebro -Dhttp.port=1234

2、访问

3、配置es地址

常见命令

1、集群检查

查询集群状态命令：

curl -XGET "http://ip:port/_cluster/health?pretty"

查询Es全局状态：

curl -XGET "http://ip:port/_cluster/stats?pretty"

查询集群设置

curl -XGET "http://ip:port/_cluster/settings?pretty"

查看集群文档总数

curl -XGET "http://ip:port/_cat/count?v"

查看集群文档总数

curl -XGET "http://ip:port/_cat/count?v"

查看集群别名组

curl -XGET "http://ip:port/_cat/aliases"

6.查看当前集群索引分片信息

curl -XGET "http://ip:port/_cat/shards?v"

注：查看某一个索引可用shards/索引名?v

7.查看集群实例存储详细信息

curl -XGET "http://ip:port/_cat/allocation?v"

8.查看当前集群的所有实例

curl -XGET "http://ip:port/_cat/nodes?v"

9.查看某索引分片转移进度

curl -XGET "http://ip:port/_cat/recovery/索引名?v"

10.查看当前集群等待任务

curl -XGET "http://ip:port/_cat/pending_tasks?v"

11.查看集群写入线程池任务

curl -XGET "http://ip:port/_cat/thread_pool/bulk?v"

12.查看集群查询线程池任务

curl -XGET "http://ip:port/_cat/thread_pool/search?v"

13.查看分片未分配的原因

curl -XGET "http://127.0.0.1:24100/_cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason" | grep UNASSIGNED

2、集群设置

设置集群分片恢复参数

curl -XPUT   "http://ip:httpport/_cluster/settings"  -H  'Content-Type: application/json' -d' { "transient": {    "cluster.routing.allocation.node_initial_primaries_recoveries":60,   "cluster.routing.allocation.node_concurrent_recoveries":30,   "cluster.routing.allocation.cluster_concurrent_rebalance":30   } }'

根据实例名称使EsNodeX实例下线：

curl -XPUT  "http://ip:httpport/_cluster/settings" -H 'Content-Type: application/json' -d' {     "transient": {         "cluster.routing.allocation.exclude._name": "EsNode2@ip"      } }'

根据ip使ES数据节点下线：

curl -XPUT  "http://ip:httpport/_cluster/settings" -H 'Content-Type: application/json' -d' {     "transient": {           "cluster.routing.allocation.exclude._ip": "ip1,ip2,ip3"      } }'

设置分片恢复过程中的最大带宽速度：

curl -XPUT "http://127.0.0.1:24100/_cluster/settings" -H 'Content-Type: application/json' -d'{ "transient":{     "indices.recovery.max_bytes_per_sec":"500mb"  }}'

重新分片为空的主分片

curl -XPOST  "http://127.0.0.1:24100/_cluster/reroute?pretty" -H 'Content-Type:application/json' -d '{   "commands": [{                "allocate_empty_primary": {                                       "index": "indexname",                                       "shard": 2,                                      "node": "EsNode1@81.20.5.24",                                      "accept_data_loss":true                                           }               }]}'

重新分配主分片，会尝试将过期副本分片分片为主。

curl -XPOST "http://127.0.0.1:24100/_cluster/reroute?pretty" -H 'Content-Type:application/json' -d '{   "commands": [{               "allocate_stale_primary": {                                        "index": "index1",                                        "shard": 2,                 "node": "EsNode1@189.39.172.103",                                        "accept_data_loss":true                                          }               }]}'

清理ES所有缓存

curl -XPOST "http://ip:port/_cache/clear"

8.关闭分片自动平衡

curl -XPUT "http://ip:port/_cluster/settings" -H 'Content-Type:application/json' -d '{   "transient":{   "cluster.routing.rebalance.enable":"none" }}'

9.手动刷新未分配的分片

curl -XPOST "http://127.0.0.1:24100/_cluster/reroute?retry_failed=true"

3、索引查看

查询索引mapping和settings

curl -XGET --tlsv1.2  --negotiate -k -u : 'https://ip:port/my_index_name?pretty'

查询索引settings

curl -XGET--tlsv1.2  --negotiate -k -u : 'https://ip:port/my_index_name/_settings?pretty'

3.查看分片未分配详细命令

curl -XGET "http://127.0.0.1:24100/_cluster/allocation/explain?pretty" -H 'Content-Type:application/json' -d '{"index": "indexname","shard": 17,"primary": true}'

4.修改索引只读字段属性为null，放开写入

curl -XPUT  "http://127.0.0.1:24100/*/_settings" -H 'Content-Type: application/json' -d '{"index.blocks.read_only_allow_delete": null}'

4、索引设置

1.关闭索引

curl -XPOST 'http://ip:port/my_index/_close?pretty'

2.打开索引

curl -XPOST 'http://ip:port/my_index/_open?pretty'

3.修改索引刷新时间：

curl -XPUT 'http://ip:port/my_index/_settings?pretty' -H 'Content-Type: application/json' -d'{"refresh_interval" : "60s"}'

4.修改translog文件保留时长，默认为12小时

curl -XPUT 'http://ip:port/my_index/_settings?pretty' -H 'Content-Type: application/json' -d'{"index.translog.retention.age" : "30m"}'

5.设置索引副本：

curl -XPUT 'http://ip:port/my_index/_settings?pretty' -H 'Content-Type: application/json' -d'{"number_of_replicas" : 1}'

6.执行refresh，将内存数据刷新到磁盘缓存

curl -XPOST 'http://ip:port/myindex/_refresh'

7.执行flush，将磁盘缓存刷新到文件系统

curl -XPOST 'https://ip:port/myindex/_flush'

8.执行synced flush，生成syncid

curl -XPOST  'http://ip:port/_flush/synced'

强制执行段合并

curl -XPOST 'http://ip:httpport/myindex/_forcemerge?only_expunge_deletes=false&max_num_segments=1&flush=true&pretty'

10.设置索引在每个esnode上的分片个数

curl -XPUT 'http://ip:httpport/myindex/_settings?pretty' -H 'Content-Type: application/json' -d'{"index.routing.allocation.total_shards_per_node" : "2"}'

配置控制段合并的refresh、merge线程数等

curl -XPUT  "http://ip:port/my_index/_settings?pretty" -H 'Content-Type: application/json' -d'{"refresh_interval": "60s","merge":{"scheduler"：{"max_merge_count" : "100","max_thread_count" : "1"},"policy":{"segments_per_tier" : "100","floor_segment" : "1m","max_merged_segment" : "2g"}}}'

12.设置索引的刷新时间和translog配置参数

注意：设置translog参数，必须先关闭索引，设置完成后再打开

代表设置所有索引，如果要设置具体某个索引，可以将替换为具体的索引名称

curl -XPUT "http://ip:httpport/*/_settings" -H 'Content-Type: application/json' -d'{ "index":{ "refresh_interval" : "60s","translog":{ "flush_threshold_size": "1GB", "sync_interval": "120s", "durability": "async"}}}'

13.限制每个索引在每个实例上的分片个数

curl -XPUT  'http://ip:httpport/myindex/_settings?pretty' -H 'Content-Type:application/json' -d '{"index.routing.allocation.total_shards_per_node":"2"}'

5、实例检查

1.查看实例安装插件

curl -XGET "http://ip:port/_cat/aliases"

2.查询指定ES实例的jvm参数：

curl -XGET 'http://ip:port/_nodes/EsNode1*/stats/jvm?pretty'curl -XGET 'http://ip:port/_nodes/EsNode1@12.40.16.156/stats/jvm?pretty'

6、设置默认分片和副本数

curl -XPUT 'http://127.0.0.1:9200/_template/template_http_request_record' -H 'Content-Type: application/json' -d '{"index_patterns": ["*"],"settings": {"number_of_shards": 1,"number_of_replicas": 0}}'

FAQ

1、Transport SSL must be enabled if security is enabled.

现象：

bootstrap check failure [1] of [1]: Transport SSL must be enabled if security is enabled. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
ERROR: Elasticsearch did not exit normally - check the logs at /data/app/elasticsearch-8.5.3/logs/cluster_prd_es.log
[2022-12-27T15:17:05,020][INFO ][o.e.n.Node               ] [server50] stopping ...
[2022-12-27T15:17:05,134][INFO ][o.e.n.Node               ] [server50] stopped
[2022-12-27T15:17:05,135][INFO ][o.e.n.Node               ] [server50] closing ...
[2022-12-27T15:17:05,160][INFO ][o.e.n.Node               ] [server50] closed
[2022-12-27T15:17:05,166][INFO ][o.e.x.m.p.NativeController] [server50] Native controller process has stopped - no new native processes can be started

ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.

解决：

xpack.security.enabled: false
xpack.security.transport.ssl.enabled: false

posted @ 2022-12-23 17:00 jluo123 阅读(44) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

低调的太阳