解决ElasticSearch的maximum shards open问题

背景：

今天，开发反馈开发环境的日志没有了。

问题现象：

kibana无最新日志。

查看elk各个容器，发现容器启动正常
64e4553efa33   kibana:7.6.0            "/usr/local/bin/dumb…"   2 months ago   Up 3 days       0.0.0.0:5601->5601/tcp, :::5601->5601/tcp                                                  kibana
9ce332c49392   logstash:7.6.0          "/usr/local/bin/dock…"   2 months ago   Up 23 minutes   0.0.0.0:5044->5044/tcp, :::5044->5044/tcp, 9600/tcp                                        logstash
0c138a5cc89b   elasticsearch:7.6.0     "/usr/local/bin/dock…"   2 months ago   Up 19 minutes   0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp                                        elasticsearch
 
 
查看filebeat服务，发现服务也正常
[root@localhost logs]$ systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
   Active: active (running) since 四 2025-02-13 13:53:28 CST; 4s ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 23647 (filebeat)
    Tasks: 19
   CGroup: /system.slice/filebeat.service
           └─23647 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat.

排查日志发现，logstash的日志报错
[2025-02-05T01:19:12,745][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"服务名_2025.02.05", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x628bd73a>], :response=>{"index"=>{"_index"=>"服务名_2025.02.05", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}

原因：

这是因为集群最大shard(分片)数不足引起的，从Elasticsearch v7.0 开始，集群中的每个节点默认限制1000个分片。

解决思路：

在服务器执行

curl -XPUT http://localhost:9200/_cluster/settings \
-u elastic:password \
-H "Content-Type: application/json" \
-d '{"transient":{"cluster":{"max_shards_per_node":10000}}}' 

返回{"acknowledged":true,"persistent":{},"transient":{"cluster":{"max_shards_per_node":"10000"}}}表示执行成功！

参考笔记：https://blog.51cto.com/u_11954248/2455379