es集群无缝迁移扩容

新节点加入老集群

cluster.name: soc-cluster
node.name: node-2test
path.data: /home/secure/elastic/data
path.logs: /home/secure/elastic/logs
network.host: 0.0.0.0
cluster.initial_master_nodes: ["node-2"]
discovery.seed_hosts: ["10.32.3.2:9300","10.32.3.18:9300"]
http.port: 9201
transport.tcp.port: 9301
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true

xpack.license.self_generated.type: basic
xpack.security.http.ssl.key: certs/instance/instance.key
xpack.security.http.ssl.certificate: certs/instance/instance.crt
xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt

xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.key: certs/instance/instance.key
xpack.security.transport.ssl.certificate: certs/instance/instance.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt

xpack.security.http.ssl.supported_protocols: TLSv1.2
xpack.security.transport.ssl.supported_protocols: TLSv1.2

cluster.name: soc-cluster
node.name: node-2
cluster.initial_master_nodes: ["node-2"]
path.data: /data/soc/elastic/data
path.logs: /data/soc/elastic/logs
network.host: 0.0.0.0
discovery.seed_hosts: ["10.32.3.2", "10.32.3.18", "10.32.3.20"]
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true

xpack.license.self_generated.type: basic
xpack.security.http.ssl.key: certs/instance/instance.key
xpack.security.http.ssl.certificate: certs/instance/instance.crt
xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt

xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.key: certs/instance/instance.key
xpack.security.transport.ssl.certificate: certs/instance/instance.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt

xpack.security.http.ssl.supported_protocols: TLSv1.2
xpack.security.transport.ssl.supported_protocols: TLSv1.2

     

     

数据迁移

      1.旧集群添加新节点后自动会触发分片均衡策略

        

         

          

       2.锁定分片只能分配到新节点上

          集群中每个节点都能尽量分配到相等的分片数量

         

           手动干预集群的分片策略

           

       3.手动驱逐旧节点上的分片到新节点上

curl -u elastic:11111 --insecure https://10.32.3.2:9200/_cat/nodes

transient   //暂时配置 重启后失效
persistent  //永久配置 重启后仍然有效

1.源集群操作
  触发集群分片均衡策略,把集群老节点上的分片全部移动到新的集群节点上
 curl -u elastic:1111 --insecure -H "Content-Type: application/json" -XPUT https://10.32.3.2:9200/_cluster/settings -d '{
  "transient" : {
     "cluster.routing.allocation.include._name" : null, //设置集群分片可以包含的节点名称
     "cluster.routing.allocation.exclude._name" : "node-2,node-18,node-20,node-4"  //设置集群分片排除的节点名称
  }
}'


在分片迁移的中间状态由于分片迁移会消耗一定的性能。因此为了稳定性考虑,可以通过如下API将迁移的并发度和迁移速度调小一点。
curl -u elastic:11111 --insecure -H "Content-Type: application/json" -XPUT https://10.32.3.2:9200/_cluster/settings -d '{ "persistent": {
            "cluster.routing.allocation.node_concurrent_recoveries": 2,
            "indices.recovery.max_bytes_per_sec": "100mb"  //设置节点的分片传输速率
 }
}'


2.集群分离
  停止集群中老节点的es服务进程
  老节点下线需要滚动进行,且每个节点的下线间隔需要大于10s,否则会有失主的风险
  原来的主节点需要最后下线
  

4.还原es集群的配置
 curl -u elastic:11111 --insecure -H "Content-Type: application/json" -XPUT https://10.32.3.2:9200/_cluster/settings -d '{
  "transient" : {
     "cluster.routing.allocation.include._name" : null,  //还原集群的默认配置
     "cluster.routing.allocation.exclude._name" : null   //还原集群的默认配置
  }
}'  

          1.把老节点上的分片迁移到新节点上

            

         

       2.新创建的分片不要再路由到老节点上

         

          等待一段时间后再观察

            

               

老节点下线

        

        

        把所有老节点下线后，集群状态正常,原来索引中的数据也可以正常查询.整个集群的数据迁移完成

elasticdump迁移数据   

      npm install -g cnpm --registry=https://registry.npm.taobao.org
      npm install elasticdump -g

      NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump --input=https://elastic:'123@456'@10.246.16661.6:9200/prob_file_description_202401 --input-cert=/home/secure/canew.crt --limit=80000  --output-cert=/home/secure/ca.crt --output=https://elastic:'aaaa@2024'@10.30.90.197:9200/full_flow_file_202401