elasticsearch备份和恢复
#elastic备份和恢复
0.备份准备
（1）.首先查看es的用户，可以在/etc/passwd中查看，一般是elasticsearch用户
（2）.然后修改脚本中给备份目录创建权限的chown为elasticsearch
（3）.给脚本curl命令添加对应的账号密码 -u 用户：密码
（4）.修改es端口为29200，改为自己环境对饮过的
（5）.然后到es部署目录中执行脚本，/usr/share/elasticsearch目录
1.部署es
（1）创建数据目录
mkdir /app/shell/es_bak/elastic/data
chmod +777 /app/shell/es_bak/elastic/data
（2）启动es服务
docker run -dit --restart=always --memory=2G --network host --name es -p 29200:29200 -p 9300:9300 -v /etc/localtime:/etc/localtime:ro -v /app/shell/es_bak/elastic/data:/usr/share/elasticsearch/data -v /app/shell/es_bak/elastic/config:/usr/share/elasticsearch/config -v /app/shell/es_bak/elastic/bak:/home/elastic/bak 192.168.176.230:8090/public/elastic:7.1.1
（3）es设置密码
[root@devops elastic]# cat espasswd.sh #!/bin/bash #设置elastic密码 /usr/bin/expect <<-EOF spawn docker exec -it es /bin/bash -c "elasticsearch-setup-passwords interactive" expect "y/N" send "y\n" expect "elastic" send "ytx@1234\n" expect "elastic" send "ytx@1234\n" expect "apm_system" send "ytx@1234\n" expect "apm_system" send "ytx@1234\n" expect "kibana" send "ytx@1234\n" expect "kibana" send "ytx@1234\n" expect "logstash_system" send "ytx@1234\n" expect "logstash_system" send "ytx@1234\n" expect "beats_system" send "ytx@1234\n" expect "beats_system" send "ytx@1234\n" expect "remote_monitoring_user" send "ytx@1234\n" expect "remote_monitoring_user" send "ytx@1234\n" expect off EOF
2.备份索引准备
（1）查看所有数据量
（2）导入测试数据
[root@devops data]# cat insert.sh #!/bin/bash #es_data.json 数据文件 #bank 索引名称 curl -H "Content-Type: application/json" -XPOST --user elastic:ytx@1234 "127.0.0.1:29200/bank/_bulk?pretty&refresh" --data-binary "@es_data.json" curl -XGET -k --user elastic:ytx@1234 "http://127.0.0.1:29200/_cat/indices?v"
（3）测试数据文件es_data.json
{"index":{"_id":"1"}}
{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}
{"index":{"_id":"6"}}
{"account_number":6,"balance":5686,"firstname":"Hattie","lastname":"Bond","age":36,"gender":"M","address":"671 Bristol Street","employer":"Netagy","email":"hattiebond@netagy.com","city":"Dante","state":"TN"}
{"index":{"_id":"13"}}
{"account_number":13,"balance":32838,"firstname":"Nanette","lastname":"Bates","age":28,"gender":"F","address":"789 Madison Street","employer":"Quility","email":"nanettebates@quility.com","city":"Nogal","state":"VA"}
{"index":{"_id":"18"}}
{"account_number":18,"balance":4180,"firstname":"Dale","lastname":"Adams","age":33,"gender":"M","address":"467 Hutchinson Court","employer":"Boink","email":"daleadams@boink.com","city":"Orick","state":"MD"}
{"index":{"_id":"20"}}
{"account_number":20,"balance":16418,"firstname":"Elinor","lastname":"Ratliff","age":36,"gender":"M","address":"282 Kings Place","employer":"Scentric","email":"elinorratliff@scentric.com","city":"Ribera","state":"WA"}
{"index":{"_id":"25"}}
{"account_number":25,"balance":40540,"firstname":"Virginia","lastname":"Ayala","age":39,"gender":"F","address":"171 Putnam Avenue","employer":"Filodyne","email":"virginiaayala@filodyne.com","city":"Nicholson","state":"PA"}
{"index":{"_id":"32"}}
{"account_number":32,"balance":48086,"firstname":"Dillard","lastname":"Mcpherson","age":34,"gender":"F","address":"702 Quentin Street","employer":"Quailcom","email":"dillardmcpherson@quailcom.com","city":"Veguita","state":"IN"}
{"index":{"_id":"37"}}
{"account_number":37,"balance":18612,"firstname":"Mcgee","lastname":"Mooney","age":39,"gender":"M","address":"826 Fillmore Place","employer":"Reversus","email":"mcgeemooney@reversus.com","city":"Tooleville","state":"OK"}
{"index":{"_id":"44"}}
{"account_number":44,"balance":34487,"firstname":"Aurelia","lastname":"Harding","age":37,"gender":"M","address":"502 Baycliff Terrace","employer":"Orbalix","email":"aureliaharding@orbalix.com","city":"Yardville","state":"DE"}
{"index":{"_id":"49"}}
{"account_number":49,"balance":29104,"firstname":"Fulton","lastname":"Holt","age":23,"gender":"F","address":"451 Humboldt Street","employer":"Anocha","email":"fultonholt@anocha.com","city":"Sunriver","state":"RI"}
{"index":{"_id":"51"}}
{"account_number":51,"balance":14097,"firstname":"Burton","lastname":"Meyers","age":31,"gender":"F","address":"334 River Street","employer":"Bezal","email":"burtonmeyers@bezal.com","city":"Jacksonburg","state":"MO"}
{"index":{"_id":"56"}}
{"account_number":56,"balance":14992,"firstname":"Josie","lastname":"Nelson","age":32,"gender":"M","address":"857 Tabor Court","employer":"Emtrac","email":"josienelson@emtrac.com","city":"Sunnyside","state":"UT"}
{"index":{"_id":"63"}}
{"account_number":63,"balance":6077,"firstname":"Hughes","lastname":"Owens","age":30,"gender":"F","address":"510 Sedgwick Street","employer":"Valpreal","email":"hughesowens@valpreal.com","city":"Guilford","state":"KS"}
{"index":{"_id":"68"}}
{"account_number":68,"balance":44214,"firstname":"Hall","lastname":"Key","age":25,"gender":"F","address":"927 Bay Parkway","employer":"Eventex","email":"hallkey@eventex.com","city":"Shawmut","state":"CA"}
{"index":{"_id":"70"}}
{"account_number":70,"balance":38172,"firstname":"Deidre","lastname":"Thompson","age":33,"gender":"F","address":"685 School Lane","employer":"Netplode","email":"deidrethompson@netplode.com","city":"Chestnut","state":"GA"}
{"index":{"_id":"75"}}
{"account_number":75,"balance":40500,"firstname":"Sandoval","lastname":"Kramer","age":22,"gender":"F","address":"166 Irvington Place","employer":"Overfork","email":"sandovalkramer@overfork.com","city":"Limestone","state":"NH"}
{"index":{"_id":"82"}}
{"account_number":82,"balance":41412,"firstname":"Concetta","lastname":"Barnes","age":39,"gender":"F","address":"195 Bayview Place","employer":"Fitcore","email":"concettabarnes@fitcore.com","city":"Summerfield","state":"NC"}
{"index":{"_id":"87"}}
{"account_number":87,"balance":1133,"firstname":"Hewitt","lastname":"Kidd","age":22,"gender":"M","address":"446 Halleck Street","employer":"Isologics","email":"hewittkidd@isologics.com","city":"Coalmont","state":"ME"}
{"index":{"_id":"94"}}
{"account_number":94,"balance":41060,"firstname":"Brittany","lastname":"Cabrera","age":30,"gender":"F","address":"183 Kathleen Court","employer":"Mixers","email":"brittanycabrera@mixers.com","city":"Cornucopia","state":"AZ"}
{"index":{"_id":"99"}}
{"account_number":99,"balance":47159,"firstname":"Ratliff","lastname":"Heath","age":39,"gender":"F","address":"806 Rockwell Place","employer":"Zappix","email":"ratliffheath@zappix.com","city":"Shaft","state":"ND"}
{"index":{"_id":"102"}}
{"account_number":102,"balance":29712,"firstname":"Dena","lastname":"Olson","age":27,"gender":"F","address":"759 Newkirk Avenue","employer":"Hinway","email":"denaolson@hinway.com","city":"Choctaw","state":"NJ"}
{"index":{"_id":"107"}}
{"account_number":107,"balance":48844,"firstname":"Randi","lastname":"Rich","age":28,"gender":"M","address":"694 Jefferson Street","employer":"Netplax","email":"randirich@netplax.com","city":"Bellfountain","state":"SC"}
{"index":{"_id":"114"}}
{"account_number":114,"balance":43045,"firstname":"Josephine","lastname":"Joseph","age":31,"gender":"F","address":"451 Oriental Court","employer":"Turnabout","email":"josephinejoseph@turnabout.com","city":"Sedley","state":"AL"}
{"index":{"_id":"119"}}
{"account_number":119,"balance":49222,"firstname":"Laverne","lastname":"Johnson","age":28,"gender":"F","address":"302 Howard Place","employer":"Senmei","email":"lavernejohnson@senmei.com","city":"Herlong","state":"DC"}
{"index":{"_id":"121"}}
{"account_number":121,"balance":19594,"firstname":"Acevedo","lastname":"Dorsey","age":32,"gender":"M","address":"479 Nova Court","employer":"Netropic","email":"acevedodorsey@netropic.com","city":"Islandia","state":"CT"}
（4）查看索引
（5）es配置文件添加备份路径
（6）重启docker
（7）拷贝备份和恢复脚本到es容器中
docker cp es_bak.sh es:/usr/share/elasticsearch/
docker cp es_rok.sh es:/usr/share/elasticsearch/
3.备份指定索引
（1）备份脚本
[root@devops shell]# cat es_bak.sh
#!/bin/bash
currentUser=$(whoami)
if [ "$currentUser" != "root" ];then
echo "当前用户为非root,请切换用户：${currentUser}"
exit 1
fi
esDir="/usr/share/elasticsearch" #es安装目录
es_backup="/home/elastic/bak" #es备份数据目录
filename=`date +%Y%m%d%H` #快照名称
ipAddress=127.0.0.1 #IP
backupBeforeDelete=yes #备份之前是否删除旧备份
echo "*******************************"
echo "* ES备份开始 *"
echo "*******************************"
#当前安装目录
currentpath=$(pwd)
echo "**当前安装目录:" ${currentpath}
#设置备份目录
echo "1.设置备份目录"
echo " * 修改elasticsearch.yml"
#查找配置文件是否存在字符串
if cat ${esDir}/config/elasticsearch.yml |grep "path.repo:"
then
#找到
echo " * elasticsearch.yml无需修改"
else
#未找到
#cat elasticsearch.yml |grep "path.repo:"
#echo "path.repo: ["'${es_backup}'"]">> ${esDir}/config/elasticsearch.yml
res=`echo $?`
if [ $res = "1" ]; then
echo -e "\033[31m修改elasticsearch.yml失败 \033[0m"
exit 1
fi
fi
echo " * 创建ES数据备份目录以及权限*"
mkdir -p ${es_backup}
chmod 755 ${es_backup}
chown elasticsearch.elasticsearch ${es_backup}
res=`echo $?`
if [ $res = "1" ]; then
echo -e "\033[31m创建ES数据备份目录失败 \033[0m"
exit 1
fi
#docker rstart es #重启docker服务加载配置文件
#检测es访问地址是否有效
esStatus=$(curl -s -m 5 -IL --user elastic:ytx@1234 http://${ipAddress}:29200|grep 200)
if [ "$esStatus" != "" ];then
echo $(date +'%Y-%m-%d %H:%M:%S') "es地址访问正常：http://${ipAddress}:29200,开始备份数据"
#创建仓库
#set -x #打印上一步命令
curl -H "Content-Type: application/json" -XPUT --user elastic:ytx@1234 'http://'${ipAddress}':29200/_snapshot/datasvr' -d ' {"type":"fs","settings":{"location":"'${es_backup}'","compress":true}}'
#set +x
res=`echo $?`
if [ $res = "1" ]; then
echo -e "\033[31m创建仓库失败 \033[0m"
exit 1
fi
echo -e " \n创建仓库成功"
#删除快照snapshot_1
curl -X DELETE --user elastic:ytx@1234 "localhost:29200/_snapshot/datasvr/snapshot_1"
echo -e " \n删除旧快照snapshot_1成功"
#创建快照snapshot_1(备份索引)
#curl -XPUT --user elastic:ytx@1234 'http://0.0.0.0:29200/_snapshot/datasvr/snapshot_2' #备份所有索引
curl -H "Content-Type:application/json" -XPUT --user elastic:ytx@1234 ''${ipAddress}':29200/_snapshot/datasvr/snapshot_1?wait_for_completion=true' -d '{"indices":"bank"}'
res=`echo $?`
if [ $res = "1" ]; then
echo -e "\033[31m 创建快照snapshot_1失败 \033[0m"
exit 1
fi
echo -e " \n创建快照snapshot_1成功"
else
echo -e "\033[31m $(date +'%Y-%m-%d %H:%M:%S') es地址访问异常：http://${ipAddress}:29200 \033[0m"
exit 1
fi
echo "*******************************"
echo "* ES备份完成 *"
echo "*******************************"
（2）执行备份脚本
cd /usr/share/elasticsearch
[root@devops elasticsearch]# ./es_bak.sh
*******************************
* ES备份开始 *
*******************************
**当前安装目录: /usr/share/elasticsearch
1.设置备份目录
* 修改elasticsearch.yml
path.repo: ["/home/elastic/bak"]
* elasticsearch.yml无需修改
* 创建ES数据备份目录以及权限*
2022-06-01 14:11:56 es地址访问正常：http://127.0.0.1:29200,开始备份数据
{"acknowledged":true}
创建仓库成功
{"error":{"root_cause":[{"type":"snapshot_missing_exception","reason":"[datasvr:snapshot_1] is missing"}],"type":"snapshot_missing_exception","reason":"[datasvr:snapshot_1] is missing"},"status":404}
删除旧快照snapshot_1成功
{"snapshot":{"snapshot":"snapshot_1","uuid":"v1sbmHO_Ql2N3eJcDLRZ3Q","version_id":7010199,"version":"7.1.1","indices":["bank"],"include_global_state":true,"state":"SUCCESS","start_time":"2022-06-01T06:11:56.606Z","start_time_in_millis":1654063916606,"end_time":"2022-06-01T06:11:57.179Z","end_time_in_millis":1654063917179,"duration_in_millis":573,"failures":[],"shards":{"total":1,"failed":0,"successful":1}}}
创建快照snapshot_1成功
*******************************
* ES备份完成 *
*******************************
（3）查看备份目录
（4）查看快照仓库
curl -XGET --user elastic:ytx@1234 'http://0.0.0.0:29200/_snapshot?pretty'
（5）查看备份数据
curl -XGET --user elastic:ytx@1234 'http://0.0.0.0:29200/_snapshot/datasvr/snapshot_1/_status'
curl -XGET --user elastic:ytx@1234 'http://0.0.0.0:29200/_snapshot/datasvr/snapshot_1'
4.删除索引bank
curl -XDELETE -k --user elastic:ytx@1234 'http://127.0.0.1:29200/bank'
5.恢复bank索引
（1）恢复脚本
[root@devops shell]# cat es_rok.sh
#!/bin/bash
currentUser=$(whoami)
if [ "$currentUser" != "root" ];then
echo "当前用户为非root,请切换用户：${currentUser}"
exit 1
fi
ipAddress=127.0.0.1 #IP
echo "*******************************"
echo "* ES恢复备份开始 *"
echo "*******************************"
#当前安装目录
currentpath=$(pwd)
echo "**当前安装目录:" ${currentpath}
#检测es访问地址是否有效
esStatus=$(curl -s -m 5 -IL --user elastic:ytx@1234 http://${ipAddress}:29200|grep 200)
if [ "$esStatus" != "" ];then
echo " * es地址访问正常：http://${ipAddress}:29200"
#快照恢复
#curl -H "Content-Type:application/json" -XPOST --user elastic:ytx@1234 ''${ipAddress}':29200/_snapshot/datasvr/snapshot_1/_restore'
curl -H "Content-Type:application/json" -XPOST --user elastic:ytx@1234 ''${ipAddress}':29200/_snapshot/datasvr/snapshot_1/_restore?wait_for_completion=true' -d '{"indices":"bank"}'
res=`echo $?`
if [ $res = "1" ]; then
echo -e "\033[31m * 快照恢复snapshot_1失败 \033[0m"
exit 1
fi
echo -e " \n * 快照恢复snapshot_1成功"
else
echo -e "\033[31m * es地址访问异常：http://${ipAddress}:29200 \033[0m"
exit 1
fi
echo "*******************************"
echo "* ES恢复备份完成 *"
echo "*******************************"
（2）恢复脚本执行
（3）完成数据恢复
6.注意：
（.security-7）索引里边包含了es的账户密码，所以在数据备份和恢复的时候不能删除它，因此我这边在做数据备份和恢复的时候也针对指定的索引去做，如果是多个索引可以使用脚本做循环。
7.报错处理
（1）备份报错
（2）解决方法：
在es配置文件中添加如下行：
vim elasticsearch.yml
添加：
path.repo: "/home/backup/elk"
重启es服务
（3）恢复报错：
（4）解决方法：
没有把所有索引删除，因为备份是全量备份，所以需要删除所有索引
curl -XDELETE -k --user elastic:ytx@1234 'http://127.0.0.1:29200/.security-7'
8.常用es命令汇总
_cat命令
#查询所有索引
curl -XGET -k --user elastic:ytx@1234 "http://127.0.0.1:29200/_cat/indices?pretty"
#查询ES全局状态
curl -XGET -k --user elastic:ytx@1234 'http://127.0.0.1:29200/_cluster/stats?pretty'
#查询es集群是否健康
curl -XGET -k --user elastic:ytx@1234 'http://127.0.0.1:29200/_cat/health?v'
#查询es集群index的整体数据量
curl -XGET -k --user elastic:ytx@1234 'http://127.0.0.1:29200/_cat/indices?v'
#查询集群线程池使用情况
curl -XGET -k --user elastic:ytx@1234 'http://127.0.0.1:29200/_cat/thread_pool?v'
查询_cat接口支持的所有的命令列表
curl -XGET -k --user elastic:ytx@1234 'http://127.0.0.1:29200/_cat/'
_nodes命令
查询指定ES实例的jvm参数 elastic是node_name
curl -XGET -k --user elastic:ytx@1234 'http://127.0.0.1:29200/_nodes/elastic/stats/jvm?pretty'
索引命令
关闭索引 my_index是索引名称
curl -XPOST -k --user elastic:ytx@1234 'http://127.0.0.1:29200/my_index/_close?pretty'
打开索引
curl -XPOST -k --user elastic:ytx@1234 'http://127.0.0.1:29200/my_index/_open?pretty'
删除索引,命令最后为索引名称，只有该索引的创建用户才可以删除索引
curl -XDELETE -k --user elastic:ytx@1234 'http://127.0.0.1:29200/bank'
查询索引mapping和settings
curl -XGET -k --user elastic:ytx@1234 'http://127.0.0.1:29200/my_index_name?pretty'
查询索引settings
curl -XGET -k --user elastic:ytx@1234 'http://127.0.0.1:29200/my_index_name/_settings?pretty'
修改索引刷新时间：
curl -XPUT --negotiate -k -u : 'https://ip:port/my_index/_settings?pretty' -H 'Content-Type: application/json' -d'{"refresh_interval" : "60s"}'
修改translog文件保留时长，默认为12小时
curl -XPUT --negotiate -k -u : 'https://ip:port/my_index/_settings?pretty' -H 'Content-Type: application/json' -d'{"index.translog.retention.age" : "30m"}'
设置索引副本：
curl -XPUT --negotiate -k -u : 'https://ip:port/my_index/_settings?pretty' -H 'Content-Type: application/json' -d'{"number_of_replicas" : 1}'
强制flush
curl -XPOST --negotiate --tlsv1.2 -k -u : 'https://ip:port/myindex/_flush'
强制refresh
curl -XPOST --negotiate --tlsv1.2 -k -u : 'https://ip:port/myindex/_refresh'
#解除只读权限
curl -XPUT "http://127.0.0.1:29200/_settings" -H 'Content-Type: application/json' -d '{"index.blocks.read_only_allow_delete":"false"}' --user elastic:ytx@1234
#查询当前索引当前类型中的所有文档
curl -XPOST --user elastic:ytx@1234 'http://127.0.0.1:29200/bank/_doc/_search'
#根据id查询数据，1是id
curl --user elastic:ytx@1234 'http://127.0.0.1:29200/bank/_doc/1'
#查看快照仓库
curl -XGET --user elastic:ytx@1234 'http://0.0.0.0:29200/_snapshot?pretty'
#查看备份数据
curl -XGET --user elastic:ytx@1234 'http://0.0.0.0:29200/_snapshot/datasvr/snapshot_1/_status'
curl -XGET --user elastic:ytx@1234 'http://0.0.0.0:29200/_snapshot/datasvr/snapshot_1'
9.参考文档
https://woj.app/3937.html
https://blog.csdn.net/liqfyiyi/article/details/120849116
