Velero云原生的灾难恢复和迁移工具

18. Velero

18.1 Velero概述

 1. Velero 是vmware开源的一个云原生的灾难恢复和迁移工具，它本身也是开源的，采用GO语言编写，可以安装的备份、恢复和迁移Kubernetes集群资源数据
2. Velero是西班牙语意思是帆船，非常符合Kubernetes社区的命名风格，Velero的开发公司Heptio,已被VMware收购
 
3. Velero支持标准的K8S集群，既可以是私有云平台也可以是公有云，除了灾难之外它还能做资源转移，支持把容器应用从一个集群迁移到另一个集群
 
4. Velero的工作方式就是把kubernetes中的数据备份到对象存储以实现高可用的持久化，默认的备份保存时间为720小时，并在需要的时候进行下载和恢复

18.2 Velero与etcd快照备份的区别

 1. etcd快照是全局备份，在即使一个资源对象需要恢复，也是需要做全局恢复到备份的状态，即会影响其它namespace中pod运行服务
 
2. Velero可以有针对性的备份，比如按照namespace单独备份、只备份单独的资源对象等，在恢复的时候只恢复单独的namespace或资源对象，而不影响其它namespace中pod运行服务
 
3. Velero支持ceph、oss等对象存储，etcd快照是一个为本地文件
 
4. Velero支持任务计划实现周期备份，但etcd快照也可以基于cronjob实现

18.3 Velero备份流程

 Velero客户端调用Kubernetes API Server创建Backup任务
Backuup 控制器基于watch机制通过API Server获取到备份任务
Backup 控制器开始执行备份动作，其会通过请求API Server获取需要备份的数据
Backup 控制器将获取到的数据备份到指定的对象存储server端

18.4 部署minio 私有共享存储

 docker run --name minio \
-p 9000:9000 \
-p 9999:9999 \
-d --restart=always \
-e "MINIO_ROOT_USER=admin" \
-e "MINIO_ROOT_PASSWORD=12345678" \
-v /data/minio/data:/data \
minio/minio server /data  \
--console-address '0.0.0.0:9999'

登录成功
18.5 配置对象存储
创建桶
这就创建好了

18.5 部署Velero

 root@deploy-harbor:/# wget https://github.com/vmware-tanzu/velero/releases/download/v1.8.1/velero-v1.8.1-linux-amd64.tar.gz
root@deploy-harbor:/velero# tar xvf velero-v1.8.1-linux-amd64.tar.gz -C /velero
root@deploy-harbor:/velero# cp velero-v1.8.1-linux-amd64/velero  /usr/local/bin/
root@deploy-harbor:/velero# velero  --help

18.6 配置velero认证环境

 #工作目录：
root@deploy-harbor:~# mkdir  /velero -p
 
#认证文件：
root@k8s-master1:/velero# vim velero-auth.txt 
#aws用户 这个key不能变value可以变  是minio的账号密码
[default]
aws_access_key_id = admin
aws_secret_access_key = 12345678

18.7 准备user-csr文件

 root@deploy-harbor:/velero# vim awsuser-csr.json
{
  "CN": "awsuser",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

18.8 准备证书签发环境

 root@deploy-harbor:/velero# apt install golang-cfssl
 
#方法一 下载改名使用
root@deploy-harbor:/velero# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64 
root@deploy-harbor:/velero# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64 
root@deploy-harbor:/velero# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64
 
root@deploy-harbor:/velero# mv cfssl-certinfo_1.6.1_linux_amd64 cfssl-certinfo
root@deploy-harbor:/velero# mv cfssl_1.6.1_linux_amd64 cfssl
root@deploy-harbor:/velero# mv cfssljson_1.6.1_linux_amd64 cfssljson
 
root@deploy-harbor:/velero# cp cfssl* /usr/local/bin/
root@deploy-harbor:/velero# chmod a+x /usr/local/bin/cfssl*
 
#方法二 使用/etc/kubeasz
/etc/kubeasz/bin/提供了这些命令拷贝即可
root@deploy-harbor:/velero# cp /etc/kubeasz/bin/cfssl* /usr/local/bin
root@deploy-harbor:/velero# chmod a+x /usr/local/bin/cfssl*
root@deploy-harbor:/velero# cp /etc/kubeasz/clusters/k8s-cluster1/ssl/ca-config.json /velero

18.9 执行证书签发

 root@deploy-harbor:/velero# ll
total 20
drwxr-xr-x  2 root root 4096 Nov 23 13:14 ./
drwxr-xr-x 24 root root 4096 Nov 23 12:24 ../
#证书签发文件
-rw-r--r--  1 root root  220 Nov 23 12:25 awsuser-csr.json
#ca证书配置文件
-rw-r--r--  1 root root  483 Nov 23 13:13 ca-config.json
#minio认证文件
-rw-r--r--  1 root root   89 Nov 23 12:25 velero-auth.txt
 
#源执行命令
1.24.x:
root@deploy-harbor:/velero# /usr/local/bin/cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=/etc/kubeasz/clusters/k8s-cluster1/ssl/ca-config.json -profile=kubernetes ./awsuser-csr.json | cfssljson -bare awsuser
 
1.23.x:
 
 
#添加了修改后 就改了一下路径
root@deploy-harbor:/velero# /usr/local/bin/cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=./ca-config.json -profile=kubernetes ./awsuser-csr.json | cfssljson -bare awsuser
2022/11/23 13:40:18 [INFO] generate received request
2022/11/23 13:40:18 [INFO] received CSR
2022/11/23 13:40:18 [INFO] generating key: rsa-2048
2022/11/23 13:40:18 [INFO] encoded CSR
2022/11/23 13:40:18 [INFO] signed certificate with serial number 204902735149897151390216548318080805156194245414
2022/11/23 13:40:18 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
 
#生成的证书文件
root@deploy-harbor:/velero# ll
total 32
drwxr-xr-x  2 root root 4096 Nov 23 13:40 ./
drwxr-xr-x 21 root root 4096 Nov 23 13:23 ../
-rw-r--r--  1 root root  997 Nov 23 13:40 awsuser.csr
-rw-r--r--  1 root root  220 Nov 23 13:23 awsuser-csr.json
#生成api-server证书路径
-rw-------  1 root root 1679 Nov 23 13:40 awsuser-key.pem
#生成api-server证书路径
-rw-r--r--  1 root root 1387 Nov 23 13:40 awsuser.pem
-rw-r--r--  1 root root  483 Nov 23 13:25 ca-config.json
-rw-r--r--  1 root root   89 Nov 23 13:23 velero-auth.txt
 
#分发证书到api-server证书路径：
root@deploy-harbor:/velero# cp awsuser-key.pem /etc/kubernetes/ssl/
root@deploy-harbor:/velero# cp awsuser.pem /etc/kubernetes/ssl/

18.10 生成集群认证config文件

 #这个IP地址可以是VIP 我这里就指定了master的IP地址了
root@deploy-harbor:/velero# export KUBE_APISERVER="https://192.168.1.70:6443"
root@deploy-harbor:/velero# kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=./awsuser.kubeconfig
 
#输出
Cluster "kubernetes" set.
 
root@deploy-harbor:/velero# ll
total 36
drwxr-xr-x  2 root root 4096 Nov 23 13:47 ./
drwxr-xr-x 21 root root 4096 Nov 23 13:23 ../
-rw-r--r--  1 root root  997 Nov 23 13:40 awsuser.csr
-rw-r--r--  1 root root  220 Nov 23 13:23 awsuser-csr.json
-rw-------  1 root root 1679 Nov 23 13:40 awsuser-key.pem
#会生成这个证书文件
-rw-------  1 root root 1938 Nov 23 13:47 awsuser.kubeconfig
-rw-r--r--  1 root root 1387 Nov 23 13:40 awsuser.pem
-rw-r--r--  1 root root  483 Nov 23 13:25 ca-config.json
-rw-r--r--  1 root root   89 Nov 23 13:23 velero-auth.txt
 
root@deploy-harbor:/velero# cat awsuser.kubeconfig 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURsRENDQW55Z0F3SUJBZ0lVTUV2SXV3UXdTL3lDMXFyUlFpZ1duVjYwRXNJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lURUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEV6QVJCZ05WQkFNVENtdDFZbVZ5CmJtVjBaWE13SUJjTk1qSXhNVEU1TURZME1qQXdXaGdQTWpFeU1qRXdNall3TmpReU1EQmFNR0V4Q3pBSkJnTlYKQkFZVEFrTk9NUkV3RHdZRFZRUUlFd2hJWVc1bldtaHZkVEVMTUFrR0ExVUVCeE1DV0ZNeEREQUtCZ05WQkFvVApBMnM0Y3pFUE1BMEdBMVVFQ3hNR1UzbHpkR1Z0TVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXQ1QkZtZkh4KzhCQXQ1OTg2QnBjV2Z0Q3VYcVoKMzlmMFBNaGhDTXNGVWtjYThjNzFxV3R0b0g4Mmk4UEpxTzdRL1FRbWF0MzdmOEdSZmkxNXd0Z0xBYUtmUXlRdgpGYlZvaks0NDhhNHRtSENtbk0zU1dvWStTU2xmbHMrYXM2cEc4SGVnSmE4WVFvR0d5bmtsb2F3dG0wVEVNcmtPCnd5MG5lbDRMcnl0TTF4QnhXek4wbGsrRlhId1o2NEkrR3JvQ2dvSm91RGtaMTNmcENycjlHUGpRelBpNE1CeVQKeXVaREFLMEE1V3FEcVhxSy9kRVJPb09yVXRoK3ROSHJCNTFqZ0QzQlJFVnZSLytBcG52VFRBM212Ym1UQVdudwo2SGVZZkdtSkpNWDRNZVBCYUIzYjVXa044cWJqckhVVXhXSmZPZG96UG1iQ3FRRmplcWRNVDh4VWZ3SURBUUFCCm8wSXdRREFPQmdOVkhROEJBZjhFQkFNQ0FRWXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVUKYk54STNOS05GeERBa1o0eUJWR1RSMmdOL2ZBd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMSTdNaUxZN0ZQawpzQ0E2b3hBVkxMZWM0Ry9lQ0wxZUJjNVVkcUdDY2RYbXQ0U2RFWTE3WWRKZ2huV1pqNGRSOTVSVFBscWtPS3hYCkI5eCt6SzZYWVVaSlo4aUEvSUtFcHlFMHZYRlBjcmZPaWdzZDdKaEhiUDFuRFpGcEErNy9XaU1DUXlwNXNXRngKTzdmYjdwVko1bnl5RHZEdHNCY0VFYW5NUHhmKzNwZG83NHFBdWowYjNsNHJyU1RjOTdOS0xyekwyQnppaEI3UQpvaGpiSHpuR1lTQkdRU2IvOW5ZU24rRzBVTG50SlFCRU8xbGdTd0RuMzJNNHd4aThUODJ5bldmYUNHaUpMOWg5CkF2U05XRVVhZmF6UDFPVGFydng3N0xnbnA4SUtGejZ2UzJSVUZJSEl4cTJrQnl4SkE4NzhTMnRTWGROWC9SSlIKam9aaXFuRWZxR1U9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://192.168.1.70:6443
  name: kubernetes
contexts: null
current-context: ""
kind: Config
preferences: {}
#这里目前是空的用户认证
users: null

设置客户端证书认证

 root@deploy-harbor:/velero# kubectl config set-credentials awsuser \
--client-certificate=/etc/kubernetes/ssl/awsuser.pem \
--client-key=/etc/kubernetes/ssl/awsuser-key.pem \
--embed-certs=true \
--kubeconfig=./awsuser.kubeconfig
 
运行上面的认证 这个用户信息就会生成了 生成的是awsuser 
私钥：certificate-authority-data
公钥：client-certificate-data
 
root@deploy-harbor:/velero# cat awsuser.kubeconfig 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURsRENDQW55Z0F3SUJBZ0lVTUV2SXV3UXdTL3lDMXFyUlFpZ1duVjYwRXNJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lURUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEV6QVJCZ05WQkFNVENtdDFZbVZ5CmJtVjBaWE13SUJjTk1qSXhNVEU1TURZME1qQXdXaGdQTWpFeU1qRXdNall3TmpReU1EQmFNR0V4Q3pBSkJnTlYKQkFZVEFrTk9NUkV3RHdZRFZRUUlFd2hJWVc1bldtaHZkVEVMTUFrR0ExVUVCeE1DV0ZNeEREQUtCZ05WQkFvVApBMnM0Y3pFUE1BMEdBMVVFQ3hNR1UzbHpkR1Z0TVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXQ1QkZtZkh4KzhCQXQ1OTg2QnBjV2Z0Q3VYcVoKMzlmMFBNaGhDTXNGVWtjYThjNzFxV3R0b0g4Mmk4UEpxTzdRL1FRbWF0MzdmOEdSZmkxNXd0Z0xBYUtmUXlRdgpGYlZvaks0NDhhNHRtSENtbk0zU1dvWStTU2xmbHMrYXM2cEc4SGVnSmE4WVFvR0d5bmtsb2F3dG0wVEVNcmtPCnd5MG5lbDRMcnl0TTF4QnhXek4wbGsrRlhId1o2NEkrR3JvQ2dvSm91RGtaMTNmcENycjlHUGpRelBpNE1CeVQKeXVaREFLMEE1V3FEcVhxSy9kRVJPb09yVXRoK3ROSHJCNTFqZ0QzQlJFVnZSLytBcG52VFRBM212Ym1UQVdudwo2SGVZZkdtSkpNWDRNZVBCYUIzYjVXa044cWJqckhVVXhXSmZPZG96UG1iQ3FRRmplcWRNVDh4VWZ3SURBUUFCCm8wSXdRREFPQmdOVkhROEJBZjhFQkFNQ0FRWXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVUKYk54STNOS05GeERBa1o0eUJWR1RSMmdOL2ZBd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMSTdNaUxZN0ZQawpzQ0E2b3hBVkxMZWM0Ry9lQ0wxZUJjNVVkcUdDY2RYbXQ0U2RFWTE3WWRKZ2huV1pqNGRSOTVSVFBscWtPS3hYCkI5eCt6SzZYWVVaSlo4aUEvSUtFcHlFMHZYRlBjcmZPaWdzZDdKaEhiUDFuRFpGcEErNy9XaU1DUXlwNXNXRngKTzdmYjdwVko1bnl5RHZEdHNCY0VFYW5NUHhmKzNwZG83NHFBdWowYjNsNHJyU1RjOTdOS0xyekwyQnppaEI3UQpvaGpiSHpuR1lTQkdRU2IvOW5ZU24rRzBVTG50SlFCRU8xbGdTd0RuMzJNNHd4aThUODJ5bldmYUNHaUpMOWg5CkF2U05XRVVhZmF6UDFPVGFydng3N0xnbnA4SUtGejZ2UzJSVUZJSEl4cTJrQnl4SkE4NzhTMnRTWGROWC9SSlIKam9aaXFuRWZxR1U9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://192.168.1.70:6443
  name: kubernetes
contexts: null
current-context: ""
kind: Config
preferences: {}
users:
- name: awsuser
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQwakNDQXJxZ0F3SUJBZ0lVSStRb0N4ZURaTTdwQTZta0xOcGc1Z3V5V3lZd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lURUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEV6QVJCZ05WQkFNVENtdDFZbVZ5CmJtVjBaWE13SUJjTk1qSXhNVEl6TVRNek5UQXdXaGdQTWpBM01qRXhNVEF4TXpNMU1EQmFNR0l4Q3pBSkJnTlYKQkFZVEFrTk9NUkF3RGdZRFZRUUlFd2RDWldsS2FXNW5NUkF3RGdZRFZRUUhFd2RDWldsS2FXNW5NUXd3Q2dZRApWUVFLRXdOck9ITXhEekFOQmdOVkJBc1RCbE41YzNSbGJURVFNQTRHQTFVRUF4TUhZWGR6ZFhObGNqQ0NBU0l3CkRRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFOODNKRE5JOEtZOGs3cjlPWVhja2JWd1pPZGsKNHZ6eFB2eE1XM0poaXg0TlNaYjlTVzhreHhjRjFsYUZIQ0dSei9TdEZwTDgvTzduR1d5RlJBVXE1K2hqRmJzdgpnOWEydTN3bmtLK3NWUGVUNlVQcGVWY0ROOUVTcFljRk5ERGd5VnQ5WGZxeUVxNkVzcU9DSVNuUXlLTzNFSURBCkdaUXFzaDBjRGYyMXU0V2JjaHBwSjJWeHRIQmhEQzdOSGROK0pmeC9ZeGU5VThSQTVCOHZQZ2g3S2tJcGtQb3MKN2Fob20ydGlwT3pkWUVTODkxZkFJekthb1BpZ0ppUWZBZGVXMUJldGl2akFZK2Z2MEJvRlNFbU5HVWs1c3kyNwo1TnlRU2RuVWJlcHk5SjY4N2hpdThlWE9aaXdsaDcxdVZaM2h0V21hS2RkT20rYlR6cmpPWUxWWER0a0NBd0VBCkFhTi9NSDB3RGdZRFZSMFBBUUgvQkFRREFnV2dNQjBHQTFVZEpRUVdNQlFHQ0NzR0FRVUZCd01CQmdnckJnRUYKQlFjREFqQU1CZ05WSFJNQkFmOEVBakFBTUIwR0ExVWREZ1FXQkJTVWhHR2hGVzNDb2tISnZrNUlFbi9kUGZURQpaREFmQmdOVkhTTUVHREFXZ0JSczNFamMwbzBYRU1DUm5qSUZVWk5IYUEzOThEQU5CZ2txaGtpRzl3MEJBUXNGCkFBT0NBUUVBV3ZMbWlid1NXV0IzYkcrejJqMi9UajY3MkxDc2xsK2JONGw4SzhVdFZyS2lVbTBzakZkaEFrRloKZHFoWi9UQXdlaDJ4U2NXUytaNU5pTkdiUk5IRFdmVXcyeTR3QldnWFFNVktEaUdZcW5iWGFYQmV4dmxpWVRycwp6aC9wS0o3UThxUjJuTVppd0NWTi90dDg2SktOaWU3Zm0rVjdSVms5WUdaZUZxOHFuWFptd09hYnlPV2FkYmY3ClVMSHM5M0dOT1B6OGF3RGNsbkxIaTZIT05YamVlQ1BXVVV5RDFQbTViVzc0ei92Rm5PQm1tdUpyZnlmYUczRE8Kdk9IOU5qellIb3RSWncwNGVQY2Zza2hNSjRwNzJuRXVmaktkdHVETDBocGZtZ0NoWk5nUGdoRkhVcjI0QnFMOQpHbTdqbDRxSGxTNWFoL2E5TXcyekJPMk9nckZtcXc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBM3pja00wandwanlUdXYwNWhkeVJ0WEJrNTJUaS9QRSsvRXhiY21HTEhnMUpsdjFKCmJ5VEhGd1hXVm9VY0laSFA5SzBXa3Z6ODd1Y1piSVZFQlNybjZHTVZ1eStEMXJhN2ZDZVFyNnhVOTVQcFErbDUKVndNMzBSS2xod1UwTU9ESlczMWQrcklTcm9TeW80SWhLZERJbzdjUWdNQVpsQ3F5SFJ3Ti9iVzdoWnR5R21rbgpaWEcwY0dFTUxzMGQwMzRsL0g5akY3MVR4RURrSHk4K0NIc3FRaW1RK2l6dHFHaWJhMktrN04xZ1JMejNWOEFqCk1wcWcrS0FtSkI4QjE1YlVGNjJLK01CajUrL1FHZ1ZJU1kwWlNUbXpMYnZrM0pCSjJkUnQ2bkwwbnJ6dUdLN3gKNWM1bUxDV0h2VzVWbmVHMWFab3AxMDZiNXRQT3VNNWd0VmNPMlFJREFRQUJBb0lCQVFDYmFBeXdHQXhUOE1ENQo0dXQxbzBkZkIwMTVQT2c4S3NvclpnNFU2SXl1ODVsVW9pdktVZFM2MXJtUHEwVVdxaW9hczVlUDVzdENtV3M5Ci90cUlyb0tmYkx6b2dnQk1NUlQvTDV5d2NrOEZ2OWtjQ3lVTHk1WEUzaktZZXFzSGpMa2tGWGlrM2UzYnkwK0UKUDBsUGo2amhNc2N1bkpBeWREZGx4TzZJd1VvNWpkOVJZTXlhY3dyV2FpTm1QZzB6YjVDVWtxYS9PUGpVamFOMQpKZUJMbGYvd0pnUnZxcXdCVHUwMzBIWGsrV1pQNk1HNTNQak0vd3RHVTZVV2hteWgwZGNmZ1c5MTRtaDdjcG5oCmlENkJaY0k1UlVGM1lSUFFBMVpnbnJSUzU5VThibHZFWG1zcTR3cDJDUFk4cUZFdEt6VHp3WkV2bkdoekNrRVIKNUM0REVaZ0JBb0dCQVBLYlpyQy9mRWxKdEp5QXFMSmVIWHV1a1RiRnJFbFhKbU0zMzRNRVVYc245QlJta210OQpPODdmSnVJemdXQjliV0xJcHJHV2czK2JsVTViejg1ZFRqSjFuUkZWZlRlM3hicXRuT3dZSVpmTXhHTzFkU0pCCmp3NzJyNlNwbmllVUYvd3ZlM1BKRE1BVDYvTjBXMmpvbzdMcTZEdEU2WU8raXR4TmQ5amVNL2ZCQW9HQkFPdUoKc1FnT2VZZU9EMS9iTXNrMjN3OS9DZDd0eGZMRGJPbHNxQTQ1YlpqbEU2OW1LYXVOYndNaWVjSmhIcmFLc3dOagpHT1BseXJ3cHhJTUhKcFhMMlFyczBSVE8yMytCbHlWbTFIeTVxb0R1T0xOaE1QN2hLaFdMSG1LQXlvV0VMQVpBCk9HajhtMG1KU3ZLMnZFT3ZyS09Ba09uMjdYK3FKTXFzVFRlejh4MFpBb0dCQU5keVdnZXNoVnN3TVV5cWxTV1IKbnUwdllaWWFoWjI3MzBOTll4aWV4UjF4cGJoWVByOW1QNjYvVFhDai9Na0xtaENYMEREMVVXSUpjTlpkZ25YcApOb0JwSWtzNmVERDY2b1dWbUQyeVVHTFdYYkdPZTY4b09UczRwMUs3cnMvSEhHWEFaUW0yQ2JTcC9HZi9kUTM5ClN5S2dxZ3U0Yndlb052amRjd2tNaTJvQkFvR0FmbkZtZGZxL1J3UCtRT2s5ZEVOZXI4VDFHWlh2aEFsaE9GWTYKU1ppT3UxdU9tOFJ1YmtmNDVtZmxrWEh0dVBUd2NNc01HNXZLb0FTVUYvc0l2YlczQ0dSbFFaUml4U3BlWjVleQpUbHFscTUwM0Z5VC8xblF0MFc3am11R2sxdFJEaGYrSHlPU0N3SkZSaU1hTWZwR2FUaFBBdDdqMWVtbHVZdGJvCmhSUzNXRkVDZ1lBZ1BWRE1hdnVPaHNwc01XRUJWOFFHZ2dRQ0kwUzVhWUpidWl5UXE2a0tnS0IvU0doNE9tSlQKK2V2Nk9WYVY1NHplRmFVRDRQeUs2OU53RFdJK3Y5cGk5ZDFPRkM2Ym9iT2VMcm12SGhSdERJOVdseVZxTklLNQpuNXZORTMxVlUrVGVMK2lIc3lvNWE2YWZPTVNkL1VPQjJxaXVidFlEai9JcWlETGRrWURLOXc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

设置上下文参数

 root@deploy-harbor:/velero# kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=awsuser \
--namespace=velero-system \
--kubeconfig=./awsuser.kubeconfig
Context "kubernetes" created.

设置默认上下文

 #设置默认上下文:
root@deploy-harbor:/velero# kubectl config use-context kubernetes --kubeconfig=awsuser.kubeconfig

k8s集群中创建awsuser账户

 root@deploy-harbor:/velero# kubectl create clusterrolebinding awsuser --clusterrole=cluster-admin --user=awsuser

创建namespace

 root@deploy-harbor:/velero# kubectl create ns velero-system

执行安装

 #--bucket 执行minio存储桶的名字 写错的话他会找不到
#--secret-file 指定密码文件
#--namespace velero-system如果你需要不同的备份项目中你就在起一个名字aliyun-system 然后http地址在指定过去
root@deploy-harbor:/velero# velero --kubeconfig  ./awsuser.kubeconfig \
	install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.3.1 \
    --bucket velerodata  \
    --secret-file ./velero-auth.txt \
    --use-volume-snapshots=false \
	--namespace velero-system \
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.1.75:9000

验证安装

 所用的镜像 # velero/velero-plugin-for-aws:v1.3.1
root@deploy-harbor:/velero# kubectl  describe pod velero-6755cb8697-phfsr -n velero-system

18.11 创建备份

 root@deploy-harbor:/velero# kubectl get pods -A
NAMESPACE              NAME                                              READY   STATUS             RESTARTS          AGE
kube-system            calico-kube-controllers-68555f5f97-mww92          1/1     Running            0                 31h
kube-system            calico-node-hxbbn                                 0/1     Running            101 (4m53s ago)   8h
kube-system            calico-node-m7xwb                                 0/1     CrashLoopBackOff   101 (4m33s ago)   8h
kubernetes-dashboard   dashboard-metrics-scraper-6dfbcf7959-55bkt        1/1     Running            4 (32h ago)       32h
kubernetes-dashboard   kubernetes-dashboard-85997c7f79-64sm8             1/1     Running            5 (32h ago)       32h
myserver               coredns-57d94f5d84-bjx7d                          1/1     Running            0                 8h
myserver               coredns-57d94f5d84-zbt5x                          1/1     Running            0                 8h
myserver               linux60-tomcat-app1-deployment-595f7ff67c-494xk   1/1     Running            0                 7h55m
myserver               linux70-nginx-deployment-55dc5fdcf9-h8kp2         1/1     Running            0                 7h55m
myserver               net-test1                                         1/1     Running            1 (33h ago)       2d6h
myserver               net-test2                                         1/1     Running            1 (33h ago)       2d7h
myserver               net-test3                                         1/1     Running            1 (33h ago)       2d5h
myserver               net-test4                                         1/1     Running            1 (33h ago)       36h
myserver               net-test5                                         1/1     Running            1 (33h ago)       35h
velero-system          velero-858b9459f9-nvbhc                           1/1     Running            2 (20m ago)       23m
 
 
#velero backup --help
#--include-namespaces 指定名称空间备份
 
DATE=`date +%Y%m%d%H%M%S`
velero backup create myserver-ns-backup-${DATE} \
--include-namespaces myserver \
--kubeconfig=./awsuser.kubeconfig \
--namespace velero-system

备份成功了针对myserver备份

18.12 恢复数据

删除前

 #我删除了整个myserver名称空间下的数据 看看是否能恢复回来
root@deploy-harbor:/velero# kubectl delete namespaces myserver

恢复-找到你要恢复的名字

根据备份文件所恢复的

 #--kubeconfig 这个是指定认证文件 也可以不使用awsuser 使用/root/.kube/config
# --wait 加上这个wait就会同步的模式 不加就是异步 异步就是请求发给api-server就完成了  同步就是必须执行完成后才可以退出
#--namespace velero-system 这个参数不要改这个是指向velero所使用的
 
velero restore create --from-backup myserver-ns-backup-20221123150613 --wait \
--kubeconfig=./awsuser.kubeconfig \
--namespace velero-system

数据一下子就回来了把所有的资源都恢复了例如 deployment svc pod 等
备份所有资源

 #区别在于没有指定固定的--include-namespaces
#myserver-ns-backup这个名字是自己起的
DATE=`date +%Y%m%d%H%M%S`
velero backup create myserver-ns-backup-${DATE} \
--kubeconfig=./awsuser.kubeconfig \
--namespace velero-system

根据namespace恢复数据

 velero restore create --from-backup myserver-ns-backup-20221123153523 --wait \
--kubeconfig=/root/.kube/config \
--include-cluster-resources=true \
--include-namespaces default \
--namespace velero-system
 
velero restore create --from-backup myserver-ns-backup-20221123153523 --wait \
--kubeconfig=/root/.kube/config \
--include-cluster-resources=true \
--include-namespaces myserver \
--namespace velero-system

18.13 备份脚本

 root@deploy-harbor:/velero# vim ns-backup.sh
#!/bin/bash
NS_NAME=`kubectl get ns | awk '{if (NR>1){print}}' | awk '{print $1}'`
DATE=`date +%Y%m%d%H%M%S`
cd /velero/
 
for i in $NS_NAME;do
velero backup create   ${i}-ns-backup-${DATE} \
--kubeconfig=/root/.kube/config \
--include-cluster-resources=true \
--include-namespaces ${i} \
--namespace velero-system
done

成功

18.14 备份指定资源对象

 velero backup create pod-backup-xxx \
include-cluster-resources=true \
--ordered-resources \
'pods=myserver/net-test1,defafut/net-test1' \
-- namespace velero-system --include-namespaces=myserver,defafut

posted @ 2022-12-06 21:21 YIDADA-SRE 阅读(1121) 评论(0) 编辑收藏举报

刷新页面返回顶部

登录后才能查看或发表评论，立即登录或者逛逛博客园首页

相关博文：

· Etcd 备份与恢复

· Secret

· 基于velero及minio实现etcd数据备份与恢复

· 10、基于velero对etcd的备份和恢复

· 使用Velero备份、恢复、迁移Kubernetes集群

阅读排行：
· DeepSeek 开源周回顾「GitHub 热点速览」
· 物流快递公司核心技术能力-地址解析分单基础技术分享
· .NET 10首个预览版发布：重大改进与新特性概览！
· AI与.NET技术实操系列（二）：开始使用ML.NET
· 单线程的Redis速度为什么快？

阅读目录(Content)

此页目录为空

YIDADA-SRE

念两句诗

Velero云原生的灾难恢复和迁移工具

18. Velero

18.1 Velero概述

18.2 Velero与etcd快照备份的区别

18.3 Velero备份流程

18.4 部署minio 私有共享存储

18.5 部署Velero

18.6 配置velero认证环境

18.7 准备user-csr文件

18.8 准备证书签发环境

18.9 执行证书签发

18.10 生成集群认证config文件

18.11 创建备份

18.12 恢复数据

18.13 备份脚本

18.14 备份指定资源对象

公告

个人信息

日历

搜索

常用链接

随笔分类

随笔档案

相册

阅读排行榜

评论排行榜

最新评论

	1. Velero 是vmware开源的一个云原生的灾难恢复和迁移工具，它本身也是开源的，采用GO语言编写，可以安装的备份、恢复和迁移Kubernetes集群资源数据
	2. Velero是西班牙语意思是帆船，非常符合Kubernetes社区的命名风格，Velero的开发公司Heptio,已被VMware收购

	3. Velero支持标准的K8S集群，既可以是私有云平台也可以是公有云，除了灾难之外它还能做资源转移，支持把容器应用从一个集群迁移到另一个集群

	4. Velero的工作方式就是把kubernetes中的数据备份到对象存储以实现高可用的持久化，默认的备份保存时间为720小时，并在需要的时候进行下载和恢复

	1. etcd快照是全局备份，在即使一个资源对象需要恢复，也是需要做全局恢复到备份的状态，即会影响其它namespace中pod运行服务

	2. Velero可以有针对性的备份，比如按照namespace单独备份、只备份单独的资源对象等，在恢复的时候只恢复单独的namespace或资源对象，而不影响其它namespace中pod运行服务

	3. Velero支持ceph、oss等对象存储，etcd快照是一个为本地文件

	4. Velero支持任务计划实现周期备份，但etcd快照也可以基于cronjob实现

	Velero客户端调用Kubernetes API Server创建Backup任务
	Backuup 控制器基于watch机制通过API Server获取到备份任务
	Backup 控制器开始执行备份动作，其会通过请求API Server获取需要备份的数据
	Backup 控制器将获取到的数据备份到指定的对象存储server端

	docker run --name minio \
	-p 9000:9000 \
	-p 9999:9999 \
	-d --restart=always \
	-e "MINIO_ROOT_USER=admin" \
	-e "MINIO_ROOT_PASSWORD=12345678" \
	-v /data/minio/data:/data \
	minio/minio server /data \
	--console-address '0.0.0.0:9999'

	root@deploy-harbor:/# wget https://github.com/vmware-tanzu/velero/releases/download/v1.8.1/velero-v1.8.1-linux-amd64.tar.gz
	root@deploy-harbor:/velero# tar xvf velero-v1.8.1-linux-amd64.tar.gz -C /velero
	root@deploy-harbor:/velero# cp velero-v1.8.1-linux-amd64/velero /usr/local/bin/
	root@deploy-harbor:/velero# velero --help

	#工作目录：
	root@deploy-harbor:~# mkdir /velero -p

	#认证文件：
	root@k8s-master1:/velero# vim velero-auth.txt
	#aws用户这个key不能变value可以变是minio的账号密码
	[default]
	aws_access_key_id = admin
	aws_secret_access_key = 12345678

	root@deploy-harbor:/velero# vim awsuser-csr.json
	{
	"CN": "awsuser",
	"hosts": [],
	"key": {
	"algo": "rsa",
	"size": 2048
	},
	"names": [
	{
	"C": "CN",
	"ST": "BeiJing",
	"L": "BeiJing",
	"O": "k8s",
	"OU": "System"
	}
	]
	}

	root@deploy-harbor:/velero# apt install golang-cfssl

	#方法一下载改名使用
	root@deploy-harbor:/velero# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64
	root@deploy-harbor:/velero# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64
	root@deploy-harbor:/velero# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64

	root@deploy-harbor:/velero# mv cfssl-certinfo_1.6.1_linux_amd64 cfssl-certinfo
	root@deploy-harbor:/velero# mv cfssl_1.6.1_linux_amd64 cfssl
	root@deploy-harbor:/velero# mv cfssljson_1.6.1_linux_amd64 cfssljson

	root@deploy-harbor:/velero# cp cfssl* /usr/local/bin/
	root@deploy-harbor:/velero# chmod a+x /usr/local/bin/cfssl*

	#方法二使用/etc/kubeasz
	/etc/kubeasz/bin/提供了这些命令拷贝即可
	root@deploy-harbor:/velero# cp /etc/kubeasz/bin/cfssl* /usr/local/bin
	root@deploy-harbor:/velero# chmod a+x /usr/local/bin/cfssl*
	root@deploy-harbor:/velero# cp /etc/kubeasz/clusters/k8s-cluster1/ssl/ca-config.json /velero

	root@deploy-harbor:/velero# ll
	total 20
	drwxr-xr-x 2 root root 4096 Nov 23 13:14 ./
	drwxr-xr-x 24 root root 4096 Nov 23 12:24 ../
	#证书签发文件
	-rw-r--r-- 1 root root 220 Nov 23 12:25 awsuser-csr.json
	#ca证书配置文件
	-rw-r--r-- 1 root root 483 Nov 23 13:13 ca-config.json
	#minio认证文件
	-rw-r--r-- 1 root root 89 Nov 23 12:25 velero-auth.txt

	#源执行命令
	1.24.x:
	root@deploy-harbor:/velero# /usr/local/bin/cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=/etc/kubeasz/clusters/k8s-cluster1/ssl/ca-config.json -profile=kubernetes ./awsuser-csr.json \| cfssljson -bare awsuser

	1.23.x:


	#添加了修改后就改了一下路径
	root@deploy-harbor:/velero# /usr/local/bin/cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=./ca-config.json -profile=kubernetes ./awsuser-csr.json \| cfssljson -bare awsuser
	2022/11/23 13:40:18 [INFO] generate received request
	2022/11/23 13:40:18 [INFO] received CSR
	2022/11/23 13:40:18 [INFO] generating key: rsa-2048
	2022/11/23 13:40:18 [INFO] encoded CSR
	2022/11/23 13:40:18 [INFO] signed certificate with serial number 204902735149897151390216548318080805156194245414
	2022/11/23 13:40:18 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
	websites. For more information see the Baseline Requirements for the Issuance and Management
	of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
	specifically, section 10.2.3 ("Information Requirements").

	#生成的证书文件
	root@deploy-harbor:/velero# ll
	total 32
	drwxr-xr-x 2 root root 4096 Nov 23 13:40 ./
	drwxr-xr-x 21 root root 4096 Nov 23 13:23 ../
	-rw-r--r-- 1 root root 997 Nov 23 13:40 awsuser.csr
	-rw-r--r-- 1 root root 220 Nov 23 13:23 awsuser-csr.json
	#生成api-server证书路径
	-rw------- 1 root root 1679 Nov 23 13:40 awsuser-key.pem
	#生成api-server证书路径
	-rw-r--r-- 1 root root 1387 Nov 23 13:40 awsuser.pem
	-rw-r--r-- 1 root root 483 Nov 23 13:25 ca-config.json
	-rw-r--r-- 1 root root 89 Nov 23 13:23 velero-auth.txt

	#分发证书到api-server证书路径：
	root@deploy-harbor:/velero# cp awsuser-key.pem /etc/kubernetes/ssl/
	root@deploy-harbor:/velero# cp awsuser.pem /etc/kubernetes/ssl/

	#这个IP地址可以是VIP 我这里就指定了master的IP地址了
	root@deploy-harbor:/velero# export KUBE_APISERVER="https://192.168.1.70:6443"
	root@deploy-harbor:/velero# kubectl config set-cluster kubernetes \
	--certificate-authority=/etc/kubernetes/ssl/ca.pem \
	--embed-certs=true \
	--server=${KUBE_APISERVER} \
	--kubeconfig=./awsuser.kubeconfig

	#输出
	Cluster "kubernetes" set.

	root@deploy-harbor:/velero# ll
	total 36
	drwxr-xr-x 2 root root 4096 Nov 23 13:47 ./
	drwxr-xr-x 21 root root 4096 Nov 23 13:23 ../
	-rw-r--r-- 1 root root 997 Nov 23 13:40 awsuser.csr
	-rw-r--r-- 1 root root 220 Nov 23 13:23 awsuser-csr.json
	-rw------- 1 root root 1679 Nov 23 13:40 awsuser-key.pem
	#会生成这个证书文件
	-rw------- 1 root root 1938 Nov 23 13:47 awsuser.kubeconfig
	-rw-r--r-- 1 root root 1387 Nov 23 13:40 awsuser.pem
	-rw-r--r-- 1 root root 483 Nov 23 13:25 ca-config.json
	-rw-r--r-- 1 root root 89 Nov 23 13:23 velero-auth.txt

	root@deploy-harbor:/velero# cat awsuser.kubeconfig
	apiVersion: v1
	clusters:
	- cluster:
	certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURsRENDQW55Z0F3SUJBZ0lVTUV2SXV3UXdTL3lDMXFyUlFpZ1duVjYwRXNJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lURUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEV6QVJCZ05WQkFNVENtdDFZbVZ5CmJtVjBaWE13SUJjTk1qSXhNVEU1TURZME1qQXdXaGdQTWpFeU1qRXdNall3TmpReU1EQmFNR0V4Q3pBSkJnTlYKQkFZVEFrTk9NUkV3RHdZRFZRUUlFd2hJWVc1bldtaHZkVEVMTUFrR0ExVUVCeE1DV0ZNeEREQUtCZ05WQkFvVApBMnM0Y3pFUE1BMEdBMVVFQ3hNR1UzbHpkR1Z0TVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXQ1QkZtZkh4KzhCQXQ1OTg2QnBjV2Z0Q3VYcVoKMzlmMFBNaGhDTXNGVWtjYThjNzFxV3R0b0g4Mmk4UEpxTzdRL1FRbWF0MzdmOEdSZmkxNXd0Z0xBYUtmUXlRdgpGYlZvaks0NDhhNHRtSENtbk0zU1dvWStTU2xmbHMrYXM2cEc4SGVnSmE4WVFvR0d5bmtsb2F3dG0wVEVNcmtPCnd5MG5lbDRMcnl0TTF4QnhXek4wbGsrRlhId1o2NEkrR3JvQ2dvSm91RGtaMTNmcENycjlHUGpRelBpNE1CeVQKeXVaREFLMEE1V3FEcVhxSy9kRVJPb09yVXRoK3ROSHJCNTFqZ0QzQlJFVnZSLytBcG52VFRBM212Ym1UQVdudwo2SGVZZkdtSkpNWDRNZVBCYUIzYjVXa044cWJqckhVVXhXSmZPZG96UG1iQ3FRRmplcWRNVDh4VWZ3SURBUUFCCm8wSXdRREFPQmdOVkhROEJBZjhFQkFNQ0FRWXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVUKYk54STNOS05GeERBa1o0eUJWR1RSMmdOL2ZBd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMSTdNaUxZN0ZQawpzQ0E2b3hBVkxMZWM0Ry9lQ0wxZUJjNVVkcUdDY2RYbXQ0U2RFWTE3WWRKZ2huV1pqNGRSOTVSVFBscWtPS3hYCkI5eCt6SzZYWVVaSlo4aUEvSUtFcHlFMHZYRlBjcmZPaWdzZDdKaEhiUDFuRFpGcEErNy9XaU1DUXlwNXNXRngKTzdmYjdwVko1bnl5RHZEdHNCY0VFYW5NUHhmKzNwZG83NHFBdWowYjNsNHJyU1RjOTdOS0xyekwyQnppaEI3UQpvaGpiSHpuR1lTQkdRU2IvOW5ZU24rRzBVTG50SlFCRU8xbGdTd0RuMzJNNHd4aThUODJ5bldmYUNHaUpMOWg5CkF2U05XRVVhZmF6UDFPVGFydng3N0xnbnA4SUtGejZ2UzJSVUZJSEl4cTJrQnl4SkE4NzhTMnRTWGROWC9SSlIKam9aaXFuRWZxR1U9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
	server: https://192.168.1.70:6443
	name: kubernetes
	contexts: null
	current-context: ""
	kind: Config
	preferences: {}
	#这里目前是空的用户认证
	users: null

	root@deploy-harbor:/velero# kubectl config set-credentials awsuser \
	--client-certificate=/etc/kubernetes/ssl/awsuser.pem \
	--client-key=/etc/kubernetes/ssl/awsuser-key.pem \
	--embed-certs=true \
	--kubeconfig=./awsuser.kubeconfig

	运行上面的认证这个用户信息就会生成了生成的是awsuser
	私钥：certificate-authority-data
	公钥：client-certificate-data

	root@deploy-harbor:/velero# cat awsuser.kubeconfig
	apiVersion: v1
	clusters:
	- cluster:
	certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURsRENDQW55Z0F3SUJBZ0lVTUV2SXV3UXdTL3lDMXFyUlFpZ1duVjYwRXNJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lURUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEV6QVJCZ05WQkFNVENtdDFZbVZ5CmJtVjBaWE13SUJjTk1qSXhNVEU1TURZME1qQXdXaGdQTWpFeU1qRXdNall3TmpReU1EQmFNR0V4Q3pBSkJnTlYKQkFZVEFrTk9NUkV3RHdZRFZRUUlFd2hJWVc1bldtaHZkVEVMTUFrR0ExVUVCeE1DV0ZNeEREQUtCZ05WQkFvVApBMnM0Y3pFUE1BMEdBMVVFQ3hNR1UzbHpkR1Z0TVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXQ1QkZtZkh4KzhCQXQ1OTg2QnBjV2Z0Q3VYcVoKMzlmMFBNaGhDTXNGVWtjYThjNzFxV3R0b0g4Mmk4UEpxTzdRL1FRbWF0MzdmOEdSZmkxNXd0Z0xBYUtmUXlRdgpGYlZvaks0NDhhNHRtSENtbk0zU1dvWStTU2xmbHMrYXM2cEc4SGVnSmE4WVFvR0d5bmtsb2F3dG0wVEVNcmtPCnd5MG5lbDRMcnl0TTF4QnhXek4wbGsrRlhId1o2NEkrR3JvQ2dvSm91RGtaMTNmcENycjlHUGpRelBpNE1CeVQKeXVaREFLMEE1V3FEcVhxSy9kRVJPb09yVXRoK3ROSHJCNTFqZ0QzQlJFVnZSLytBcG52VFRBM212Ym1UQVdudwo2SGVZZkdtSkpNWDRNZVBCYUIzYjVXa044cWJqckhVVXhXSmZPZG96UG1iQ3FRRmplcWRNVDh4VWZ3SURBUUFCCm8wSXdRREFPQmdOVkhROEJBZjhFQkFNQ0FRWXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVUKYk54STNOS05GeERBa1o0eUJWR1RSMmdOL2ZBd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMSTdNaUxZN0ZQawpzQ0E2b3hBVkxMZWM0Ry9lQ0wxZUJjNVVkcUdDY2RYbXQ0U2RFWTE3WWRKZ2huV1pqNGRSOTVSVFBscWtPS3hYCkI5eCt6SzZYWVVaSlo4aUEvSUtFcHlFMHZYRlBjcmZPaWdzZDdKaEhiUDFuRFpGcEErNy9XaU1DUXlwNXNXRngKTzdmYjdwVko1bnl5RHZEdHNCY0VFYW5NUHhmKzNwZG83NHFBdWowYjNsNHJyU1RjOTdOS0xyekwyQnppaEI3UQpvaGpiSHpuR1lTQkdRU2IvOW5ZU24rRzBVTG50SlFCRU8xbGdTd0RuMzJNNHd4aThUODJ5bldmYUNHaUpMOWg5CkF2U05XRVVhZmF6UDFPVGFydng3N0xnbnA4SUtGejZ2UzJSVUZJSEl4cTJrQnl4SkE4NzhTMnRTWGROWC9SSlIKam9aaXFuRWZxR1U9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
	server: https://192.168.1.70:6443
	name: kubernetes
	contexts: null
	current-context: ""
	kind: Config
	preferences: {}
	users:
	- name: awsuser
	user:
	client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQwakNDQXJxZ0F3SUJBZ0lVSStRb0N4ZURaTTdwQTZta0xOcGc1Z3V5V3lZd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lURUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEV6QVJCZ05WQkFNVENtdDFZbVZ5CmJtVjBaWE13SUJjTk1qSXhNVEl6TVRNek5UQXdXaGdQTWpBM01qRXhNVEF4TXpNMU1EQmFNR0l4Q3pBSkJnTlYKQkFZVEFrTk9NUkF3RGdZRFZRUUlFd2RDWldsS2FXNW5NUkF3RGdZRFZRUUhFd2RDWldsS2FXNW5NUXd3Q2dZRApWUVFLRXdOck9ITXhEekFOQmdOVkJBc1RCbE41YzNSbGJURVFNQTRHQTFVRUF4TUhZWGR6ZFhObGNqQ0NBU0l3CkRRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFOODNKRE5JOEtZOGs3cjlPWVhja2JWd1pPZGsKNHZ6eFB2eE1XM0poaXg0TlNaYjlTVzhreHhjRjFsYUZIQ0dSei9TdEZwTDgvTzduR1d5RlJBVXE1K2hqRmJzdgpnOWEydTN3bmtLK3NWUGVUNlVQcGVWY0ROOUVTcFljRk5ERGd5VnQ5WGZxeUVxNkVzcU9DSVNuUXlLTzNFSURBCkdaUXFzaDBjRGYyMXU0V2JjaHBwSjJWeHRIQmhEQzdOSGROK0pmeC9ZeGU5VThSQTVCOHZQZ2g3S2tJcGtQb3MKN2Fob20ydGlwT3pkWUVTODkxZkFJekthb1BpZ0ppUWZBZGVXMUJldGl2akFZK2Z2MEJvRlNFbU5HVWs1c3kyNwo1TnlRU2RuVWJlcHk5SjY4N2hpdThlWE9aaXdsaDcxdVZaM2h0V21hS2RkT20rYlR6cmpPWUxWWER0a0NBd0VBCkFhTi9NSDB3RGdZRFZSMFBBUUgvQkFRREFnV2dNQjBHQTFVZEpRUVdNQlFHQ0NzR0FRVUZCd01CQmdnckJnRUYKQlFjREFqQU1CZ05WSFJNQkFmOEVBakFBTUIwR0ExVWREZ1FXQkJTVWhHR2hGVzNDb2tISnZrNUlFbi9kUGZURQpaREFmQmdOVkhTTUVHREFXZ0JSczNFamMwbzBYRU1DUm5qSUZVWk5IYUEzOThEQU5CZ2txaGtpRzl3MEJBUXNGCkFBT0NBUUVBV3ZMbWlid1NXV0IzYkcrejJqMi9UajY3MkxDc2xsK2JONGw4SzhVdFZyS2lVbTBzakZkaEFrRloKZHFoWi9UQXdlaDJ4U2NXUytaNU5pTkdiUk5IRFdmVXcyeTR3QldnWFFNVktEaUdZcW5iWGFYQmV4dmxpWVRycwp6aC9wS0o3UThxUjJuTVppd0NWTi90dDg2SktOaWU3Zm0rVjdSVms5WUdaZUZxOHFuWFptd09hYnlPV2FkYmY3ClVMSHM5M0dOT1B6OGF3RGNsbkxIaTZIT05YamVlQ1BXVVV5RDFQbTViVzc0ei92Rm5PQm1tdUpyZnlmYUczRE8Kdk9IOU5qellIb3RSWncwNGVQY2Zza2hNSjRwNzJuRXVmaktkdHVETDBocGZtZ0NoWk5nUGdoRkhVcjI0QnFMOQpHbTdqbDRxSGxTNWFoL2E5TXcyekJPMk9nckZtcXc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
	client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBM3pja00wandwanlUdXYwNWhkeVJ0WEJrNTJUaS9QRSsvRXhiY21HTEhnMUpsdjFKCmJ5VEhGd1hXVm9VY0laSFA5SzBXa3Z6ODd1Y1piSVZFQlNybjZHTVZ1eStEMXJhN2ZDZVFyNnhVOTVQcFErbDUKVndNMzBSS2xod1UwTU9ESlczMWQrcklTcm9TeW80SWhLZERJbzdjUWdNQVpsQ3F5SFJ3Ti9iVzdoWnR5R21rbgpaWEcwY0dFTUxzMGQwMzRsL0g5akY3MVR4RURrSHk4K0NIc3FRaW1RK2l6dHFHaWJhMktrN04xZ1JMejNWOEFqCk1wcWcrS0FtSkI4QjE1YlVGNjJLK01CajUrL1FHZ1ZJU1kwWlNUbXpMYnZrM0pCSjJkUnQ2bkwwbnJ6dUdLN3gKNWM1bUxDV0h2VzVWbmVHMWFab3AxMDZiNXRQT3VNNWd0VmNPMlFJREFRQUJBb0lCQVFDYmFBeXdHQXhUOE1ENQo0dXQxbzBkZkIwMTVQT2c4S3NvclpnNFU2SXl1ODVsVW9pdktVZFM2MXJtUHEwVVdxaW9hczVlUDVzdENtV3M5Ci90cUlyb0tmYkx6b2dnQk1NUlQvTDV5d2NrOEZ2OWtjQ3lVTHk1WEUzaktZZXFzSGpMa2tGWGlrM2UzYnkwK0UKUDBsUGo2amhNc2N1bkpBeWREZGx4TzZJd1VvNWpkOVJZTXlhY3dyV2FpTm1QZzB6YjVDVWtxYS9PUGpVamFOMQpKZUJMbGYvd0pnUnZxcXdCVHUwMzBIWGsrV1pQNk1HNTNQak0vd3RHVTZVV2hteWgwZGNmZ1c5MTRtaDdjcG5oCmlENkJaY0k1UlVGM1lSUFFBMVpnbnJSUzU5VThibHZFWG1zcTR3cDJDUFk4cUZFdEt6VHp3WkV2bkdoekNrRVIKNUM0REVaZ0JBb0dCQVBLYlpyQy9mRWxKdEp5QXFMSmVIWHV1a1RiRnJFbFhKbU0zMzRNRVVYc245QlJta210OQpPODdmSnVJemdXQjliV0xJcHJHV2czK2JsVTViejg1ZFRqSjFuUkZWZlRlM3hicXRuT3dZSVpmTXhHTzFkU0pCCmp3NzJyNlNwbmllVUYvd3ZlM1BKRE1BVDYvTjBXMmpvbzdMcTZEdEU2WU8raXR4TmQ5amVNL2ZCQW9HQkFPdUoKc1FnT2VZZU9EMS9iTXNrMjN3OS9DZDd0eGZMRGJPbHNxQTQ1YlpqbEU2OW1LYXVOYndNaWVjSmhIcmFLc3dOagpHT1BseXJ3cHhJTUhKcFhMMlFyczBSVE8yMytCbHlWbTFIeTVxb0R1T0xOaE1QN2hLaFdMSG1LQXlvV0VMQVpBCk9HajhtMG1KU3ZLMnZFT3ZyS09Ba09uMjdYK3FKTXFzVFRlejh4MFpBb0dCQU5keVdnZXNoVnN3TVV5cWxTV1IKbnUwdllaWWFoWjI3MzBOTll4aWV4UjF4cGJoWVByOW1QNjYvVFhDai9Na0xtaENYMEREMVVXSUpjTlpkZ25YcApOb0JwSWtzNmVERDY2b1dWbUQyeVVHTFdYYkdPZTY4b09UczRwMUs3cnMvSEhHWEFaUW0yQ2JTcC9HZi9kUTM5ClN5S2dxZ3U0Yndlb052amRjd2tNaTJvQkFvR0FmbkZtZGZxL1J3UCtRT2s5ZEVOZXI4VDFHWlh2aEFsaE9GWTYKU1ppT3UxdU9tOFJ1YmtmNDVtZmxrWEh0dVBUd2NNc01HNXZLb0FTVUYvc0l2YlczQ0dSbFFaUml4U3BlWjVleQpUbHFscTUwM0Z5VC8xblF0MFc3am11R2sxdFJEaGYrSHlPU0N3SkZSaU1hTWZwR2FUaFBBdDdqMWVtbHVZdGJvCmhSUzNXRkVDZ1lBZ1BWRE1hdnVPaHNwc01XRUJWOFFHZ2dRQ0kwUzVhWUpidWl5UXE2a0tnS0IvU0doNE9tSlQKK2V2Nk9WYVY1NHplRmFVRDRQeUs2OU53RFdJK3Y5cGk5ZDFPRkM2Ym9iT2VMcm12SGhSdERJOVdseVZxTklLNQpuNXZORTMxVlUrVGVMK2lIc3lvNWE2YWZPTVNkL1VPQjJxaXVidFlEai9JcWlETGRrWURLOXc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

	root@deploy-harbor:/velero# kubectl config set-context kubernetes \
	--cluster=kubernetes \
	--user=awsuser \
	--namespace=velero-system \
	--kubeconfig=./awsuser.kubeconfig
	Context "kubernetes" created.

	#设置默认上下文:
	root@deploy-harbor:/velero# kubectl config use-context kubernetes --kubeconfig=awsuser.kubeconfig

	#--bucket 执行minio存储桶的名字写错的话他会找不到
	#--secret-file 指定密码文件
	#--namespace velero-system如果你需要不同的备份项目中你就在起一个名字aliyun-system 然后http地址在指定过去
	root@deploy-harbor:/velero# velero --kubeconfig ./awsuser.kubeconfig \
	install \
	--provider aws \
	--plugins velero/velero-plugin-for-aws:v1.3.1 \
	--bucket velerodata \
	--secret-file ./velero-auth.txt \
	--use-volume-snapshots=false \
	--namespace velero-system \
	--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.1.75:9000

	所用的镜像 # velero/velero-plugin-for-aws:v1.3.1
	root@deploy-harbor:/velero# kubectl describe pod velero-6755cb8697-phfsr -n velero-system

	root@deploy-harbor:/velero# kubectl get pods -A
	NAMESPACE NAME READY STATUS RESTARTS AGE
	kube-system calico-kube-controllers-68555f5f97-mww92 1/1 Running 0 31h
	kube-system calico-node-hxbbn 0/1 Running 101 (4m53s ago) 8h
	kube-system calico-node-m7xwb 0/1 CrashLoopBackOff 101 (4m33s ago) 8h
	kubernetes-dashboard dashboard-metrics-scraper-6dfbcf7959-55bkt 1/1 Running 4 (32h ago) 32h
	kubernetes-dashboard kubernetes-dashboard-85997c7f79-64sm8 1/1 Running 5 (32h ago) 32h
	myserver coredns-57d94f5d84-bjx7d 1/1 Running 0 8h
	myserver coredns-57d94f5d84-zbt5x 1/1 Running 0 8h
	myserver linux60-tomcat-app1-deployment-595f7ff67c-494xk 1/1 Running 0 7h55m
	myserver linux70-nginx-deployment-55dc5fdcf9-h8kp2 1/1 Running 0 7h55m
	myserver net-test1 1/1 Running 1 (33h ago) 2d6h
	myserver net-test2 1/1 Running 1 (33h ago) 2d7h
	myserver net-test3 1/1 Running 1 (33h ago) 2d5h
	myserver net-test4 1/1 Running 1 (33h ago) 36h
	myserver net-test5 1/1 Running 1 (33h ago) 35h
	velero-system velero-858b9459f9-nvbhc 1/1 Running 2 (20m ago) 23m


	#velero backup --help
	#--include-namespaces 指定名称空间备份

	DATE=`date +%Y%m%d%H%M%S`
	velero backup create myserver-ns-backup-${DATE} \
	--include-namespaces myserver \
	--kubeconfig=./awsuser.kubeconfig \
	--namespace velero-system

	#我删除了整个myserver名称空间下的数据看看是否能恢复回来
	root@deploy-harbor:/velero# kubectl delete namespaces myserver

	#--kubeconfig 这个是指定认证文件也可以不使用awsuser 使用/root/.kube/config
	# --wait 加上这个wait就会同步的模式不加就是异步异步就是请求发给api-server就完成了同步就是必须执行完成后才可以退出
	#--namespace velero-system 这个参数不要改这个是指向velero所使用的

	velero restore create --from-backup myserver-ns-backup-20221123150613 --wait \
	--kubeconfig=./awsuser.kubeconfig \
	--namespace velero-system

	#区别在于没有指定固定的--include-namespaces
	#myserver-ns-backup这个名字是自己起的
	DATE=`date +%Y%m%d%H%M%S`
	velero backup create myserver-ns-backup-${DATE} \
	--kubeconfig=./awsuser.kubeconfig \
	--namespace velero-system

	velero restore create --from-backup myserver-ns-backup-20221123153523 --wait \
	--kubeconfig=/root/.kube/config \
	--include-cluster-resources=true \
	--include-namespaces default \
	--namespace velero-system

	velero restore create --from-backup myserver-ns-backup-20221123153523 --wait \
	--kubeconfig=/root/.kube/config \
	--include-cluster-resources=true \
	--include-namespaces myserver \
	--namespace velero-system

	root@deploy-harbor:/velero# vim ns-backup.sh
	#!/bin/bash
	NS_NAME=`kubectl get ns \| awk '{if (NR>1){print}}' \| awk '{print $1}'`
	DATE=`date +%Y%m%d%H%M%S`
	cd /velero/

	for i in $NS_NAME;do
	velero backup create ${i}-ns-backup-${DATE} \
	--kubeconfig=/root/.kube/config \
	--include-cluster-resources=true \
	--include-namespaces ${i} \
	--namespace velero-system
	done

	velero backup create pod-backup-xxx \
	include-cluster-resources=true \
	--ordered-resources \
	'pods=myserver/net-test1,defafut/net-test1' \
	-- namespace velero-system --include-namespaces=myserver,defafut