1、Velero简介

Velero 是vmware开源的一个云原生的灾难恢复和迁移工具,它本身也是开源的,采用Go语言编写,可以安全的备份、恢复和迁移Kubernetes集群资源数据;官网。Velero 是西班牙语意思是帆船,非常符合Kubernetes社区的命名风格,Velero的开发公司Heptio,已被VMware收购。Velero 支持标准的K8S集群,既可以是私有云平台也可以是公有云,除了灾备之外它还能做资源移转,支持把容器应用从一个集群迁移到另一个集群。Velero 的工作方式就是把kubernetes中的数据备份到对象存储以实现高可用和持久化,默认的备份保存时间为720小时,并在需要的时候进行下载和恢复。

2、Velero与etcd快照备份的区别

  • etcd 快照是全局完成备份(类似于MySQL全部备份),即使需要恢复一个资源对象(类似于只恢复MySQL的一个库),但是也需要做全局恢复到备份的状态(类似于MySQL的全库恢复),即会影响其它namespace中pod运行服务(类似于会影响MySQL其它数据库的数据)。
  • Velero可以有针对性的备份,比如按照namespace单独备份、只备份单独的资源对象等,在恢复的时候可以根据备份只恢复单独的namespace或资源对象,而不影响其它namespace中pod运行服务。
  • velero支持ceph、oss等对象存储,etcd 快照是一个为本地文件。
  • velero支持任务计划实现周期备份,但etcd 快照也可以基于cronjob实现。
  • velero支持对AWS EBS创建快照及还原

3、velero整体架构

基于velero及minio实现etcd数据备份与恢复-LMLPHP

4、velero备份流程

基于velero及minio实现etcd数据备份与恢复-LMLPHP

5、对象存储minio部署

5.1、创建数据目录

root@harbor:~# mkdir -p /data/minio

5.2、创建minio容器

下载镜像

root@harbor:~# docker pull minio/minio:RELEASE.2023-08-31T15-31-16Z
RELEASE.2023-08-31T15-31-16Z: Pulling from minio/minio
0c10cd59e10e: Pull complete 
b55c0ddd1333: Pull complete 
4aade59ba7c6: Pull complete 
7c45df1e40d6: Pull complete 
adedf83b12e0: Pull complete 
bc9f33183b0c: Pull complete 
Digest: sha256:76868af456548aab229762d726271b0bf8604a500416b3e9bdcb576940742cda
Status: Downloaded newer image for minio/minio:RELEASE.2023-08-31T15-31-16Z
docker.io/minio/minio:RELEASE.2023-08-31T15-31-16Z
root@harbor:~# 

创建minio容器

root@harbor:~# docker run --name minio \
> -p 9000:9000 \
> -p 9999:9999 \
> -d --restart=always \
> -e "MINIO_ROOT_USER=admin" \
> -e "MINIO_ROOT_PASSWORD=12345678" \
> -v /data/minio/data:/data \
> minio/minio:RELEASE.2023-08-31T15-31-16Z server /data \
> --console-address '0.0.0.0:9999'
ba5e511da5f30a17614d719979e28066788ca7520d87c67077a38389e70423f1
root@harbor:~# 

5.3、minio web界面登录

基于velero及minio实现etcd数据备份与恢复-LMLPHP
基于velero及minio实现etcd数据备份与恢复-LMLPHP

5.4、minio 创建bucket

基于velero及minio实现etcd数据备份与恢复-LMLPHP

5.5、minio 验证bucket

基于velero及minio实现etcd数据备份与恢复-LMLPHP

6、在master节点部署velero

6.1、下载velero客户端工具

下载velero客户端工具

root@k8s-master01:/usr/local/src# wget https://github.com/vmware-tanzu/velero/releases/download/v1.11.1/velero-v1.11.1-linux-amd64.tar.gz

解压压缩包

root@k8s-master01:/usr/local/src# ll
total 99344
drwxr-xr-x  3 root root     4096 Sep  2 12:38 ./
drwxr-xr-x 10 root root     4096 Feb 17  2023 ../
drwxr-xr-x  2 root root     4096 Oct 21  2015 bin/
-rw-r--r--  1 root root 64845365 May 31 13:21 buildkit-v0.11.6.linux-amd64.tar.gz
-rw-r--r--  1 root root 36864459 Sep  2 12:31 velero-v1.11.1-linux-amd64.tar.gz
root@k8s-master01:/usr/local/src# tar xf velero-v1.11.1-linux-amd64.tar.gz 
root@k8s-master01:/usr/local/src# ll
total 99348
drwxr-xr-x  4 root root     4096 Sep  2 12:39 ./
drwxr-xr-x 10 root root     4096 Feb 17  2023 ../
drwxr-xr-x  2 root root     4096 Oct 21  2015 bin/
-rw-r--r--  1 root root 64845365 May 31 13:21 buildkit-v0.11.6.linux-amd64.tar.gz
drwxr-xr-x  3 root root     4096 Sep  2 12:39 velero-v1.11.1-linux-amd64/
-rw-r--r--  1 root root 36864459 Sep  2 12:31 velero-v1.11.1-linux-amd64.tar.gz
root@k8s-master01:/usr/local/src# 

将velero二进制文件移动至/usr/local/bin

root@k8s-master01:/usr/local/src# ll velero-v1.11.1-linux-amd64
total 83780
drwxr-xr-x 3 root root     4096 Sep  2 12:39 ./
drwxr-xr-x 4 root root     4096 Sep  2 12:39 ../
-rw-r--r-- 1 root root    10255 Dec 13  2022 LICENSE
drwxr-xr-x 4 root root     4096 Sep  2 12:39 examples/
-rwxr-xr-x 1 root root 85765416 Jul 25 08:43 velero*
root@k8s-master01:/usr/local/src# cp velero-v1.11.1-linux-amd64/velero /usr/local/bin/
root@k8s-master01:/usr/local/src# 

验证velero命令是否可执行?

root@k8s-master01:/usr/local/src# velero --help
Velero is a tool for managing disaster recovery, specifically for Kubernetes
cluster resources. It provides a simple, configurable, and operationally robust
way to back up your application state and associated data.

If you're familiar with kubectl, Velero supports a similar model, allowing you to
execute commands such as 'velero get backup' and 'velero create schedule'. The same
operations can also be performed as 'velero backup get' and 'velero schedule create'.

Usage:
  velero [command]

Available Commands:
  backup            Work with backups
  backup-location   Work with backup storage locations
  bug               Report a Velero bug
  client            Velero client related commands
  completion        Generate completion script
  create            Create velero resources
  debug             Generate debug bundle
  delete            Delete velero resources
  describe          Describe velero resources
  get               Get velero resources
  help              Help about any command
  install           Install Velero
  plugin            Work with plugins
  repo              Work with repositories
  restore           Work with restores
  schedule          Work with schedules
  snapshot-location Work with snapshot locations
  uninstall         Uninstall Velero
  version           Print the velero version and associated image

Flags:
      --add_dir_header                   If true, adds the file directory to the header of the log messages
      --alsologtostderr                  log to standard error as well as files (no effect when -logtostderr=true)
      --colorized optionalBool           Show colored output in TTY. Overrides 'colorized' value from $HOME/.config/velero/config.json if present. Enabled by default
      --features stringArray             Comma-separated list of features to enable for this Velero process. Combines with values from $HOME/.config/velero/config.json if present
  -h, --help                             help for velero
      --kubeconfig string                Path to the kubeconfig file to use to talk to the Kubernetes apiserver. If unset, try the environment variable KUBECONFIG, as well as in-cluster configuration
      --kubecontext string               The context to use to talk to the Kubernetes apiserver. If unset defaults to whatever your current-context is (kubectl config current-context)
      --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                   If non-empty, write log files in this directory (no effect when -logtostderr=true)
      --log_file string                  If non-empty, use this log file (no effect when -logtostderr=true)
      --log_file_max_size uint           Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --logtostderr                      log to standard error instead of files (default true)
  -n, --namespace string                 The namespace in which Velero should operate (default "velero")
      --one_output                       If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)
      --skip_headers                     If true, avoid header prefixes in the log messages
      --skip_log_headers                 If true, avoid headers when opening log files (no effect when -logtostderr=true)
      --stderrthreshold severity         logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=false) (default 2)
  -v, --v Level                          number for the log level verbosity
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging

Use "velero [command] --help" for more information about a command.
root@k8s-master01:/usr/local/src# 

6.2、配置velero认证环境

6.2.1、创建velero工作目录

root@k8s-master01:/usr/local/src# mkdir  /data/velero -p
root@k8s-master01:/usr/local/src# cd /data/velero/
root@k8s-master01:/data/velero# ll
total 8
drwxr-xr-x 2 root root 4096 Sep  2 12:42 ./
drwxr-xr-x 3 root root 4096 Sep  2 12:42 ../
root@k8s-master01:/data/velero# 

6.2.2、创建访问minio的认证文件

root@k8s-master01:/data/velero# ll
total 12
drwxr-xr-x 2 root root 4096 Sep  2 12:43 ./
drwxr-xr-x 3 root root 4096 Sep  2 12:42 ../
-rw-r--r-- 1 root root   69 Sep  2 12:43 velero-auth.txt
root@k8s-master01:/data/velero# cat velero-auth.txt 
[default]
aws_access_key_id = admin
aws_secret_access_key = 12345678
root@k8s-master01:/data/velero# 

6.2.3、准备user-csr文件

root@k8s-master01:/data/velero# ll 
total 16
drwxr-xr-x 2 root root 4096 Sep  2 12:48 ./
drwxr-xr-x 3 root root 4096 Sep  2 12:42 ../
-rw-r--r-- 1 root root  222 Sep  2 12:48 awsuser-csr.json
-rw-r--r-- 1 root root   69 Sep  2 12:43 velero-auth.txt
root@k8s-master01:/data/velero# cat awsuser-csr.json 
{
  "CN": "awsuser",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "SiChuan",
      "L": "GuangYuan",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
root@k8s-master01:/data/velero# 

6.2.4、准备证书签发环境

安装证书签发工具

root@k8s-master01:/data/velero# apt install golang-cfssl

下载cfssl

root@k8s-master01:/data/velero# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64

下载cfssljson

root@k8s-master01:/data/velero# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64 

cfssl-certinfo

root@k8s-master01:/data/velero# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64

重命名

root@k8s-master01:/data/velero# ll
total 40248
drwxr-xr-x 2 root root     4096 Sep  2 12:57 ./
drwxr-xr-x 3 root root     4096 Sep  2 12:42 ../
-rw-r--r-- 1 root root      222 Sep  2 12:48 awsuser-csr.json
-rw-r--r-- 1 root root 13502544 Aug 31 03:00 cfssl-certinfo_1.6.1_linux_amd64
-rw-r--r-- 1 root root 16659824 Aug 31 03:00 cfssl_1.6.1_linux_amd64
-rw-r--r-- 1 root root 11029744 Aug 31 03:00 cfssljson_1.6.1_linux_amd64
-rw-r--r-- 1 root root       69 Sep  2 12:43 velero-auth.txt
root@k8s-master01:/data/velero# mv cfssl-certinfo_1.6.1_linux_amd64 cfssl-certinfo
root@k8s-master01:/data/velero# mv cfssl_1.6.1_linux_amd64 cfssl
root@k8s-master01:/data/velero# mv cfssljson_1.6.1_linux_amd64 cfssljson
root@k8s-master01:/data/velero# ll
total 40248
drwxr-xr-x 2 root root     4096 Sep  2 12:58 ./
drwxr-xr-x 3 root root     4096 Sep  2 12:42 ../
-rw-r--r-- 1 root root      222 Sep  2 12:48 awsuser-csr.json
-rw-r--r-- 1 root root 16659824 Aug 31 03:00 cfssl
-rw-r--r-- 1 root root 13502544 Aug 31 03:00 cfssl-certinfo
-rw-r--r-- 1 root root 11029744 Aug 31 03:00 cfssljson
-rw-r--r-- 1 root root       69 Sep  2 12:43 velero-auth.txt
root@k8s-master01:/data/velero# 

移动二进制文件至/usr/local/bin/

root@k8s-master01:/data/velero# cp cfssl-certinfo cfssl cfssljson /usr/local/bin/

添加可执行权限

root@k8s-master01:/data/velero# chmod  a+x /usr/local/bin/cfssl*
root@k8s-master01:/data/velero# ll /usr/local/bin/cfssl*
-rwxr-xr-x 1 root root 16659824 Sep  2 12:59 /usr/local/bin/cfssl*
-rwxr-xr-x 1 root root 13502544 Sep  2 12:59 /usr/local/bin/cfssl-certinfo*
-rwxr-xr-x 1 root root 11029744 Sep  2 12:59 /usr/local/bin/cfssljson*
root@k8s-master01:/data/velero# 

6.2.5、执行证书签发

复制部署k8s集群的ca-config.json文件至/data/velero

root@k8s-deploy:~# scp /etc/kubeasz/clusters/k8s-cluster01/ssl/ca-config.json 192.168.0.31:/data/velero
ca-config.json                                                                                                       100%  459   203.8KB/s   00:00    
root@k8s-deploy:~# 

验证ca-config.json是否正常复制

root@k8s-master01:/data/velero# ll 
total 40252
drwxr-xr-x 2 root root     4096 Sep  2 13:03 ./
drwxr-xr-x 3 root root     4096 Sep  2 12:42 ../
-rw-r--r-- 1 root root      222 Sep  2 12:48 awsuser-csr.json
-rw-r--r-- 1 root root      459 Sep  2 13:03 ca-config.json
-rw-r--r-- 1 root root 16659824 Aug 31 03:00 cfssl
-rw-r--r-- 1 root root 13502544 Aug 31 03:00 cfssl-certinfo
-rw-r--r-- 1 root root 11029744 Aug 31 03:00 cfssljson
-rw-r--r-- 1 root root       69 Sep  2 12:43 velero-auth.txt
root@k8s-master01:/data/velero# 

签发证书

root@k8s-master01:/data/velero# cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=./ca-config.json -profile=kubernetes ./awsuser-csr.json | cfssljson -bare awsuser
2023/09/02 13:05:37 [INFO] generate received request
2023/09/02 13:05:37 [INFO] received CSR
2023/09/02 13:05:37 [INFO] generating key: rsa-2048
2023/09/02 13:05:38 [INFO] encoded CSR
2023/09/02 13:05:38 [INFO] signed certificate with serial number 309924608852958492895277791638870960844710474947
2023/09/02 13:05:38 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

6.2.6、验证证书

root@k8s-master01:/data/velero# ll
total 40264
drwxr-xr-x 2 root root     4096 Sep  2 13:05 ./
drwxr-xr-x 3 root root     4096 Sep  2 12:42 ../
-rw-r--r-- 1 root root      222 Sep  2 12:48 awsuser-csr.json
-rw------- 1 root root     1679 Sep  2 13:05 awsuser-key.pem
-rw-r--r-- 1 root root     1001 Sep  2 13:05 awsuser.csr
-rw-r--r-- 1 root root     1391 Sep  2 13:05 awsuser.pem
-rw-r--r-- 1 root root      459 Sep  2 13:03 ca-config.json
-rw-r--r-- 1 root root 16659824 Aug 31 03:00 cfssl
-rw-r--r-- 1 root root 13502544 Aug 31 03:00 cfssl-certinfo
-rw-r--r-- 1 root root 11029744 Aug 31 03:00 cfssljson
-rw-r--r-- 1 root root       69 Sep  2 12:43 velero-auth.txt
root@k8s-master01:/data/velero# 

6.2.7、分发证书到api-server证书路径

root@k8s-master01:/data/velero# cp awsuser-key.pem /etc/kubernetes/ssl/
root@k8s-master01:/data/velero# cp awsuser.pem /etc/kubernetes/ssl/
root@k8s-master01:/data/velero# ll /etc/kubernetes/ssl/
total 48
drwxr-xr-x 2 root root 4096 Sep  2 13:07 ./
drwxr-xr-x 3 root root 4096 Apr 22 14:56 ../
-rw-r--r-- 1 root root 1679 Apr 22 14:54 aggregator-proxy-key.pem
-rw-r--r-- 1 root root 1387 Apr 22 14:54 aggregator-proxy.pem
-rw------- 1 root root 1679 Sep  2 13:07 awsuser-key.pem
-rw-r--r-- 1 root root 1391 Sep  2 13:07 awsuser.pem
-rw-r--r-- 1 root root 1679 Apr 22 14:10 ca-key.pem
-rw-r--r-- 1 root root 1310 Apr 22 14:10 ca.pem
-rw-r--r-- 1 root root 1679 Apr 22 14:56 kubelet-key.pem
-rw-r--r-- 1 root root 1460 Apr 22 14:56 kubelet.pem
-rw-r--r-- 1 root root 1679 Apr 22 14:54 kubernetes-key.pem
-rw-r--r-- 1 root root 1655 Apr 22 14:54 kubernetes.pem
root@k8s-master01:/data/velero#

6.3、生成k8s集群认证config文件

root@k8s-master01:/data/velero# export KUBE_APISERVER="https://192.168.0.111:6443"                                                                     root@k8s-master01:/data/velero# kubectl config set-cluster kubernetes \
> --certificate-authority=/etc/kubernetes/ssl/ca.pem \
> --embed-certs=true \
> --server=${KUBE_APISERVER} \
> --kubeconfig=./awsuser.kubeconfig
Cluster "kubernetes" set.
root@k8s-master01:/data/velero# ll
total 40268
drwxr-xr-x 2 root root     4096 Sep  2 13:12 ./
drwxr-xr-x 3 root root     4096 Sep  2 12:42 ../
-rw-r--r-- 1 root root      222 Sep  2 12:48 awsuser-csr.json
-rw------- 1 root root     1679 Sep  2 13:05 awsuser-key.pem
-rw-r--r-- 1 root root     1001 Sep  2 13:05 awsuser.csr
-rw------- 1 root root     1951 Sep  2 13:12 awsuser.kubeconfig
-rw-r--r-- 1 root root     1391 Sep  2 13:05 awsuser.pem
-rw-r--r-- 1 root root      459 Sep  2 13:03 ca-config.json
-rw-r--r-- 1 root root 16659824 Aug 31 03:00 cfssl
-rw-r--r-- 1 root root 13502544 Aug 31 03:00 cfssl-certinfo
-rw-r--r-- 1 root root 11029744 Aug 31 03:00 cfssljson
-rw-r--r-- 1 root root       69 Sep  2 12:43 velero-auth.txt
root@k8s-master01:/data/velero# cat awsuser.kubeconfig 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURtakNDQW9LZ0F3SUJBZ0lVTW01blNKSUtCdGNmeXY3MVlZZy91QlBsT3JZd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pERUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEZqQVVCZ05WQkFNVERXdDFZbVZ5CmJtVjBaWE10WTJFd0lCY05Nak13TkRJeU1UTXpNekF3V2hnUE1qRXlNekF6TWpreE16TXpNREJhTUdReEN6QUoKQmdOVkJBWVRBa05PTVJFd0R3WURWUVFJRXdoSVlXNW5XbWh2ZFRFTE1Ba0dBMVVFQnhNQ1dGTXhEREFLQmdOVgpCQW9UQTJzNGN6RVBNQTBHQTFVRUN4TUdVM2x6ZEdWdE1SWXdGQVlEVlFRREV3MXJkV0psY201bGRHVnpMV05oCk1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBcTRmdWtncjl2ditQWVVtQmZnWjUKTVJIOTZRekErMVgvZG5hUlpzN1lPZjZMaEZ5ZWJxUTFlM3k2bmN3Tk90WUkyemJ3SVJKL0c3YTNsTSt0Qk5sTQpwdE5Db1lxalF4WVY2YkpOcGNIRFJldTY0Z1BYcHhHY1FNZGE2Q1VhVTBrNENMZ0I2ZGx1OE8rUTdaL1dNeWhTClZQMWp5dEpnK1I4UGZRUWVzdnlTanBzaUM4cmdUQjc2VWU0ZXJqaEFwb2JSbzRILzN2cGhVUXRLNTBQSWVVNlgKTnpuTVNONmdLMXRqSjZPSStlVkE1dWdTTnFOc3FVSXFHWmhmZXZSeFBhNzVBbDhrbmRxc3cyTm5WSFFOZmpGUApZR3lNOFlncllUWm9sa2RGYk9Wb2g0U3pncTFnclc0dzBpMnpySVlJTzAzNTBEODh4RFRGRTBka3FPSlRVb0JyCmtRSURBUUFCbzBJd1FEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlYKSFE0RUZnUVU5SjZoekJaOTNZMklac1ZYYUYwZk1uZ0crS1V3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUZLNwpjZ3l3UnI4aWt4NmpWMUYwVUNJRGxEN0FPQ3dTcE1Odithd1Zyd2k4Mk5xL3hpL2RjaGU1TjhJUkFEUkRQTHJUClRRS2M4M2FURXM1dnpKczd5Nnl6WHhEbUZocGxrY3NoenVhQkdFSkhpbGpuSHJ0Z09tL1ZQck5QK3hhWXdUNHYKZFNOdEIrczgxNGh6OWhaSitmTHRMb1RBS2tMUjVMRjkyQjF2c0JsVnlkaUhLSnF6MCtORkdJMzdiY1pvc0cxdwpwbVpROHgyWUFxWHE2VFlUQnoxLzR6UGlSM3FMQmxtRkNMZVJCa1RJb2VhUkFxU2ZkeDRiVlhGeTlpQ1lnTHU4CjVrcmQzMEdmZU5pRUpZVWJtZzNxcHNVSUlQTmUvUDdHNU0raS9GSlpDcFBOQ3Y4aS9MQ0Z2cVhPbThvYmdYYm8KeDNsZWpWVlZ6eG9yNEtOd3pUZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://192.168.0.111:6443
  name: kubernetes
contexts: null
current-context: ""
kind: Config
preferences: {}
users: null
root@k8s-master01:/data/velero# 

6.3.1、设置客户端证书认证

root@k8s-master01:/data/velero# kubectl config set-credentials awsuser \
> --client-certificate=/etc/kubernetes/ssl/awsuser.pem \
> --client-key=/etc/kubernetes/ssl/awsuser-key.pem \
> --embed-certs=true \
> --kubeconfig=./awsuser.kubeconfig
User "awsuser" set.
root@k8s-master01:/data/velero# cat awsuser.kubeconfig 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURtakNDQW9LZ0F3SUJBZ0lVTW01blNKSUtCdGNmeXY3MVlZZy91QlBsT3JZd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pERUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEZqQVVCZ05WQkFNVERXdDFZbVZ5CmJtVjBaWE10WTJFd0lCY05Nak13TkRJeU1UTXpNekF3V2hnUE1qRXlNekF6TWpreE16TXpNREJhTUdReEN6QUoKQmdOVkJBWVRBa05PTVJFd0R3WURWUVFJRXdoSVlXNW5XbWh2ZFRFTE1Ba0dBMVVFQnhNQ1dGTXhEREFLQmdOVgpCQW9UQTJzNGN6RVBNQTBHQTFVRUN4TUdVM2x6ZEdWdE1SWXdGQVlEVlFRREV3MXJkV0psY201bGRHVnpMV05oCk1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBcTRmdWtncjl2ditQWVVtQmZnWjUKTVJIOTZRekErMVgvZG5hUlpzN1lPZjZMaEZ5ZWJxUTFlM3k2bmN3Tk90WUkyemJ3SVJKL0c3YTNsTSt0Qk5sTQpwdE5Db1lxalF4WVY2YkpOcGNIRFJldTY0Z1BYcHhHY1FNZGE2Q1VhVTBrNENMZ0I2ZGx1OE8rUTdaL1dNeWhTClZQMWp5dEpnK1I4UGZRUWVzdnlTanBzaUM4cmdUQjc2VWU0ZXJqaEFwb2JSbzRILzN2cGhVUXRLNTBQSWVVNlgKTnpuTVNONmdLMXRqSjZPSStlVkE1dWdTTnFOc3FVSXFHWmhmZXZSeFBhNzVBbDhrbmRxc3cyTm5WSFFOZmpGUApZR3lNOFlncllUWm9sa2RGYk9Wb2g0U3pncTFnclc0dzBpMnpySVlJTzAzNTBEODh4RFRGRTBka3FPSlRVb0JyCmtRSURBUUFCbzBJd1FEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlYKSFE0RUZnUVU5SjZoekJaOTNZMklac1ZYYUYwZk1uZ0crS1V3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUZLNwpjZ3l3UnI4aWt4NmpWMUYwVUNJRGxEN0FPQ3dTcE1Odithd1Zyd2k4Mk5xL3hpL2RjaGU1TjhJUkFEUkRQTHJUClRRS2M4M2FURXM1dnpKczd5Nnl6WHhEbUZocGxrY3NoenVhQkdFSkhpbGpuSHJ0Z09tL1ZQck5QK3hhWXdUNHYKZFNOdEIrczgxNGh6OWhaSitmTHRMb1RBS2tMUjVMRjkyQjF2c0JsVnlkaUhLSnF6MCtORkdJMzdiY1pvc0cxdwpwbVpROHgyWUFxWHE2VFlUQnoxLzR6UGlSM3FMQmxtRkNMZVJCa1RJb2VhUkFxU2ZkeDRiVlhGeTlpQ1lnTHU4CjVrcmQzMEdmZU5pRUpZVWJtZzNxcHNVSUlQTmUvUDdHNU0raS9GSlpDcFBOQ3Y4aS9MQ0Z2cVhPbThvYmdYYm8KeDNsZWpWVlZ6eG9yNEtOd3pUZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://192.168.0.111:6443
  name: kubernetes
contexts: null
current-context: ""
kind: Config
preferences: {}
users:
- name: awsuser
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQxekNDQXIrZ0F3SUJBZ0lVTmttQUJ6ZjVhdCtoZC9vYmtONXBVV3JWOU1Nd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pERUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEZqQVVCZ05WQkFNVERXdDFZbVZ5CmJtVjBaWE10WTJFd0lCY05Nak13T1RBeU1UTXdNVEF3V2hnUE1qQTNNekE0TWpBeE16QXhNREJhTUdReEN6QUoKQmdOVkJBWVRBa05PTVJBd0RnWURWUVFJRXdkVGFVTm9kV0Z1TVJJd0VBWURWUVFIRXdsSGRXRnVaMWwxWVc0eApEREFLQmdOVkJBb1RBMnM0Y3pFUE1BMEdBMVVFQ3hNR1UzbHpkR1Z0TVJBd0RnWURWUVFERXdkaGQzTjFjMlZ5Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBeVU3ZWtvQ0ZFS0Jnd3Z1SU12ekkKSHNqRmFZNzNmTm5aWVhqU0lsVEJKeDNqY1dYVGh1eno5a013WktPRFNybmxWcTF0SnZ1dHRVNWpCaHRielJKOAorVVFYTkFhTVYxOFhVaGdvSmJZaHRCWStpSGhjK1dBNTYwaEEybEJaaFU2RGZzQjNVam9RbjNKdU02YUQ0eHBECjNIZG1TUGJ0am0xRkVWaTFkVHpSeVhDSWxrTkJFR3hLam5MSjZ5dC9YcnVuNW9wdjBudE9jQWw0VWJSWHFGejMKaTlBS3ArOUhENUV6bE5QaVUwY1FlZkxERGEwRXp3N1NyaDFpNG9rdnhVSnhyd0FhcTdaK1Q5blVRWkV5a0RpNQpuVG1NNlNucEh5aFltNW5sd2FRdUZvekY2bWt4UzFPYXJJMStiZGhYSHU1T3ViYWJKOXY0bVA5TDhYRFY3TDd6CkxRSURBUUFCbzM4d2ZUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0hRWURWUjBsQkJZd0ZBWUlLd1lCQlFVSEF3RUcKQ0NzR0FRVUZCd01DTUF3R0ExVWRFd0VCL3dRQ01BQXdIUVlEVlIwT0JCWUVGREtyYkphanpDTTNzM2ZHUzBtUwpLV0lHbm5XM01COEdBMVVkSXdRWU1CYUFGUFNlb2N3V2ZkMk5pR2JGVjJoZEh6SjRCdmlsTUEwR0NTcUdTSWIzCkRRRUJDd1VBQTRJQkFRQXd4b043eUNQZzFRQmJRcTNWT1JYUFVvcXRDVjhSUHFjd3V4bTJWZkVmVmdPMFZYanQKTHR5aEl2RDlubEsyazNmTFpWTVc2MWFzbVhtWkttUTh3YkZtL1RieE83ZkdJSWdpSzJKOGpWWHZYRnhNeExZNQpRVjcvd3QxUUluWjJsTjBsM0c3TGhkYjJ4UjFORmd1eWNXdWtWV3JKSWtpcU1Ma0lOLzdPSFhtSFZXazV1a1ZlCmNoYmVIdnJSSXRRNHBPYjlFZVgzTUxiZXBkRjJ4TWs5NmZrVXJGWmhKYWREVnB5NXEwbHFpUVJkMVpIWk4xSkMKWVBrZGRXdVQxbHNXaWJzN3BWTHRXMXlnV0JlS2hKQ0FVeTlUZEZ1WEt1QUZvdVJKUUJWQUs4dTFHRU1aL2JEYgp2eXRxN2N6ZndWOFNreVNpKzZHcldZRVBXUXllUEVEWjBPU1oKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBeVU3ZWtvQ0ZFS0Jnd3Z1SU12eklIc2pGYVk3M2ZOblpZWGpTSWxUQkp4M2pjV1hUCmh1eno5a013WktPRFNybmxWcTF0SnZ1dHRVNWpCaHRielJKOCtVUVhOQWFNVjE4WFVoZ29KYllodEJZK2lIaGMKK1dBNTYwaEEybEJaaFU2RGZzQjNVam9RbjNKdU02YUQ0eHBEM0hkbVNQYnRqbTFGRVZpMWRUelJ5WENJbGtOQgpFR3hLam5MSjZ5dC9YcnVuNW9wdjBudE9jQWw0VWJSWHFGejNpOUFLcCs5SEQ1RXpsTlBpVTBjUWVmTEREYTBFCnp3N1NyaDFpNG9rdnhVSnhyd0FhcTdaK1Q5blVRWkV5a0RpNW5UbU02U25wSHloWW01bmx3YVF1Rm96RjZta3gKUzFPYXJJMStiZGhYSHU1T3ViYWJKOXY0bVA5TDhYRFY3TDd6TFFJREFRQUJBb0lCQVFDTWVnb2RSNndUcHliKwp5WklJcXBkbnpBamVtWktnd0ZET2tRWnFTS1NsREZsY0Y0ZWRueHE3WGJXV2RQZzRuREtxNHNqSnJGVlNzUW12CkNFWnVlNWxVUkt6QWRGVlkzeFdpQnhOMUJYek5jN3hkZFVqRUNOOUNEYUNiOS9nUWEzS2RiK2VVTE1yT3lZYVgKYW5xY2J3YXVBWEFTT0tZYmZxcjA2T2R2a1dwLzIxWnF2bnAvdmZrN3dIYzduUktLWmt1ZVg0bVExdFFqdVBoZQpDQXN5WWZOeWM0VjVyUDF1K3AzTGU4Ly9sTXZQZ0wydFBib3NaaGYvM0dCUGpPZHFabVdnL0R5blhzN21qcnhqCng2OHJOcHIxU2ZhQUNOMjNQdE9HbXcreXh4NjdENTNUSVJUaXZIY2Izd0FIMnNRdkVzbG9HN0lMU0d2THJ1S3IKS0c2RkQwb0JBb0dCQVBsWFdydWxQa3B6bzl5ZUJnd3hudmlyN2x2THZrVkV1Q3ZmRVg2MHViYm5HOVVsZm1BQgpEaVduOFcvUkVHVjE0cFBtcjQ2eE5QLzkrb1p3cDNRNUMzbFNocENVWEVxZjVHTzBUSXdSb1NVdndKcUo2UHc0Cm4yb0xEbXBNS3k5bkZEcFFCQTFoYUZSQnZJd3ZxOXdHc0NmK3Fyc3pTNHM2bHp1Qm1KVDZXYUdCQW9HQkFNNnYKSWJrSXJnVW54NlpuNngyRjVmSEQ0YVFydWRLbWFScnN3NzV4dFY0ZVlEdU1ZVVIrYWxwY2llVDZ4Y1Z3NUFvbQp6Q2o1VUNsejZJZ3pJc2MyTGRMR3JOeDVrcUFTMzA1K0UxaVdEeGh6UG44bUhETkI2NGY5WTVYdjJ6bm9maWVsCmNKd2pBaE5OZlR1ck45ODR5RXpQL0tHa1NsbGNxdHFsOVF6VVZrK3RBb0dCQU81c0RGTy85NmRqbW4yY0VYWloKZ0lTU2l2TDJDUFBkZVNwaVBDMW5qT29MWmI3VUFscTB4NTFVVVBhMTk3SzlIYktGZEx2Q1VVYXp5bm9CZ080TwptaDBodjVEQ2ZOblN1S1pxUW9QeFc2RGVYNUttYXNXN014eEloRGs2cWxUQ2dVSWRQeks0UVBYSWdnMmVpL3h4CjNNSHhyN29mbTQzL3NacnlHai9pZ0JDQkFvR0FWb3BzRTE3NEJuNldrUzIzKzUraUhXNElYOFpUUTBtY2ZzS2UKWDNLYkgzS1dscmg3emNNazR2c1dYZ05HcGhwVDBaQlhNZHphWE5FRWoycmg2QW5lZS8vbVIxYThOenhQdGowQgorcml5VDJtSnhKRi9nMUxadlJJekRZZm1Ba1EvOW5mR1JBcEFoemFOOWxzRnhQaXduY0VFcGVYMW41ODJodUN3ClQ1UGxJKzBDZ1lFQTN1WmptcXl1U0ZsUGR0Q3NsN1ZDUWc4K0N6L1hBTUNQZGt0SmF1bng5VWxtWVZXVFQzM2oKby9uVVRPVHY1TWZPTm9wejVYOXM4SCsyeXFOdWpna2NHZmFTeHFTNlBkbWNhcTJGMTYxTDdTR0JDb2w1MVQ5ZwpXQkRObnlqOFprSkQxd2pRNkNDWG4zNDZIMS9YREZjbmhnc2c2UHRjTGh3RC8yS0l3eFVmdzFBPQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
root@k8s-master01:/data/velero# 

6.3.2、设置上下文参数

root@k8s-master01:/data/velero# kubectl config set-context kubernetes \
> --cluster=kubernetes \
> --user=awsuser \
> --namespace=velero-system \
> --kubeconfig=./awsuser.kubeconfig
Context "kubernetes" created.
root@k8s-master01:/data/velero# cat awsuser.kubeconfig 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURtakNDQW9LZ0F3SUJBZ0lVTW01blNKSUtCdGNmeXY3MVlZZy91QlBsT3JZd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pERUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEZqQVVCZ05WQkFNVERXdDFZbVZ5CmJtVjBaWE10WTJFd0lCY05Nak13TkRJeU1UTXpNekF3V2hnUE1qRXlNekF6TWpreE16TXpNREJhTUdReEN6QUoKQmdOVkJBWVRBa05PTVJFd0R3WURWUVFJRXdoSVlXNW5XbWh2ZFRFTE1Ba0dBMVVFQnhNQ1dGTXhEREFLQmdOVgpCQW9UQTJzNGN6RVBNQTBHQTFVRUN4TUdVM2x6ZEdWdE1SWXdGQVlEVlFRREV3MXJkV0psY201bGRHVnpMV05oCk1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBcTRmdWtncjl2ditQWVVtQmZnWjUKTVJIOTZRekErMVgvZG5hUlpzN1lPZjZMaEZ5ZWJxUTFlM3k2bmN3Tk90WUkyemJ3SVJKL0c3YTNsTSt0Qk5sTQpwdE5Db1lxalF4WVY2YkpOcGNIRFJldTY0Z1BYcHhHY1FNZGE2Q1VhVTBrNENMZ0I2ZGx1OE8rUTdaL1dNeWhTClZQMWp5dEpnK1I4UGZRUWVzdnlTanBzaUM4cmdUQjc2VWU0ZXJqaEFwb2JSbzRILzN2cGhVUXRLNTBQSWVVNlgKTnpuTVNONmdLMXRqSjZPSStlVkE1dWdTTnFOc3FVSXFHWmhmZXZSeFBhNzVBbDhrbmRxc3cyTm5WSFFOZmpGUApZR3lNOFlncllUWm9sa2RGYk9Wb2g0U3pncTFnclc0dzBpMnpySVlJTzAzNTBEODh4RFRGRTBka3FPSlRVb0JyCmtRSURBUUFCbzBJd1FEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlYKSFE0RUZnUVU5SjZoekJaOTNZMklac1ZYYUYwZk1uZ0crS1V3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUZLNwpjZ3l3UnI4aWt4NmpWMUYwVUNJRGxEN0FPQ3dTcE1Odithd1Zyd2k4Mk5xL3hpL2RjaGU1TjhJUkFEUkRQTHJUClRRS2M4M2FURXM1dnpKczd5Nnl6WHhEbUZocGxrY3NoenVhQkdFSkhpbGpuSHJ0Z09tL1ZQck5QK3hhWXdUNHYKZFNOdEIrczgxNGh6OWhaSitmTHRMb1RBS2tMUjVMRjkyQjF2c0JsVnlkaUhLSnF6MCtORkdJMzdiY1pvc0cxdwpwbVpROHgyWUFxWHE2VFlUQnoxLzR6UGlSM3FMQmxtRkNMZVJCa1RJb2VhUkFxU2ZkeDRiVlhGeTlpQ1lnTHU4CjVrcmQzMEdmZU5pRUpZVWJtZzNxcHNVSUlQTmUvUDdHNU0raS9GSlpDcFBOQ3Y4aS9MQ0Z2cVhPbThvYmdYYm8KeDNsZWpWVlZ6eG9yNEtOd3pUZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://192.168.0.111:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    namespace: velero-system
    user: awsuser
  name: kubernetes
current-context: ""
kind: Config
preferences: {}
users:
- name: awsuser
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQxekNDQXIrZ0F3SUJBZ0lVTmttQUJ6ZjVhdCtoZC9vYmtONXBVV3JWOU1Nd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pERUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEZqQVVCZ05WQkFNVERXdDFZbVZ5CmJtVjBaWE10WTJFd0lCY05Nak13T1RBeU1UTXdNVEF3V2hnUE1qQTNNekE0TWpBeE16QXhNREJhTUdReEN6QUoKQmdOVkJBWVRBa05PTVJBd0RnWURWUVFJRXdkVGFVTm9kV0Z1TVJJd0VBWURWUVFIRXdsSGRXRnVaMWwxWVc0eApEREFLQmdOVkJBb1RBMnM0Y3pFUE1BMEdBMVVFQ3hNR1UzbHpkR1Z0TVJBd0RnWURWUVFERXdkaGQzTjFjMlZ5Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBeVU3ZWtvQ0ZFS0Jnd3Z1SU12ekkKSHNqRmFZNzNmTm5aWVhqU0lsVEJKeDNqY1dYVGh1eno5a013WktPRFNybmxWcTF0SnZ1dHRVNWpCaHRielJKOAorVVFYTkFhTVYxOFhVaGdvSmJZaHRCWStpSGhjK1dBNTYwaEEybEJaaFU2RGZzQjNVam9RbjNKdU02YUQ0eHBECjNIZG1TUGJ0am0xRkVWaTFkVHpSeVhDSWxrTkJFR3hLam5MSjZ5dC9YcnVuNW9wdjBudE9jQWw0VWJSWHFGejMKaTlBS3ArOUhENUV6bE5QaVUwY1FlZkxERGEwRXp3N1NyaDFpNG9rdnhVSnhyd0FhcTdaK1Q5blVRWkV5a0RpNQpuVG1NNlNucEh5aFltNW5sd2FRdUZvekY2bWt4UzFPYXJJMStiZGhYSHU1T3ViYWJKOXY0bVA5TDhYRFY3TDd6CkxRSURBUUFCbzM4d2ZUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0hRWURWUjBsQkJZd0ZBWUlLd1lCQlFVSEF3RUcKQ0NzR0FRVUZCd01DTUF3R0ExVWRFd0VCL3dRQ01BQXdIUVlEVlIwT0JCWUVGREtyYkphanpDTTNzM2ZHUzBtUwpLV0lHbm5XM01COEdBMVVkSXdRWU1CYUFGUFNlb2N3V2ZkMk5pR2JGVjJoZEh6SjRCdmlsTUEwR0NTcUdTSWIzCkRRRUJDd1VBQTRJQkFRQXd4b043eUNQZzFRQmJRcTNWT1JYUFVvcXRDVjhSUHFjd3V4bTJWZkVmVmdPMFZYanQKTHR5aEl2RDlubEsyazNmTFpWTVc2MWFzbVhtWkttUTh3YkZtL1RieE83ZkdJSWdpSzJKOGpWWHZYRnhNeExZNQpRVjcvd3QxUUluWjJsTjBsM0c3TGhkYjJ4UjFORmd1eWNXdWtWV3JKSWtpcU1Ma0lOLzdPSFhtSFZXazV1a1ZlCmNoYmVIdnJSSXRRNHBPYjlFZVgzTUxiZXBkRjJ4TWs5NmZrVXJGWmhKYWREVnB5NXEwbHFpUVJkMVpIWk4xSkMKWVBrZGRXdVQxbHNXaWJzN3BWTHRXMXlnV0JlS2hKQ0FVeTlUZEZ1WEt1QUZvdVJKUUJWQUs4dTFHRU1aL2JEYgp2eXRxN2N6ZndWOFNreVNpKzZHcldZRVBXUXllUEVEWjBPU1oKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBeVU3ZWtvQ0ZFS0Jnd3Z1SU12eklIc2pGYVk3M2ZOblpZWGpTSWxUQkp4M2pjV1hUCmh1eno5a013WktPRFNybmxWcTF0SnZ1dHRVNWpCaHRielJKOCtVUVhOQWFNVjE4WFVoZ29KYllodEJZK2lIaGMKK1dBNTYwaEEybEJaaFU2RGZzQjNVam9RbjNKdU02YUQ0eHBEM0hkbVNQYnRqbTFGRVZpMWRUelJ5WENJbGtOQgpFR3hLam5MSjZ5dC9YcnVuNW9wdjBudE9jQWw0VWJSWHFGejNpOUFLcCs5SEQ1RXpsTlBpVTBjUWVmTEREYTBFCnp3N1NyaDFpNG9rdnhVSnhyd0FhcTdaK1Q5blVRWkV5a0RpNW5UbU02U25wSHloWW01bmx3YVF1Rm96RjZta3gKUzFPYXJJMStiZGhYSHU1T3ViYWJKOXY0bVA5TDhYRFY3TDd6TFFJREFRQUJBb0lCQVFDTWVnb2RSNndUcHliKwp5WklJcXBkbnpBamVtWktnd0ZET2tRWnFTS1NsREZsY0Y0ZWRueHE3WGJXV2RQZzRuREtxNHNqSnJGVlNzUW12CkNFWnVlNWxVUkt6QWRGVlkzeFdpQnhOMUJYek5jN3hkZFVqRUNOOUNEYUNiOS9nUWEzS2RiK2VVTE1yT3lZYVgKYW5xY2J3YXVBWEFTT0tZYmZxcjA2T2R2a1dwLzIxWnF2bnAvdmZrN3dIYzduUktLWmt1ZVg0bVExdFFqdVBoZQpDQXN5WWZOeWM0VjVyUDF1K3AzTGU4Ly9sTXZQZ0wydFBib3NaaGYvM0dCUGpPZHFabVdnL0R5blhzN21qcnhqCng2OHJOcHIxU2ZhQUNOMjNQdE9HbXcreXh4NjdENTNUSVJUaXZIY2Izd0FIMnNRdkVzbG9HN0lMU0d2THJ1S3IKS0c2RkQwb0JBb0dCQVBsWFdydWxQa3B6bzl5ZUJnd3hudmlyN2x2THZrVkV1Q3ZmRVg2MHViYm5HOVVsZm1BQgpEaVduOFcvUkVHVjE0cFBtcjQ2eE5QLzkrb1p3cDNRNUMzbFNocENVWEVxZjVHTzBUSXdSb1NVdndKcUo2UHc0Cm4yb0xEbXBNS3k5bkZEcFFCQTFoYUZSQnZJd3ZxOXdHc0NmK3Fyc3pTNHM2bHp1Qm1KVDZXYUdCQW9HQkFNNnYKSWJrSXJnVW54NlpuNngyRjVmSEQ0YVFydWRLbWFScnN3NzV4dFY0ZVlEdU1ZVVIrYWxwY2llVDZ4Y1Z3NUFvbQp6Q2o1VUNsejZJZ3pJc2MyTGRMR3JOeDVrcUFTMzA1K0UxaVdEeGh6UG44bUhETkI2NGY5WTVYdjJ6bm9maWVsCmNKd2pBaE5OZlR1ck45ODR5RXpQL0tHa1NsbGNxdHFsOVF6VVZrK3RBb0dCQU81c0RGTy85NmRqbW4yY0VYWloKZ0lTU2l2TDJDUFBkZVNwaVBDMW5qT29MWmI3VUFscTB4NTFVVVBhMTk3SzlIYktGZEx2Q1VVYXp5bm9CZ080TwptaDBodjVEQ2ZOblN1S1pxUW9QeFc2RGVYNUttYXNXN014eEloRGs2cWxUQ2dVSWRQeks0UVBYSWdnMmVpL3h4CjNNSHhyN29mbTQzL3NacnlHai9pZ0JDQkFvR0FWb3BzRTE3NEJuNldrUzIzKzUraUhXNElYOFpUUTBtY2ZzS2UKWDNLYkgzS1dscmg3emNNazR2c1dYZ05HcGhwVDBaQlhNZHphWE5FRWoycmg2QW5lZS8vbVIxYThOenhQdGowQgorcml5VDJtSnhKRi9nMUxadlJJekRZZm1Ba1EvOW5mR1JBcEFoemFOOWxzRnhQaXduY0VFcGVYMW41ODJodUN3ClQ1UGxJKzBDZ1lFQTN1WmptcXl1U0ZsUGR0Q3NsN1ZDUWc4K0N6L1hBTUNQZGt0SmF1bng5VWxtWVZXVFQzM2oKby9uVVRPVHY1TWZPTm9wejVYOXM4SCsyeXFOdWpna2NHZmFTeHFTNlBkbWNhcTJGMTYxTDdTR0JDb2w1MVQ5ZwpXQkRObnlqOFprSkQxd2pRNkNDWG4zNDZIMS9YREZjbmhnc2c2UHRjTGh3RC8yS0l3eFVmdzFBPQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
root@k8s-master01:/data/velero# 

6.3.3、设置默认上下文

root@k8s-master01:/data/velero# kubectl config use-context kubernetes --kubeconfig=awsuser.kubeconfig
Switched to context "kubernetes".
root@k8s-master01:/data/velero# cat awsuser.kubeconfig             
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURtakNDQW9LZ0F3SUJBZ0lVTW01blNKSUtCdGNmeXY3MVlZZy91QlBsT3JZd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pERUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEZqQVVCZ05WQkFNVERXdDFZbVZ5CmJtVjBaWE10WTJFd0lCY05Nak13TkRJeU1UTXpNekF3V2hnUE1qRXlNekF6TWpreE16TXpNREJhTUdReEN6QUoKQmdOVkJBWVRBa05PTVJFd0R3WURWUVFJRXdoSVlXNW5XbWh2ZFRFTE1Ba0dBMVVFQnhNQ1dGTXhEREFLQmdOVgpCQW9UQTJzNGN6RVBNQTBHQTFVRUN4TUdVM2x6ZEdWdE1SWXdGQVlEVlFRREV3MXJkV0psY201bGRHVnpMV05oCk1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBcTRmdWtncjl2ditQWVVtQmZnWjUKTVJIOTZRekErMVgvZG5hUlpzN1lPZjZMaEZ5ZWJxUTFlM3k2bmN3Tk90WUkyemJ3SVJKL0c3YTNsTSt0Qk5sTQpwdE5Db1lxalF4WVY2YkpOcGNIRFJldTY0Z1BYcHhHY1FNZGE2Q1VhVTBrNENMZ0I2ZGx1OE8rUTdaL1dNeWhTClZQMWp5dEpnK1I4UGZRUWVzdnlTanBzaUM4cmdUQjc2VWU0ZXJqaEFwb2JSbzRILzN2cGhVUXRLNTBQSWVVNlgKTnpuTVNONmdLMXRqSjZPSStlVkE1dWdTTnFOc3FVSXFHWmhmZXZSeFBhNzVBbDhrbmRxc3cyTm5WSFFOZmpGUApZR3lNOFlncllUWm9sa2RGYk9Wb2g0U3pncTFnclc0dzBpMnpySVlJTzAzNTBEODh4RFRGRTBka3FPSlRVb0JyCmtRSURBUUFCbzBJd1FEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlYKSFE0RUZnUVU5SjZoekJaOTNZMklac1ZYYUYwZk1uZ0crS1V3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUZLNwpjZ3l3UnI4aWt4NmpWMUYwVUNJRGxEN0FPQ3dTcE1Odithd1Zyd2k4Mk5xL3hpL2RjaGU1TjhJUkFEUkRQTHJUClRRS2M4M2FURXM1dnpKczd5Nnl6WHhEbUZocGxrY3NoenVhQkdFSkhpbGpuSHJ0Z09tL1ZQck5QK3hhWXdUNHYKZFNOdEIrczgxNGh6OWhaSitmTHRMb1RBS2tMUjVMRjkyQjF2c0JsVnlkaUhLSnF6MCtORkdJMzdiY1pvc0cxdwpwbVpROHgyWUFxWHE2VFlUQnoxLzR6UGlSM3FMQmxtRkNMZVJCa1RJb2VhUkFxU2ZkeDRiVlhGeTlpQ1lnTHU4CjVrcmQzMEdmZU5pRUpZVWJtZzNxcHNVSUlQTmUvUDdHNU0raS9GSlpDcFBOQ3Y4aS9MQ0Z2cVhPbThvYmdYYm8KeDNsZWpWVlZ6eG9yNEtOd3pUZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://192.168.0.111:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    namespace: velero-system
    user: awsuser
  name: kubernetes
current-context: kubernetes
kind: Config
preferences: {}
users:
- name: awsuser
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQxekNDQXIrZ0F3SUJBZ0lVTmttQUJ6ZjVhdCtoZC9vYmtONXBVV3JWOU1Nd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pERUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEZqQVVCZ05WQkFNVERXdDFZbVZ5CmJtVjBaWE10WTJFd0lCY05Nak13T1RBeU1UTXdNVEF3V2hnUE1qQTNNekE0TWpBeE16QXhNREJhTUdReEN6QUoKQmdOVkJBWVRBa05PTVJBd0RnWURWUVFJRXdkVGFVTm9kV0Z1TVJJd0VBWURWUVFIRXdsSGRXRnVaMWwxWVc0eApEREFLQmdOVkJBb1RBMnM0Y3pFUE1BMEdBMVVFQ3hNR1UzbHpkR1Z0TVJBd0RnWURWUVFERXdkaGQzTjFjMlZ5Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBeVU3ZWtvQ0ZFS0Jnd3Z1SU12ekkKSHNqRmFZNzNmTm5aWVhqU0lsVEJKeDNqY1dYVGh1eno5a013WktPRFNybmxWcTF0SnZ1dHRVNWpCaHRielJKOAorVVFYTkFhTVYxOFhVaGdvSmJZaHRCWStpSGhjK1dBNTYwaEEybEJaaFU2RGZzQjNVam9RbjNKdU02YUQ0eHBECjNIZG1TUGJ0am0xRkVWaTFkVHpSeVhDSWxrTkJFR3hLam5MSjZ5dC9YcnVuNW9wdjBudE9jQWw0VWJSWHFGejMKaTlBS3ArOUhENUV6bE5QaVUwY1FlZkxERGEwRXp3N1NyaDFpNG9rdnhVSnhyd0FhcTdaK1Q5blVRWkV5a0RpNQpuVG1NNlNucEh5aFltNW5sd2FRdUZvekY2bWt4UzFPYXJJMStiZGhYSHU1T3ViYWJKOXY0bVA5TDhYRFY3TDd6CkxRSURBUUFCbzM4d2ZUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0hRWURWUjBsQkJZd0ZBWUlLd1lCQlFVSEF3RUcKQ0NzR0FRVUZCd01DTUF3R0ExVWRFd0VCL3dRQ01BQXdIUVlEVlIwT0JCWUVGREtyYkphanpDTTNzM2ZHUzBtUwpLV0lHbm5XM01COEdBMVVkSXdRWU1CYUFGUFNlb2N3V2ZkMk5pR2JGVjJoZEh6SjRCdmlsTUEwR0NTcUdTSWIzCkRRRUJDd1VBQTRJQkFRQXd4b043eUNQZzFRQmJRcTNWT1JYUFVvcXRDVjhSUHFjd3V4bTJWZkVmVmdPMFZYanQKTHR5aEl2RDlubEsyazNmTFpWTVc2MWFzbVhtWkttUTh3YkZtL1RieE83ZkdJSWdpSzJKOGpWWHZYRnhNeExZNQpRVjcvd3QxUUluWjJsTjBsM0c3TGhkYjJ4UjFORmd1eWNXdWtWV3JKSWtpcU1Ma0lOLzdPSFhtSFZXazV1a1ZlCmNoYmVIdnJSSXRRNHBPYjlFZVgzTUxiZXBkRjJ4TWs5NmZrVXJGWmhKYWREVnB5NXEwbHFpUVJkMVpIWk4xSkMKWVBrZGRXdVQxbHNXaWJzN3BWTHRXMXlnV0JlS2hKQ0FVeTlUZEZ1WEt1QUZvdVJKUUJWQUs4dTFHRU1aL2JEYgp2eXRxN2N6ZndWOFNreVNpKzZHcldZRVBXUXllUEVEWjBPU1oKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBeVU3ZWtvQ0ZFS0Jnd3Z1SU12eklIc2pGYVk3M2ZOblpZWGpTSWxUQkp4M2pjV1hUCmh1eno5a013WktPRFNybmxWcTF0SnZ1dHRVNWpCaHRielJKOCtVUVhOQWFNVjE4WFVoZ29KYllodEJZK2lIaGMKK1dBNTYwaEEybEJaaFU2RGZzQjNVam9RbjNKdU02YUQ0eHBEM0hkbVNQYnRqbTFGRVZpMWRUelJ5WENJbGtOQgpFR3hLam5MSjZ5dC9YcnVuNW9wdjBudE9jQWw0VWJSWHFGejNpOUFLcCs5SEQ1RXpsTlBpVTBjUWVmTEREYTBFCnp3N1NyaDFpNG9rdnhVSnhyd0FhcTdaK1Q5blVRWkV5a0RpNW5UbU02U25wSHloWW01bmx3YVF1Rm96RjZta3gKUzFPYXJJMStiZGhYSHU1T3ViYWJKOXY0bVA5TDhYRFY3TDd6TFFJREFRQUJBb0lCQVFDTWVnb2RSNndUcHliKwp5WklJcXBkbnpBamVtWktnd0ZET2tRWnFTS1NsREZsY0Y0ZWRueHE3WGJXV2RQZzRuREtxNHNqSnJGVlNzUW12CkNFWnVlNWxVUkt6QWRGVlkzeFdpQnhOMUJYek5jN3hkZFVqRUNOOUNEYUNiOS9nUWEzS2RiK2VVTE1yT3lZYVgKYW5xY2J3YXVBWEFTT0tZYmZxcjA2T2R2a1dwLzIxWnF2bnAvdmZrN3dIYzduUktLWmt1ZVg0bVExdFFqdVBoZQpDQXN5WWZOeWM0VjVyUDF1K3AzTGU4Ly9sTXZQZ0wydFBib3NaaGYvM0dCUGpPZHFabVdnL0R5blhzN21qcnhqCng2OHJOcHIxU2ZhQUNOMjNQdE9HbXcreXh4NjdENTNUSVJUaXZIY2Izd0FIMnNRdkVzbG9HN0lMU0d2THJ1S3IKS0c2RkQwb0JBb0dCQVBsWFdydWxQa3B6bzl5ZUJnd3hudmlyN2x2THZrVkV1Q3ZmRVg2MHViYm5HOVVsZm1BQgpEaVduOFcvUkVHVjE0cFBtcjQ2eE5QLzkrb1p3cDNRNUMzbFNocENVWEVxZjVHTzBUSXdSb1NVdndKcUo2UHc0Cm4yb0xEbXBNS3k5bkZEcFFCQTFoYUZSQnZJd3ZxOXdHc0NmK3Fyc3pTNHM2bHp1Qm1KVDZXYUdCQW9HQkFNNnYKSWJrSXJnVW54NlpuNngyRjVmSEQ0YVFydWRLbWFScnN3NzV4dFY0ZVlEdU1ZVVIrYWxwY2llVDZ4Y1Z3NUFvbQp6Q2o1VUNsejZJZ3pJc2MyTGRMR3JOeDVrcUFTMzA1K0UxaVdEeGh6UG44bUhETkI2NGY5WTVYdjJ6bm9maWVsCmNKd2pBaE5OZlR1ck45ODR5RXpQL0tHa1NsbGNxdHFsOVF6VVZrK3RBb0dCQU81c0RGTy85NmRqbW4yY0VYWloKZ0lTU2l2TDJDUFBkZVNwaVBDMW5qT29MWmI3VUFscTB4NTFVVVBhMTk3SzlIYktGZEx2Q1VVYXp5bm9CZ080TwptaDBodjVEQ2ZOblN1S1pxUW9QeFc2RGVYNUttYXNXN014eEloRGs2cWxUQ2dVSWRQeks0UVBYSWdnMmVpL3h4CjNNSHhyN29mbTQzL3NacnlHai9pZ0JDQkFvR0FWb3BzRTE3NEJuNldrUzIzKzUraUhXNElYOFpUUTBtY2ZzS2UKWDNLYkgzS1dscmg3emNNazR2c1dYZ05HcGhwVDBaQlhNZHphWE5FRWoycmg2QW5lZS8vbVIxYThOenhQdGowQgorcml5VDJtSnhKRi9nMUxadlJJekRZZm1Ba1EvOW5mR1JBcEFoemFOOWxzRnhQaXduY0VFcGVYMW41ODJodUN3ClQ1UGxJKzBDZ1lFQTN1WmptcXl1U0ZsUGR0Q3NsN1ZDUWc4K0N6L1hBTUNQZGt0SmF1bng5VWxtWVZXVFQzM2oKby9uVVRPVHY1TWZPTm9wejVYOXM4SCsyeXFOdWpna2NHZmFTeHFTNlBkbWNhcTJGMTYxTDdTR0JDb2w1MVQ5ZwpXQkRObnlqOFprSkQxd2pRNkNDWG4zNDZIMS9YREZjbmhnc2c2UHRjTGh3RC8yS0l3eFVmdzFBPQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
root@k8s-master01:/data/velero# 

6.3.4、k8s集群中创建awsuser账户

root@k8s-master01:/data/velero# kubectl create clusterrolebinding awsuser --clusterrole=cluster-admin --user=awsuser
clusterrolebinding.rbac.authorization.k8s.io/awsuser created
root@k8s-master01:/data/velero# kubectl get clusterrolebinding -A|grep awsuser
awsuser                                                ClusterRole/cluster-admin                                          47s
root@k8s-master01:/data/velero# 

6.3.5、验证证书的可用性

root@k8s-master01:/data/velero# kubectl --kubeconfig ./awsuser.kubeconfig get nodes
NAME           STATUS                     ROLES    AGE    VERSION
192.168.0.31   Ready,SchedulingDisabled   master   132d   v1.26.4
192.168.0.32   Ready,SchedulingDisabled   master   132d   v1.26.4
192.168.0.33   Ready,SchedulingDisabled   master   132d   v1.26.4
192.168.0.34   Ready                      node     132d   v1.26.4
192.168.0.35   Ready                      node     132d   v1.26.4
192.168.0.36   Ready                      node     132d   v1.26.4
root@k8s-master01:/data/velero# kubectl --kubeconfig ./awsuser.kubeconfig get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS       AGE
calico-kube-controllers-5456dd947c-pwl2n   1/1     Running   31 (79m ago)   132d
calico-node-4zmb4                          1/1     Running   26 (79m ago)   132d
calico-node-7lc66                          1/1     Running   28 (79m ago)   132d
calico-node-bkhkd                          1/1     Running   28 (13d ago)   132d
calico-node-mw49k                          1/1     Running   28 (79m ago)   132d
calico-node-v726r                          1/1     Running   26 (79m ago)   132d
calico-node-x9r7h                          1/1     Running   28 (79m ago)   132d
coredns-77879dc67d-k9ztn                   1/1     Running   4 (79m ago)    27d
coredns-77879dc67d-qwb48                   1/1     Running   4 (79m ago)    27d
snapshot-controller-0                      1/1     Running   28 (79m ago)   132d
root@k8s-master01:/data/velero# 

6.3.6、k8s集群中创建namespace

root@k8s-master01:/data/velero# kubectl create ns velero-system
namespace/velero-system created
root@k8s-master01:/data/velero# kubectl get ns
NAME              STATUS   AGE
argocd            Active   129d
default           Active   132d
kube-node-lease   Active   132d
kube-public       Active   132d
kube-system       Active   132d
magedu            Active   90d
myserver          Active   98d
velero-system     Active   5s
root@k8s-master01:/data/velero#

6.4、执行安装velero服务端

root@k8s-master01:/data/velero# velero --kubeconfig  ./awsuser.kubeconfig \
>     install \
>     --provider aws \
>     --plugins velero/velero-plugin-for-aws:v1.5.5 \
>     --bucket velerodata  \
>     --secret-file ./velero-auth.txt \
>     --use-volume-snapshots=false \
>     --namespace velero-system \
> --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.0.42:9000
CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource
CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource client
CustomResourceDefinition/backuprepositories.velero.io: created
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: attempting to create resource client
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: attempting to create resource client
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: attempting to create resource client
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero-system: attempting to create resource
Namespace/velero-system: attempting to create resource client
Namespace/velero-system: already exists, proceeding
Namespace/velero-system: created
ClusterRoleBinding/velero-velero-system: attempting to create resource
ClusterRoleBinding/velero-velero-system: attempting to create resource client
ClusterRoleBinding/velero-velero-system: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: attempting to create resource client
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: attempting to create resource client
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: attempting to create resource client
BackupStorageLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: attempting to create resource client
Deployment/velero: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero-system' to view the status.
root@k8s-master01:/data/velero# 

6.5、验证安装velero服务端

root@k8s-master01:/data/velero# kubectl get pod -n velero-system
NAME                      READY   STATUS    RESTARTS   AGE
velero-5d675548c4-2dx8d   1/1     Running   0          105s
root@k8s-master01:/data/velero#

7、velero备份数据

7.1、对default ns进行备份

root@k8s-master01:/data/velero# DATE=`date +%Y%m%d%H%M%S`
root@k8s-master01:/data/velero#  velero backup create default-backup-${DATE} \
> --include-cluster-resources=true \
> --include-namespaces default \
> --kubeconfig=./awsuser.kubeconfig \
> --namespace velero-system
Backup request "default-backup-20230902133242" submitted successfully.
Run `velero backup describe default-backup-20230902133242` or `velero backup logs default-backup-20230902133242` for more details.
root@k8s-master01:/data/velero# 

验证备份

root@k8s-master01:/data/velero# velero backup describe default-backup-20230902133242 --kubeconfig=./awsuser.kubeconfig --namespace velero-system
Name:         default-backup-20230902133242
Namespace:    velero-system
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.26.4
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=26

Phase:  Completed


Namespaces:
  Included:  default
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  included

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  1h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2023-09-02 13:33:01 +0000 UTC
Completed:  2023-09-02 13:33:09 +0000 UTC

Expiration:  2023-10-02 13:33:01 +0000 UTC

Total items to be backed up:  288
Items backed up:              288

Velero-Native Snapshots: <none included>
root@k8s-master01:/data/velero# 

minio验证备份数据
基于velero及minio实现etcd数据备份与恢复-LMLPHP
删除pod并验证数据恢复

  • 删除pod
root@k8s-master01:/data/velero# kubectl get pods 
NAME   READY   STATUS    RESTARTS      AGE
bash   1/1     Running   5 (93m ago)   27d
root@k8s-master01:/data/velero# kubectl delete pod bash -n default 
pod "bash" deleted
root@k8s-master01:/data/velero# kubectl get pods                   
No resources found in default namespace.
root@k8s-master01:/data/velero# 
  • 恢复pod
root@k8s-master01:/data/velero# velero restore create --from-backup default-backup-20230902133242 --wait --kubeconfig=./awsuser.kubeconfig --namespace velero-system
Restore request "default-backup-20230902133242-20230902134421" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
..............................
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe default-backup-20230902133242-20230902134421` and `velero restore logs default-backup-20230902133242-20230902134421`.
root@k8s-master01:/data/velero# 
  • 验证pod
root@k8s-master01:/data/velero# kubectl get pods 
NAME   READY   STATUS    RESTARTS   AGE
bash   1/1     Running   0          77s
root@k8s-master01:/data/velero# kubectl exec -it bash bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[root@bash ~]# ping www.baidu.com
PING www.a.shifen.com (14.119.104.189) 56(84) bytes of data.
64 bytes from 14.119.104.189 (14.119.104.189): icmp_seq=1 ttl=53 time=42.5 ms
64 bytes from 14.119.104.189 (14.119.104.189): icmp_seq=2 ttl=53 time=42.2 ms
^C
--- www.a.shifen.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 42.234/42.400/42.567/0.264 ms
[root@bash ~]# 

对myserver ns进行备份

root@k8s-master01:/data/velero# DATE=`date +%Y%m%d%H%M%S`
root@k8s-master01:/data/velero# velero backup create myserver-ns-backup-${DATE} \
> --include-cluster-resources=true \
> --include-namespaces myserver \
> --kubeconfig=/root/.kube/config \
> --namespace velero-system
Backup request "myserver-ns-backup-20230902134938" submitted successfully.
Run `velero backup describe myserver-ns-backup-20230902134938` or `velero backup logs myserver-ns-backup-20230902134938` for more details.
root@k8s-master01:/data/velero# 

minio验证备份
基于velero及minio实现etcd数据备份与恢复-LMLPHP
删除pod并验证恢复

root@k8s-master01:/data/velero# DATE=`date +%Y%m%d%H%M%S`
root@k8s-master01:/data/velero# velero backup create myserver-ns-backup-${DATE} \
> --include-cluster-resources=true \
> --include-namespaces myserver \
> --kubeconfig=/root/.kube/config \
> --namespace velero-system
Backup request "myserver-ns-backup-20230902134938" submitted successfully.
Run `velero backup describe myserver-ns-backup-20230902134938` or `velero backup logs myserver-ns-backup-20230902134938` for more details.
root@k8s-master01:/data/velero# kubectl get pods -n myserver 
NAME                                                  READY   STATUS    RESTARTS        AGE
myserver-myapp-deployment-name-6965765b9c-h4kj6       1/1     Running   13 (104m ago)   98d
myserver-myapp-frontend-deployment-6bd57599f4-8zw5s   1/1     Running   13 (104m ago)   98d
myserver-myapp-frontend-deployment-6bd57599f4-j276c   1/1     Running   13 (104m ago)   98d
myserver-myapp-frontend-deployment-6bd57599f4-p76bw   1/1     Running   13 (13d ago)    98d
root@k8s-master01:/data/velero# kubectl get deployments.apps -n myserver 
NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
myserver-myapp-deployment-name       1/1     1            1           98d
myserver-myapp-frontend-deployment   3/3     3            3           98d
root@k8s-master01:/data/velero# kubectl delete deployments.apps myserver-myapp-deployment-name -n myserver 
deployment.apps "myserver-myapp-deployment-name" deleted
root@k8s-master01:/data/velero# kubectl get pods -n myserver 
NAME                                                  READY   STATUS    RESTARTS        AGE
myserver-myapp-frontend-deployment-6bd57599f4-8zw5s   1/1     Running   13 (106m ago)   98d
myserver-myapp-frontend-deployment-6bd57599f4-j276c   1/1     Running   13 (106m ago)   98d
myserver-myapp-frontend-deployment-6bd57599f4-p76bw   1/1     Running   13 (13d ago)    98d
root@k8s-master01:/data/velero# velero restore create --from-backup myserver-ns-backup-20230902134938 --wait \
> --kubeconfig=./awsuser.kubeconfig \
> --namespace velero-system
Restore request "myserver-ns-backup-20230902134938-20230902135401" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
...............................
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe myserver-ns-backup-20230902134938-20230902135401` and `velero restore logs myserver-ns-backup-20230902134938-20230902135401`.
root@k8s-master01:/data/velero# kubectl get deployments.apps -n myserver 
NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
myserver-myapp-deployment-name       0/1     1            0           37s
myserver-myapp-frontend-deployment   3/3     3            3           98d
root@k8s-master01:/data/velero# kubectl get pods -n myserver                 
NAME                                                  READY   STATUS            RESTARTS        AGE
myserver-myapp-deployment-name-6965765b9c-h4kj6       0/1     PodInitializing   0               69s
myserver-myapp-frontend-deployment-6bd57599f4-8zw5s   1/1     Running           13 (108m ago)   98d
myserver-myapp-frontend-deployment-6bd57599f4-j276c   1/1     Running           13 (108m ago)   98d
myserver-myapp-frontend-deployment-6bd57599f4-p76bw   1/1     Running           13 (13d ago)    98d
root@k8s-master01:/data/velero#

7.2、备份指定资源对象

备份指定namespace中的pod或特定资源

root@k8s-master01:/data/velero# DATE=`date +%Y%m%d%H%M%S`
root@k8s-master01:/data/velero#  velero backup create pod-backup-${DATE} --include-cluster-resources=true \
>  --ordered-resources 'pods=default/bash,magedu/ubuntu1804,magedu/mysql-0;deployments.apps=myserver/myserver-myapp-frontend-deployment,magedu/wordpress-app-deployment;services=myserver/myserver-myapp-service-name,magedu/mysql,magedu/zookeeper' \
>  --namespace velero-system --include-namespaces=myserver,magedu,default
Backup request "pod-backup-20230902141842" submitted successfully.
Run `velero backup describe pod-backup-20230902141842` or `velero backup logs pod-backup-20230902141842` for more details.
root@k8s-master01:/data/velero# 

删除pod并验证恢复

  • 删除资源
root@k8s-master01:/data/velero# kubectl get deployments.apps -n magedu 
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
magedu-consumer-deployment     3/3     3            3           22d
magedu-dubboadmin-deployment   1/1     1            1           22d
magedu-provider-deployment     3/3     3            3           22d
wordpress-app-deployment       1/1     1            1           13d
zookeeper1                     1/1     1            1           90d
zookeeper2                     1/1     1            1           90d
zookeeper3                     1/1     1            1           90d
root@k8s-master01:/data/velero# kubectl delete deployments.apps wordpress-app-deployment -n magedu 
deployment.apps "wordpress-app-deployment" deleted
root@k8s-master01:/data/velero# kubectl get deployments.apps -n magedu 
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
magedu-consumer-deployment     3/3     3            3           22d
magedu-dubboadmin-deployment   1/1     1            1           22d
magedu-provider-deployment     3/3     3            3           22d
zookeeper1                     1/1     1            1           90d
zookeeper2                     1/1     1            1           90d
zookeeper3                     1/1     1            1           90d
root@k8s-master01:/data/velero# kubectl get svc -n magedu 
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                        AGE
magedu-consumer-server      NodePort    10.100.208.121   <none>        80:49630/TCP                                   22d
magedu-dubboadmin-service   NodePort    10.100.244.92    <none>        80:31080/TCP                                   22d
magedu-provider-spec        NodePort    10.100.187.168   <none>        80:44873/TCP                                   22d
mysql                       ClusterIP   None             <none>        3306/TCP                                       79d
mysql-0                     ClusterIP   None             <none>        3306/TCP                                       13d
mysql-read                  ClusterIP   10.100.15.127    <none>        3306/TCP                                       79d
redis                       ClusterIP   None             <none>        6379/TCP                                       88d
redis-access                NodePort    10.100.117.185   <none>        6379:36379/TCP                                 88d
wordpress-app-spec          NodePort    10.100.189.214   <none>        80:30031/TCP,443:30033/TCP                     13d
zookeeper                   ClusterIP   10.100.237.95    <none>        2181/TCP                                       90d
zookeeper1                  NodePort    10.100.63.118    <none>        2181:32181/TCP,2888:30541/TCP,3888:31200/TCP   90d
zookeeper2                  NodePort    10.100.199.43    <none>        2181:32182/TCP,2888:32670/TCP,3888:32264/TCP   90d
zookeeper3                  NodePort    10.100.41.9      <none>        2181:32183/TCP,2888:31329/TCP,3888:32546/TCP   90d
root@k8s-master01:/data/velero# kubectl delete svc mysql -n magedu 
service "mysql" deleted
root@k8s-master01:/data/velero# kubectl delete svc zookeeper -n magedu      
service "zookeeper" deleted
root@k8s-master01:/data/velero# kubectl get svc -n magedu 
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                        AGE
magedu-consumer-server      NodePort    10.100.208.121   <none>        80:49630/TCP                                   22d
magedu-dubboadmin-service   NodePort    10.100.244.92    <none>        80:31080/TCP                                   22d
magedu-provider-spec        NodePort    10.100.187.168   <none>        80:44873/TCP                                   22d
mysql-0                     ClusterIP   None             <none>        3306/TCP                                       13d
mysql-read                  ClusterIP   10.100.15.127    <none>        3306/TCP                                       79d
redis                       ClusterIP   None             <none>        6379/TCP                                       88d
redis-access                NodePort    10.100.117.185   <none>        6379:36379/TCP                                 88d
wordpress-app-spec          NodePort    10.100.189.214   <none>        80:30031/TCP,443:30033/TCP                     13d
zookeeper1                  NodePort    10.100.63.118    <none>        2181:32181/TCP,2888:30541/TCP,3888:31200/TCP   90d
zookeeper2                  NodePort    10.100.199.43    <none>        2181:32182/TCP,2888:32670/TCP,3888:32264/TCP   90d
zookeeper3                  NodePort    10.100.41.9      <none>        2181:32183/TCP,2888:31329/TCP,3888:32546/TCP   90d
root@k8s-master01:/data/velero#
  • 恢复资源
root@k8s-master01:/data/velero# velero restore create --from-backup pod-backup-20230902141842 --wait \
> --kubeconfig=./awsuser.kubeconfig \
> --namespace velero-system
Restore request "pod-backup-20230902141842-20230902142341" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
............................................
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe pod-backup-20230902141842-20230902142341` and `velero restore logs pod-backup-20230902141842-20230902142341`.
root@k8s-master01:/data/velero# 
  • 验证对应资源是否恢复
root@k8s-master01:/data/velero# kubectl get kubectl get deployments.apps -n magedu 
error: the server doesn't have a resource type "kubectl"
root@k8s-master01:/data/velero# kubectl get deployments.apps -n magedu 
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
magedu-consumer-deployment     3/3     3            3           22d
magedu-dubboadmin-deployment   1/1     1            1           22d
magedu-provider-deployment     3/3     3            3           22d
wordpress-app-deployment       1/1     1            1           2m57s
zookeeper1                     1/1     1            1           90d
zookeeper2                     1/1     1            1           90d
zookeeper3                     1/1     1            1           90d
root@k8s-master01:/data/velero# kubectl get pods -n magedu 
NAME                                            READY   STATUS      RESTARTS        AGE
magedu-consumer-deployment-798c7d785b-fp4b9     1/1     Running     3 (140m ago)    22d
magedu-consumer-deployment-798c7d785b-wmv9p     1/1     Running     3 (140m ago)    22d
magedu-consumer-deployment-798c7d785b-zqm74     1/1     Running     3 (13d ago)     22d
magedu-dubboadmin-deployment-798c4dfdd8-kvfvh   1/1     Running     3 (140m ago)    22d
magedu-provider-deployment-6fccc6d9f5-k6z7m     1/1     Running     3 (140m ago)    22d
magedu-provider-deployment-6fccc6d9f5-nl4zd     1/1     Running     3 (140m ago)    22d
magedu-provider-deployment-6fccc6d9f5-p94rb     1/1     Running     3 (140m ago)    22d
mysql-0                                         2/2     Running     12 (140m ago)   79d
mysql-1                                         2/2     Running     12 (140m ago)   79d
mysql-2                                         2/2     Running     12 (140m ago)   79d
redis-0                                         1/1     Running     8 (13d ago)     88d
redis-1                                         1/1     Running     8 (140m ago)    88d
redis-2                                         1/1     Running     8 (140m ago)    88d
redis-3                                         1/1     Running     8 (13d ago)     87d
redis-4                                         1/1     Running     8 (140m ago)    88d
redis-5                                         1/1     Running     8 (140m ago)    88d
ubuntu1804                                      0/1     Completed   0               88d
wordpress-app-deployment-64c956bf9c-6qp8q       2/2     Running     0               3m31s
zookeeper1-675c5477cb-vmwwq                     1/1     Running     10 (13d ago)    90d
zookeeper2-759fb6c6f-7jktr                      1/1     Running     10 (140m ago)   90d
zookeeper3-5c78bb5974-vxpbh                     1/1     Running     10 (140m ago)   90d
root@k8s-master01:/data/velero# kubectl svc -n magedu
error: unknown command "svc" for "kubectl"

Did you mean this?
        set
root@k8s-master01:/data/velero# kubectl get svc -n magedu 
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                        AGE
magedu-consumer-server      NodePort    10.100.208.121   <none>        80:49630/TCP                                   22d
magedu-dubboadmin-service   NodePort    10.100.244.92    <none>        80:31080/TCP                                   22d
magedu-provider-spec        NodePort    10.100.187.168   <none>        80:44873/TCP                                   22d
mysql                       ClusterIP   None             <none>        3306/TCP                                       4m6s
mysql-0                     ClusterIP   None             <none>        3306/TCP                                       13d
mysql-read                  ClusterIP   10.100.15.127    <none>        3306/TCP                                       79d
redis                       ClusterIP   None             <none>        6379/TCP                                       88d
redis-access                NodePort    10.100.117.185   <none>        6379:36379/TCP                                 88d
wordpress-app-spec          NodePort    10.100.189.214   <none>        80:30031/TCP,443:30033/TCP                     13d
zookeeper                   ClusterIP   10.100.177.73    <none>        2181/TCP                                       4m6s
zookeeper1                  NodePort    10.100.63.118    <none>        2181:32181/TCP,2888:30541/TCP,3888:31200/TCP   90d
zookeeper2                  NodePort    10.100.199.43    <none>        2181:32182/TCP,2888:32670/TCP,3888:32264/TCP   90d
zookeeper3                  NodePort    10.100.41.9      <none>        2181:32183/TCP,2888:31329/TCP,3888:32546/TCP   90d
root@k8s-master01:/data/velero# 

7.3、批量备份所有namespace

root@k8s-master01:/data/velero# cat all-ns-backup.sh
#!/bin/bash
NS_NAME=`kubectl get ns | awk '{if (NR>2){print}}' | awk '{print $1}'`
DATE=`date +%Y%m%d%H%M%S`
cd /data/velero/
for i in $NS_NAME;do
velero backup create ${i}-ns-backup-${DATE} \
--include-cluster-resources=true \
--include-namespaces ${i} \
--kubeconfig=/root/.kube/config \
--namespace velero-system
done
root@k8s-master01:/data/velero# 

执行脚本备份

root@k8s-master01:/data/velero# bash all-ns-backup.sh
Backup request "default-ns-backup-20230902143131" submitted successfully.
Run `velero backup describe default-ns-backup-20230902143131` or `velero backup logs default-ns-backup-20230902143131` for more details.
Backup request "kube-node-lease-ns-backup-20230902143131" submitted successfully.
Run `velero backup describe kube-node-lease-ns-backup-20230902143131` or `velero backup logs kube-node-lease-ns-backup-20230902143131` for more details.
Backup request "kube-public-ns-backup-20230902143131" submitted successfully.
Run `velero backup describe kube-public-ns-backup-20230902143131` or `velero backup logs kube-public-ns-backup-20230902143131` for more details.
Backup request "kube-system-ns-backup-20230902143131" submitted successfully.
Run `velero backup describe kube-system-ns-backup-20230902143131` or `velero backup logs kube-system-ns-backup-20230902143131` for more details.
Backup request "magedu-ns-backup-20230902143131" submitted successfully.
Run `velero backup describe magedu-ns-backup-20230902143131` or `velero backup logs magedu-ns-backup-20230902143131` for more details.
Backup request "myserver-ns-backup-20230902143131" submitted successfully.
Run `velero backup describe myserver-ns-backup-20230902143131` or `velero backup logs myserver-ns-backup-20230902143131` for more details.
Backup request "velero-system-ns-backup-20230902143131" submitted successfully.
Run `velero backup describe velero-system-ns-backup-20230902143131` or `velero backup logs velero-system-ns-backup-20230902143131` for more details.
root@k8s-master01:/data/velero#

在minio验证备份
基于velero及minio实现etcd数据备份与恢复-LMLPHP

09-03 06:47