備份和遷移 Kubernetes 利器：Velero

你是否在運(yùn)維kubernetes集群中有過(guò)這樣的經(jīng)歷：

?個(gè)新?把某個(gè)namespace點(diǎn)擊刪除，導(dǎo)致這下?所有的資源全部丟失，只能?步?步的重新部署。新搭建集群，為了保證環(huán)境盡可能?致，只能從?集群拿出來(lái)yaml?件在新集群中瘋狂apply。令?抓狂的瞬間隨之?來(lái)的就是浪費(fèi)?好?春的搬磚時(shí)光。

現(xiàn)在已經(jīng)開源了很多集群資源對(duì)象備份的?具，把這些?具利?起來(lái)讓你的?作事半功倍，不在苦逼加班。

1

集群備份?較

1.etcd備份

etcd備份可以實(shí)現(xiàn)K8S集群的備份，但是這種備份?般是全局的，可以恢復(fù)到集群某?時(shí)刻的狀態(tài)，? 法精確到恢復(fù)某?資源對(duì)象，?般使?快照的形式進(jìn)?備份和恢復(fù)。

# 備份#!/usr/bin/env bashdate;CACERT="/opt/kubernetes/ssl/ca.pem"CERT="/opt/kubernetes/ssl/server.pem"EKY="/opt/kubernetes/ssl/server-key.pem"ENDPOINTS="192.168.1.36:2379" ETCDCTL_API=3 etcdctl \--cacert="${CACERT}" --cert="${CERT}" --key="${EKY}" \--endpoints=${ENDPOINTS} \snapshot save /data/etcd_backup_dir/etcd-snapshot-`date +%Y%m%d`.db # 備份保留30天find /data/etcd_backup_dir/ -name *.db -mtime +30 -exec rm -f {} \;

2.資源對(duì)象備份

對(duì)于更?粒度的劃分到每種資源對(duì)象的備份，對(duì)于誤刪除了某種namespace或deployment以及集群遷 移就很有?了。現(xiàn)在開源?具有很多都提供了這樣的功能，?如Velero, PX-Backup，Kasten。

你是否在運(yùn)維kubernetes集群中有過(guò)這樣的經(jīng)歷：

?個(gè)新?把某個(gè)namespace點(diǎn)擊刪除，導(dǎo)致這下?所有的資源全部丟失，只能?步?步的重新部署。新搭建集群，為了保證環(huán)境盡可能?致，只能從?集群拿出來(lái)yaml?件在新集群中瘋狂apply。令?抓狂的瞬間隨之?來(lái)的就是浪費(fèi)?好?春的搬磚時(shí)光。

現(xiàn)在已經(jīng)開源了很多集群資源對(duì)象備份的?具，把這些?具利?起來(lái)讓你的?作事半功倍，不在苦逼加班。

1

集群備份?較

1.etcd備份

# 備份#!/usr/bin/env bashdate;CACERT="/opt/kubernetes/ssl/ca.pem"CERT="/opt/kubernetes/ssl/server.pem"EKY="/opt/kubernetes/ssl/server-key.pem"ENDPOINTS="192.168.1.36:2379" ETCDCTL_API=3 etcdctl \--cacert="${CACERT}" --cert="${CERT}" --key="${EKY}" \--endpoints=${ENDPOINTS} \snapshot save /data/etcd_backup_dir/etcd-snapshot-`date +%Y%m%d`.db # 備份保留30天find /data/etcd_backup_dir/ -name *.db -mtime +30 -exec rm -f {} \;

velero:

Velero is an open source tool to safely backup and restore, perform disaster recovery, andmigrate Kubernetes cluster resources and persistent volumes.

Built from the ground up for Kubernetes, PX-Backup delivers enterprise-grade applicationand data protection with fast recovery at the click of a button

urpose-built for Kubernetes, Kasten K10 provides enterprise operations teams an easy-touse, scalable, and secure system for backup/restore, disaster recovery, and mobility ofKubernetes applications.

2

velero

官?介紹的velero提到了以上三個(gè)功能，主要就是備份恢復(fù)和遷移。

可以看到創(chuàng)建了很多crd，并最終在veleronamespace下將應(yīng)?跑起來(lái)了。其實(shí)從crd的命名上就可以 看出他?概有哪些?途了。

2.定時(shí)備份

對(duì)于運(yùn)維?員來(lái)說(shuō)，對(duì)外提供?個(gè)集群的穩(wěn)定性保證是必不可少的，這就需要我們開啟定時(shí)備份功能。通過(guò)命令?能夠開始定時(shí)任務(wù)，指定那么分區(qū)，保留多少時(shí)間的備份數(shù)據(jù)，每隔多?時(shí)間進(jìn)?備份?次。

Examples: # Create a backup every 6 hours velero create schedule NAME --schedule="0 */6 * * *" # Create a backup every 6 hours with the @every notation velero create schedule NAME --schedule="@every 6h" # Create a daily backup of the web namespace velero create schedule NAME --schedule="@every 24h" --include-namespaces web # Create a weekly backup, each living for 90 days (2160 hours) velero create schedule NAME --schedule="@every 168h" --ttl 2160h0m0s

velero create schedule 360cloud --schedule="@every 24h" --ttl 2160h0m0sSchedule "360cloud" created successfully.[root@xxxxx ~]# kubectl get schedules --all-namespacesNAMESPACE NAME AGEvelero 360cloud 40s[root@xxxxx ~]# kubectl get schedules -n velero 360cloud -o yamlapiVersion: velero.io/v1kind: Schedulemetadata: generation: 3 name: 360cloud namespace: velero resourceVersion: "18164238" selfLink: /apis/velero.io/v1/namespaces/velero/schedules/360cloud uid: 7c04af34-1529-4b48-a3d1-d2f5e98de328spec: schedule: '@every 24h' template: hooks: {} includedNamespaces: - '*' ttl: 2160h0m0sstatus: lastBackup: "2021-03-07T08:18:49Z" phase: Enabled

3.集群遷移備份

對(duì)于我們要遷移部分的資源對(duì)象，可能并沒(méi)有進(jìn)?定時(shí)備份，可能有了定時(shí)備份，但是想要最新的數(shù)據(jù)。那么備份?個(gè)?次性的數(shù)據(jù)?來(lái)遷移就好了。

velero backup create test01 --include-namespaces defaultBackup request "test01" submitted successfully.Run `velero backup describe test01` or `velero backup logs test01` for moredetails.[root@xxxxx ~]# velero backup describe test01Name: test01Namespace: veleroLabels: velero.io/storage-location=defaultAnnotations: velero.io/source-cluster-k8s-gitversion=v1.19.7 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=19Phase: InProgressErrors: 0Warnings: 0Namespaces: Included: default Excluded: Resources: Included: * Excluded: Cluster-scoped: autoLabel selector: Storage Location: defaultVelero-Native Snapshot PVs: autoTTL: 720h0m0sHooks: Backup Format Version: 1.1.0Started: 2021-03-07 16:44:52 +0800 CSTCompleted: Expiration: 2021-04-06 16:44:52 +0800 CSTVelero-Native Snapshots:

[root@xxxxx ~]# velero restore create --from-backup test01Restore request "test01-20210307164809" submitted successfully.Run `velero restore describe test01-20210307164809` or `velero restore logstest01-20210307164809` for more details.[root@xxxxx ~]# kuebctl ^C[root@xxxxx ~]# kubectl get podNAME READY STATUS RESTARTS AGEnginx-6799fc88d8-4bnfg 0/1 ContainerCreating 0 6snginx-6799fc88d8-cq82j 0/1 ContainerCreating 0 6snginx-6799fc88d8-f6qsx 0/1 ContainerCreating 0 6snginx-6799fc88d8-gq2xt 0/1 ContainerCreating 0 6snginx-6799fc88d8-j5fc7 0/1 ContainerCreating 0 6snginx-6799fc88d8-kvvx6 0/1 ContainerCreating 0 5snginx-6799fc88d8-pccc4 0/1 ContainerCreating 0 5snginx-6799fc88d8-q2fnt 0/1 ContainerCreating 0 4snginx-6799fc88d8-r9dqn 0/1 ContainerCreating 0 4snginx-6799fc88d8-zqv6v 0/1 ContainerCreating 0 4s

s3中的存儲(chǔ)記錄：

3

PVC的備份遷移

velero install --use-restic

apiVersion: v1kind: Podmetadata: annotations: backup.velero.io/backup-volumes: mypvc name: rbd-testspec: containers: - name: web-server image: nginx volumeMounts: - name: mypvc mountPath: /var/lib/www/html volumes: - name: mypvc persistentVolumeClaim: claimName: rbd-pvc-zhf readOnly: false

可以通過(guò) opt-in , opt-out 的形式，為pod添加注解來(lái)進(jìn)?選擇需要備份的pod中的volume。

velero backup create testpvc05 --snapshot-volumes=true --include-namespacesdefaultBackup request "testpvc05" submitted successfully.Run `velero backup describe testpvc05` or `velero backup logs testpvc05` formore details.[root@xxxx ceph]# velero backup describe testpvc05Name: testpvc05Namespace: veleroLabels: velero.io/storage-location=defaultAnnotations: velero.io/source-cluster-k8s-gitversion=v1.19.7 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=19Phase: CompletedErrors: 0Warnings: 0Namespaces: Included: default Excluded: Resources: Included: * Excluded: Cluster-scoped: autoLabel selector: Storage Location: defaultVelero-Native Snapshot PVs: trueTTL: 720h0m0sHooks: Backup Format Version: 1.1.0Started: 2021-03-10 15:11:26 +0800 CSTCompleted: 2021-03-10 15:11:36 +0800 CSTExpiration: 2021-04-09 15:11:26 +0800 CSTTotal items to be backed up: 92Items backed up: 92Velero-Native Snapshots: Restic Backups (specify --details for more information): Completed: 1

[root@xxxxxx ceph]# kubectl delete pod rbd-testpod "rbd-test" deletedkubectl delete pvc[root@p48453v ceph]# kubectl delete pvc rbd-pvc-zhfpersistentvolumeclaim "rbd-pvc-zhf" deleted

[root@xxxxx ceph]# velero restore create testpvc05 --restore-volumes=true--from-backup testpvc05Restore request "testpvc05" submitted successfully.Run `velero restore describe testpvc05` or `velero restore logs testpvc05` formore details.[root@xxxxxx ceph]#[root@xxxxxx ceph]# kuebctl^C[root@xxxxxx ceph]# kubectl get podNAME READY STATUS RESTARTS AGEnginx-6799fc88d8-4bnfg 1/1 Running 0 2d22hrbd-test 0/1 Init:0/1 0 6s

數(shù)據(jù)恢復(fù)顯示

[root@xxxxxx ceph]# kubectl exec rbd-test sh -- ls -l /var/lib/www/htmltotal 20drwx------ 2 root root 16384 Mar 10 06:31 lost+found-rw-r--r-- 1 root root 13 Mar 10 07:11 zheng.txt[root@xxxxxx ceph]# kubectl exec rbd-test sh -- cat/var/lib/www/html/zheng.txtzhenghongfei[root@xxxxx ceph]#

4

HOOK

metadata: name: nginx-deployment namespace: nginx-examplespec: replicas: selector: matchLabels: app: nginx template: metadata: labels: app: nginx annotations: pre.hook.backup.velero.io/container: fsfreeze pre.hook.backup.velero.io/command: '["/sbin/fsfreeze", "--freeze","/var/log/nginx"]' post.hook.backup.velero.io/container: fsfreeze post.hook.backup.velero.io/command: '["/sbin/fsfreeze", "--unfreeze","/var/log/nginx"]'

引導(dǎo)使?前置和后置掛鉤凍結(jié)?件系統(tǒng)。凍結(jié)?件系統(tǒng)有助于確保所有掛起的磁盤IO操作在拍攝快照之 前已經(jīng)完成。

當(dāng)然我們可以使?這種?式執(zhí)?備份mysql或其他的?件,但是只建議使???件會(huì)備份恢復(fù)，針對(duì)于 pod進(jìn)?備份恢復(fù)。

5

探究備份實(shí)現(xiàn)

collector := &itemCollector{ log: log, backupRequest: backupRequest, discoveryHelper: kb.discoveryHelper, dynamicFactory: kb.dynamicFactory, cohabitatingResources: cohabitatingResources(), dir: tempDir, } items := collector.getAllItems()

調(diào)?函數(shù)

func (kb *kubernetesBackupper) backupItem(log logrus.FieldLogger, grschema.GroupResource, itemBackupper *itemBackupper, unstructured*unstructured.Unstructured, preferredGVR schema.GroupVersionResource) bool { backedUpItem, err := itemBackupper.backupItem(log, unstructured, gr,preferredGVR) if aggregate, ok := err.(kubeerrs.Aggregate); ok { log.WithField("name", unstructured.GetName()).Infof("%d errors encounteredbackup up item", len(aggregate.Errors())) // log each error separately so we get error location info in the log, andan // accurate count of errors for _, err = range aggregate.Errors() { log.WithError(err).WithField("name",unstructured.GetName()).Error("Error backing up item") } return false } if err != nil { log.WithError(err).WithField("name", unstructured.GetName()).Error("Errorbacking up item") return false } return backedUpItem}

client, err :=ib.dynamicFactory.ClientForGroupVersionResource(gvr.GroupVersion(), resource,additionalItem.Namespace) if err != nil { return nil, err } item, err := client.Get(additionalItem.Name, metav1.GetOptions{})

log.Debugf("Resource %s/%s, version= %s, preferredVersion=%s",groupResource.String(), name, version, preferredVersion) if version == preferredVersion { if namespace != "" { filePath = filepath.Join(velerov1api.ResourcesDir,groupResource.String(), velerov1api.NamespaceScopedDir, namespace,name+".json")PX-Backup kanisterhttps://github.com/vmware-tanzu/velerohttps://portworx.com/https://www.kasten.io/https://github.com/kanisterio/kanisterhttps://duyanghao.github.io/kubernetes-ha-and-bur/https://blog.kubernauts.io/backup-and-restore-of-kubernetes-applications-using-heptios-velerowith-restic-and-rook-ceph-as-2e8df15b1487 } else { filePath = filepath.Join(velerov1api.ResourcesDir,groupResource.String(), velerov1api.ClusterScopedDir, name+".json") } hdr = &tar.Header{ Name: filePath, Size: int64(len(itemBytes)), Typeflag: tar.TypeReg, Mode: 0755, ModTime: time.Now(), } if err := ib.tarWriter.WriteHeader(hdr); err != nil { return false, errors.WithStack(err) } if _, err := ib.tarWriter.Write(itemBytes); err != nil { return false, errors.WithStack(err) }}

5

其他的備份工具

PX-Backup 需要交費(fèi)的產(chǎn)品，??幣玩家可以更加強(qiáng)?。kanister更傾向于數(shù)據(jù)上的存儲(chǔ)和恢復(fù)，?如etcd的snap，mongo等。

參考鏈接：

https://github.com/vmware-tanzu/velero https://portworx.com/ https://www.kasten.io/ https://github.com/kanisterio/kanister https://duyanghao.github.io/kubernetes-ha-and-bur/ https://blog.kubernauts.io/backup-and-restore-of-kubernetes-applications-using-heptios-velerowith-restic-and-rook-ceph-as-2e8df15b1487

Kubernetes 運(yùn)維

版權(quán)聲明：本文內(nèi)容由網(wǎng)絡(luò)用戶投稿，版權(quán)歸原作者所有，本站不擁有其著作權(quán)，亦不承擔(dān)相應(yīng)法律責(zé)任。如果您發(fā)現(xiàn)本站中有涉嫌抄襲或描述失實(shí)的內(nèi)容，請(qǐng)聯(lián)系我們jiasou666@gmail.com 處理，核實(shí)后本網(wǎng)站將在24小時(shí)內(nèi)刪除侵權(quán)內(nèi)容。