From f684e16defbb1e3982ff6f184f79562184d6358e Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Wed, 21 Aug 2024 16:13:20 +0800 Subject: [PATCH] data mover ms doc Signed-off-by: Lyndon-Li --- .../docs/main/csi-snapshot-data-movement.md | 167 +++--------------- ...ata-movement-pod-resource-configuration.md | 4 + site/data/docs/main-toc.yml | 2 + 3 files changed, 33 insertions(+), 140 deletions(-) create mode 100644 site/content/docs/main/data-movement-pod-resource-configuration.md diff --git a/site/content/docs/main/csi-snapshot-data-movement.md b/site/content/docs/main/csi-snapshot-data-movement.md index 31419e3ee..740f14486 100644 --- a/site/content/docs/main/csi-snapshot-data-movement.md +++ b/site/content/docs/main/csi-snapshot-data-movement.md @@ -18,7 +18,7 @@ On the other hand, there are quite some cases that CSI snapshot is not available CSI Snapshot Data Movement supports both built-in data mover and customized data movers. For the details of how Velero works with customized data movers, check the [Volume Snapshot Data Movement design][1]. Velero provides a built-in data mover which uses Velero built-in uploaders (at present the available uploader is Kopia uploader) to read the snapshot data and write to the Unified Repository (by default implemented by Kopia repository). -The way for Velero built-in data mover to access the snapshot data is based on the hostpath access by Velero node-agent, so the node-agent pods need to run as root user and even under privileged mode in some environments, as same as [File System Backup][3]. +Velero built-in data mover restores both volume data and metadata, so the data mover pods need to run as root user. ## Setup CSI Snapshot Data Movement @@ -31,139 +31,13 @@ The way for Velero built-in data mover to access the snapshot data is based on t ### Install Velero Node Agent -Velero Node Agent is a Kubernetes daemonset that hosts Velero data movement modules, i.e., data mover controller, uploader & repository. +Velero Node Agent is a Kubernetes daemonset that hosts Velero data movement controllers and launch data mover pods. If you are using Velero built-in data mover, Node Agent must be installed. To install Node Agent, use the `--use-node-agent` flag. ``` velero install --use-node-agent ``` -### Configure Node Agent DaemonSet spec - -After installation, some PaaS/CaaS platforms based on Kubernetes also require modifications the node-agent DaemonSet spec. -The steps in this section are only needed if you are installing on RancherOS, Nutanix, OpenShift, OpenShift on IBM Cloud, VMware Tanzu Kubernetes Grid -Integrated Edition (formerly VMware Enterprise PKS), or Microsoft Azure. - - -**RancherOS** - - -Update the host path for volumes in the node-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to -`/opt/rke/var/lib/kubelet/pods`. - -```yaml -hostPath: - path: /var/lib/kubelet/pods -``` - -to - -```yaml -hostPath: - path: /opt/rke/var/lib/kubelet/pods -``` - -**Nutanix** - -Update the host path for volumes in the node-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to -`/var/nutanix/var/lib/kubelet`. - -```yaml -hostPath: - path: /var/lib/kubelet/pods -``` - -to - -```yaml -hostPath: - path: /var/nutanix/var/lib/kubelet -``` - -**OpenShift** - - -To mount the correct hostpath to pods volumes, run the node-agent pod in `privileged` mode. - -1. Add the `velero` ServiceAccount to the `privileged` SCC: - - ``` - oc adm policy add-scc-to-user privileged -z velero -n velero - ``` - -2. Install Velero with the '--privileged-node-agent' option to request a privileged mode: - - ``` - velero install --use-node-agent --privileged-node-agent - ``` - -If node-agent is not running in a privileged mode, it will not be able to access snapshot volumes within the mounted -hostpath directory because of the default enforced SELinux mode configured in the host system level. You can -[create a custom SCC][6] to relax the -security in your cluster so that node-agent pods are allowed to use the hostPath volume plugin without granting -them access to the `privileged` SCC. - -By default a userland openshift namespace will not schedule pods on all nodes in the cluster. - -To schedule on all nodes the namespace needs an annotation: - -``` -oc annotate namespace openshift.io/node-selector="" -``` - -This should be done before velero installation. - -Or the ds needs to be deleted and recreated: - -``` -oc get ds node-agent -o yaml -n > ds.yaml -oc annotate namespace openshift.io/node-selector="" -oc create -n -f ds.yaml -``` - -**OpenShift on IBM Cloud** - - -Update the host path and mount path for volumes in the node-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/plugins` to -`/var/data/kubelet/plugins`. - -```yaml -hostPath: - path: /var/lib/kubelet/plugins -``` - -to - -```yaml -hostPath: - path: /var/data/kubelet/plugins -``` - -and - -```yaml -- name: host-plugins - mountPath: /var/lib/kubelet/plugins -``` - -to - -```yaml -- name: host-plugins - mountPath: /var/data/kubelet/plugins -``` - -**VMware Tanzu Kubernetes Grid Integrated Edition (formerly VMware Enterprise PKS)** - -You need to enable the `Allow Privileged` option in your plan configuration so that Velero is able to mount the hostpath. - -The hostPath should be changed from `/var/lib/kubelet/pods` to `/var/vcap/data/kubelet/pods` - -```yaml -hostPath: - path: /var/vcap/data/kubelet/pods -``` - ### Configure A Backup Storage Location At present, Velero backup repository supports object storage as the backup storage. Velero gets the parameters from the @@ -338,17 +212,17 @@ If you are using a customized data mover, follow the data mover's instruction fo ## How backup and restore work CSI snapshot data movement is a combination of CSI snapshot and data movement, which is jointly executed by Velero server, CSI plugin and the data mover. -This section lists some general concept of how CSI snapshot data movement backup and restore work. For the detailed mechanisms and workflows, you can check the [Volume Snapshot Data Movement design][1]. +This section lists some general concept of how CSI snapshot data movement backup and restore work. For the detailed mechanisms and workflows, you can check the [Volume Snapshot Data Movement design][1] and [VGDP Micro Service For Volume Snapshot Data Movement design][18]. ### Custom resource and controllers Velero has three custom resource definitions and associated controllers: - `DataUpload` - represents a data upload of a volume snapshot. The CSI plugin creates one `DataUpload` per CSI snapshot. Data movers need to handle these CRs to finish the data upload process. -Velero built-in data mover runs a controller for this resource on each node (in node-agent daemonset). Controllers from different nodes may handle one CR in different phases, but finally the data transfer is done by one single controller which will call uploaders from the backend. +Velero built-in data mover runs a controller for this resource on each node (in node-agent daemonset). Controllers from different nodes may handle one CR in different phases, but finally the data transfer is done by a data mover pod in one node. - `DataDownload` - represents a data download of a volume snapshot. The CSI plugin creates one `DataDownload` per volume to be restored. Data movers need to handle these CRs to finish the data upload process. -Velero built-in data mover runs a controller for this resource on each node (in node-agent daemonset). Controllers from different nodes may handle one CR in different phases, but finally the data transfer is done by one single controller which will call uploaders from the backend. +Velero built-in data mover runs a controller for this resource on each node (in node-agent daemonset). Controllers from different nodes may handle one CR in different phases, but finally the data transfer is done by a data mover pod in one node. - `BackupRepository` - represents/manages the lifecycle of Velero's backup repositories. Velero creates a backup repository per namespace when the first CSI snapshot backup/restore for a namespace is requested. You can see information about your Velero's backup repositories by running `velero repo get`. This CR is used by Velero built-in data movers, customized data movers may or may not use it. @@ -371,8 +245,10 @@ Velero backs up resources for CSI snapshot data movement backup in the same way - Velero built-in data mover creates a volume from the CSI snapshot and transfer the data to the backup storage according to the backup storage location defined by users. - After the volume is created from the CSI snapshot, Velero built-in data mover waits for Kubernetes to provision the volume, this may take some time varying from storage providers, but if the provision cannot be finished in a given time, Velero built-in data mover will cancel this `DataUpload` CR. The timeout is configurable through a node-agent's parameter `data-mover-prepare-timeout`, the default value is 30 minutes. +- Velero built-in data mover launches a data mover pod to transfer the data from the provisioned volume to the backup storage. - When the data transfer completes or any error happens, Velero built-in data mover sets the `DataUpload` CR to the terminal state, either `Completed` or `Failed`. - Velero built-in data mover also monitors the cancellation request to the `DataUpload` CR, once that happens, it cancels its ongoing activities, cleans up the intermediate resources and set the `DataUpload` CR to `Cancelled`. +- Throughout the data transfer, Velero built-in data mover monitors the status of the data mover pod and delete it after `DataUpload` CR is set to the terminal state. ### Restore @@ -389,9 +265,10 @@ Velero restores resources for CSI snapshot data movement restore in the same way - Velero built-in data mover creates a volume with the same specification of the source volume. - Velero built-in data mover waits for Kubernetes to provision the volume, this may take some time varying from storage providers, but if the provision cannot be finished in a given time, Velero built-in data mover will cancel this `DataDownload` CR. The timeout is configurable through the same node-agent's parameter `data-mover-prepare-timeout`. -- After the volume is provisioned, Velero built-in data mover starts to transfer the data from the backup storage according to the backup storage location defined by users. +- After the volume is provisioned, Velero built-in data mover starts a data mover pod to transfer the data from the backup storage according to the backup storage location defined by users. - When the data transfer completes or any error happens, Velero built-in data mover sets the `DataDownload` CR to the terminal state, either `Completed` or `Failed`. - Velero built-in data mover also monitors the cancellation request to the `DataDownload` CR, once that happens, it cancels its ongoing activities, cleans up the intermediate resources and set the `DataDownload` CR to `Cancelled`. +- Throughout the data transfer, Velero built-in data mover monitors the status of the data mover pod and delete it after `DataDownload` CR is set to the terminal state. ### Backup Deletion When a backup is created, a snapshot is saved into the repository for the volume data. The snapshot is a reference to the volume data saved in the repository. @@ -424,22 +301,28 @@ kubectl -n velero get datauploads -l velero.io/backup-name=YOUR_BACKUP_NAME -w kubectl -n velero get datadownloads -l velero.io/restore-name=YOUR_RESTORE_NAME -w ``` +### Restart and resume +When Velero server is restarted, if the resource backup/restore has completed, so the backup/restore has excceds `InProgress` status and is waiting for the completion of the data movements, Velero will recapture the status of the running data movements and resume the execution. +When node-agent is restarted, if the `DataUpload`/`DataDownload` is in `InProgress` status, Velero recapture the status of the running data mover pod and resume the execution. +When node-agent is restarted, if the `DataUpload`/`DataDownload` is in `New` or `Prepared` status, the data mover pod has not started, Velero processes it as normal cases, or the restart doesn't affect the execution. + ### Cancellation At present, Velero backup and restore doesn't support end to end cancellation that is launched by users. However, Velero cancels the `DataUpload`/`DataDownload` in below scenarios automatically: -- When Velero server is restarted -- When node-agent is restarted +- When Velero server is restarted and the backup/restore is in `InProgress` status +- When node-agent is restarted and the `DataUpload`/`DataDownload` is in `Accepted` status +- When node-agent is restarted and the resume of an existing `DataUpload`/`DataDownload` that is in `InProgress` status fails - When an ongoing backup/restore is deleted - When a backup/restore does not finish before the item operation timeout (default value is `4 hours`) Customized data movers that support cancellation could cancel their ongoing tasks and clean up any intermediate resources. If you are using Velero built-in data mover, the cancellation is supported. ### Support ReadOnlyRootFilesystem setting -When the Velero server/node-agent pod's SecurityContext sets the `ReadOnlyRootFileSystem` parameter to true, the Velero server/node-agent pod's filesystem is running in read-only mode. Then the backup/restore may fail, because the uploader/repository needs to write some cache and configuration data into the pod's root filesystem. +When the Velero server pod's SecurityContext sets the `ReadOnlyRootFileSystem` parameter to true, the Velero server pod's filesystem is running in read-only mode. Then the backup deletion may fail, because the repository needs to write some cache and configuration data into the pod's root filesystem. ``` -Errors: Velero: name: /mongodb-0 message: /Error backing up item error: /failed to wait BackupRepository: backup repository is not ready: error to connect to backup repo: error to connect repo with storage: error to connect to repository: unable to write config file: unable to create config directory: mkdir /home/cnb/udmrepo: read-only file system name: /mongodb-1 message: /Error backing up item error: /failed to wait BackupRepository: backup repository is not ready: error to connect to backup repo: error to connect repo with storage: error to connect to repository: unable to write config file: unable to create config directory: mkdir /home/cnb/udmrepo: read-only file system name: /mongodb-2 message: /Error backing up item error: /failed to wait BackupRepository: backup repository is not ready: error to connect to backup repo: error to connect repo with storage: error to connect to repository: unable to write config file: unable to create config directory: mkdir /home/cnb/udmrepo: read-only file system Cluster: +Errors: /error to connect repo with storage: error to connect to repository: unable to write config file: unable to create config directory: mkdir /home/cnb/udmrepo: read-only file system ``` The workaround is making those directories as ephemeral k8s volumes, then those directories are not counted as pod's root filesystem. @@ -473,15 +356,17 @@ spec: ...... ``` +At present, Velero doesn't allow to set `ReadOnlyRootFileSystem` parameter to data mover pods, so the root filesystem for the data mover pods are always writable. + ### Resource Consumption Both the uploader and repository consume remarkable CPU/memory during the backup/restore, especially for massive small files or large backup size cases. -For Velero built-in data mover, Velero uses [BestEffort as the QoS][13] for node-agent pods (so no CPU/memory request/limit is set), so that backups/restores wouldn't fail due to resource throttling in any cases. -If you want to constraint the CPU/memory usage, you need to [customize the resource limits][11]. The CPU/memory consumption is always related to the scale of data to be backed up/restored, refer to [Performance Guidance][12] for more details, so it is highly recommended that you perform your own testing to find the best resource limits for your data. +For Velero built-in data mover, Velero uses [BestEffort as the QoS][13] for data mover pods (so no CPU/memory request/limit is set), so that backups/restores wouldn't fail due to resource throttling in any cases. +If you want to constraint the CPU/memory usage, you need to [customize data mover pod resource limits][11]. The CPU/memory consumption is always related to the scale of data to be backed up/restored, refer to [Performance Guidance][12] for more details, so it is highly recommended that you perform your own testing to find the best resource limits for your data. During the restore, the repository may also cache data/metadata so as to reduce the network footprint and speed up the restore. The repository uses its own policy to store and clean up the cache. -For Kopia repository, the cache is stored in the node-agent pod's root file system and the cleanup is triggered for the data/metadata that are older than 10 minutes (not configurable at present). So you should prepare enough disk space, otherwise, the node-agent pod may be evicted due to running out of the ephemeral storage. +For Kopia repository, the cache is stored in the data mover pod's root file system. Velero allows you to configure a limit of the cache size so that the data mover pod won't be evicted due to running out of the ephemeral storage. For more details, check [Backup Repository Configuration][17]. ### Node Selection @@ -507,10 +392,12 @@ advanced configuration options for `BackupPVC`, allowing users to fine-tune acce [8]: api-types/backupstoragelocation.md [9]: supported-providers.md [10]: restore-reference.md#changing-pv/pvc-Storage-Classes -[11]: customize-installation.md#customize-resource-requests-and-limits +[11]: data-movement-pod-resource-configuration.md [12]: performance-guidance.md [13]: https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/ [14]: node-agent-concurrency.md [15]: data-movement-backup-node-selection.md [16]: data-movement-backup-pvc-configuration.md +[17]: backup-repository-configuration.md +[18]: https://github.com/vmware-tanzu/velero/pull/7576 diff --git a/site/content/docs/main/data-movement-pod-resource-configuration.md b/site/content/docs/main/data-movement-pod-resource-configuration.md new file mode 100644 index 000000000..e35bc2433 --- /dev/null +++ b/site/content/docs/main/data-movement-pod-resource-configuration.md @@ -0,0 +1,4 @@ +--- +title: "Data Movement Pod Resource Configuration" +layout: docs +--- \ No newline at end of file diff --git a/site/data/docs/main-toc.yml b/site/data/docs/main-toc.yml index ec5f04604..372a612d1 100644 --- a/site/data/docs/main-toc.yml +++ b/site/data/docs/main-toc.yml @@ -55,6 +55,8 @@ toc: url: /node-agent-concurrency - page: Backup PVC Configuration url: /data-movement-backup-pvc-configuration + - page: Data Movement Pod Resource Configuration + url: /data-movement-pod-resource-configuration - page: Verifying Self-signed Certificates url: /self-signed-certificates - page: Changing RBAC permissions