mirror of
https://github.com/vmware-tanzu/velero.git
synced 2026-09-29 19:25:49 +00:00
Merge pull request #8144 from Lyndon-Li/data-mover-ms-doc
Data mover micro service doc
This commit is contained in:
Shubham Pampattiwar
@@ -0,0 +1 @@
|
||||
Add document for data mover micro service
|
||||
@@ -18,7 +18,7 @@ On the other hand, there are quite some cases that CSI snapshot is not available
|
||||
|
||||
CSI Snapshot Data Movement supports both built-in data mover and customized data movers. For the details of how Velero works with customized data movers, check the [Volume Snapshot Data Movement design][1]. Velero provides a built-in data mover which uses Velero built-in uploaders (at present the available uploader is Kopia uploader) to read the snapshot data and write to the Unified Repository (by default implemented by Kopia repository).
|
||||
|
||||
The way for Velero built-in data mover to access the snapshot data is based on the hostpath access by Velero node-agent, so the node-agent pods need to run as root user and even under privileged mode in some environments, as same as [File System Backup][3].
|
||||
Velero built-in data mover restores both volume data and metadata, so the data mover pods need to run as root user.
|
||||
|
||||
## Setup CSI Snapshot Data Movement
|
||||
|
||||
@@ -31,139 +31,13 @@ The way for Velero built-in data mover to access the snapshot data is based on t
|
||||
|
||||
### Install Velero Node Agent
|
||||
|
||||
Velero Node Agent is a Kubernetes daemonset that hosts Velero data movement modules, i.e., data mover controller, uploader & repository.
|
||||
Velero Node Agent is a Kubernetes daemonset that hosts Velero data movement controllers and launches data mover pods.
|
||||
If you are using Velero built-in data mover, Node Agent must be installed. To install Node Agent, use the `--use-node-agent` flag.
|
||||
|
||||
```
|
||||
velero install --use-node-agent
|
||||
```
|
||||
|
||||
### Configure Node Agent DaemonSet spec
|
||||
|
||||
After installation, some PaaS/CaaS platforms based on Kubernetes also require modifications the node-agent DaemonSet spec.
|
||||
The steps in this section are only needed if you are installing on RancherOS, Nutanix, OpenShift, OpenShift on IBM Cloud, VMware Tanzu Kubernetes Grid
|
||||
Integrated Edition (formerly VMware Enterprise PKS), or Microsoft Azure.
|
||||
|
||||
|
||||
**RancherOS**
|
||||
|
||||
|
||||
Update the host path for volumes in the node-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to
|
||||
`/opt/rke/var/lib/kubelet/pods`.
|
||||
|
||||
```yaml
|
||||
hostPath:
|
||||
path: /var/lib/kubelet/pods
|
||||
```
|
||||
|
||||
to
|
||||
|
||||
```yaml
|
||||
hostPath:
|
||||
path: /opt/rke/var/lib/kubelet/pods
|
||||
```
|
||||
|
||||
**Nutanix**
|
||||
|
||||
Update the host path for volumes in the node-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/pods` to
|
||||
`/var/nutanix/var/lib/kubelet`.
|
||||
|
||||
```yaml
|
||||
hostPath:
|
||||
path: /var/lib/kubelet/pods
|
||||
```
|
||||
|
||||
to
|
||||
|
||||
```yaml
|
||||
hostPath:
|
||||
path: /var/nutanix/var/lib/kubelet
|
||||
```
|
||||
|
||||
**OpenShift**
|
||||
|
||||
|
||||
To mount the correct hostpath to pods volumes, run the node-agent pod in `privileged` mode.
|
||||
|
||||
1. Add the `velero` ServiceAccount to the `privileged` SCC:
|
||||
|
||||
```
|
||||
oc adm policy add-scc-to-user privileged -z velero -n velero
|
||||
```
|
||||
|
||||
2. Install Velero with the '--privileged-node-agent' option to request a privileged mode:
|
||||
|
||||
```
|
||||
velero install --use-node-agent --privileged-node-agent
|
||||
```
|
||||
|
||||
If node-agent is not running in a privileged mode, it will not be able to access snapshot volumes within the mounted
|
||||
hostpath directory because of the default enforced SELinux mode configured in the host system level. You can
|
||||
[create a custom SCC][6] to relax the
|
||||
security in your cluster so that node-agent pods are allowed to use the hostPath volume plugin without granting
|
||||
them access to the `privileged` SCC.
|
||||
|
||||
By default a userland openshift namespace will not schedule pods on all nodes in the cluster.
|
||||
|
||||
To schedule on all nodes the namespace needs an annotation:
|
||||
|
||||
```
|
||||
oc annotate namespace <velero namespace> openshift.io/node-selector=""
|
||||
```
|
||||
|
||||
This should be done before velero installation.
|
||||
|
||||
Or the ds needs to be deleted and recreated:
|
||||
|
||||
```
|
||||
oc get ds node-agent -o yaml -n <velero namespace> > ds.yaml
|
||||
oc annotate namespace <velero namespace> openshift.io/node-selector=""
|
||||
oc create -n <velero namespace> -f ds.yaml
|
||||
```
|
||||
|
||||
**OpenShift on IBM Cloud**
|
||||
|
||||
|
||||
Update the host path and mount path for volumes in the node-agent DaemonSet in the Velero namespace from `/var/lib/kubelet/plugins` to
|
||||
`/var/data/kubelet/plugins`.
|
||||
|
||||
```yaml
|
||||
hostPath:
|
||||
path: /var/lib/kubelet/plugins
|
||||
```
|
||||
|
||||
to
|
||||
|
||||
```yaml
|
||||
hostPath:
|
||||
path: /var/data/kubelet/plugins
|
||||
```
|
||||
|
||||
and
|
||||
|
||||
```yaml
|
||||
- name: host-plugins
|
||||
mountPath: /var/lib/kubelet/plugins
|
||||
```
|
||||
|
||||
to
|
||||
|
||||
```yaml
|
||||
- name: host-plugins
|
||||
mountPath: /var/data/kubelet/plugins
|
||||
```
|
||||
|
||||
**VMware Tanzu Kubernetes Grid Integrated Edition (formerly VMware Enterprise PKS)**
|
||||
|
||||
You need to enable the `Allow Privileged` option in your plan configuration so that Velero is able to mount the hostpath.
|
||||
|
||||
The hostPath should be changed from `/var/lib/kubelet/pods` to `/var/vcap/data/kubelet/pods`
|
||||
|
||||
```yaml
|
||||
hostPath:
|
||||
path: /var/vcap/data/kubelet/pods
|
||||
```
|
||||
|
||||
### Configure A Backup Storage Location
|
||||
|
||||
At present, Velero backup repository supports object storage as the backup storage. Velero gets the parameters from the
|
||||
@@ -338,17 +212,17 @@ If you are using a customized data mover, follow the data mover's instruction fo
|
||||
## How backup and restore work
|
||||
|
||||
CSI snapshot data movement is a combination of CSI snapshot and data movement, which is jointly executed by Velero server, CSI plugin and the data mover.
|
||||
This section lists some general concept of how CSI snapshot data movement backup and restore work. For the detailed mechanisms and workflows, you can check the [Volume Snapshot Data Movement design][1].
|
||||
This section lists some general concept of how CSI snapshot data movement backup and restore work. For the detailed mechanisms and workflows, you can check the [Volume Snapshot Data Movement design][1] and [VGDP Micro Service For Volume Snapshot Data Movement design][18].
|
||||
|
||||
### Custom resource and controllers
|
||||
|
||||
Velero has three custom resource definitions and associated controllers:
|
||||
|
||||
- `DataUpload` - represents a data upload of a volume snapshot. The CSI plugin creates one `DataUpload` per CSI snapshot. Data movers need to handle these CRs to finish the data upload process.
|
||||
Velero built-in data mover runs a controller for this resource on each node (in node-agent daemonset). Controllers from different nodes may handle one CR in different phases, but finally the data transfer is done by one single controller which will call uploaders from the backend.
|
||||
Velero built-in data mover runs a controller for this resource on each node (in node-agent daemonset). Controllers from different nodes may handle one CR in different phases, but finally the data transfer is done by a data mover pod in one node.
|
||||
|
||||
- `DataDownload` - represents a data download of a volume snapshot. The CSI plugin creates one `DataDownload` per volume to be restored. Data movers need to handle these CRs to finish the data upload process.
|
||||
Velero built-in data mover runs a controller for this resource on each node (in node-agent daemonset). Controllers from different nodes may handle one CR in different phases, but finally the data transfer is done by one single controller which will call uploaders from the backend.
|
||||
Velero built-in data mover runs a controller for this resource on each node (in node-agent daemonset). Controllers from different nodes may handle one CR in different phases, but finally the data transfer is done by a data mover pod in one node.
|
||||
|
||||
- `BackupRepository` - represents/manages the lifecycle of Velero's backup repositories. Velero creates a backup repository per namespace when the first CSI snapshot backup/restore for a namespace is requested. You can see information about your Velero's backup repositories by running `velero repo get`.
|
||||
This CR is used by Velero built-in data movers, customized data movers may or may not use it.
|
||||
@@ -371,8 +245,10 @@ Velero backs up resources for CSI snapshot data movement backup in the same way
|
||||
|
||||
- Velero built-in data mover creates a volume from the CSI snapshot and transfer the data to the backup storage according to the backup storage location defined by users.
|
||||
- After the volume is created from the CSI snapshot, Velero built-in data mover waits for Kubernetes to provision the volume, this may take some time varying from storage providers, but if the provision cannot be finished in a given time, Velero built-in data mover will cancel this `DataUpload` CR. The timeout is configurable through a node-agent's parameter `data-mover-prepare-timeout`, the default value is 30 minutes.
|
||||
- Velero built-in data mover launches a data mover pod to transfer the data from the provisioned volume to the backup storage.
|
||||
- When the data transfer completes or any error happens, Velero built-in data mover sets the `DataUpload` CR to the terminal state, either `Completed` or `Failed`.
|
||||
- Velero built-in data mover also monitors the cancellation request to the `DataUpload` CR, once that happens, it cancels its ongoing activities, cleans up the intermediate resources and set the `DataUpload` CR to `Cancelled`.
|
||||
- Throughout the data transfer, Velero built-in data mover monitors the status of the data mover pod and deletes it after `DataUpload` CR is set to the terminal state.
|
||||
|
||||
### Restore
|
||||
|
||||
@@ -389,9 +265,10 @@ Velero restores resources for CSI snapshot data movement restore in the same way
|
||||
|
||||
- Velero built-in data mover creates a volume with the same specification of the source volume.
|
||||
- Velero built-in data mover waits for Kubernetes to provision the volume, this may take some time varying from storage providers, but if the provision cannot be finished in a given time, Velero built-in data mover will cancel this `DataDownload` CR. The timeout is configurable through the same node-agent's parameter `data-mover-prepare-timeout`.
|
||||
- After the volume is provisioned, Velero built-in data mover starts to transfer the data from the backup storage according to the backup storage location defined by users.
|
||||
- After the volume is provisioned, Velero built-in data mover starts a data mover pod to transfer the data from the backup storage according to the backup storage location defined by users.
|
||||
- When the data transfer completes or any error happens, Velero built-in data mover sets the `DataDownload` CR to the terminal state, either `Completed` or `Failed`.
|
||||
- Velero built-in data mover also monitors the cancellation request to the `DataDownload` CR, once that happens, it cancels its ongoing activities, cleans up the intermediate resources and set the `DataDownload` CR to `Cancelled`.
|
||||
- Throughout the data transfer, Velero built-in data mover monitors the status of the data mover pod and deletes it after `DataDownload` CR is set to the terminal state.
|
||||
|
||||
### Backup Deletion
|
||||
When a backup is created, a snapshot is saved into the repository for the volume data. The snapshot is a reference to the volume data saved in the repository.
|
||||
@@ -424,22 +301,28 @@ kubectl -n velero get datauploads -l velero.io/backup-name=YOUR_BACKUP_NAME -w
|
||||
kubectl -n velero get datadownloads -l velero.io/restore-name=YOUR_RESTORE_NAME -w
|
||||
```
|
||||
|
||||
### Restart and resume
|
||||
When Velero server is restarted, if the resource backup/restore has completed, so the backup/restore has excceded `InProgress` status and is waiting for the completion of the data movements, Velero will recapture the status of the running data movements and resume the execution.
|
||||
When node-agent is restarted, if the `DataUpload`/`DataDownload` is in `InProgress` status, Velero recaptures the status of the running data mover pod and resume the execution.
|
||||
When node-agent is restarted, if the `DataUpload`/`DataDownload` is in `New` or `Prepared` status, the data mover pod has not started, Velero processes it as normal cases, or the restart doesn't affect the execution.
|
||||
|
||||
### Cancellation
|
||||
|
||||
At present, Velero backup and restore doesn't support end to end cancellation that is launched by users.
|
||||
However, Velero cancels the `DataUpload`/`DataDownload` in below scenarios automatically:
|
||||
- When Velero server is restarted
|
||||
- When node-agent is restarted
|
||||
- When Velero server is restarted and the backup/restore is in `InProgress` status
|
||||
- When node-agent is restarted and the `DataUpload`/`DataDownload` is in `Accepted` status
|
||||
- When node-agent is restarted and the resume of an existing `DataUpload`/`DataDownload` that is in `InProgress` status fails
|
||||
- When an ongoing backup/restore is deleted
|
||||
- When a backup/restore does not finish before the item operation timeout (default value is `4 hours`)
|
||||
|
||||
Customized data movers that support cancellation could cancel their ongoing tasks and clean up any intermediate resources. If you are using Velero built-in data mover, the cancellation is supported.
|
||||
|
||||
### Support ReadOnlyRootFilesystem setting
|
||||
When the Velero server/node-agent pod's SecurityContext sets the `ReadOnlyRootFileSystem` parameter to true, the Velero server/node-agent pod's filesystem is running in read-only mode. Then the backup/restore may fail, because the uploader/repository needs to write some cache and configuration data into the pod's root filesystem.
|
||||
When the Velero server pod's SecurityContext sets the `ReadOnlyRootFileSystem` parameter to true, the Velero server pod's filesystem is running in read-only mode. Then the backup deletion may fail, because the repository needs to write some cache and configuration data into the pod's root filesystem.
|
||||
|
||||
```
|
||||
Errors: Velero: name: /mongodb-0 message: /Error backing up item error: /failed to wait BackupRepository: backup repository is not ready: error to connect to backup repo: error to connect repo with storage: error to connect to repository: unable to write config file: unable to create config directory: mkdir /home/cnb/udmrepo: read-only file system name: /mongodb-1 message: /Error backing up item error: /failed to wait BackupRepository: backup repository is not ready: error to connect to backup repo: error to connect repo with storage: error to connect to repository: unable to write config file: unable to create config directory: mkdir /home/cnb/udmrepo: read-only file system name: /mongodb-2 message: /Error backing up item error: /failed to wait BackupRepository: backup repository is not ready: error to connect to backup repo: error to connect repo with storage: error to connect to repository: unable to write config file: unable to create config directory: mkdir /home/cnb/udmrepo: read-only file system Cluster: <none>
|
||||
Errors: /error to connect repo with storage: error to connect to repository: unable to write config file: unable to create config directory: mkdir /home/cnb/udmrepo: read-only file system
|
||||
```
|
||||
|
||||
The workaround is making those directories as ephemeral k8s volumes, then those directories are not counted as pod's root filesystem.
|
||||
@@ -473,15 +356,17 @@ spec:
|
||||
......
|
||||
```
|
||||
|
||||
At present, Velero doesn't allow to set `ReadOnlyRootFileSystem` parameter to data mover pods, so the root filesystem for the data mover pods are always writable.
|
||||
|
||||
### Resource Consumption
|
||||
|
||||
Both the uploader and repository consume remarkable CPU/memory during the backup/restore, especially for massive small files or large backup size cases.
|
||||
|
||||
For Velero built-in data mover, Velero uses [BestEffort as the QoS][13] for node-agent pods (so no CPU/memory request/limit is set), so that backups/restores wouldn't fail due to resource throttling in any cases.
|
||||
If you want to constraint the CPU/memory usage, you need to [customize the resource limits][11]. The CPU/memory consumption is always related to the scale of data to be backed up/restored, refer to [Performance Guidance][12] for more details, so it is highly recommended that you perform your own testing to find the best resource limits for your data.
|
||||
For Velero built-in data mover, Velero uses [BestEffort as the QoS][13] for data mover pods (so no CPU/memory request/limit is set), so that backups/restores wouldn't fail due to resource throttling in any cases.
|
||||
If you want to constraint the CPU/memory usage, you need to [Customize Data Mover Pod Resource Limits][11]. The CPU/memory consumption is always related to the scale of data to be backed up/restored, refer to [Performance Guidance][12] for more details, so it is highly recommended that you perform your own testing to find the best resource limits for your data.
|
||||
|
||||
During the restore, the repository may also cache data/metadata so as to reduce the network footprint and speed up the restore. The repository uses its own policy to store and clean up the cache.
|
||||
For Kopia repository, the cache is stored in the node-agent pod's root file system. Velero allows you to configure a limit of the cache size so that the data mover pod won't be evicted due to running out of the ephemeral storage. For more details, check [Backup Repository Configuration][17].
|
||||
For Kopia repository, the cache is stored in the data mover pod's root file system. Velero allows you to configure a limit of the cache size so that the data mover pod won't be evicted due to running out of the ephemeral storage. For more details, check [Backup Repository Configuration][17].
|
||||
|
||||
### Node Selection
|
||||
|
||||
@@ -502,15 +387,16 @@ advanced configuration options for `BackupPVC`, allowing users to fine-tune acce
|
||||
[3]: file-system-backup.md
|
||||
[4]: https://kubernetes.io/blog/2020/12/10/kubernetes-1.20-volume-snapshot-moves-to-ga/
|
||||
[5]: https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation
|
||||
[6]: https://docs.openshift.com/container-platform/latest/authentication/managing-security-context-constraints.html
|
||||
[7]: https://docs.microsoft.com/en-us/azure/aks/azure-files-dynamic-pv
|
||||
[8]: api-types/backupstoragelocation.md
|
||||
[9]: supported-providers.md
|
||||
[10]: restore-reference.md#changing-pv/pvc-Storage-Classes
|
||||
[11]: customize-installation.md#customize-resource-requests-and-limits
|
||||
[11]: data-movement-pod-resource-configuration.md
|
||||
[12]: performance-guidance.md
|
||||
[13]: https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/
|
||||
[14]: node-agent-concurrency.md
|
||||
[15]: data-movement-backup-node-selection.md
|
||||
[16]: data-movement-backup-pvc-configuration.md
|
||||
[17]: backup-repository-configuration.md
|
||||
[18]: https://github.com/vmware-tanzu/velero/pull/7576
|
||||
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
title: "Data Movement Pod Resource Configuration"
|
||||
layout: docs
|
||||
---
|
||||
|
||||
During [CSI Snapshot Data Movement][1], Velero built-in data mover launches data mover pods to to run the data transfer. While the data transfer is a time and resource consuming activity.
|
||||
|
||||
Velero built-in data mover by default uses the [BestEffort QoS][2] for the data mover pods, which guarantees the best performance of the data movement activities. On the other hand, it may take lots of cluster resource, i.e., CPU, memory, and how many resources are taken is decided by the concurrency and the scale of data to be moved.
|
||||
|
||||
If the cluster nodes don't have sufficient resource, Velero also allows you to customize the resources for the data mover pods.
|
||||
Note: If less resources are assigned to data mover pods, the data movement activities may take longer time; or the data mover pods may be OOM killed if the assigned memory resource doesn't meet the requirements. Consequently, the dataUpload/dataDownload may run longer or fail.
|
||||
|
||||
Refer to [Performance Guidance][3] for a guidance of performance vs. resource usage, and it is highly recommended that you perform your own testing to find the best resource limits for your data.
|
||||
|
||||
Velero introduces a new section in the node-agent configMap, called ```podResources```, through which you can set customized resources configurations for data mover pods.
|
||||
If it is not there, a configMap should be created manually. The configMap should be in the same namespace where Velero is installed. If multiple Velero instances are installed in different namespaces, there should be one configMap in each namespace which applies to node-agent in that namespace only. The name of the configMap should be specified in the node-agent server parameter ```--node-agent-config```.
|
||||
Node-agent server checks these configurations at startup time. Therefore, you could edit this configMap any time, but in order to make the changes effective, node-agent server needs to be restarted.
|
||||
|
||||
### Sample
|
||||
Here is a sample of the configMap with ```podResources```:
|
||||
```json
|
||||
{
|
||||
"podResources": {
|
||||
"cpuRequest": "1000m",
|
||||
"cpuLimit": "1000m",
|
||||
"memoryRequest": "512Mi",
|
||||
"memoryLimit": "1Gi"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The string values in ```podResources``` must match Kubernetes Quantity expressions; for each resource, the "request" value must not be larger than the "limit" value. Otherwise, if any one of the values fail, the entire ```podResources``` configuration will be ignored (so the default policy will be used).
|
||||
|
||||
To create the configMap, save something like the above sample to a json file and then run below command:
|
||||
```
|
||||
kubectl create cm node-agent-config -n velero --from-file=<json file name>
|
||||
```
|
||||
|
||||
To provide the configMap to node-agent, edit the node-agent daemonset and add the ```- --node-agent-config``` argument to the spec:
|
||||
1. Open the node-agent daemonset spec
|
||||
```
|
||||
kubectl edit ds node-agent -n velero
|
||||
```
|
||||
2. Add ```- --node-agent-config``` to ```spec.template.spec.containers```
|
||||
```
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- args:
|
||||
- --node-agent-config=<configMap name>
|
||||
```
|
||||
|
||||
[1]: csi-snapshot-data-movement.md
|
||||
[2]: https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/
|
||||
[3]: performance-guidance.md
|
||||
@@ -53,8 +53,10 @@ toc:
|
||||
url: /csi-snapshot-data-movement
|
||||
- page: Node-agent Concurrency
|
||||
url: /node-agent-concurrency
|
||||
- page: Backup PVC Configuration
|
||||
- page: Data Movement Backup PVC Configuration
|
||||
url: /data-movement-backup-pvc-configuration
|
||||
- page: Data Movement Pod Resource Configuration
|
||||
url: /data-movement-pod-resource-configuration
|
||||
- page: Backup Repository Configuration
|
||||
url: /backup-repository-configuration
|
||||
- page: Verifying Self-signed Certificates
|
||||
|
||||
Reference in New Issue
Block a user