From f5e2552c5a686e3b3b8875feeac601d4ab2c9b87 Mon Sep 17 00:00:00 2001 From: Ming Qiu Date: Thu, 11 Apr 2024 02:40:02 +0000 Subject: [PATCH] Add maintenance job doc Signed-off-by: Ming Qiu --- site/content/docs/main/api-types/restore.md | 2 ++ .../docs/main/customize-installation.md | 14 ++++++-- .../docs/main/repository-maintenance.md | 36 +++++++++++++++++++ site/content/docs/main/restore-reference.md | 6 ++++ site/content/docs/main/velero-install.md | 6 +++- 5 files changed, 60 insertions(+), 4 deletions(-) create mode 100644 site/content/docs/main/repository-maintenance.md diff --git a/site/content/docs/main/api-types/restore.md b/site/content/docs/main/api-types/restore.md index 95e8f6c9a..7b388d882 100644 --- a/site/content/docs/main/api-types/restore.md +++ b/site/content/docs/main/api-types/restore.md @@ -43,6 +43,8 @@ spec: uploaderConfig: # WriteSparseFiles is a flag to indicate whether write files sparsely or not writeSparseFiles: true + # ParallelFilesDownload is the concurrency number setting for restore + parallelFilesDownload: 10 # Array of namespaces to include in the restore. If unspecified, all namespaces are included. # Optional. includedNamespaces: diff --git a/site/content/docs/main/customize-installation.md b/site/content/docs/main/customize-installation.md index 1b017fe40..8a08fca7f 100644 --- a/site/content/docs/main/customize-installation.md +++ b/site/content/docs/main/customize-installation.md @@ -95,7 +95,7 @@ the config file setting. ## Customize resource requests and limits -At installation, You could set resource requests and limits for the Velero pod and the node-agent pod, if you are using the [File System Backup][3] or [CSI Snapshot Data Movement][12]. +At installation, You could set resource requests and limits for the Velero pod, the node-agent pod and the [repository maintenance job][14], if you are using the [File System Backup][3] or [CSI Snapshot Data Movement][12]. {{< table caption="Velero Customize resource requests and limits defaults" >}} |Setting|Velero pod defaults|node-agent pod defaults| @@ -107,7 +107,10 @@ At installation, You could set resource requests and limits for the Velero pod a {{< /table >}} For Velero pod, through testing, the Velero maintainers have found these defaults work well when backing up and restoring 1000 or less resources. -For node-agent pod, by default it doesn't have CPU/memory request/limit, so that the backups/restores won't break due to resource throttling. The Velero maintainers have also done some [Performance Tests][13] to show the relationship of CPU/memory usage and the scale of data being backed up/restored. +For node-agent pod, by default it doesn't have CPU/memory request/limit, so that the backups/restores won't break due to resource throttling. The Velero maintainers have also done some [Performance Tests][13] to show the relationship of CPU/memory usage and the scale of data being backed up/restored. + +For repository maintenance job, it's no limit on resources by default. You could configure the job resource limitation based on target data to be backed up, some further settings please refer to [repository maintenance job][14]. + You don't have to change the defaults all the time, but if you need, it's recommended that you perform your own testing to find the best resource limits for your clusters and resources. ### Install with custom resource requests and limits @@ -125,7 +128,11 @@ velero install \ [--node-agent-pod-cpu-request ] \ [--node-agent-pod-mem-request ] \ [--node-agent-pod-cpu-limit ] \ - [--node-agent-pod-mem-limit ] + [--node-agent-pod-mem-limit ] \ + [--maintenance-job-cpu-request ] \ + [--maintenance-job-mem-request ] \ + [--maintenance-job-cpu-limit ] \ + [--maintenance-job-mem-limit ] ``` ### Update resource requests and limits after install @@ -423,3 +430,4 @@ If you get an error like `complete:13: command not found: compdef`, then add the [11]: https://github.com/vmware-tanzu/velero/blob/main/pkg/apis/velero/v1/constants.go [12]: csi-snapshot-data-movement.md [13]: performance-guidance.md +[14]: repository-maintenance.md diff --git a/site/content/docs/main/repository-maintenance.md b/site/content/docs/main/repository-maintenance.md new file mode 100644 index 000000000..43c25bdde --- /dev/null +++ b/site/content/docs/main/repository-maintenance.md @@ -0,0 +1,36 @@ +--- +title: "Repository Maintenance" +layout: docs +--- + +From v1.14 on, Velero decouples repository maintenance from the Velero server by launching a k8s job to do maintenance when needed, to mitigate the impact on the Velero server during backups. + +Before v1.14.0, Velero performs periodic maintenance on the repository within Velero server pod, this operation may consume significant CPU and memory resources in some cases, leading to Velero server being killed by OOM. Now Velero will launch independent k8s jobs to do the maintenance in Velero installation namespace. + +For repository maintenance jobs, there's no limit on resources by default. You could configure the job resource limitation based on target data to be backed up. + +## Settings +### Resource Limitation +You can customize the maintenance job resource requests and limit when using the [velero install][1] CLI command. + +### Log +Maintenance job inherits the log level and log format settings from the Velero server, so if the Velero server enabled the debug log, the maintenance job will also open the debug level log. + +### Num of Keeping Latest Maintenance Jobs +Velero will keep one specific number of the latest maintenance jobs for each repository. By default, we only keep 3 latest maintenance jobs for each repository, and Velero support configures this setting by the below command when Velero installs: + +```bash +velero install --keep-latest-maintenance-jobs +``` + +### Default Repository Maintenance Frequency +The frequency of running maintenance jobs could be set by the below command when Velero is installed: +```bash +velero install --default-repo-maintain-frequency +``` +For Kopia the default maintenance frequency is 1 hour, and Restic is 7 * 24 hours. + +### Others +Maintenance jobs will inherit the labels, annotations, tolerations, affinity, nodeSelector, service account, image, environment variables, cloud-credentials etc. from Velero deployment. + +[1]: velero-install.md#usage \ No newline at end of file diff --git a/site/content/docs/main/restore-reference.md b/site/content/docs/main/restore-reference.md index 6de606c8e..e4e3c7822 100644 --- a/site/content/docs/main/restore-reference.md +++ b/site/content/docs/main/restore-reference.md @@ -279,6 +279,12 @@ If using fs-restore or CSI snapshot data movements, it's supported to write spar velero restore create --from-backup --write-sparse-files --wait ``` +## Parallel Files Download +If using fs-restore or CSI snapshot data movements, it's possible to configure one option for parallel file downloads during the restore by Kopia uploader using the command below: +```bash +velero restore create --from-backup --parallel-files-download --wait +``` + ## Removing a Restore object There are two ways to delete a Restore object: diff --git a/site/content/docs/main/velero-install.md b/site/content/docs/main/velero-install.md index 9dba90d12..ac5b4def3 100644 --- a/site/content/docs/main/velero-install.md +++ b/site/content/docs/main/velero-install.md @@ -26,7 +26,11 @@ velero install \ [--node-agent-pod-cpu-request ] \ [--node-agent-pod-mem-request ] \ [--node-agent-pod-cpu-limit ] \ - [--node-agent-pod-mem-limit ] + [--node-agent-pod-mem-limit ] \ + [--maintenance-job-cpu-request ] \ + [--maintenance-job-mem-request ] \ + [--maintenance-job-cpu-limit ] \ + [--maintenance-job-mem-limit ] ``` The values for the resource requests and limits flags follow the same format as [Kubernetes resource requirements][3]