mirror of
https://github.com/vmware-tanzu/velero.git
synced 2026-09-29 03:05:54 +00:00
Xun Jiang/Bruce JiangandGitHub
00788053ae
Run the E2E test on kind / setup-test-matrix (push) Failing after 3s
Scorecard supply-chain security / Scorecard analysis (push) Skipped
e2e-test-kind.yaml / extract (push) Failing after 8s
Run the E2E test on kind / get-go-version (push) Failing after 9s
Run the E2E test on kind / build (push) Skipped
Run the E2E test on kind / run-e2e-test (push) Skipped
push.yml / extract (push) Failing after 6s
Main CI / get-go-version (push) Failing after 7s
Main CI / Build (push) Skipped
Signed-off-by: Xun Jiang <xun.jiang@broadcom.com>
271 lines
13 KiB
Markdown
271 lines
13 KiB
Markdown
---
|
|
title: "Repository Maintenance"
|
|
layout: docs
|
|
---
|
|
|
|
From v1.14 on, Velero decouples repository maintenance from the Velero server by launching a k8s job to do maintenance when needed, to mitigate the impact on the Velero server during backups.
|
|
|
|
Before v1.14.0, Velero performs periodic maintenance on the repository within Velero server pod, this operation may consume significant CPU and memory resources in some cases, leading to Velero server being killed by OOM. Now Velero will launch independent k8s jobs to do the maintenance in Velero installation namespace.
|
|
|
|
For repository maintenance jobs, there's no limit on resources by default. You could configure the job resource limitation based on target data to be backed up.
|
|
|
|
From v1.15 and on, Velero introduces a new ConfigMap, specified by `velero server --repo-maintenance-job-configmap` parameter, to set repository maintenance Job configuration, including Node Affinity and resources. The old `velero server` parameters ( `--maintenance-job-cpu-request`, `--maintenance-job-mem-request`, `--maintenance-job-cpu-limit`, `--maintenance-job-mem-limit`, and `--keep-latest-maintenance-jobs`) introduced in v1.14 are deprecated, and will be deleted in v1.17.
|
|
|
|
The users can specify the ConfigMap name during velero installation by CLI:
|
|
`velero install --repo-maintenance-job-configmap=<ConfigMap-Name>`
|
|
|
|
## Settings
|
|
### Resource Limitation and Node Affinity
|
|
Those are specified by the ConfigMap specified by `velero server --repo-maintenance-job-configmap` parameter.
|
|
|
|
This ConfigMap content is a Map.
|
|
If there is a key value as `global` in the map, the key's value is applied to all BackupRepositories maintenance jobs that cannot find their own specific configuration in the ConfigMap.
|
|
The other keys in the map is the combination of three elements of a BackupRepository, because those three keys can identify a unique BackupRepository:
|
|
* The namespace in which BackupRepository backs up volume data.
|
|
* The BackupRepository referenced BackupStorageLocation's name.
|
|
* The BackupRepository's type. Possible value is `kopia`.
|
|
|
|
If there is a key match with BackupRepository, the key's value is applied to the BackupRepository's maintenance jobs.
|
|
By this way, it's possible to let user configure before the BackupRepository is created.
|
|
This is especially convenient for administrator configuring during the Velero installation.
|
|
For example, the following BackupRepository's key should be `test-default-kopia`.
|
|
|
|
``` yaml
|
|
- apiVersion: velero.io/v1
|
|
kind: BackupRepository
|
|
metadata:
|
|
generateName: test-default-kopia-
|
|
labels:
|
|
velero.io/repository-type: kopia
|
|
velero.io/storage-location: default
|
|
velero.io/volume-namespace: test
|
|
name: test-default-kopia-kgt6n
|
|
namespace: velero
|
|
spec:
|
|
backupStorageLocation: default
|
|
maintenanceFrequency: 1h0m0s
|
|
repositoryType: kopia
|
|
volumeNamespace: test
|
|
```
|
|
|
|
You can still customize the maintenance job resource requests and limit when using the [velero install][1] CLI command.
|
|
|
|
The `LoadAffinity` structure is reused from design [node-agent affinity configuration][2].
|
|
|
|
### Affinity Example
|
|
It's possible that the users want to choose nodes that match condition A or condition B to run the job.
|
|
For example, the user want to let the nodes is in a specified machine type or the nodes locate in the us-central1-x zones to run the job.
|
|
This can be done by adding multiple entries in the `LoadAffinity` array.
|
|
|
|
The sample of the ```repo-maintenance-job-configmap``` ConfigMap for the above scenario is as below:
|
|
``` bash
|
|
cat <<EOF > repo-maintenance-job-config.yaml
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: repo-maintenance-job-config
|
|
namespace: velero
|
|
data:
|
|
global: |
|
|
{
|
|
"podResources": {
|
|
"cpuRequest": "100m",
|
|
"cpuLimit": "200m",
|
|
"ephemeralStorageRequest": "5Gi",
|
|
"ephemeralStorageLimit": "10Gi",
|
|
"memoryRequest": "100Mi",
|
|
"memoryLimit": "200Mi"
|
|
},
|
|
"keepLatestMaintenanceJobs": 1,
|
|
"loadAffinity": [
|
|
{
|
|
"nodeSelector": {
|
|
"matchExpressions": [
|
|
{
|
|
"key": "topology.kubernetes.io/zone",
|
|
"operator": "In",
|
|
"values": [
|
|
"us-central1-a",
|
|
"us-central1-b",
|
|
"us-central1-c"
|
|
]
|
|
}
|
|
]
|
|
}
|
|
}
|
|
]
|
|
}
|
|
kibishii-default-kopia: |
|
|
{
|
|
"podResources": {
|
|
"cpuRequest": "200m",
|
|
"cpuLimit": "400m",
|
|
"ephemeralStorageRequest": "5Gi",
|
|
"ephemeralStorageLimit": "10Gi",
|
|
"memoryRequest": "200Mi",
|
|
"memoryLimit": "400Mi"
|
|
},
|
|
"keepLatestMaintenanceJobs": 2
|
|
}
|
|
EOF
|
|
```
|
|
Notice: although loadAffinity is an array, Velero only takes the first element of the array.
|
|
|
|
This sample showcases how to use affinity configuration:
|
|
- matchLabels: maintenance job runs on nodes located in `us-central1-a`, `us-central1-b` and `us-central1-c`.
|
|
|
|
To create the configMap, users need to save something like the above sample to a json file and then run below command:
|
|
```
|
|
kubectl apply -f repo-maintenance-job-config.yaml
|
|
```
|
|
|
|
### Log
|
|
Maintenance job inherits the log level and log format settings from the Velero server, so if the Velero server enabled the debug log, the maintenance job will also open the debug level log.
|
|
|
|
### Num of Keeping Latest Maintenance Jobs
|
|
Velero will keep one specific number of the latest maintenance jobs for each repository. By default, we only keep 3 latest maintenance jobs for each repository, and Velero support configures this setting by the below command when Velero installs:
|
|
|
|
```bash
|
|
velero install --keep-latest-maintenance-jobs <NUM>
|
|
```
|
|
|
|
### Default Repository Maintenance Frequency
|
|
The frequency of running maintenance jobs could be set by the below command when Velero is installed:
|
|
```bash
|
|
velero install --default-repo-maintain-frequency <DURATION>
|
|
```
|
|
For Kopia the default maintenance frequency is 1 hour.
|
|
|
|
### Full Maintenance Interval customization
|
|
See [backup repository configuration][3]
|
|
|
|
### Maintenance History
|
|
You can view the maintenance history by describing the corresponding backupRepository CR:
|
|
|
|
```
|
|
Status:
|
|
Last Maintenance Time: <timestamp>
|
|
Recent Maintenance:
|
|
Complete Timestamp: <timestamp>
|
|
Result: Succeeded
|
|
Start Timestamp: <timestamp>
|
|
Complete Timestamp: <timestamp>
|
|
Result: Succeeded
|
|
Start Timestamp: <timestamp>
|
|
Message: <error message>
|
|
Result: Failed
|
|
Start Timestamp: <timestamp>
|
|
```
|
|
|
|
- `Last Maintenance Time` indicates the time of the latest successful maintenance job
|
|
- `Recent Maintenance` keeps the status of the recent 3 maintenance jobs, including its start time, result (succeeded/failed), completion time (if the maintenance job succeeded), or error message (if the maintenance failed)
|
|
|
|
### Others
|
|
Maintenance jobs will inherit nodeSelector, service account, image, environment variables, cloud-credentials, priorityClassName etc. from Velero deployment.
|
|
|
|
For labels and annotations, maintenance jobs do NOT inherit all labels and annotations from the Velero deployment. Instead, they include:
|
|
|
|
**Labels:**
|
|
|
|
* `velero.io/repo-name: <repository-name>` - automatically added to identify which repository they are maintaining
|
|
* Only specific [third-party labels][4] from the Velero server deployment that are in the predefined list, currently limited to:
|
|
* `azure.workload.identity/use`
|
|
* Any custom labels explicitly configured in the repository maintenance job ConfigMap
|
|
|
|
**Annotations:**
|
|
|
|
* Only specific [third-party annotations][5] from the Velero server deployment that are in the predefined list, currently limited to:
|
|
* `iam.amazonaws.com/role`
|
|
* Any custom annotations explicitly configured in the repository maintenance job ConfigMap
|
|
|
|
**Important:** Other labels and annotations from the Velero deployment are NOT inherited by maintenance jobs. This is by design to ensure only specific labels and annotations required for cloud provider identity systems are propagated.
|
|
|
|
**Tolerations:**
|
|
|
|
Similarly, maintenance jobs do NOT inherit all tolerations from the Velero deployment. Instead, they include:
|
|
* The required Windows toleration (`os=windows:NoSchedule`) for backward compatibility
|
|
* Only specific [third-party tolerations][6] from the Velero server deployment that are in the predefined list, currently limited to:
|
|
* `kubernetes.azure.com/scalesetpriority`
|
|
* `CriticalAddonsOnly`
|
|
* Any custom tolerations explicitly configured in the repository maintenance job ConfigMap
|
|
|
|
Maintenance jobs will not run for backup repositories whose backup storage location is set as readOnly.
|
|
|
|
#### Priority Class Configuration
|
|
Maintenance jobs can be configured with a specific priority class through the repository maintenance job ConfigMap. The priority class name should be specified in the global configuration section:
|
|
|
|
```json
|
|
{
|
|
"global": {
|
|
"priorityClassName": "low-priority",
|
|
"podResources": {
|
|
"cpuRequest": "100m",
|
|
"memoryRequest": "128Mi"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
Note that priority class configuration is only read from the global configuration section, ensuring all maintenance jobs use the same priority class regardless of which repository they are maintaining.
|
|
|
|
#### Tolerations Configuration
|
|
Maintenance jobs can be configured with customized tolerations through the repository maintenance job ConfigMap to allow scheduling on nodes with custom taints (such as dedicated node groups managed by Karpenter). The tolerations should be specified in the global configuration section:
|
|
|
|
```json
|
|
{
|
|
"global": {
|
|
"tolerations": [
|
|
{
|
|
"key": "dedicated",
|
|
"operator": "Equal",
|
|
"value": "maintenance",
|
|
"effect": "NoSchedule"
|
|
}
|
|
]
|
|
}
|
|
}
|
|
```
|
|
|
|
Note that tolerations configuration is only read from the global configuration section, ensuring all maintenance jobs use the same tolerations regardless of which repository they are maintaining.
|
|
|
|
Configured tolerations are merged with (and deduplicated against) the Windows toleration (`os=windows:NoSchedule`) and in-tree third-party tolerations inherited from the Velero deployment (`kubernetes.azure.com/scalesetpriority` and `CriticalAddonsOnly`).
|
|
Deduplication applies only to exact duplicate tolerations with the same `key`, `operator`, `value`, and `effect`; configured tolerations do not override different inherited tolerations based only on matching keys or partial field matches.
|
|
|
|
#### Pod Labels and Annotations Configuration
|
|
Maintenance jobs can be configured with customized labels (`podLabels`) and annotations (`podAnnotations`) through the repository maintenance job ConfigMap to support third-party integrations, monitoring, and environment-specific requirements.
|
|
|
|
Both labels and annotations must be specified in the `global` configuration section:
|
|
|
|
```json
|
|
{
|
|
"global": {
|
|
"podLabels": {
|
|
"environment": "production",
|
|
"team": "storage"
|
|
},
|
|
"podAnnotations": {
|
|
"vault.hashicorp.com/agent-inject": "true",
|
|
"prometheus.io/scrape": "true"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
Note that `podLabels` and `podAnnotations` are only read from the global configuration section, ensuring all maintenance jobs use the same labels and annotations regardless of which repository they are maintaining.
|
|
|
|
**Label Handling Rules:**
|
|
* Velero automatically adds the `velero.io/repo-name: <repository-name>` label to identify the repository being maintained; this reserved label cannot be overridden by user configuration.
|
|
* If `podLabels` is configured in the ConfigMap, user-provided labels are applied. Note that user-provided `podLabels` supersede the in-tree third-party labels from the Velero deployment. If you need both custom labels and in-tree third-party labels (e.g. `azure.workload.identity/use`), you must explicitly include them in the `podLabels` configuration.
|
|
* If `podLabels` is not configured, Velero automatically inherits the allowed [third-party labels][4] from the Velero deployment.
|
|
|
|
**Annotation Handling Rules:**
|
|
* If `podAnnotations` is configured in the ConfigMap, user-provided annotations are applied. Note that user-provided `podAnnotations` supersede the in-tree third-party annotations from the Velero deployment. If you need both custom annotations and in-tree third-party annotations (e.g. `iam.amazonaws.com/role`), you must explicitly include them in the `podAnnotations` configuration.
|
|
* If `podAnnotations` is not configured, Velero automatically inherits the allowed [third-party annotations][5] from the Velero deployment.
|
|
|
|
[1]: velero-install.md#usage
|
|
[2]: node-agent-concurrency.md
|
|
[3]: backup-repository-configuration.md#full-maintenance-interval-customization
|
|
[4]: https://github.com/velero-io/velero/blob/d5a2e7e6b9512e8ba52ec269ed5ce9a0fa23548c/pkg/util/third_party.go#L19-L21
|
|
[5]: https://github.com/velero-io/velero/blob/d5a2e7e6b9512e8ba52ec269ed5ce9a0fa23548c/pkg/util/third_party.go#L23-L25
|
|
[6]: https://github.com/velero-io/velero/blob/d5a2e7e6b9512e8ba52ec269ed5ce9a0fa23548c/pkg/util/third_party.go#L27-L30
|