diff --git a/changelogs/unreleased/9528-Lyndon-Li b/changelogs/unreleased/9528-Lyndon-Li new file mode 100644 index 000000000..52bb4a8f3 --- /dev/null +++ b/changelogs/unreleased/9528-Lyndon-Li @@ -0,0 +1 @@ +Add block data mover design for block level incremental backup by integrating with Kubernetes CBT \ No newline at end of file diff --git a/design/block-data-mover/block-data-mover.md b/design/block-data-mover/block-data-mover.md index 2a7d8c2d9..28bdc7b11 100644 --- a/design/block-data-mover/block-data-mover.md +++ b/design/block-data-mover/block-data-mover.md @@ -7,23 +7,29 @@ **Velero Generic Data Path (VGDP)**: VGDP is the collective of modules that is introduced in [Unified Repository design][1]. Velero uses these modules to finish data transfer for various purposes (i.e., PodVolume backup/restore, Volume Snapshot Data Movement). VGDP modules include uploaders and the backup repository. **Velero Built-in Data Mover (VBDM)**: VBDM, which is introduced in [Volume Snapshot Data Movement design][2] and [Unified Repository design][1], is the built-in data mover shipped along with Velero, it includes Velero data mover controllers and VGDP. **Data Mover Pods**: Intermediate pods which hold VGDP and complete the data transfer. See [VGDP Micro Service for Volume Snapshot Data Movement][3] for details. -**Change Block Tracking (CBT)**: CBT is the mechanism to track changed blocks, so that backups could back up the changed data only. CBT usually provides by the comupting/storage plateform. +**Change Block Tracking (CBT)**: CBT is the mechanism to track changed blocks, so that backups could back up the changed data only. CBT usually provides by the comupting/storage platform. **TCO**: Total Cost of Ownership. This is a general criteria for products/solutions, but also means a lot for BR solutions. For example, this means what kind of backup storage (and its cost) it requires, the retention policy of backup copies, the ways to remove backup data redundancy, etc. ## Background -At present, [CSI Snapshot Data Movement][2] which is implemented by VBDM, ships a file system uploader, so the backup/restore is done from file system only. -However, once possible, block level backup/restore is better than file system level backup/restore: -- Block level backup could leverage CBT to process minimual size of data so significantly reduce the overhead to network, backup repository and backup storage. As a result, TCO is significantly reduced. +Kubernetes supports two kinds of volume mode, `FileSystem` and `Block`, for persistent volumes. Underlyingly, the storage could use a block storage to provision either `FileSystem` mode or `Block` mode volumes; and the storage could use a file storage to provision `FileSystem` mode volumes. +For volumes provisioned by block storage, they could be backed up/restored from the block level, regardless the volume mode of the persistent volume. +On the other hand, as long as the data could be accessed from the file system, a backup/restore could be conducted from the file system level. That is to say `FileSystem` mode volumes could be backed up/restored from the file system level, regardless of the backend storage type. +Then if a `FileSystem` mode volume is provisioned by a block storage, the volume could be backed up/restored either from the file system level or block level. + +For Velero, [CSI Snapshot Data Movement][2] which is implemented by VBDM, ships a file system uploader, so the backup/restore is done from file system only. + +Once possible, block level backup/restore is better than file system level backup/restore: +- Block level backup could leverage CBT to process minimal size of data, so it significantly reduces the overhead to network, backup repository and backup storage. As a result, TCO is significantly reduced. - Block level backup/restore is performant in throughput and resource consumption, because it doesn't need to handle the complexity of the file system, especially for the case that huge number of small files in the file system. - Block level backup/restore is less OS dependent because the uploader doesn't need the OS to be aware of the file system in the volume. -At present, [Kubernetes CBT API][4] is mature and close to Beta stage. Many plateform/storage has supported/is going to support it. +At present, [Kubernetes CBT API][4] is mature and close to Beta stage. Many platform/storage has supported/is going to support it. Therefore, it is very important for Velero to deliver the block level backup/restore and recommend users to use it over the file system data mover as long as: - The volume is backed by block storage so block level access is possible -- The plateform supports CBT +- The platform supports CBT -Meanwhile, file system data mover is still valuable for below scenarios: +Meanwhile, file system level backup/restore is still valuable for below scenarios: - The volume is backed by file storage, e.g., AWS EFS, Azure File, CephFS, VKS File Volume, etc. - The volume is backed by block storage but CBT is not available - The volume doesn't support CSI snapshot, so Velero fs-backup method is used @@ -32,6 +38,8 @@ There are rich features delivered with VGDP, VBDM and [VGDP micro service][3], t Velero VBDM supports linux and Windows nodes, however, Windows container doesn't support block mode volumes, so backing up/restoring from Windows nodes is not supported until Windows container removes this limitation. As a result, if there are both linux and Windows nodes in the cluster, block data mover can only run in linux nodes. +Both the Kubernetes CBT service and Velero work in the boundary of the cluster, even though the backend storage may be shared by multiple clusters, Velero can only protection workloads in the same cluster where it is running. + ## Goals Add a block data mover to VBDM and support block level backup/restore for [CSI Snapshot Data Movement][2], which includes: @@ -40,6 +48,7 @@ Add a block data mover to VBDM and support block level backup/restore for [CSI S - Support block level restore from full/incremental backup for both `FileSystem` and `Block` mode volumes - Support block level backup/restore for both linux and Windows workloads - Support all existing features, i.e., load concurrency, node selection, cache volume, deduplication, compression, encryption, etc. for the block data mover +- Support volumes processed from file system level and block level in the same backup/restore ## Non-Goals @@ -83,16 +92,69 @@ Below are the major changes based on the existing VBDM: ## Detailed Design ### Selectable Data Mover Type + +#### Per Backup Selection At present, the backup accepts a `DataMover` parameter and when its value is empty or `velero`, VBDM is used. After block data mover is introduced, VBDM will have two types of data movers, Velero file system data mover and Velero block data mover. A new type string `velero-block` is introduced for Velero block data mover, that is, when `DataMover` is set as `velero-block`, Velero block data mover is used. Another new value `velero-fs` is introduced for Velero file system data mover, that is, when `DataMover` is set as `velero-fs`, Velero file system data mover is used. -For backwards compatibility consideration, `velero` is preserved a valid value, it refers to the default data mover, and the default data mover may change among releases. -When `DataMover` is empty or `velero`, the default data mover is used. At present, Velero file system data mover is the default data mover; we can change the default one to Velero block data mover in future releases. +For backwards compatibility consideration, `velero` is preserved a valid value, it refers to the default data mover, and the default data mover may change among releases. At present, Velero file system data mover is the default data mover; we can change the default one to Velero block data mover in future releases. + +#### Volume Policy +It is a valid case that users have multiple volumes in a single backup, while they want to use Velero file system data mover for some of the volumes and use Velero block data mover for some others. +To meet this requirement, a combined solution of Per Backup Selection and Volume Policy is used. + +Here are the data structs for VolumePolicy: +```go +type volPolicy struct { + action Action + conditions []volumeCondition +} + +type volumeCondition interface { + match(v *structuredVolume) bool + validate() error +} + +type structuredVolume struct { + capacity resource.Quantity + storageClass string + nfs *nFSVolumeSource + csi *csiVolumeSource + volumeType SupportedVolume + pvcLabels map[string]string + pvcPhase string +} + +type Action struct { + Type VolumeActionType `yaml:"type"` + Parameters map[string]any `yaml:"parameters,omitempty"` +} + +const ( + ConfigmapRefType string = "configmap" + Skip VolumeActionType = "skip" + FSBackup VolumeActionType = "fs-backup" + Snapshot VolumeActionType = "snapshot" +) +``` + +`action.parameters` is used to provide extra information of the action. This is an ideal place to differentate Velero file system data mover and Velero block data mover. +Therefore, Velero built-in data mover will support `dataMover` key in `parameters`, with the value either `velero-fs` or `velero-block`. While `velero-fs` and `velero-block` are with the same meaning with Per Backup Selection. + +As an example, here is how a user might use both `velero-block` and `velero-fs` in a single backup: +- Users set `DataMover` parameter for the backup as `velero-block` +- Users add a record into Volume Policy, make `conditions` to filter the volumes they want to backup through Velero file system data mover, make `action.type` as `snapshot` and insert a record into `action.parameter` as `dataMover:velero-fs` + +In this way, all volumes matched by `conditions` will be backed up with Velero file system data mover; while the others will fallback to the per backup method Velero block data mover. + +Vice versa, users could set the per backup method as file system data mover and select volumes for Velero block data mover. + +The selected data mover for each volume should be recorded to `volumeInfo.json`. ### Controllers Backup controller and Restore controller are kept as is, async operations are still used to interact with VBDM with block data mover. -DataUpload controller and DataDownload controller are almost kpet as is, with some minor changes to handle the data mover type and backup type appropriately and convey it to the exposers. With [VGDP Micro Service][3], the controllers are almost isolated from VGDP, so no major changes are required. +DataUpload controller and DataDownload controller are almost kept as is, with some minor changes to handle the data mover type and backup type appropriately and convey it to the exposers. With [VGDP Micro Service][3], the controllers are almost isolated from VGDP, so no major changes are required. ### Exposer @@ -107,7 +169,7 @@ However, Kubernetes doesn't allow to bound a PV to a PVC with mismatch volume mo Therefore, the workflow of ***Finish Volume Readiness*** as introduced in [Volume Snapshot Data Movement design][2] is changed as below: - When restore completes and restorePV is created, set restorePV's `deletionPolicy` to `Retain` -- Create another rebindPV and copy restorePV's `volumeHandle` +- Create another rebindPV and copy restorePV's `volumeHandle` but the `volumeMode` matches to the targetPVC - Delete restorePV - Set the rebindPV's claim reference (the ```claimRef``` filed) to targetPVC - Add the ```velero.io/dynamic-pv-restore``` label to the rebindPV @@ -184,7 +246,7 @@ To support non-Kopia uploader to save snapshots to Unified Repo, snapshot relate ListSnapshot(ctx context.Context, source string) ([]Snapshot, error) ``` -To support non-Kopia uploader to save metadata, which is used to describe the the backed up objects, some metadata related methods will be added to `BackupRepo` interface: +To support non-Kopia uploader to save metadata, which is used to describe the backed up objects, some metadata related methods will be added to `BackupRepo` interface: ```go // WriteMetadata writes metadata to the repo, metadata is used to describe data, e.g., file system // dirs are saved as metadata @@ -220,12 +282,12 @@ The Incremental Aware Object Extension uses fix-sized splitter for deduplication The chunk size is intentionally chosen as 1MB, reasons: - 1MB is a multiply of 4KB for file systems or common block sizes for raw block device usages - 1MB is the start boundary of partitions for modern operating systems, for both MBR and GPT, so partition metadata could be isolated to a separate chunk -- The more chunks are there, the more indexes in the repository, 1MB is a modreate value regarding to the overhead of indexes for Kopia repository +- The more chunks are there, the more indexes in the repository, 1MB is a moderate value regarding to the overhead of indexes for Kopia repository #### Benefits Since the existing block address table(BAT) of CAOS is reused and kept as is, it brings below benefits: - All the entries are still managed by Kopia CAOS, so Velero doesn't need to keep an extra data -- The objects written by Velero block uploader is still recgnozible by Kopia, for both full backup and incremental backup +- The objects written by Velero block uploader is still recognizable by Kopia, for both full backup and incremental backup - The existing data management in Kopia repository still works for objects generated by Velero block uploader, e.g., snapshot GC, repository maintenance, etc. Most importantly, this solution is super perfromant: @@ -234,7 +296,7 @@ Most importantly, this solution is super perfromant: #### Uploader behavior The block uploader's skippable write must also be aligned to this 1MB boundary, because Incremental Aware Object Extension needs to clone the entries that have been skipped from the parent object. -File system uploader is still using variable-sized deduplication, it is fine to keep data from the two uploaders into the same Kopia repository, though normally they won't be mutrally deduplicated. +File system uploader is still using variable-sized deduplication, it is fine to keep data from the two uploaders into the same Kopia repository, though normally they won't be mutually deduplicated. Volume could be resized; and volume size may not be aligned to 1MB boundary. The uploader need to handle the resize appropriately since Incremental Aware Object Extension cannot copy a BAT entry partially. #### CBT Layer @@ -280,6 +342,32 @@ Therefore, ChangeId is saved together with the repository snapshot. The data mover always queries parent snapshot from Unified Repo together with the ChangeId. In this way, no mismatch would happen. Inside the uploader, the upper layer (DataUpload controller) could also provide the ChangeId as a mechanism of double confirmation. The received ChangeId would be re-evaluated against the one in the provided snapshot. +For Kubernetes API, changeId is represented by `BaseSnapshotId`. +changeId retrieval is storage specific, generally, it is retrieved from the `SnapshotHandle` of the VolumeSnapshotContent object; however, storages may also refer to other places to retrieve the changeId. +That is, `SnapshotHandle` and changeId may be two different values, in this case, the both values need to be preserved. + +#### Volume Snapshot retention +Storages/CSI drivers may support the changeId differently based on the storage's capabilities: +1. In order to calculate the changes, some storages require the parent snapshot mapping to the changeId always exists at the time of `GetMetadataDelta` is called, then the parent snapshot can NOT be deleted as long as there are incremental backups based on it. +2. Some storages don't require the parent snapshot itself at the time of calculating changes, then parent snapshot could be deleted immediately after the parent backup completes. + +The existing exposer works perfectly with Case 1, that is, the snapshot is always deleted when the backup completes. +However, for Case 2, since the snapshot must be retained, the exposer needs changes as below: +- At the end of each backup, keep the current VolumeSnapshot's `deletionPolicy` as `Retain`, then when the VolumeSnapshot is deleted at the end of the backup, the cureent snapshot is retained in the storage +- `GetMetadataDelta` is called with `BaseSnapshotId` set as the preserved changeId +- When deleting a backup, a VolumeSnapshot-VolumeSnapshotContent pair is rebuilt with `deletionPolicy` as `delete` and `snapshotHandle` as the preserved one +- Then the rebuilt VolumeSnapshot is deleted so that the volume snapshot is deleted from the storage + +There is no way to automatically detect which way a specific volume support, so an interface is exposed to users to set the volume snapshot retention method. +The interface could be added to the `Action.Parameters` of Volume Policy. By default, Velero block data mover takes Way 1, so volume snapshot is never retained; if users specify `RetainSnapshot` parameter, Way 2 will be taken. +```go +type Action struct { + Type VolumeActionType `yaml:"type"` + Parameters map[string]any `yaml:"parameters,omitempty"` +} +``` +In this way, users could specify --- for storage class "xxx" or CSI driver "yyy", backup through CSI snapshot with Velero block data mover and retain the snapshot. + #### Incremental Size By the end of the backup, incremental size is also returned by the uploader, as same as Velero file system uploader. The size indicates how much data are unquie so processed by the uploader, based on the provided CBT. @@ -293,8 +381,9 @@ There are some occasions that the incremental backup won't continue, so the data Incremental backup could continue when volume is resized. Block uploader supports to write disk with arbibrary size. When volume resize happens, block uploader needs to handle it appropriately in below ways: -- The new boundary should always be Max(oldSize, newSize) -- Always rewrite the tail between RoundDownTo1M(newSize) and newSize +- Loop with CBT to the boundary of RoundDownTo1M(newSize) +- Read data between RoundDownTo1M(newSize) and newSize to get the tail data +- Call `WriteAt` from offset RoundDownTo1M(newSize) with the tail data That is to say, the tail must be rewrite since Incremental Aware Object Extension cannot copy BAT entries halfly for the incremental backup. @@ -309,7 +398,7 @@ Sequaltial read/write of the volume is always optimized, there is no prove that ### Progress Report Progress report outside of the data mover will reuse the existing mechanism. -Innside the data mover, progress update is embedded to the uploader writer. +Inside the data mover, progress update is embedded to the uploader writer. The progress struct is kept as is, Velero block data mover still supports `TotalBytes` and `BytesDone`: ```go type Progress struct { @@ -325,7 +414,7 @@ For many reasons, a periodical full backup is required: Therefore, backup type (full/incremental) should be supported in Velero's manual backup and backup schedule. Backup type will also be added to `volumeInfo.json` to support observability purposes. -Backup TTL is still used for users to specify a backup's retention time. By default, both full and incremetnal backups are with 30 days retention, even though this is not so reasonable for the full backups. This could be enhanced when Velero supports sophisticated retention policy. +Backup TTL is still used for users to specify a backup's retention time. By default, both full and incremental backups are with 30 days retention, even though this is not so reasonable for the full backups. This could be enhanced when Velero supports sophisticated retention policy. As a workaround, users could create two schedules for the same scope of backup, one is for full backups, with less frequency and longer backup TTL; the other one is for incremental backups, with normal frequency and shorter backup TTL. #### File System Data Mover @@ -336,14 +425,6 @@ Therefore, to solve this problem and to make it align with Velero block data mov At present, the data path for Velero file system data mover has already supported it, we only need to expose this functionality to users. -### Data Mover Selection -At present, in the Backup CRD, there is a `DataMover` field for users to specify the data mover to be used. This field will be kept, once it is set, all the data movements in the backup will use the specified data mover. - -However, this is not enough. In one backup, users may include volumes backed by both block storage and file system storage. -To meet this requirement, VolumePolicy will be enhanced to support the `DataMover` action. - -The selected data mover for each volume should be updated to `volumeInfo.json`. - ### Backup Describe Backup type should be added to backup description, this value could be retrieved from `volumeInfo.json`. The `DataMover` item in the existing backup description should be updated to reflect the actual data mover completing the backup, this information could be retrieved from `volumeInfo.json`. @@ -426,7 +507,7 @@ When the parameter is not specified, by default, Velero goes with incremental ba -[1]: Implemented/unified-repo-and-kopia-integration/unified-repo-and-kopia-integration.md -[2]: Implemented/volume-snapshot-data-movement/volume-snapshot-data-movement.md -[3]: Implemented/vgdp-micro-service/vgdp-micro-service.md +[1]: ../Implemented/unified-repo-and-kopia-integration/unified-repo-and-kopia-integration.md +[2]: ../Implemented/volume-snapshot-data-movement/volume-snapshot-data-movement.md +[3]: ../Implemented/vgdp-micro-service/vgdp-micro-service.md [4]: https://kubernetes.io/blog/2025/09/25/csi-changed-block-tracking/ \ No newline at end of file