Kefu Chai 8080658df7 backup_task: remove a component once it is uploaded ...

Previously, during backup, SSTable components are preserved in the
snapshot directory even after being uploaded. This leads to redundant
uploads in case of failed backups or restarts, wasting time and
resources (S3 API calls).

This change

- adds an optional query parameter named "move_files" to
  "/storage_service/backup" API. if it is set to "true", SSTable
  components are removed once they are backed up to object storage.
- conditionally removes SSTable components from the snapshot directory once
  they are successfully uploaded to the target location. This prevents
  re-uploading the same files and reduces disk usage.

This change only "Refs" #20655, because, we can move further optimize
the backup process, consider:

- Sending HEAD requests to S3 to check for existing files before uploading.
- Implementing support for resuming partially uploaded files.

Fixes #21799
Refs #20655

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

2025-01-22 11:17:01 +08:00

..

__init__.py

[test.py] Refactor alternator, nodetool, rest_api

2024-06-13 13:56:10 +02:00

conftest.py

test.py: Create central conftest.

2024-11-24 20:09:48 +02:00

suite.yaml

test/object_store: Enable tablets to match production settings

2024-11-28 10:00:45 +08:00

test_backup.py

backup_task: remove a component once it is uploaded

2025-01-22 11:17:01 +08:00

test_basic.py

treewide: Rename sstable registry location field to be owner

2024-10-11 14:11:28 +03:00