mirror of
https://github.com/scylladb/scylladb.git
synced 2026-05-14 03:42:14 +00:00
150345cc52
This series adds per-test bucket isolation to all S3 and GCS object storage tests. Previously, every test shared a single pre-created bucket, which meant tests could interfere with each other through leftover objects and could not run concurrently across multiple `test.py` processes without risking collisions. New `create_bucket`, `delete_bucket`, and `delete_bucket_with_objects` methods on `s3::client`, following the existing `make_request` pattern. `create_bucket` handles the `BUCKET_ALREADY_OWNED_BY_YOU` error gracefully. A new `s3_test_fixture` RAII class for C++ Boost tests that creates a uniquely-named bucket on construction (derived from the Boost test name and pid) and tears down everything — objects, bucket, client — on destruction. All S3 tests in `s3_test.cc` are migrated to use it, removing manual `deferred_delete_object` and `deferred_close` boilerplate. The minio server policy is broadened to allow dynamic bucket creation/deletion. A `client::make` overload that accepts a custom `retry_strategy`, used in tests with a fast 1ms retry delay instead of exponential backoff, significantly reducing test runtime for transient errors during bucket lifecycle operations. Python-side (`test/cluster/object_store`): each pytest fixture (`object_storage`, `s3_storage`, `s3_server`) now creates a unique bucket per test function via `create_test_bucket()` and destroys it on teardown. Bucket names are sanitized from the pytest node name with a short UUID suffix for uniqueness. Object storage helpers (`S3Server`, `MinioWrapper`, `GSFront`, `GSServerImpl`, factory functions, CQL helpers, `s3_server` fixture) are extracted from `test/cluster/object_store/conftest.py` into a shared `test/pylib/object_storage.py` module, eliminating duplication across test suites. The conftest becomes a thin re-export wrapper. Old class names are preserved as aliases for backward compatibility. | Test Name | new test specific retry strategy execution time (ms) | original execution time (ms) | Δ (ms) | Speedup | |--------------------------------------------------------------|----------------:|-------------:|---------:|--------:| | test_client_upload_file_multi_part_with_remainder_proxy | 19,261 | 61,395 | −42,134 | **3.2×** | | test_client_upload_file_multi_part_without_remainder_proxy | 16,901 | 53,688 | −36,787 | **3.2×** | | test_client_upload_file_single_part_proxy | 3,478 | 6,789 | −3,311 | **2.0×** | | test_client_multipart_copy_upload_proxy | 1,303 | 1,619 | −316 | 1.2× | | test_client_put_get_object_proxy | 150 | 365 | −215 | **2.4×** | | test_client_readable_file_stream_proxy | 125 | 327 | −202 | **2.6×** | | test_small_object_copy_proxy | 205 | 389 | −184 | 1.9× | | test_client_put_get_tagging_proxy | 181 | 350 | −169 | 1.9× | | test_client_multipart_upload_proxy | 1,252 | 1,416 | −164 | 1.1× | | test_client_list_objects_proxy | 729 | 881 | −152 | 1.2× | | test_chunked_download_data_source_with_delays_proxy | 830 | 960 | −130 | 1.2× | | test_client_readable_file_proxy | 148 | 279 | −131 | 1.9× | | test_client_upload_file_multi_part_with_remainder_minio | 3,358 | 3,170 | +188 | 0.9× | | test_client_upload_file_multi_part_without_remainder_minio | 3,131 | 2,929 | +202 | 0.9× | | test_client_upload_file_single_part_minio | 519 | 421 | +98 | 0.8× | | test_download_data_source_proxy | 180 | 237 | −57 | 1.3× | | test_client_list_objects_incomplete_proxy | 590 | 641 | −51 | 1.1× | | test_large_object_copy_proxy | 952 | 991 | −39 | 1.0× | | test_client_multipart_upload_fallback_proxy | 148 | 185 | −37 | 1.3× | | test_client_multipart_copy_upload_minio | 641 | 674 | −33 | 1.1× | No backport needed — this is a test infrastructure improvement with no production code impact beyond the new `s3::client` methods. Closes scylladb/scylladb#29508 * github.com:scylladb/scylladb: test: extract object storage helpers to test/pylib/object_storage.py test: add per-test bucket isolation to object_store fixtures s3: add client::make overload with custom retry strategy test: add s3_test_fixture and migrate tests to per-bucket isolation s3: add create_bucket and delete_bucket to client