77121247bd
During tablet migration with S3-backed storage, every streaming attempt fails with `std::logic_error("unsupported operation on s3 readable file")`. The tablet load balancer continuously retries, and after prolonged failure the cluster degrades — S3 objects start returning 404, write timeouts cascade, and the cluster becomes effectively non-functional.
Two bugs in `sstable_stream_sink_impl`:
1. **`output()` used `open_file()` + `make_file_output_stream()`** to create the writable stream for receiving SSTable data on the follower side. For object storage backends, `open_file()` ignores write flags and always returns a read-only `readable_file`. Writing through it throws `std::logic_error("unsupported operation on s3 readable file")`, breaking all tablet migration streaming with S3-backed storage.
2. **`load_metadata()` used `file_exists()` on a local filesystem path** to check whether the Scylla metadata component exists before loading it. For S3-backed storage this always returns false. When encryption is enabled, each streamed SSTable component gets a fresh encryption key (because the previous key from scylla metadata is never loaded from S3). Only the last key survives, making all other components unreadable — decryption with the wrong key produces garbage, leading to parse failures or OOM crashes on `sst->load()`.
- Replace `open_file()` + `make_file_output_stream()` with `make_component_sink()` + `output_stream()`, which correctly produces an upload sink for S3/GCS and a `file_data_sink` for local filesystem. This is the same mechanism already used by `save_metadata()` in the same class.
- Replace `file_exists(local_path)` with `_sst->_storage->exists()` which correctly checks for the component on whatever backend the SSTable uses.
- C++ unit tests (`test_stream_sink_write_local`, `test_stream_sink_write_s3`, `test_stream_sink_write_gs`) in `file_stream_test.cc` exercise `sstable_stream_sink_impl::output()` with all storage backends.
- Parametrize the existing `test_file_streaming_respects_encryption` Python integration test over local, S3, and GCS backends using a new `storage` fixture in `test/cluster/conftest.py`. The S3/GCS variants exercise the `load_metadata` fix — without it, streaming fails with `malformed_sstable_exception` (decryption with the wrong key) and the migration never completes.
- Add reusable `make_cfg()` and `make_ks_opts()` helpers to `test/cluster/util.py` for building storage-parametrized server config and keyspace CQL options.
Fixes: https://scylladb.atlassian.net/browse/SCYLLADB-1704
Fixes: https://scylladb.atlassian.net/browse/SCYLLADB-1829
Backport is not needed since no release supports object-storage-backed keyspaces.
Closes scylladb/scylladb#29698
* github.com:scylladb/scylladb:
test: parametrize encrypted streaming test with storage backends
sstables: fix load_metadata to check S3 for scylla metadata component
test: verify sstable stream sink writes to object storage
sstables: use make_component_sink for stream sink output
Scylla in-source tests.
For details on how to run the tests, see docs/dev/testing.md
Shared C++ utils, libraries are in lib/, for Python - pylib/
alternator - Python tests which connect to a single server and use the DynamoDB API unit, boost, raft - unit tests in C++ cqlpy - Python tests which connect to a single server and use CQL topology* - tests that set up clusters and add/remove nodes cql - approval tests that use CQL and pre-recorded output rest_api - tests for Scylla REST API Port 9000 scylla-gdb - tests for scylla-gdb.py helper script nodetool - tests for C++ implementation of nodetool
If you can use an existing folder, consider adding your test to it. New folders should be used for new large categories/subsystems, or when the test environment is significantly different from some existing suite, e.g. you plan to start scylladb with different configuration, and you intend to add many tests and would like them to reuse an existing Scylla cluster (clusters can be reused for tests within the same folder).
To add a new folder, create a new directory, and then
copy & edit its suite.ini.