9adfb91f46
Currently, to stream data from sstable component the sstables code uses file_data_source_impl. In case the component is on S3, the s3::readable_file is put into that data source. The data source is configured with 128k buffers and at most 4 read-ahead-s. With that configuration, downloading full object from S3 becomes too slow -- GET-ing file with 128k requests is not nice even with 4 parallel read-ahead-s. Better solution for S3 downloading is to request way larger chunk with one GET and then produce smaller, 128k or alike, buffers upon data arrival. This is what the newly introduced data source impl does -- it spawns a background GET and lets the upper input stream read buffers directly from the arriving body. This PR doesn't yet make sstable layer use the new sink, just introduces it and adds unit and perf tests. Testing |Test|Download speed, MB/s| |-|-| |file_input_stream (*), 1 socket | 4.996| |file_input_stream (*), 2 sockets | 9.403| |s3_data_source (**) | 93.164| (*) The file_input_stream test renders 128k GETs and is configured to issue at most 4 read-ahead-s (**) The s3_data_source uses at most 1 socket regardless of what perf-test configures it to refs: #22458 Closes scylladb/scylladb#22907 * github.com:scylladb/scylladb: test: Extend s3-perf test with stream download one test/perf: Tune-up s3 test options parsing test: Add unit test for newly introduced download source s3/client: Introduce data_source_impl for object downloading s3/client: Detach format_range_header() helper
Scylla in-source tests.
For details on how to run the tests, see docs/dev/testing.md
Shared C++ utils, libraries are in lib/, for Python - pylib/
alternator - Python tests which connect to a single server and use the DynamoDB API unit, boost, raft - unit tests in C++ cqlpy - Python tests which connect to a single server and use CQL topology* - tests that set up clusters and add/remove nodes cql - approval tests that use CQL and pre-recorded output rest_api - tests for Scylla REST API Port 9000 scylla-gdb - tests for scylla-gdb.py helper script nodetool - tests for C++ implementation of nodetool
If you can use an existing folder, consider adding your test to it. New folders should be used for new large categories/subsystems, or when the test environment is significantly different from some existing suite, e.g. you plan to start scylladb with different configuration, and you intend to add many tests and would like them to reuse an existing Scylla cluster (clusters can be reused for tests within the same folder).
To add a new folder, create a new directory, and then
copy & edit its suite.ini.