mirrors/scylladb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0ade15df33ecdd12dab396331debf82ae55d7bc7
scylladb/utils
History
Ernest Zaslavsky 30199552ac s3_client: Mitigate connection exhaustion in download_source 
The existing `download_source` implementation optimizes performance
by keeping the connection to S3 open and draining data directly from
the socket. While this eliminates the overhead (60-100ms) of repeatedly
establishing new connections, it leads to rapid exhaustion of client-
side connections.

On a single shard, two `mx_readers` for load and stream are enough to
trigger this issue. Since each client typically holds two connections,
readers keeping index and data sources open can cause deadlocks where
processes stall due to unavailable connections.

Introduce `chunked_download_source`, a new S3 download method built on
`download_source`, to dynamically manage connections:

- Buffers data in 5MiB chunks using a producer-consumer model
- Closes connections once buffers reach capacity, returning them to
  the pool for other clients
- Uses a filling fiber that resumes fetching once buffers are
  consumed from the queue

Performance remains comparable to `download_source`, achieving
95MiB/s for sequential 1GiB downloads from S3. However, preloading
large chunks may cause read amplification.

Fixes: https://github.com/scylladb/scylladb/issues/23785

Closes scylladb/scylladb#23880
2025-06-10 12:58:24 +03:00
..
abi
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
arch/powerpc/crc32-vpmsum
…
gz
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
lsa
utils/lsa/chunked_managed_vector: fix the calculation of max_chunk_capacity()
2025-04-28 12:30:13 +02:00
rest
encryption/utils: Move encryption httpclient to "general" REST client
2025-05-30 12:21:51 +03:00
s3
s3_client: Mitigate connection exhaustion in download_source
2025-06-10 12:58:24 +03:00
advanced_rpc_compressor_protocol.hh
utils: add advanced_rpc_compressor
2024-12-23 23:37:02 +01:00
advanced_rpc_compressor.cc
treewide: use angle brackets for including seastar headers
2025-03-17 10:03:06 +02:00
advanced_rpc_compressor.hh
utils: use std::to_underlying() when appropriate
2025-02-19 12:12:28 +03:00
alien_worker.cc
utils: introduce alien_worker
2024-12-23 23:37:02 +01:00
alien_worker.hh
utils: introduce alien_worker
2024-12-23 23:37:02 +01:00
allocation_strategy.hh
utils/logalloc: enforce LSA allocation size limits
2025-05-29 13:05:08 +02:00
amortized_reserve.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
anchorless_list.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
array-search.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
array-search.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
ascii.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
ascii.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
assert.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
atomic_vector.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
aws_sigv4.cc
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
aws_sigv4.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
base64.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
base64.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
big_decimal.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
big_decimal.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
bit_cast.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
bloom_calculations.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
bloom_calculations.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
bloom_filter.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
bloom_filter.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
bounded_stats_deque.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
bptree.hh
B+tree: Clean const_iterator->iterator conversion
2025-02-26 23:17:27 +02:00
buffer_input_stream.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
buffer_input_stream.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
buffer_view-to-managed_bytes_view.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
build_id.cc
…
build_id.hh
…
cached_file_stats.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
cached_file.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
checked-file-impl.hh
code: Move checked-file-impl.hh to util/
2025-03-06 10:22:05 +02:00
chunked_vector.hh
utils: chunked_vector: add swap() method
2025-05-14 16:19:40 +03:00
class_registrator.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
clmul.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
CMakeLists.txt
utils::http::dns_connection_factory: Use a shared certificate_credentials
2025-06-10 11:20:21 +03:00
coarse_steady_clock.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
collection-concepts.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
compact-radix-tree.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
composite_abort_source.hh
…
config_file_impl.hh
tree: Remove unused boost headers
2025-02-25 10:32:32 +03:00
config_file.cc
tree: Remove unused boost headers
2025-02-25 10:32:32 +03:00
config_file.hh
Merge 'config: prevent SIGHUP from changing non-liveupdatable parameters' from Andrzej Jackowski
2025-02-06 11:33:59 +03:00
contiguous_shared_buffer.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
coroutine.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
crc.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
cross-shard-barrier.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
data_input.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
date.h
date: do not put space before literal operator
2024-10-21 11:21:52 +03:00
dict_trainer.cc
utils/hashers: add get_sha256()
2025-04-01 00:07:28 +02:00
dict_trainer.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
digest_algorithm.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
digester.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
directories.cc
tree: Fix polymorphic exception handling by using references
2025-02-26 23:15:16 +02:00
directories.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
disk_space_monitor.cc
disk_space_monitor: add space_source_registration
2025-06-04 16:25:24 +03:00
disk_space_monitor.hh
disk_space_monitor: add space_source_registration
2025-06-04 16:25:24 +03:00
disk-error-handler.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
disk-error-handler.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
div_ceil.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
double-decker.hh
B+tree: Clean const_iterator->iterator conversion
2025-02-26 23:17:27 +02:00
dynamic_bitset.cc
treewide: migrate from boost::adaptors::reversed to std::views::reverse
2025-01-07 13:22:00 +02:00
dynamic_bitset.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
entangled.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
enum_option.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
error_injection.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
error_injection.hh
test: Verify partitioned set store split and unsplit correctly
2025-04-29 15:47:33 -03:00
estimated_histogram.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
exception_container.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
exceptions.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
exceptions.hh
exceptions: Add try_catch_nested to universally handle nested exceptions of the same type.
2025-03-26 11:15:13 +01:00
exponential_backoff_retry.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
extremum_tracking.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
file_lock.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
file_lock.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
flush_queue.hh
utils: flush_queue: use named_gate
2025-04-12 11:47:02 +03:00
fragment_range.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
fragmented_temporary_buffer.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
from_chars_exactly.hh
utils: introduce from_chars_exactly()
2025-01-09 11:14:49 +02:00
hash.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
hashers.cc
utils/hashers: add get_sha256()
2025-04-01 00:07:28 +02:00
hashers.hh
utils/hashers: add get_sha256()
2025-04-01 00:07:28 +02:00
hashing.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
histogram_metrics_helper.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
histogram_metrics_helper.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
histogram.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
http.cc
utils::http::dns_connection_factory: Use a shared certificate_credentials
2025-06-10 11:20:21 +03:00
http.hh
utils::http::dns_connection_factory: Use a shared certificate_credentials
2025-06-10 11:20:21 +03:00
human_readable.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
human_readable.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
i_filter.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
i_filter.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
immutable-collection.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
input_stream.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
int_range.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
intrusive_btree.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
intrusive-array.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
io-wrappers.cc
utils: Add "io-wrappers", useful IO helper types
2025-03-20 14:54:09 +00:00
io-wrappers.hh
utils: Add "io-wrappers", useful IO helper types
2025-03-20 14:54:09 +00:00
labels.cc
Adding the __level and features labels
2025-03-03 16:58:38 +02:00
labels.hh
Adding the __level and features labels
2025-03-03 16:58:38 +02:00
large_bitset.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
large_bitset.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
latency.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
lexicographical_compare.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
like_matcher.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
like_matcher.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
limiting_data_source.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
limiting_data_source.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
linearizing_input_stream.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
lister.cc
code: Move checked-file-impl.hh to util/
2025-03-06 10:22:05 +02:00
lister.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
loading_cache.hh
utils::loading_cache: gracefully skip timer if gate closed
2025-04-30 16:43:22 +03:00
loading_shared_values.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
log_heap.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
log.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
logalloc.cc
Merge 'utils/logalloc: enforce the max contiguous allocation size limit' from Michał Chojnowski
2025-05-29 22:11:41 +03:00
logalloc.hh
Merge 'utils/logalloc: enforce the max contiguous allocation size limit' from Michał Chojnowski
2025-05-29 22:11:41 +03:00
lru.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
managed_bytes.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
managed_bytes.hh
managed_bytes: in the copy constructor, respect the target preferred allocation size
2025-04-16 22:06:06 +02:00
managed_ref.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
managed_vector.hh
utils/lsa/chunked_managed_vector: fix the calculation of max_chunk_capacity()
2025-04-28 12:30:13 +02:00
maybe_yield.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
memory_data_sink.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
memory_limit_reached.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
multiprecision_int.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
multiprecision_int.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
murmur_hash.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
murmur_hash.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
mutable_view.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
neat-object-id.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
observable.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
on_internal_error.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
on_internal_error.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
only_on_shard0.hh
storage_service, tablets: Collect per-node capacity in load_stats
2025-03-06 12:17:32 +01:00
overloaded_functor.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
phased_barrier.hh
utils: phased_barrier, pluggable: use named gate
2025-04-12 11:47:00 +03:00
pluggable.hh
utils: phased_barrier, pluggable: use named gate
2025-04-12 11:47:00 +03:00
preempt.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
pretty_printers.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
pretty_printers.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
rate_limiter.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
rate_limiter.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
recent_entries_map.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
refcounted.hh
utils: add advanced_rpc_compressor
2024-12-23 23:37:02 +01:00
reservoir_sampling.hh
utils: introduce reservoir_sampling
2024-12-23 23:37:02 +01:00
result_combinators.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
result_loop.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
result_try.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
result.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
reusable_buffer.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
rjson.cc
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
rjson.hh
utils/rjson: streaming_writer: add RawValue()
2025-02-17 08:01:38 -05:00
rpc_utils.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
runtime.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
runtime.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
sequenced_set.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
sequential_producer.hh
Introduce LDAP role manager & saslauthd authenticator
2025-01-12 14:50:29 +02:00
serialization.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
serialized_action.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
shared_dict.hh
utils/hashers: add get_sha256()
2025-04-01 00:07:28 +02:00
simple_hashers.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
small_vector.hh
utils: small_vector: expose internal_capacity()
2024-12-24 12:19:20 +02:00
sorting.hh
utils/sorting: Fix VerticesContainer concept constraints
2025-02-26 23:23:53 +02:00
stall_free.hh
tree: correct misspellings
2025-01-26 15:54:06 +02:00
stream_compressor.cc
utils/stream_compressor: allocate memory for zstd compressors externally
2025-05-27 12:43:11 +03:00
stream_compressor.hh
utils/stream_compressor: allocate memory for zstd compressors externally
2025-05-27 12:43:11 +03:00
streaming_histogram.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
tagged_integer.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
throttle.hh
…
to_string.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
to_string.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
top_k.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
tuple_utils.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
unconst.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
unique_view.hh
utils: implement drop-in replacement for replacing boost::adaptors::uniqued
2025-01-21 16:24:45 +08:00
updateable_value.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
updateable_value.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
user_provided_param.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
utf8.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
utf8.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
UUID_gen.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
UUID_gen.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
uuid.cc
tree: Remove unused boost headers
2025-02-25 10:32:32 +03:00
UUID.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
value_or_reference.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
variant_element.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
vle.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
xx_hasher.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00