mirrors/scylladb
1
0
Fork 0
mirror of https://github.com/scylladb/scylladb.git synced 2026-05-30 03:30:49 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
b7400d20dd884ca64730f291f258fe412f781edd
scylladb/utils
History
Pavel Emelyanov 4b13b24695 Merge 's3: make S3 connection pool size configurable per scheduling group' from Ernest Zaslavsky 
The S3 client creates a separate HTTP connection pool per scheduling group. Previously, the pool size was hardcoded as shares/100, yielding 1-10 connections. This was not tunable and could under-provision connections for groups with low share counts.
**Changes**
- A missing include (short_streams.hh) in sstables_loader.cc is added first to fix CMake builds where the header is not transitively included.
- The hardcoded per-share divisor is replaced with a per-shard connection budget. The new `object_storage_connections_per_shard` config option (default 128) specifies the total number of connections available on each shard. Connections are distributed proportionally across scheduling groups based on their shares: `max_connections = budget * group_shares / total_shares`. Remainder connections are assigned to the group with the most shares. When a new scheduling group client is created, all existing groups are rebalanced via `set_maximum_connections`. Creation and rebalance are serialized with a semaphore to prevent concurrent rebalances from racing.
- The config option is made live-updateable: a `storage_manager` observer propagates changes to all existing S3 clients, triggering rebalance under the same semaphore.
Fixes: https://scylladb.atlassian.net/browse/SCYLLADB-1704
No backport needed since this change affects KS on object storage which is not operational yet.

Closes scylladb/scylladb#29719

* github.com:scylladb/scylladb:
  s3: make connections_per_shard live-updateable
  s3: distribute connection pool proportionally across scheduling groups
2026-05-21 12:12:36 +03:00
..
abi
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
azure/identity
tree: add missing -present to copyright headers
2026-05-21 10:57:42 +02:00
gcp
Update seastar submodule
2026-05-20 13:47:12 +03:00
gz
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
lsa
tree: add missing -present to copyright headers
2026-05-21 10:57:42 +02:00
rest
tree: add missing -present to copyright headers
2026-05-21 10:57:42 +02:00
s3
s3: make connections_per_shard live-updateable
2026-05-20 20:45:14 +03:00
abstract_formatter.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
alien_worker.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
alien_worker.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
allocation_strategy.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
amortized_reserve.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
anchorless_list.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
array-search.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
array-search.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
ascii.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
ascii.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
assert.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
atomic_vector.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
aws_sigv4.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
aws_sigv4.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
base64.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
base64.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
big_decimal.cc
vector_search: fix decimal/varint precision loss in filter value_to_json()
2026-05-18 17:07:26 +03:00
big_decimal.hh
vector_search: fix decimal/varint precision loss in filter value_to_json()
2026-05-18 17:07:26 +03:00
bit_cast.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
bloom_calculations.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
bloom_calculations.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
bloom_filter.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
bloom_filter.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
bounded_stats_deque.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
bptree.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
buffer_view-to-managed_bytes_view.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
build_id.cc
treewide: change assert() to SCYLLA_ASSERT()
2024-08-05 08:23:35 +03:00
build_id.hh
…
cached_file_stats.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
cached_file.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
checked-file-impl.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
chunked_vector.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
class_registrator.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
clmul.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
CMakeLists.txt
object_storage: add retryable machinery to object storage
2026-02-22 14:00:44 +02:00
coarse_steady_clock.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
collection-concepts.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
compact-radix-tree.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
composite_abort_source.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
config_file_impl.hh
Merge 'Make inclusion of config.hh cheaper' from Nadav Har'El
2026-04-15 14:40:15 +03:00
config_file.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
config_file.hh
Merge 'Make inclusion of config.hh cheaper' from Nadav Har'El
2026-04-15 14:40:15 +03:00
contiguous_shared_buffer.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
coroutine.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
crc.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
cross-shard-barrier.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
crypt_sha512.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
crypt_sha512.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
data_input.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
date.h
date: do not put space before literal operator
2024-10-21 11:21:52 +03:00
digest_algorithm.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
digester.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
directories.cc
sstables: refactor parse_path() to return std::expected<> instead of throwing
2026-05-11 11:58:14 +03:00
directories.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
disk_space_monitor.cc
Merge 'streaming: add oos protection in mutation based streaming' from Łukasz Paszkowski
2026-04-20 17:56:36 +03:00
disk_space_monitor.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
disk-error-handler.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
disk-error-handler.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
div_ceil.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
double-decker.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
dynamic_bitset.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
dynamic_bitset.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
entangled.hh
tree: add missing -present to copyright headers
2026-05-21 10:57:42 +02:00
enum_option.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
error_injection.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
error_injection.hh
utils/error_injection: add waiters() API
2026-04-30 11:45:12 +02:00
estimated_histogram.hh
treewide: fix spelling errors.
2026-04-21 18:20:26 +03:00
exception_container.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
exceptions.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
exceptions.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
exponential_backoff_retry.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
extremum_tracking.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
file_lock.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
file_lock.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
flush_queue.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
fragment_range.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
fragmented_temporary_buffer.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
from_chars_exactly.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
hash.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
hashers.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
hashers.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
hashing.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
histogram_metrics_helper.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
histogram_metrics_helper.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
histogram.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
http_client_error_processing.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
http_client_error_processing.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
http.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
http.hh
Update seastar submodule
2026-05-20 13:47:12 +03:00
human_readable.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
human_readable.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
i_filter.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
i_filter.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
immutable-collection.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
input_stream.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
int_range.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
interval.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
intrusive_btree.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
intrusive-array.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
io-wrappers.cc
utils: Replace local memory sink/source with seastar equivalents
2026-05-12 08:47:43 +03:00
io-wrappers.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
labels.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
labels.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
large_bitset.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
large_bitset.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
latency.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
lexicographical_compare.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
like_matcher.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
like_matcher.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
linearizing_input_stream.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
lister.cc
Update seastar submodule
2025-12-30 19:37:13 +03:00
lister.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
loading_cache.hh
utils: loading_cache: add insert() that is a no-op when caching is disabled
2026-04-30 16:51:23 +02:00
loading_shared_values.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
log_heap.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
log.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
logalloc.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
logalloc.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
lru_string_map.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
lru.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
managed_bytes.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
managed_bytes.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
managed_ref.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
managed_string.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
managed_vector.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
map_difference.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
maybe_yield.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
memory_data_sink.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
memory_limit_reached.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
multiprecision_int.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
multiprecision_int.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
murmur_hash.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
murmur_hash.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
mutable_view.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
neat-object-id.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
observable.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
on_internal_error.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
on_internal_error.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
only_on_shard0.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
overloaded_functor.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
phased_barrier.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
pluggable.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
preempt.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
pretty_printers.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
pretty_printers.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
rate_limiter.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
rate_limiter.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
recent_entries_map.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
refcounted.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
replicator.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
reservoir_sampling.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
result_combinators.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
result_loop.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
result_try.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
result.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
reusable_buffer.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
rjson.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
rjson.hh
rjson: Add helpers for conversions to dht::token and sstable_id
2026-05-12 10:40:21 +03:00
rolling_max_tracker.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
rpc_utils.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
runtime.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
runtime.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
scoped_item_list.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
seekable_source.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
sequenced_set.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
sequential_producer.hh
tree: add missing -present to copyright headers
2026-05-21 10:57:42 +02:00
serialization.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
serialized_action.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
simple_hashers.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
simple_value_with_expiry.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
small_vector.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
sorting.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
stall_free.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
streaming_histogram.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
tagged_integer.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
throttle.hh
treewide: change assert() to SCYLLA_ASSERT()
2024-08-05 08:23:35 +03:00
to_string.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
to_string.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
top_k.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
tuple_utils.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
unconst.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
unique_view.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
updateable_value.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
updateable_value.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
upload_progress.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
user_provided_param.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
utf8.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
utf8.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
UUID_gen.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
UUID_gen.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
uuid.cc
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
UUID.hh
utils: Add UUID::is_name_based()
2026-04-17 20:58:39 +03:00
value_or_reference.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
variant_element.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00
vle.hh
tree: add missing -present to copyright headers
2026-05-21 10:57:42 +02:00
xx_hasher.hh
LICENSE: Update to version 1.1
2026-04-12 19:46:33 +03:00