mirrors/scylladb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a78cc17bd451cf66bc45b366e417073f47bd526c
scylladb/utils
History
Botond Dénes ddf8547f25 Merge 'Add concurrency control and workload isolation for S3 client' from Pavel Emelyanov 
In its current state s3 client uses a single default-configured http client thus making different sched classes' workload compete with each other for sockets to make requests on. There's an attempt to handle that in upload-sink implementation that limits itself with some small number of concurrent PUT requests, but that doesn't help much as many sinks don't share this limit.

This PR makes S3 client maintain a set of http clients, one per sched-group, configures maximum number of TCP connections proportional to group's shares and removes the artificial limit from sinks thus making them share the group's http concurrency limit.

As a side effect, the upload-sink fixes the no-writes-after-flush protection -- if it's violated, write will result in exception, while currently it just hangs on a semaphore forever.

fixes: #13458
fixes: #13320
fixes: #13021

Closes #14187

* github.com:scylladb/scylladb:
  s3/client: Replace skink flush semaphore with gate
  s3/client: Configure different max-connections on http clients
  s3/client: Maintain several http clients on-board
  s3/client: Remove now unused http reference from sink and file
  s3/client: Add make_request() method
2023-06-20 07:09:21 +03:00
..
abi
utils/abi: don't use the deprecated std::unexpected_handler
2022-11-27 12:25:20 +02:00
arch/powerpc/crc32-vpmsum
…
gz
utils/gs/barrett.hh: aarch64: s/brarett/barrett/
2022-11-17 11:09:59 +02:00
lsa
utils/chunked_managed_vector: use operator<=> when appropriate
2023-04-28 15:59:08 +03:00
s3
s3/client: Replace skink flush semaphore with gate
2023-06-08 18:38:57 +03:00
allocation_strategy.hh
code: Convert nothrow construction assertions into concepts
2022-02-24 19:44:50 +03:00
amortized_reserve.hh
db: range_tombstone_list: Avoid quadratic behavior when applying
2022-08-05 20:34:07 +03:00
anchorless_list.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
array-search.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
array-search.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
ascii.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
ascii.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
atomic_vector.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
aws_sigv4.cc
utils/aws: Export timepoint formatter
2023-04-17 11:14:45 +03:00
aws_sigv4.hh
utils/aws: Brush up the aws_sigv4.hh header
2023-04-17 11:16:45 +03:00
base64.cc
utils: throw error on malformed input in base64 decode
2023-01-16 14:36:23 +01:00
base64.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
big_decimal.cc
treewide: do not include unused header
2023-05-07 19:01:29 +03:00
big_decimal.hh
utils: big_decimal: replace compare() with <=> operator
2023-04-15 12:52:30 +08:00
bit_cast.hh
utils: bit_cast: require TriviallyCopyable To
2022-08-08 08:02:27 +03:00
bloom_calculations.cc
bloom_filter: mark internal help function static
2023-03-24 15:41:45 +08:00
bloom_calculations.hh
bloom_filter: add more constness to false positive rate tables
2023-03-24 15:41:45 +08:00
bloom_filter.cc
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
bloom_filter.hh
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
bounded_stats_deque.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
bptree.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
buffer_input_stream.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
buffer_input_stream.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
buffer_view-to-managed_bytes_view.hh
utils/managed_bytes, serializer: add conversion between buffer_view<bytes_ostream> and managed_bytes_view
2023-05-07 17:17:34 +03:00
build_id.cc
build_id: cache the value
2022-06-02 11:21:05 +03:00
build_id.hh
treewide: extent copyright statements to present day
2021-06-06 19:18:49 +03:00
cached_file.hh
code: Switch to seastar API level 7
2023-06-06 13:29:16 +03:00
chunked_vector.hh
utils: to_string: generalize range helpers
2023-05-02 10:48:46 +03:00
class_registrator.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
clmul.hh
utils: clmul: add 64-bit constexpr clmul
2022-11-12 17:04:05 +02:00
CMakeLists.txt
utils: move generation-number to gms
2023-04-23 08:37:32 +03:00
coarse_steady_clock.hh
logalloc: prevent false positives in reclaim_timer
2022-10-02 13:41:40 +03:00
collection-concepts.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
compact-radix-tree.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
config_file_impl.hh
utils: s/std::regex/boost::regex/
2023-04-06 09:51:28 -04:00
config_file.cc
treewide: do not include unused header
2023-05-07 19:01:29 +03:00
config_file.hh
utils: config_file: add a space after =
2023-03-29 19:22:21 +08:00
coroutine.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
crc.hh
code: Convert is_integral assertions to concepts
2022-02-24 19:44:29 +03:00
cross-shard-barrier.hh
cross-shard-barrier: Capture shared barrier in complete
2022-09-16 08:21:02 +03:00
data_input.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
date.h
date: use std::in_range() to check for invalid year
2023-02-15 22:56:49 +08:00
digest_algorithm.hh
utils: move hashing related files to utils/ module
2023-02-17 07:19:52 +02:00
digester.hh
utils: move hashing related files to utils/ module
2023-02-17 07:19:52 +02:00
directories.cc
Update seastar submodule
2023-06-07 20:25:49 +03:00
directories.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
disk-error-handler.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
disk-error-handler.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
div_ceil.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
double-decker.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
dynamic_bitset.cc
utils: dynamic_bitset: mark functions noexcept
2022-07-27 13:32:36 +03:00
dynamic_bitset.hh
utils: dynamic_bitset: mark functions noexcept
2022-07-27 13:32:36 +03:00
entangled.hh
compile: Fix headers so that *-headers targets compile cleanly.
2022-03-25 16:19:26 +02:00
enum_option.hh
config: fix printing of experimental feature list
2022-07-11 09:17:30 +02:00
error_injection.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
error_injection.hh
treewide: use system-#include (angle brackets) for seastar
2022-04-26 14:46:42 +03:00
estimated_histogram.hh
estimated_histogram: add missing getter method
2022-07-26 15:59:33 +03:00
exception_container.hh
exception_container: switch impl to std::shared_ptr and make copyable
2022-02-22 16:08:52 +01:00
exceptions.cc
utils/exceptions.cc: don't rethrow in is_timeout_exception
2022-07-05 16:41:09 +02:00
exceptions.hh
exceptions: Mark storage_io_error::code() with noexcept
2022-10-03 18:50:06 +03:00
exponential_backoff_retry.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
extremum_tracking.hh
min_max_tracker: mark functions noexcept
2022-07-27 13:17:27 +03:00
fb_utilities.hh
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
file_lock.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
file_lock.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
flush_queue.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
fmt-compat.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
fragment_range.hh
utils: fragment_range: specialize fmt::formatter<FragmentedView>
2023-04-11 16:09:38 +03:00
fragmented_temporary_buffer.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
hash.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
hashers.cc
utils: hashing: add simple_xx_hasher
2023-04-24 14:06:43 +03:00
hashers.hh
utils: hashers: add HasherReturning concept
2023-04-24 14:06:40 +03:00
hashing.hh
utils: hashers: add HasherReturning concept
2023-04-24 14:06:40 +03:00
histogram_metrics_helper.cc
histogram_metrics_helper.hh: Add to_metrics_summary function
2022-07-27 16:58:52 +03:00
histogram_metrics_helper.hh
histogram_metrics_helper.hh: Add to_metrics_summary function
2022-07-27 16:58:52 +03:00
histogram.hh
utils/histogram: drop defaulted copy ctor and assignment operator
2023-02-14 19:22:19 +08:00
human_readable.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
human_readable.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
i_filter.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
i_filter.hh
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
immutable-collection.hh
compile: Fix headers so that *-headers targets compile cleanly.
2022-03-25 16:19:26 +02:00
in.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
input_stream.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
int_range.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
intrusive_btree.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
intrusive-array.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
large_bitset.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
large_bitset.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
latency.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
lexicographical_compare.hh
treewide: use std::lexicographical_compare_threeway
2023-04-21 14:28:18 +03:00
like_matcher.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
like_matcher.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
limiting_data_source.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
limiting_data_source.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
linearizing_input_stream.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
lister.cc
lister: Make lister::dir_entry_types an enum_set
2022-11-17 19:01:45 +03:00
lister.hh
lister: Make lister::dir_entry_types an enum_set
2022-11-17 19:01:45 +03:00
loading_cache.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
loading_shared_values.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
log_heap.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
logalloc.cc
utils: logalloc: correct asan_interface.h location
2023-05-29 23:03:25 +03:00
logalloc.hh
utils/logalloc: add use_standard_allocator_segment_pool_backend()
2022-09-16 13:02:40 +03:00
lru.hh
row_cache, lru: Introduce evict_shallow()
2023-01-27 21:56:31 +01:00
managed_bytes.cc
utils: managed_bytes: add bidirectional conversion between bytes_opt and managed_bytes_opt
2023-05-07 17:16:38 +03:00
managed_bytes.hh
utils/managed_bytes, serializer: add conversion between buffer_view<bytes_ostream> and managed_bytes_view
2023-05-07 17:17:34 +03:00
managed_ref.hh
code: Convert nothrow construction assertions into concepts
2022-02-24 19:44:50 +03:00
managed_vector.hh
code: Convert nothrow construction assertions into concepts
2022-02-24 19:44:50 +03:00
maybe_yield.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
memory_data_sink.hh
memory_data_sink: Add move ctor
2023-03-20 07:55:20 +02:00
multiprecision_int.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
multiprecision_int.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
murmur_hash.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
murmur_hash.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
mutable_view.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
neat-object-id.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
observable.hh
utils: observer: qualify seastar::noncopyable_function
2022-11-28 21:58:30 +02:00
overloaded_functor.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
phased_barrier.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
preempt.hh
mutation_partition_v2: Accept arbitrary preemption source in apply_monotonically()
2023-01-27 21:56:31 +01:00
ranges.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
rate_limiter.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
rate_limiter.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
recent_entries_map.hh
utils: introduce recent_entries_map datatype to track least recent visited entries.
2023-02-03 19:04:32 +01:00
result_combinators.hh
result_combinators: add result_wrap_unpack
2022-02-22 16:25:21 +01:00
result_loop.hh
utils: result_loop: remove invalid and incorrect constraint
2022-04-18 12:27:18 +03:00
result_try.hh
util/result_try: reference this explicitly
2023-02-28 21:56:55 +08:00
result.hh
utils/result.hh: add missing header includes for boost.outcome
2022-03-16 12:28:47 +03:00
reusable_buffer.hh
utils: redesign reusable_buffer
2023-04-26 22:09:17 +02:00
rjson.cc
utils: yield when streaming json in print()
2023-01-23 13:46:06 +01:00
rjson.hh
alternator: evaluate expressions as false for stored malformed binary
2023-01-16 15:15:27 +01:00
runtime.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
runtime.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
sequenced_set.hh
sequenced_set: add extract_vector method
2023-05-21 11:33:38 +04:00
serialization.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
serialized_action.hh
serialized_action: make serialized_action abortable
2023-05-11 16:31:23 +03:00
simple_hashers.hh
utils: hashing: add simple_xx_hasher
2023-04-24 14:06:43 +03:00
small_vector.hh
utils: to_string: generalize range helpers
2023-05-02 10:48:46 +03:00
stall_free.hh
clear_gently: remove noexcept for rvalue references overload
2023-05-24 12:08:19 +04:00
streaming_histogram.hh
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
tagged_integer.hh
utils: tagged_integer: implement std::numeric_limits::{min,max}
2023-05-15 10:19:39 +03:00
to_string.cc
to_string: add print operators for std::{weak,partial}_ordering
2023-02-15 11:09:04 +02:00
to_string.hh
utils: to_string: get rid of utils::join
2023-05-02 10:59:58 +03:00
top_k.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
tuple_utils.hh
storage_proxy.cc: refactor encode_replica_exception_for_rpc
2023-06-15 15:52:50 +04:00
unconst.hh
db: Introduce mutation_partition_v2
2023-01-27 19:15:39 +01:00
updateable_value.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
updateable_value.hh
utils/updateable_value: add transforming_value_updater
2022-10-05 10:52:49 +03:00
utf8.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
utf8.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
UUID_gen.cc
utils: move hashing related files to utils/ module
2023-02-17 07:19:52 +02:00
UUID_gen.hh
utils: UUID_gen: accept decimicroseconds in min_time_UUID
2023-04-21 10:33:02 +02:00
uuid.cc
utils: UUID: specialize fmt::formatter for UUID and tagged_uuid<>
2023-03-20 14:25:45 +08:00
UUID.hh
utils: UUID: drop uint64_t_tri_compare()
2023-05-07 18:07:49 +03:00
variant_element.hh
utils/variant_element: simplify is_variant_element with right fold
2022-11-27 16:34:34 +02:00
vle.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
xx_hasher.hh
utils: move hashing related files to utils/ module
2023-02-17 07:19:52 +02:00