mirrors/scylladb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f3ecd279f2f363c01fb16a47e67835ec315f05ab
scylladb/utils
History
Avi Kivity 1da6a939fe Merge 'Track memory usage of S3 object uploads' from Pavel Emelyanov 
The S3 uploading sink needs to collect buffers internally before sending them out, because the minimal upload-able part size is 5Mb. When the necessary amount of bytes is accumulated, the part uploading fibers starts in the background. On flush the sink waits for all the fibers to complete and handles failure of any.

Uploading parallelism is nowadays limited by the means of the http client max-connections parameter. However, when a part uploading fibers waits for it connection it keeps the 5Mb+ buffers on the request's body, so even though the number of uploading parts is limited, the number of _waiting_ parts is effectively not.

This PR adds a shard-wide limiter on the number of background buffers S3 clients (and theirs http clients) may use.

Closes scylladb/scylladb#15497

* github.com:scylladb/scylladb:
  s3::client: Track memory in client uploads
  code: Configure s3 clients' memory usage
  s3::client: Construct client with shared semaphore
  sstables::storage_manager: Introduce config
2023-09-21 18:24:42 +03:00
..
abi
util: replace <tab> with spaces
2023-09-11 14:38:46 +03:00
arch/powerpc/crc32-vpmsum
…
gz
utils/gs/barrett.hh: aarch64: s/brarett/barrett/
2022-11-17 11:09:59 +02:00
lsa
utils/chunked_managed_vector: use operator<=> when appropriate
2023-04-28 15:59:08 +03:00
s3
Merge 'Track memory usage of S3 object uploads' from Pavel Emelyanov
2023-09-21 18:24:42 +03:00
allocation_strategy.hh
lsa, test: Extend memory footprint test with per-type total sizes
2023-09-18 11:23:18 +02:00
amortized_reserve.hh
db: range_tombstone_list: Avoid quadratic behavior when applying
2022-08-05 20:34:07 +03:00
anchorless_list.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
array-search.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
array-search.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
ascii.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
ascii.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
atomic_vector.hh
utils: atomic_vector: mark for_each functions as const
2023-09-04 16:14:38 +03:00
aws_sigv4.cc
utils/aws: Export timepoint formatter
2023-04-17 11:14:45 +03:00
aws_sigv4.hh
utils/aws: Brush up the aws_sigv4.hh header
2023-04-17 11:16:45 +03:00
base64.cc
utils: throw error on malformed input in base64 decode
2023-01-16 14:36:23 +01:00
base64.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
big_decimal.cc
treewide: do not include unused header
2023-05-07 19:01:29 +03:00
big_decimal.hh
utils: big_decimal: replace compare() with <=> operator
2023-04-15 12:52:30 +08:00
bit_cast.hh
utils: bit_cast: require TriviallyCopyable To
2022-08-08 08:02:27 +03:00
bloom_calculations.cc
bloom_filter: mark internal help function static
2023-03-24 15:41:45 +08:00
bloom_calculations.hh
bloom_filter: add more constness to false positive rate tables
2023-03-24 15:41:45 +08:00
bloom_filter.cc
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
bloom_filter.hh
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
bounded_stats_deque.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
bptree.hh
bptree: Replace assert with static_assert
2023-08-06 16:36:12 +03:00
buffer_input_stream.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
buffer_input_stream.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
buffer_view-to-managed_bytes_view.hh
utils/managed_bytes, serializer: add conversion between buffer_view<bytes_ostream> and managed_bytes_view
2023-05-07 17:17:34 +03:00
build_id.cc
build_id: cache the value
2022-06-02 11:21:05 +03:00
build_id.hh
treewide: extent copyright statements to present day
2021-06-06 19:18:49 +03:00
cached_file_stats.hh
utils: cached_file: deglobalize cached_file metrics
2023-09-01 22:34:41 +02:00
cached_file.hh
utils: cached_file: deglobalize cached_file metrics
2023-09-01 22:34:41 +02:00
chunked_vector.hh
utils: to_string: generalize range helpers
2023-05-02 10:48:46 +03:00
class_registrator.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
clmul.hh
utils: clmul: add 64-bit constexpr clmul
2022-11-12 17:04:05 +02:00
CMakeLists.txt
build: cmake: include pretty_printers.cc in util
2023-07-11 09:16:33 +03:00
coarse_steady_clock.hh
logalloc: prevent false positives in reclaim_timer
2022-10-02 13:41:40 +03:00
collection-concepts.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
compact-radix-tree.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
config_file_impl.hh
db: config: add live_updatable_config_params_changeable_via_cql option
2023-08-16 17:56:27 +03:00
config_file.cc
treewide: do not include unused header
2023-05-07 19:01:29 +03:00
config_file.hh
utils/config_file: document config_file::value_status
2023-09-18 16:20:06 +03:00
coroutine.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
crc.hh
code: Convert is_integral assertions to concepts
2022-02-24 19:44:29 +03:00
cross-shard-barrier.hh
cross-shard-barrier: Capture shared barrier in complete
2022-09-16 08:21:02 +03:00
data_input.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
date.h
date: use std::in_range() to check for invalid year
2023-02-15 22:56:49 +08:00
digest_algorithm.hh
utils: move hashing related files to utils/ module
2023-02-17 07:19:52 +02:00
digester.hh
utils: move hashing related files to utils/ module
2023-02-17 07:19:52 +02:00
directories.cc
utils/directories: verify_owner_and_mode: add recursive flag
2023-07-31 16:01:43 +03:00
directories.hh
utils/directories: verify_owner_and_mode: add recursive flag
2023-07-31 16:01:43 +03:00
disk-error-handler.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
disk-error-handler.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
div_ceil.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
double-decker.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
dynamic_bitset.cc
utils: dynamic_bitset: mark functions noexcept
2022-07-27 13:32:36 +03:00
dynamic_bitset.hh
utils: dynamic_bitset: mark functions noexcept
2022-07-27 13:32:36 +03:00
entangled.hh
compile: Fix headers so that *-headers targets compile cleanly.
2022-03-25 16:19:26 +02:00
enum_option.hh
config: fix printing of experimental feature list
2022-07-11 09:17:30 +02:00
error_injection.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
error_injection.hh
utils: error injection: check if it is an ongoing one-shot injection in is_enabled
2023-07-13 15:56:33 +02:00
estimated_histogram.hh
estimated_histogram: do not use dynamic format_string
2023-07-08 15:10:41 +03:00
exception_container.hh
exception_container: switch impl to std::shared_ptr and make copyable
2022-02-22 16:08:52 +01:00
exceptions.cc
util: replace <tab> with spaces
2023-09-11 14:38:46 +03:00
exceptions.hh
exceptions: Mark storage_io_error::code() with noexcept
2022-10-03 18:50:06 +03:00
exponential_backoff_retry.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
extremum_tracking.hh
min_max_tracker: mark functions noexcept
2022-07-27 13:17:27 +03:00
fb_utilities.hh
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
file_lock.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
file_lock.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
flush_queue.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
fmt-compat.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
fragment_range.hh
utils: fragment_range: specialize fmt::formatter<FragmentedView>
2023-04-11 16:09:38 +03:00
fragmented_temporary_buffer.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
hash.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
hashers.cc
utils: hashing: add simple_xx_hasher
2023-04-24 14:06:43 +03:00
hashers.hh
utils: hashers: add HasherReturning concept
2023-04-24 14:06:40 +03:00
hashing.hh
utils: hashers: add HasherReturning concept
2023-04-24 14:06:40 +03:00
histogram_metrics_helper.cc
histogram_metrics_helper.hh: Add to_metrics_summary function
2022-07-27 16:58:52 +03:00
histogram_metrics_helper.hh
histogram_metrics_helper.hh: Add to_metrics_summary function
2022-07-27 16:58:52 +03:00
histogram.hh
utils/histogram: drop defaulted copy ctor and assignment operator
2023-02-14 19:22:19 +08:00
http.hh
utils: export dns_connection_factory from s3/client.cc to http.hh
2023-09-14 05:25:14 -04:00
human_readable.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
human_readable.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
i_filter.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
i_filter.hh
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
immutable-collection.hh
compile: Fix headers so that *-headers targets compile cleanly.
2022-03-25 16:19:26 +02:00
in.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
input_stream.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
int_range.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
intrusive_btree.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
intrusive-array.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
large_bitset.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
large_bitset.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
latency.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
lexicographical_compare.hh
treewide: use std::lexicographical_compare_threeway
2023-04-21 14:28:18 +03:00
like_matcher.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
like_matcher.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
limiting_data_source.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
limiting_data_source.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
linearizing_input_stream.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
lister.cc
lister: Make lister::dir_entry_types an enum_set
2022-11-17 19:01:45 +03:00
lister.hh
utils/lister: Limit the API of scan_dir() to fs::path
2023-09-05 20:50:42 +03:00
loading_cache.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
loading_shared_values.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
log_heap.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
logalloc.cc
utils/logalloc: introduce logalloc::bad_alloc
2023-09-21 17:43:53 +03:00
logalloc.hh
lsa, test: Extend memory footprint test with per-type total sizes
2023-09-18 11:23:18 +02:00
lru.hh
config: add index_cache_fraction
2023-09-01 22:34:23 +02:00
managed_bytes.cc
utils: managed_bytes: add bidirectional conversion between bytes_opt and managed_bytes_opt
2023-05-07 17:16:38 +03:00
managed_bytes.hh
utils/managed_bytes, serializer: add conversion between buffer_view<bytes_ostream> and managed_bytes_view
2023-05-07 17:17:34 +03:00
managed_ref.hh
code: Convert nothrow construction assertions into concepts
2022-02-24 19:44:50 +03:00
managed_vector.hh
code: Convert nothrow construction assertions into concepts
2022-02-24 19:44:50 +03:00
maybe_yield.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
memory_data_sink.hh
memory_data_sink: Add move ctor
2023-03-20 07:55:20 +02:00
multiprecision_int.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
multiprecision_int.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
murmur_hash.cc
build: disable implicit fallthrough
2023-07-10 19:36:06 +02:00
murmur_hash.hh
build: disable implicit fallthrough
2023-07-10 19:36:06 +02:00
mutable_view.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
neat-object-id.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
observable.hh
utils: observer: qualify seastar::noncopyable_function
2022-11-28 21:58:30 +02:00
overloaded_functor.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
phased_barrier.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
preempt.hh
mutation_partition_v2: Accept arbitrary preemption source in apply_monotonically()
2023-01-27 21:56:31 +01:00
pretty_printers.cc
utils: drop operator<< for pretty printers
2023-07-17 14:02:13 +08:00
pretty_printers.hh
utils: drop operator<< for pretty printers
2023-07-17 14:02:13 +08:00
ranges.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
rate_limiter.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
rate_limiter.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
recent_entries_map.hh
utils: introduce recent_entries_map datatype to track least recent visited entries.
2023-02-03 19:04:32 +01:00
result_combinators.hh
result_combinators: add result_wrap_unpack
2022-02-22 16:25:21 +01:00
result_loop.hh
utils: result_loop: remove invalid and incorrect constraint
2022-04-18 12:27:18 +03:00
result_try.hh
util/result_try: reference this explicitly
2023-02-28 21:56:55 +08:00
result.hh
utils/result.hh: add missing header includes for boost.outcome
2022-03-16 12:28:47 +03:00
reusable_buffer.hh
utils: redesign reusable_buffer
2023-04-26 22:09:17 +02:00
rjson.cc
utils: yield when streaming json in print()
2023-01-23 13:46:06 +01:00
rjson.hh
alternator: evaluate expressions as false for stored malformed binary
2023-01-16 15:15:27 +01:00
rpc_utils.hh
storage_proxy: handler responses, use pointers to default constructed values instead of nulls
2023-06-26 11:10:38 +03:00
runtime.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
runtime.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
sequenced_set.hh
sequenced_set: add extract_vector method
2023-05-21 11:33:38 +04:00
serialization.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
serialized_action.hh
serialized_action: make serialized_action abortable
2023-05-11 16:31:23 +03:00
simple_hashers.hh
utils: hashing: add simple_xx_hasher
2023-04-24 14:06:43 +03:00
small_vector.hh
utils: to_string: generalize range helpers
2023-05-02 10:48:46 +03:00
stall_free.hh
clear_gently: remove noexcept for rvalue references overload
2023-05-24 12:08:19 +04:00
streaming_histogram.hh
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
tagged_integer.hh
utils: tagged_integer: implement std::numeric_limits::{min,max}
2023-05-15 10:19:39 +03:00
throttle.hh
test: Extract throttle object to separate header
2023-06-21 00:58:24 +02:00
to_string.cc
to_string: add print operators for std::{weak,partial}_ordering
2023-02-15 11:09:04 +02:00
to_string.hh
utils: to_string: get rid of utils::join
2023-05-02 10:59:58 +03:00
top_k.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
tuple_utils.hh
storage_proxy.cc: refactor encode_replica_exception_for_rpc
2023-06-15 15:52:50 +04:00
unconst.hh
db: Introduce mutation_partition_v2
2023-01-27 19:15:39 +01:00
updateable_value.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
updateable_value.hh
utils/updateable_value: add transforming_value_updater
2022-10-05 10:52:49 +03:00
utf8.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
utf8.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
UUID_gen.cc
utils: move hashing related files to utils/ module
2023-02-17 07:19:52 +02:00
UUID_gen.hh
test: use correct timestamp resolution in test_group0_history_clearing_old_entries
2023-07-13 10:33:52 +03:00
uuid.cc
utils: UUID: specialize fmt::formatter for UUID and tagged_uuid<>
2023-03-20 14:25:45 +08:00
UUID.hh
utils: UUID: make default tagged_uuid ctor constexpr
2023-07-31 19:29:18 +03:00
variant_element.hh
utils/variant_element: simplify is_variant_element with right fold
2022-11-27 16:34:34 +02:00
vle.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
xx_hasher.hh
utils: move hashing related files to utils/ module
2023-02-17 07:19:52 +02:00