mirrors/scylladb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fc5306c5e8d274a8cd0f4722fceaa4896f40d18e
scylladb/utils
History
Pavel Emelyanov fc5306c5e8 s3::client: Track memory in client uploads 
When uploading an object part, client spawns a background fiber that
keeps the buffers with data on the http request's write_body() lambda
capture. This generates unbound usage of memory with uploaded buffers
which is not nice. Even though s3 client is limited with http's client
max-connections parallelism, waiting for the available connection still
happens with buffers held in memory.

This patch makes the client claim the background memory from the
provided semaphore (which, in turn, sits on the shard-wide storage
manager instance). Once body writing is complete, the claimed units are
returned back to the semaphore allowing for more background writes.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-09-20 17:50:29 +03:00
..
abi
util: replace <tab> with spaces
2023-09-11 14:38:46 +03:00
arch/powerpc/crc32-vpmsum
…
gz
utils/gs/barrett.hh: aarch64: s/brarett/barrett/
2022-11-17 11:09:59 +02:00
lsa
utils/chunked_managed_vector: use operator<=> when appropriate
2023-04-28 15:59:08 +03:00
s3
s3::client: Track memory in client uploads
2023-09-20 17:50:29 +03:00
allocation_strategy.hh
lsa, test: Extend memory footprint test with per-type total sizes
2023-09-18 11:23:18 +02:00
amortized_reserve.hh
db: range_tombstone_list: Avoid quadratic behavior when applying
2022-08-05 20:34:07 +03:00
anchorless_list.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
array-search.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
array-search.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
ascii.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
ascii.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
atomic_vector.hh
utils: atomic_vector: mark for_each functions as const
2023-09-04 16:14:38 +03:00
aws_sigv4.cc
utils/aws: Export timepoint formatter
2023-04-17 11:14:45 +03:00
aws_sigv4.hh
utils/aws: Brush up the aws_sigv4.hh header
2023-04-17 11:16:45 +03:00
base64.cc
utils: throw error on malformed input in base64 decode
2023-01-16 14:36:23 +01:00
base64.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
big_decimal.cc
treewide: do not include unused header
2023-05-07 19:01:29 +03:00
big_decimal.hh
utils: big_decimal: replace compare() with <=> operator
2023-04-15 12:52:30 +08:00
bit_cast.hh
utils: bit_cast: require TriviallyCopyable To
2022-08-08 08:02:27 +03:00
bloom_calculations.cc
bloom_filter: mark internal help function static
2023-03-24 15:41:45 +08:00
bloom_calculations.hh
bloom_filter: add more constness to false positive rate tables
2023-03-24 15:41:45 +08:00
bloom_filter.cc
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
bloom_filter.hh
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
bounded_stats_deque.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
bptree.hh
bptree: Replace assert with static_assert
2023-08-06 16:36:12 +03:00
buffer_input_stream.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
buffer_input_stream.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
buffer_view-to-managed_bytes_view.hh
utils/managed_bytes, serializer: add conversion between buffer_view<bytes_ostream> and managed_bytes_view
2023-05-07 17:17:34 +03:00
build_id.cc
build_id: cache the value
2022-06-02 11:21:05 +03:00
build_id.hh
…
cached_file_stats.hh
utils: cached_file: deglobalize cached_file metrics
2023-09-01 22:34:41 +02:00
cached_file.hh
utils: cached_file: deglobalize cached_file metrics
2023-09-01 22:34:41 +02:00
chunked_vector.hh
utils: to_string: generalize range helpers
2023-05-02 10:48:46 +03:00
class_registrator.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
clmul.hh
utils: clmul: add 64-bit constexpr clmul
2022-11-12 17:04:05 +02:00
CMakeLists.txt
build: cmake: include pretty_printers.cc in util
2023-07-11 09:16:33 +03:00
coarse_steady_clock.hh
logalloc: prevent false positives in reclaim_timer
2022-10-02 13:41:40 +03:00
collection-concepts.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
compact-radix-tree.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
config_file_impl.hh
db: config: add live_updatable_config_params_changeable_via_cql option
2023-08-16 17:56:27 +03:00
config_file.cc
treewide: do not include unused header
2023-05-07 19:01:29 +03:00
config_file.hh
utils/config_file: document config_file::value_status
2023-09-18 16:20:06 +03:00
coroutine.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
crc.hh
code: Convert is_integral assertions to concepts
2022-02-24 19:44:29 +03:00
cross-shard-barrier.hh
cross-shard-barrier: Capture shared barrier in complete
2022-09-16 08:21:02 +03:00
data_input.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
date.h
date: use std::in_range() to check for invalid year
2023-02-15 22:56:49 +08:00
digest_algorithm.hh
utils: move hashing related files to utils/ module
2023-02-17 07:19:52 +02:00
digester.hh
utils: move hashing related files to utils/ module
2023-02-17 07:19:52 +02:00
directories.cc
utils/directories: verify_owner_and_mode: add recursive flag
2023-07-31 16:01:43 +03:00
directories.hh
utils/directories: verify_owner_and_mode: add recursive flag
2023-07-31 16:01:43 +03:00
disk-error-handler.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
disk-error-handler.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
div_ceil.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
double-decker.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
dynamic_bitset.cc
utils: dynamic_bitset: mark functions noexcept
2022-07-27 13:32:36 +03:00
dynamic_bitset.hh
utils: dynamic_bitset: mark functions noexcept
2022-07-27 13:32:36 +03:00
entangled.hh
compile: Fix headers so that *-headers targets compile cleanly.
2022-03-25 16:19:26 +02:00
enum_option.hh
config: fix printing of experimental feature list
2022-07-11 09:17:30 +02:00
error_injection.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
error_injection.hh
utils: error injection: check if it is an ongoing one-shot injection in is_enabled
2023-07-13 15:56:33 +02:00
estimated_histogram.hh
estimated_histogram: do not use dynamic format_string
2023-07-08 15:10:41 +03:00
exception_container.hh
exception_container: switch impl to std::shared_ptr and make copyable
2022-02-22 16:08:52 +01:00
exceptions.cc
util: replace <tab> with spaces
2023-09-11 14:38:46 +03:00
exceptions.hh
exceptions: Mark storage_io_error::code() with noexcept
2022-10-03 18:50:06 +03:00
exponential_backoff_retry.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
extremum_tracking.hh
min_max_tracker: mark functions noexcept
2022-07-27 13:17:27 +03:00
fb_utilities.hh
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
file_lock.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
file_lock.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
flush_queue.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
fmt-compat.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
fragment_range.hh
utils: fragment_range: specialize fmt::formatter<FragmentedView>
2023-04-11 16:09:38 +03:00
fragmented_temporary_buffer.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
hash.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
hashers.cc
utils: hashing: add simple_xx_hasher
2023-04-24 14:06:43 +03:00
hashers.hh
utils: hashers: add HasherReturning concept
2023-04-24 14:06:40 +03:00
hashing.hh
utils: hashers: add HasherReturning concept
2023-04-24 14:06:40 +03:00
histogram_metrics_helper.cc
histogram_metrics_helper.hh: Add to_metrics_summary function
2022-07-27 16:58:52 +03:00
histogram_metrics_helper.hh
histogram_metrics_helper.hh: Add to_metrics_summary function
2022-07-27 16:58:52 +03:00
histogram.hh
utils/histogram: drop defaulted copy ctor and assignment operator
2023-02-14 19:22:19 +08:00
http.hh
utils: export dns_connection_factory from s3/client.cc to http.hh
2023-09-14 05:25:14 -04:00
human_readable.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
human_readable.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
i_filter.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
i_filter.hh
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
immutable-collection.hh
compile: Fix headers so that *-headers targets compile cleanly.
2022-03-25 16:19:26 +02:00
in.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
input_stream.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
int_range.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
intrusive_btree.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
intrusive-array.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
large_bitset.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
large_bitset.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
latency.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
lexicographical_compare.hh
treewide: use std::lexicographical_compare_threeway
2023-04-21 14:28:18 +03:00
like_matcher.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
like_matcher.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
limiting_data_source.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
limiting_data_source.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
linearizing_input_stream.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
lister.cc
lister: Make lister::dir_entry_types an enum_set
2022-11-17 19:01:45 +03:00
lister.hh
utils/lister: Limit the API of scan_dir() to fs::path
2023-09-05 20:50:42 +03:00
loading_cache.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
loading_shared_values.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
log_heap.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
logalloc.cc
lsa, test: Extend memory footprint test with per-type total sizes
2023-09-18 11:23:18 +02:00
logalloc.hh
lsa, test: Extend memory footprint test with per-type total sizes
2023-09-18 11:23:18 +02:00
lru.hh
config: add index_cache_fraction
2023-09-01 22:34:23 +02:00
managed_bytes.cc
utils: managed_bytes: add bidirectional conversion between bytes_opt and managed_bytes_opt
2023-05-07 17:16:38 +03:00
managed_bytes.hh
utils/managed_bytes, serializer: add conversion between buffer_view<bytes_ostream> and managed_bytes_view
2023-05-07 17:17:34 +03:00
managed_ref.hh
code: Convert nothrow construction assertions into concepts
2022-02-24 19:44:50 +03:00
managed_vector.hh
code: Convert nothrow construction assertions into concepts
2022-02-24 19:44:50 +03:00
maybe_yield.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
memory_data_sink.hh
memory_data_sink: Add move ctor
2023-03-20 07:55:20 +02:00
multiprecision_int.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
multiprecision_int.hh
treewide: use defaulted operator!=() and operator==()
2023-04-27 10:24:46 +03:00
murmur_hash.cc
build: disable implicit fallthrough
2023-07-10 19:36:06 +02:00
murmur_hash.hh
build: disable implicit fallthrough
2023-07-10 19:36:06 +02:00
mutable_view.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
neat-object-id.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
observable.hh
utils: observer: qualify seastar::noncopyable_function
2022-11-28 21:58:30 +02:00
overloaded_functor.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
phased_barrier.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
preempt.hh
mutation_partition_v2: Accept arbitrary preemption source in apply_monotonically()
2023-01-27 21:56:31 +01:00
pretty_printers.cc
utils: drop operator<< for pretty printers
2023-07-17 14:02:13 +08:00
pretty_printers.hh
utils: drop operator<< for pretty printers
2023-07-17 14:02:13 +08:00
ranges.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
rate_limiter.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
rate_limiter.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
recent_entries_map.hh
utils: introduce recent_entries_map datatype to track least recent visited entries.
2023-02-03 19:04:32 +01:00
result_combinators.hh
result_combinators: add result_wrap_unpack
2022-02-22 16:25:21 +01:00
result_loop.hh
utils: result_loop: remove invalid and incorrect constraint
2022-04-18 12:27:18 +03:00
result_try.hh
util/result_try: reference this explicitly
2023-02-28 21:56:55 +08:00
result.hh
utils/result.hh: add missing header includes for boost.outcome
2022-03-16 12:28:47 +03:00
reusable_buffer.hh
utils: redesign reusable_buffer
2023-04-26 22:09:17 +02:00
rjson.cc
utils: yield when streaming json in print()
2023-01-23 13:46:06 +01:00
rjson.hh
alternator: evaluate expressions as false for stored malformed binary
2023-01-16 15:15:27 +01:00
rpc_utils.hh
storage_proxy: handler responses, use pointers to default constructed values instead of nulls
2023-06-26 11:10:38 +03:00
runtime.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
runtime.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
sequenced_set.hh
sequenced_set: add extract_vector method
2023-05-21 11:33:38 +04:00
serialization.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
serialized_action.hh
serialized_action: make serialized_action abortable
2023-05-11 16:31:23 +03:00
simple_hashers.hh
utils: hashing: add simple_xx_hasher
2023-04-24 14:06:43 +03:00
small_vector.hh
utils: to_string: generalize range helpers
2023-05-02 10:48:46 +03:00
stall_free.hh
clear_gently: remove noexcept for rvalue references overload
2023-05-24 12:08:19 +04:00
streaming_histogram.hh
treewide: remove empty comments in top-of-files
2022-05-13 07:11:58 +02:00
tagged_integer.hh
utils: tagged_integer: implement std::numeric_limits::{min,max}
2023-05-15 10:19:39 +03:00
throttle.hh
test: Extract throttle object to separate header
2023-06-21 00:58:24 +02:00
to_string.cc
to_string: add print operators for std::{weak,partial}_ordering
2023-02-15 11:09:04 +02:00
to_string.hh
utils: to_string: get rid of utils::join
2023-05-02 10:59:58 +03:00
top_k.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
tuple_utils.hh
storage_proxy.cc: refactor encode_replica_exception_for_rpc
2023-06-15 15:52:50 +04:00
unconst.hh
db: Introduce mutation_partition_v2
2023-01-27 19:15:39 +01:00
updateable_value.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
updateable_value.hh
utils/updateable_value: add transforming_value_updater
2022-10-05 10:52:49 +03:00
utf8.cc
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
utf8.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
UUID_gen.cc
utils: move hashing related files to utils/ module
2023-02-17 07:19:52 +02:00
UUID_gen.hh
test: use correct timestamp resolution in test_group0_history_clearing_old_entries
2023-07-13 10:33:52 +03:00
uuid.cc
utils: UUID: specialize fmt::formatter for UUID and tagged_uuid<>
2023-03-20 14:25:45 +08:00
UUID.hh
utils: UUID: make default tagged_uuid ctor constexpr
2023-07-31 19:29:18 +03:00
variant_element.hh
utils/variant_element: simplify is_variant_element with right fold
2022-11-27 16:34:34 +02:00
vle.hh
treewide: use Software Package Data Exchange (SPDX) license identifiers
2022-01-18 12:15:18 +01:00
xx_hasher.hh
utils: move hashing related files to utils/ module
2023-02-17 07:19:52 +02:00