mirrors/scylladb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d126ea09ba0d4cd51cf96fb1e72f032fa9009bf3
scylladb/utils
History
Kefu Chai 55777812d4 s3/client: Optimize file streaming with zero-copy multipart uploads 
When streaming files using multipart upload, switch from using
`output_stream::write(const char*, size_t)` to passing buffer objects
directly to `output_stream::write()`. This eliminates unnecessary memory
copying that occurred when the original implementation had to
defensively copy data before sending.

The buffer objects can now be safely reused by the output stream instead
of creating deep copies, which should improve performance by reducing
memory operations during S3 file uploads.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#23567
2025-04-07 12:50:06 +03:00
..
abi
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
arch/powerpc/crc32-vpmsum
…
gz
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
lsa
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
s3
s3/client: Optimize file streaming with zero-copy multipart uploads
2025-04-07 12:50:06 +03:00
advanced_rpc_compressor_protocol.hh
utils: add advanced_rpc_compressor
2024-12-23 23:37:02 +01:00
advanced_rpc_compressor.cc
treewide: use angle brackets for including seastar headers
2025-03-17 10:03:06 +02:00
advanced_rpc_compressor.hh
utils: use std::to_underlying() when appropriate
2025-02-19 12:12:28 +03:00
alien_worker.cc
utils: introduce alien_worker
2024-12-23 23:37:02 +01:00
alien_worker.hh
utils: introduce alien_worker
2024-12-23 23:37:02 +01:00
allocation_strategy.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
amortized_reserve.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
anchorless_list.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
array-search.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
array-search.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
ascii.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
ascii.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
assert.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
atomic_vector.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
aws_sigv4.cc
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
aws_sigv4.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
base64.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
base64.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
big_decimal.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
big_decimal.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
bit_cast.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
bloom_calculations.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
bloom_calculations.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
bloom_filter.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
bloom_filter.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
bounded_stats_deque.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
bptree.hh
B+tree: Clean const_iterator->iterator conversion
2025-02-26 23:17:27 +02:00
buffer_input_stream.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
buffer_input_stream.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
buffer_view-to-managed_bytes_view.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
build_id.cc
treewide: change assert() to SCYLLA_ASSERT()
2024-08-05 08:23:35 +03:00
build_id.hh
…
cached_file_stats.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
cached_file.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
checked-file-impl.hh
code: Move checked-file-impl.hh to util/
2025-03-06 10:22:05 +02:00
chunked_vector.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
class_registrator.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
clmul.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
CMakeLists.txt
Merge 'Fix EAR not applied on write to S3 (but on read).' from Calle Wilund
2025-03-24 10:12:46 +03:00
coarse_steady_clock.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
collection-concepts.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
compact-radix-tree.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
composite_abort_source.hh
treewide: change assert() to SCYLLA_ASSERT()
2024-08-05 08:23:35 +03:00
config_file_impl.hh
tree: Remove unused boost headers
2025-02-25 10:32:32 +03:00
config_file.cc
tree: Remove unused boost headers
2025-02-25 10:32:32 +03:00
config_file.hh
Merge 'config: prevent SIGHUP from changing non-liveupdatable parameters' from Andrzej Jackowski
2025-02-06 11:33:59 +03:00
contiguous_shared_buffer.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
coroutine.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
crc.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
cross-shard-barrier.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
data_input.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
date.h
date: do not put space before literal operator
2024-10-21 11:21:52 +03:00
dict_trainer.cc
utils/hashers: add get_sha256()
2025-04-01 00:07:28 +02:00
dict_trainer.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
digest_algorithm.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
digester.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
directories.cc
tree: Fix polymorphic exception handling by using references
2025-02-26 23:15:16 +02:00
directories.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
disk_space_monitor.cc
config, disk_space_monitor: Allow overriding capacity via config
2025-03-06 13:35:37 +01:00
disk_space_monitor.hh
config, disk_space_monitor: Allow overriding capacity via config
2025-03-06 13:35:37 +01:00
disk-error-handler.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
disk-error-handler.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
div_ceil.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
double-decker.hh
B+tree: Clean const_iterator->iterator conversion
2025-02-26 23:17:27 +02:00
dynamic_bitset.cc
treewide: migrate from boost::adaptors::reversed to std::views::reverse
2025-01-07 13:22:00 +02:00
dynamic_bitset.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
entangled.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
enum_option.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
error_injection.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
error_injection.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
estimated_histogram.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
exception_container.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
exceptions.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
exceptions.hh
exceptions: Add try_catch_nested to universally handle nested exceptions of the same type.
2025-03-26 11:15:13 +01:00
exponential_backoff_retry.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
extremum_tracking.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
file_lock.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
file_lock.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
flush_queue.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
fragment_range.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
fragmented_temporary_buffer.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
from_chars_exactly.hh
utils: introduce from_chars_exactly()
2025-01-09 11:14:49 +02:00
hash.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
hashers.cc
utils/hashers: add get_sha256()
2025-04-01 00:07:28 +02:00
hashers.hh
utils/hashers: add get_sha256()
2025-04-01 00:07:28 +02:00
hashing.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
histogram_metrics_helper.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
histogram_metrics_helper.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
histogram.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
http.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
human_readable.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
human_readable.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
i_filter.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
i_filter.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
immutable-collection.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
input_stream.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
int_range.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
intrusive_btree.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
intrusive-array.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
io-wrappers.cc
utils: Add "io-wrappers", useful IO helper types
2025-03-20 14:54:09 +00:00
io-wrappers.hh
utils: Add "io-wrappers", useful IO helper types
2025-03-20 14:54:09 +00:00
labels.cc
Adding the __level and features labels
2025-03-03 16:58:38 +02:00
labels.hh
Adding the __level and features labels
2025-03-03 16:58:38 +02:00
large_bitset.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
large_bitset.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
latency.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
lexicographical_compare.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
like_matcher.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
like_matcher.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
limiting_data_source.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
limiting_data_source.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
linearizing_input_stream.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
lister.cc
code: Move checked-file-impl.hh to util/
2025-03-06 10:22:05 +02:00
lister.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
loading_cache.hh
utils: loading_cache: make clock_type a template parameter
2025-01-23 09:28:08 +02:00
loading_shared_values.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
log_heap.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
log.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
logalloc.cc
utils/logalloc.cc label metrics with basic_level
2025-03-03 16:58:38 +02:00
logalloc.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
lru.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
managed_bytes.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
managed_bytes.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
managed_ref.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
managed_vector.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
maybe_yield.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
memory_data_sink.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
memory_limit_reached.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
multiprecision_int.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
multiprecision_int.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
murmur_hash.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
murmur_hash.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
mutable_view.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
neat-object-id.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
observable.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
on_internal_error.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
on_internal_error.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
only_on_shard0.hh
storage_service, tablets: Collect per-node capacity in load_stats
2025-03-06 12:17:32 +01:00
overloaded_functor.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
phased_barrier.hh
utils: phased_barrier: add close() method
2024-12-26 06:54:07 +02:00
pluggable.hh
utils: add class pluggable
2025-03-05 08:25:50 +02:00
preempt.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
pretty_printers.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
pretty_printers.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
rate_limiter.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
rate_limiter.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
recent_entries_map.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
refcounted.hh
utils: add advanced_rpc_compressor
2024-12-23 23:37:02 +01:00
reservoir_sampling.hh
utils: introduce reservoir_sampling
2024-12-23 23:37:02 +01:00
result_combinators.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
result_loop.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
result_try.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
result.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
reusable_buffer.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
rjson.cc
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
rjson.hh
utils/rjson: streaming_writer: add RawValue()
2025-02-17 08:01:38 -05:00
rpc_utils.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
runtime.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
runtime.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
sequenced_set.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
sequential_producer.hh
Introduce LDAP role manager & saslauthd authenticator
2025-01-12 14:50:29 +02:00
serialization.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
serialized_action.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
shared_dict.hh
utils/hashers: add get_sha256()
2025-04-01 00:07:28 +02:00
simple_hashers.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
small_vector.hh
utils: small_vector: expose internal_capacity()
2024-12-24 12:19:20 +02:00
sorting.hh
utils/sorting: Fix VerticesContainer concept constraints
2025-02-26 23:23:53 +02:00
stall_free.hh
tree: correct misspellings
2025-01-26 15:54:06 +02:00
stream_compressor.cc
utils: correct misspellings
2025-01-02 16:47:57 +02:00
stream_compressor.hh
utils: correct misspellings
2025-01-02 16:47:57 +02:00
streaming_histogram.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
tagged_integer.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
throttle.hh
treewide: change assert() to SCYLLA_ASSERT()
2024-08-05 08:23:35 +03:00
to_string.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
to_string.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
top_k.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
tuple_utils.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
unconst.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
unique_view.hh
utils: implement drop-in replacement for replacing boost::adaptors::uniqued
2025-01-21 16:24:45 +08:00
updateable_value.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
updateable_value.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
user_provided_param.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
utf8.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
utf8.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
UUID_gen.cc
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
UUID_gen.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
uuid.cc
tree: Remove unused boost headers
2025-02-25 10:32:32 +03:00
UUID.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00
value_or_reference.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
variant_element.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
vle.hh
treewide: relicense to ScyllaDB-Source-Available-1.0
2024-12-18 17:45:13 +02:00
xx_hasher.hh
utils: do not include unused headers
2025-01-14 07:56:39 -05:00