mirrors/scylladb
1
0
Fork 0
mirror of https://github.com/scylladb/scylladb.git synced 2026-06-01 04:26:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

sstables: Switch data and index sink to use jumbo uploader

Browse Source 
These two can grow large. Non-jumbo sink is effectively limited with
10000 parts, since each is ~5Mb the maximum uploadable data/index
happens to be 50Gb which is too small.

Other components shouldn't grow that big and continue using simple and a
bit faster uploading sink.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
This commit is contained in:
Pavel Emelyanov
2023-04-19 15:00:30 +03:00
parent b3df2d0db0
commit b58ad040d2
1 changed files with 2 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
sstables/storage.cc
View File

@@ -494,7 +494,8 @@ future<file> s3_storage::open_component(const sstable& sst, component_type type,
future<data_sink> s3_storage::make_data_or_index_sink(sstable& sst, component_type type, io_priority_class pc) {
assert(type == component_type::Data || type == component_type::Index);
co_await ensure_remote_prefix(sst);
co_return _client->make_upload_sink(make_s3_object_name(sst, type));
// FIXME: if we have file size upper bound upfront, it's better to use make_upload_sink() instead
co_return _client->make_upload_jumbo_sink(make_s3_object_name(sst, type));
}
future<data_sink> s3_storage::make_component_sink(sstable& sst, component_type type, open_flags oflags, file_output_stream_options options) {