mirrors/scylladb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge 'Reserve IOCBs for tool applications' from Botond Dénes

Browse Source 
Artifact tests have been failing since the switch to the native nodetool, because ScyllaDB doesn't leave any IOCBs for tools. On some setups it will consume all of them and then nodetool and any other native app will refuse to start because it will fail to allocate IOCBs.
This PR fixes this by making use of the freshly introduced `--reserve-io-control-blocks` seastar option, to reserve IOCBs for tool applications. Since the `linux-aio` and `epoll` reactor backends require quite a bit of these, we enable the `io_uring` reactor backend and switch tools to use this backend instead. The `io_uring` reactor backend needs just 2 IOCBs to function, so the reserve of 10 IOCBs set up in this PR is good for running 5 tool applications in parallel, which should be more than enough.

Fixes: https://github.com/scylladb/scylladb/issues/19185

The problem this PR fixes has a manual workaround (and is rare to begin with), no backport needed.

Closes scylladb/scylladb#21527

* github.com:scylladb/scylladb:
  main: configure a reserve IOCB for scylla-nodetool and friends
  configure: enable the io_uring backend
  main: use configure seastar defaults via app_template::seastar_options
This commit is contained in:
Avi Kivity
2024-12-09 19:22:19 +02:00
parent a9c244ddf7 ed08709e75
commit 1bac6b75dc
4 changed files with 35 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
CMakeLists.txt
View File

@@ -91,7 +91,7 @@ else()
set(Seastar_APPS ON CACHE BOOL "" FORCE)
set(Seastar_EXCLUDE_APPS_FROM_ALL ON CACHE BOOL "" FORCE)
set(Seastar_EXCLUDE_TESTS_FROM_ALL ON CACHE BOOL "" FORCE)
set(Seastar_IO_URING OFF CACHE BOOL "" FORCE)
set(Seastar_IO_URING ON CACHE BOOL "" FORCE)
set(Seastar_SCHEDULING_GROUPS_COUNT 16 CACHE STRING "" FORCE)
set(Seastar_UNUSED_RESULT_ERROR ON CACHE BOOL "" FORCE)
add_subdirectory(seastar)

2
configure.py
View File

@@ -1708,7 +1708,7 @@ def configure_seastar(build_dir, mode, mode_config):
'-DSeastar_UNUSED_RESULT_ERROR=ON',
'-DCMAKE_EXPORT_COMPILE_COMMANDS=ON',
'-DSeastar_SCHEDULING_GROUPS_COUNT=16',
'-DSeastar_IO_URING=OFF', # io_uring backend is not stable enough
'-DSeastar_IO_URING=ON',
]
if args.stack_guards is not None:

31
main.cc
View File

@@ -662,7 +662,7 @@ static int scylla_main(int ac, char** av) {
try {
runtime::init_uptime();
std::setvbuf(stdout, nullptr, _IOLBF, 1000);
app_template::config app_cfg;
app_template::seastar_options app_cfg;
app_cfg.name = "Scylla";
app_cfg.description =
R"(scylla - NoSQL data store using the seastar framework
@@ -677,17 +677,38 @@ For more information about individual tools, run: scylla {tool_name} --help
To start the scylla server proper, simply invoke as: scylla server (or just scylla).
)";
app_cfg.default_task_quota = 500us;
#ifdef DEBUG
// Increase the task quota to improve work:poll ratio in slow debug mode.
app_cfg.default_task_quota = 5ms;
app_cfg.reactor_opts.task_quota_ms.set_default_value(5);
#else
app_cfg.reactor_opts.task_quota_ms.set_default_value(0.5);
#endif
app_cfg.auto_handle_sigint_sigterm = false;
app_cfg.max_networking_aio_io_control_blocks = 50000;
app_cfg.reactor_opts.max_networking_io_control_blocks.set_default_value(50000);
{
const auto candidates = app_cfg.reactor_opts.reactor_backend.get_candidate_names();
// We don't wan't ScyllaDB to run with the io_uring backend.
// So select the default reactor backend explicitely here.
if (std::ranges::contains(candidates, "linux-aio")) {
app_cfg.reactor_opts.reactor_backend.select_default_candidate("linux-aio");
} else {
app_cfg.reactor_opts.reactor_backend.select_default_candidate("epoll");
}
// Leave some reserve IOCBs for scylla-nodetool and other native tool apps.
if (std::ranges::contains(candidates, "io_uring")) {
app_cfg.reactor_opts.reserve_io_control_blocks.set_default_value(10);
} else {
startlog.warn("Need to leave extra IOCBs in reserve for tools because the io_uring reactor backend is not available."
" Tools will fall-back to the epoll reactor backend, which requires more IOCBs to function.");
app_cfg.reactor_opts.reserve_io_control_blocks.set_default_value(1024);
}
}
// We need to have the entire app config to run the app, but we need to
// run the app to read the config file with UDF specific options so that
// we know whether we need to reserve additional memory for UDFs.
app_cfg.reserve_additional_memory_per_shard = db::config::wasm_udf_reserved_memory;
app_cfg.smp_opts.reserve_additional_memory_per_shard = db::config::wasm_udf_reserved_memory;
app_template app(std::move(app_cfg));
auto ext = std::make_shared<db::extensions>();

8
tools/utils.cc
View File

@@ -147,7 +147,13 @@ void configure_tool_mode(app_template::seastar_options& opts, const sstring& log
opts.reactor_opts.relaxed_dma.set_value();
opts.reactor_opts.unsafe_bypass_fsync.set_value(true);
opts.reactor_opts.kernel_page_cache.set_value(true);
opts.reactor_opts.reactor_backend.select_candidate("epoll");
if (std::ranges::contains(opts.reactor_opts.reactor_backend.get_candidate_names(), "io_uring")) {
opts.reactor_opts.reactor_backend.select_candidate("io_uring");
} else {
// On some systems (e.g. docker), io_uring is not available.
// In this case fall back to epoll.
opts.reactor_opts.reactor_backend.select_candidate("epoll");
}
opts.smp_opts.thread_affinity.set_value(false);
opts.smp_opts.mbind.set_value(false);
opts.smp_opts.smp.set_value(1);