main, test: adjust number of networking iocbs

Seastar's default limit of 10,000 iocbs per shard is too low for
some workload (it places an upper bound on the number of idle
connections, above which a crash occurs). Use the new Seastar
feature to raise the default to 50000.

Also multiply the global reservation by 5, and round it upwards
so the number is less weird. This prevents io_setup() from failing.

For tests, the reservation is reduced since they don't create large
numbers of connections. This reduces surprise test failures when they
are run on machines that haven't been adjusted.

Fixes #9051

Closes #9052

dist/common/sysctl.d/99-scylla-aio.conf

@@ -1,2 +1,2 @@
 # Raise max AIO events
-fs.aio-max-nr = 5578536
+fs.aio-max-nr = 30000000

main.cc

@@ -438,6 +438,7 @@ int main(int ac, char** av) {
     app_cfg.name = "Scylla";
     app_cfg.default_task_quota = 500us;
     app_cfg.auto_handle_sigint_sigterm = false;
+    app_cfg.max_networking_aio_io_control_blocks = 50000;
     app_template app(std::move(app_cfg));
 
     auto ext = std::make_shared<db::extensions>();

test.py

@@ -292,7 +292,8 @@ class Test:
 
 
 class UnitTest(Test):
-    standard_args = shlex.split("--overprovisioned --unsafe-bypass-fsync 1 --kernel-page-cache 1 --blocked-reactor-notify-ms 2000000 --collectd 0")
+    standard_args = shlex.split("--overprovisioned --unsafe-bypass-fsync 1 --kernel-page-cache 1 --blocked-reactor-notify-ms 2000000 --collectd 0"
+                                " --max-networking-io-control-blocks=100")
 
     def __init__(self, test_no, shortname, args, suite, mode, options):
         super().__init__(test_no, shortname, suite, mode, options)

test/cql-pytest/run.py

@@ -185,6 +185,7 @@ def run_scylla_cmd(pid, dir):
         '--smp', '2',
         '-m', '1G',
         '--overprovisioned',
+        '--max-networking-io-control-blocks', '100',
         '--unsafe-bypass-fsync', '1',
         '--kernel-page-cache', '1',
         '--api-address', ip,

test/redis/run

@@ -70,6 +70,7 @@ ln -s "$SCYLLA" "$SCYLLA_LINK"
         --ring-delay-ms 0 --collectd 0 \
         --smp 2 -m 1G \
         --overprovisioned --unsafe-bypass-fsync 1 --kernel-page-cache 1 \
+        --max-networking-io-control-blocks 100 \
         --api-address $SCYLLA_IP \
         --rpc-address $SCYLLA_IP \
         --listen-address $SCYLLA_IP \