From ebda2fd4dbba66a1cea210759b3393e04086d41a Mon Sep 17 00:00:00 2001
From: Avi Kivity <avi@scylladb.com>
Date: Mon, 2 Aug 2021 11:37:57 +0300
Subject: [PATCH] test: cql_test_env: increase file descriptor limit

It was observed that since fce124bd9029a87 ('Merge "Introduce
flat_mutation_reader_v2" from Tomasz') database_test takes much longer.
This is expected since it now runs the upgrade/downgrade reader tests
on all existing tests. It was also observed that in a similar time frame
database_test sometimes times our on test machines, taking much
longer than usual, even with the extra work for testing reader
upgrade/downgrade.

In an attempt to reproduce, I noticed ti failing on EMFILE (too many
open file descriptors). I saw that tests usually use ~100 open file
descriptors, while the default limit is 1024.

I suspect we have runaway concurrency, but I was not able to pinpoint the
cause. It could be compaction lagging behind, or cleanup work for
deleting tables (the test
test_database_with_data_in_sstables_is_a_mutation_source creates and
deletes many tables).

As a stopgap solution to unblock the tests, this patch raises the file
descriptor limit in the way recommended by [1]. While tests shouldn't
use so many descriptors, I ran out of ideas about how to plug the hole.

Note that main() does something similar, through more elaborate since
it needs to communicate to users. See ec60f44b6493 ("main: improve
process file limit handling").

[1] http://0pointer.net/blog/file-descriptor-limits.html

Closes #9121
---
 test/lib/cql_test_env.cc | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/test/lib/cql_test_env.cc b/test/lib/cql_test_env.cc
index c0e4274755..92d6b998c8 100644
--- a/test/lib/cql_test_env.cc
+++ b/test/lib/cql_test_env.cc
@@ -69,6 +69,8 @@
 #include "db/sstables-format-selector.hh"
 #include "repair/row_level.hh"
 #include "debug.hh"
+#include <sys/time.h>
+#include <sys/resource.h>
 
 using namespace std::chrono_literals;
 
@@ -171,6 +173,22 @@ private:
         }
         return ::make_shared<service::query_state>(_core_local.local().client_state, empty_service_permit());
     }
+    static void adjust_rlimit() {
+        // Tests should use 1024 file descriptors, but don't punish them
+        // with weird behavior if they do.
+        //
+        // Since this more of a courtesy, don't make the situation worse if
+        // getrlimit/setrlimit fail for some reason.
+        struct rlimit lim;
+        int r = getrlimit(RLIMIT_NOFILE, &lim);
+        if (r == -1) {
+            return;
+        }
+        if (lim.rlim_cur < lim.rlim_max) {
+            lim.rlim_cur = lim.rlim_max;
+            setrlimit(RLIMIT_NOFILE, &lim);
+        }
+    }
 public:
     single_node_cql_env(
             sharded<database>& db,
@@ -189,7 +207,9 @@ public:
             , _mnotifier(mnotifier)
             , _sl_controller(sl_controller)
             , _mm(mm)
-    { }
+    {
+        adjust_rlimit();
+    }
 
     virtual future<::shared_ptr<cql_transport::messages::result_message>> execute_cql(sstring_view text) override {
         testlog.trace("{}(\"{}\")", __FUNCTION__, text);