compaction_manager: delay initialization of the compaction manager.

If the compaction manager is started, compactions may start (this is
regardless of whether or not we trigger them). The problem with that is
that they start at a time in which we are flushing the commitlog and the
initialization procedure waits for the commitlog to be fully flushed and
the resulting memtables flushed before we move on.

Because there are no incoming writes, the amount of shares in memtable
flushes decrease as memory used decreases and that can cause the startup
procedure to take a long time.

We have recently started to bump the shares manually for manual flushes.
While that guarantees that we will not drive the shares to zero, I will
make the argument that we can do better by making sure that those things
are, at this point, running alone: user experience is affected by
startup times and the bump we give to user-triggered operations will
only do so much. Even if we increase the shares a lot flushes will still
be fighting for resources with compactions and startup will take longer
than it could.

By making sure that flushes are this point running alone we improve the
user experience by making sure the startup is as fast as it can be.

Signed-off-by: Glauber Costa <glauber@scylladb.com>

database.cc

@@ -2228,7 +2228,6 @@ database::database(const db::config& cfg, database_config dbcfg)
     , _result_memory_limiter(dbcfg.available_memory / 10)
 {
     local_schema_registry().init(*this); // TODO: we're never unbound.
-    _compaction_manager->start();
     setup_metrics();
 
     _row_cache_tracker.set_compaction_scheduling_group(dbcfg.memory_compaction_scheduling_group);

main.cc

@@ -685,6 +685,11 @@ int main(int ac, char** av) {
                     cl->delete_segments(std::move(paths));
                 }
             }
+
+            db.invoke_on_all([&proxy] (database& db) {
+                db.get_compaction_manager().start();
+            }).get();
+
             // If the same sstable is shared by several shards, it cannot be
             // deleted until all shards decide to compact it. So we want to
             // start these compactions now. Note we start compacting only after