cli: Add the --workdir|-W option

When starting scylla daemon as non-root the initialization fails
because standard /var/lib/scylla is not accessible by regular users.
Making the default dir accessible for user is not very convenient
either, as it will cause conflicts if two or more instances of scylla
are in use.

This problem can be resolved by specifying --commitlog-directory,
--data-file-directories, etc on start, but it's too much typing. I
propose to revive Nadav's --home option that allows to move all the
directories under the same prefix in one go.

Unlike Nadav's approach the --workdir option doesn't do any tricky
manipulations with existing directories. Insead, as Pekka suggested,
the individual directories are placed under the workir if and only
if the respective option is NOT provided. Otherwise the directory
configuration is taken as is regardless of whether its absolute or
relative path.

The values substutution is done early on start. Avi suggested that
this is unsafe wrt HUP config re-read and proper paths must be
resolved on the fly, but this patch doesn't address that yet, here's
why.

First of all, the respective options are MustRestart now and the
substitution is done before HUP handler is installed.

Next, commitlog and data_file values are copied on start, so marking
the options as LiveUpdate won't make any effect.

Finally, the existing named_value::operator() returns a reference,
so returning a calculated (and thus temporary) value is not possible
(from my current understanding, correct me if I'm wrong). Thus if we
want the *_directory() to return calculated value all callers of them
must be patched to call something different (e.g. *_directory.get() ?)
which will lead to more confusion and errors.

Changes v3:
 - the option is --workdir back again
 - the existing *directory are only affected if unset
 - default config doesn't have any of these set
 - added the short -W alias

Changes v2:
 - the option is --home now
 - all other paths are changed to be relative

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Message-Id: <20191119130059.18066-1-xemul@scylladb.com>

README.md

@@ -27,10 +27,10 @@ Please see [HACKING.md](HACKING.md) for detailed information on building and dev
 
 ```
 
-* run Scylla with one CPU and ./tmp as data directory
+* run Scylla with one CPU and ./tmp as work directory
 
 ```
-./build/release/scylla --datadir tmp --commitlog-directory tmp --smp 1
+./build/release/scylla --workdir tmp --smp 1
 ```
 
 * For more run options:

conf/scylla.yaml

@@ -25,15 +25,19 @@
 # multiple tokens per node, see http://cassandra.apache.org/doc/latest/operating
 num_tokens: 256
 
+# Directory where Scylla should store all its files, which are commitlog,
+# data, hints, view_hints and saved_caches subdirectories. All of these
+# subs can be overriden by the respective options below.
+# If unset, the value defaults to /var/lib/scylla
+# workdir: /var/lib/scylla
+
 # Directory where Scylla should store data on disk.
-# If not set, the default directory is /var/lib/scylla/data.
-data_file_directories:
-    - /var/lib/scylla/data
+# data_file_directories:
+#    - /var/lib/scylla/data
 
 # commit log.  when running on magnetic HDD, this should be a
 # separate spindle than the data directories.
-# If not set, the default directory is /var/lib/scylla/commitlog.
-commitlog_directory: /var/lib/scylla/commitlog
+# commitlog_directory: /var/lib/scylla/commitlog
 
 # commitlog_sync may be either "periodic" or "batch."
 #

db/config.cc

@@ -198,15 +198,17 @@ db::config::config(std::shared_ptr<db::extensions> exts)
     )
     /* Default directories */
     /* If you have changed any of the default directories during installation, make sure you have root access and set these properties: */
-    , commitlog_directory(this, "commitlog_directory", value_status::Used, "/var/lib/scylla/commitlog",
+    , work_directory(this, "workdir,W", value_status::Used, "/var/lib/scylla",
+        "The directory in which Scylla will put all its subdirectories. The location of individual subdirs can be overriden by the respective *_directory options.")
+    , commitlog_directory(this, "commitlog_directory", value_status::Used, "",
         "The directory where the commit log is stored. For optimal write performance, it is recommended the commit log be on a separate disk partition (ideally, a separate physical device) from the data file directories.")
-    , data_file_directories(this, "data_file_directories", value_status::Used, { "/var/lib/scylla/data" },
+    , data_file_directories(this, "data_file_directories", value_status::Used, { },
         "The directory location where table data (SSTables) is stored")
-    , hints_directory(this, "hints_directory", value_status::Used, "/var/lib/scylla/hints",
+    , hints_directory(this, "hints_directory", value_status::Used, "",
         "The directory where hints files are stored if hinted handoff is enabled.")
-    , view_hints_directory(this, "view_hints_directory", value_status::Used, "/var/lib/scylla/view_hints",
+    , view_hints_directory(this, "view_hints_directory", value_status::Used, "",
         "The directory where materialized-view updates are stored while a view replica is unreachable.")
-    , saved_caches_directory(this, "saved_caches_directory", value_status::Unused, "/var/lib/scylla/saved_caches",
+    , saved_caches_directory(this, "saved_caches_directory", value_status::Unused, "",
         "The directory location where table key and row caches are stored.")
     /* Commonly used properties */
     /* Properties most frequently used when configuring Scylla. */
@@ -743,6 +745,28 @@ db::config::config()
 db::config::~config()
 {}
 
+void db::config::setup_directories() {
+    maybe_in_workdir(commitlog_directory, "commitlog");
+    maybe_in_workdir(data_file_directories, "data");
+    maybe_in_workdir(hints_directory, "hints");
+    maybe_in_workdir(view_hints_directory, "view_hints");
+    maybe_in_workdir(saved_caches_directory, "saved_caches");
+}
+
+void db::config::maybe_in_workdir(named_value<sstring>& to, const char* sub) {
+    if (!to.is_set()) {
+        to(work_directory() + "/" + sub);
+    }
+}
+
+void db::config::maybe_in_workdir(named_value<string_list>& tos, const char* sub) {
+    if (!tos.is_set()) {
+        string_list n;
+        n.push_back(work_directory() + "/" + sub);
+        tos(n);
+    }
+}
+
 const sstring db::config::default_tls_priority("SECURE128:-VERS-TLS1.0");
 
 namespace utils {

db/config.hh

@@ -82,6 +82,7 @@ public:
 
     // Throws exception if experimental feature is disabled.
     void check_experimental(const sstring& what) const;
+    void setup_directories();
 
     /**
      * Scans the environment variables for configuration files directory
@@ -112,6 +113,7 @@ public:
     named_value<sstring> listen_address;
     named_value<sstring> listen_interface;
     named_value<bool> listen_interface_prefer_ipv6;
+    named_value<sstring> work_directory;
     named_value<sstring> commitlog_directory;
     named_value<string_list> data_file_directories;
     named_value<sstring> hints_directory;
@@ -332,6 +334,9 @@ private:
     log_legacy_value<std::unordered_map<sstring, seastar::log_level>> logger_log_level;
     log_legacy_value<bool> log_to_stdout, log_to_syslog;
 
+    void maybe_in_workdir(named_value<sstring>&, const char*);
+    void maybe_in_workdir(named_value<string_list>&, const char*);
+
     std::shared_ptr<db::extensions> _extensions;
 };
 

main.cc

@@ -535,6 +535,7 @@ int main(int ac, char** av) {
             ::stop_signal stop_signal; // we can move this earlier to support SIGINT during initialization
             read_config(opts, *cfg).get();
             configurable::init_all(opts, *cfg, *ext).get();
+            cfg->setup_directories();
 
             // We're writing to a non-atomic variable here. But bool writes are atomic
             // in all supported architectures, and the broadcast_to_all_shards().get() below