db/config: add auto_snapshot_ttl

Add a live-updateable configuration option to set auto_snapshot_ttl.
It will be used in the following patches to set an
expiration time on the auto-snapshot, and then
to automatically delete it when it expires.

Fixes SCYLLADB-191

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

conf/scylla.yaml

@@ -372,7 +372,16 @@ commitlog_total_space_in_mb: -1
 # or dropping of column families. The STRONGLY advised default of true 
 # should be used to provide data safety. If you set this flag to false, you will
 # lose data on truncation or drop.
-# auto_snapshot: true
+#
+# `auto_snapshot_ttl` specifies the time-to-live (TTL) for automatic snapshots in seconds.
+# A value of 0 means snapshots are kept indefinitely.
+# The auto-snapshot will be deleted automatically when it expires.
+# It is a good practice to set `auto_snapshot_ttl` to a reasonable time allowing backup
+# of the auto-snapshot, but to have a safety net that will automatically clean up stale snapshots
+# to reclaim their disk space.
+#
+auto_snapshot: true
+auto_snapshot_ttl: 864000
 
 # When executing a scan, within or across a partition, we need to keep the
 # tombstones seen in memory so we can return them to the coordinator, which

db/config.cc

@@ -1055,6 +1055,8 @@ db::config::config(std::shared_ptr<db::extensions> exts)
     */
     , auto_snapshot(this, "auto_snapshot", liveness::LiveUpdate, value_status::Used, true,
         "Enable or disable whether a snapshot is taken of the data before keyspace truncation or dropping of tables. To prevent data loss, using the default setting is strongly advised. If you set to false, you will lose data on truncation or drop.")
+    , auto_snapshot_ttl(this, "auto_snapshot_ttl", liveness::LiveUpdate, value_status::Used, 0,
+        "The time-to-live (TTL) for automatic snapshots in seconds. A value of 0 means snapshots are kept indefinitely.")
     /**
     * @Group Key caches and global row properties
     * @GroupDescription When creating or modifying tables, you enable or disable the key cache (partition key cache) or row cache for that table by setting the caching parameter. Other row and key cache tuning and configuration options are set at the global (node) level. Cassandra uses these settings to automatically distribute memory for each table on the node based on the overall workload and specific table usage. You can also configure the save periods for these caches globally.

db/config.hh

@@ -301,6 +301,7 @@ public:
     named_value<sstring> partitioner;
     named_value<uint16_t> storage_port;
     named_value<bool> auto_snapshot;
+    named_value<uint32_t> auto_snapshot_ttl;
     named_value<uint32_t> key_cache_keys_to_save;
     named_value<uint32_t> key_cache_save_period;
     named_value<uint32_t> key_cache_size_in_mb;

docs/kb/snapshots.rst

@@ -35,6 +35,12 @@ Apart from *planned backup* procedure described above, and as a safeguard from *
 
 The default setting for the ``auto_snapshot`` flag in ``/etc/scylla/scylla.yaml`` file is ``true``. It is **not** recommended to set it to ``false``, unless there is a good backup and recovery strategy in place.
 
+The automatically created snapshots remain on local storage until they are backed-up using the ``move_files`` option or they are explicitly deleted.
+Otherwise, stale snapshots might cause a node to run out of storage space by holding up to the SSTables in the snapshot after they are deleted from the table (by compaction, tablet-migration, TRUNCATE, DROP TABLE, and so on).
+The ``auto_snapshot_ttl`` configuration option can be used to automatically delete stale snapshots.
+
+The default setting for the ``auto_snapshot_ttl`` option in ``/etc/scylla/scylla.yaml`` file is ``864000`` seconds (10 days). It is recommended to set it to a reasonable time allowing backup of the auto-snapshot, yet have a safety net that will automatically clean up stale snapshots to reclaim their disk space.
+
 Snapshot Creation
 -----------------