There Are A Lot Of Single SSTable Compactions That Don't Compact Anything
=========================================================================

This guide describes what to do if you start having many tombstones compactions that don't compact anything.

Phenomena
^^^^^^^^^

ScyllaDB's CPU utilization is unexpectedly high and we see too many compactions while there are not many
WRITE/UPDATE/DELETE/TTLed operations and you see many messages like this in the syslog:

.. code-block:: shell

    compaction - Compacted 1 SSTables to ... (~100% of original) ...

Problem
^^^^^^^

ScyllaDB SSTables can have expired or soon to be expired tombstones in them and there is a need to clean them up eventually.
Tombstones can be generated by DELETE operations, TTL data, insertion of null fields or usage of collections.
Compactions will get rid of expired tombstones, but if there are no compactions currently happening, ScyllaDB may apply
heuristics to force compactions on a lone table that has a certain ratio of expired tombstones.

To validate that the SSTable being compacted indeed has tombstones:

.. code-block:: shell

    sstablemetadata /var/lib/scylla/data/some_ks/some_cf-UUID/some_ks-ka-*-Data.db | grep "Estimated droppable tombstones:"

The output may be something like this:

.. code-block:: shell

    Estimated droppable tombstones: 0.08672144669324656
    Estimated droppable tombstones: 0.08935147496746765
    Estimated droppable tombstones: 0.11850671228977766
    Estimated droppable tombstones: 0.08511697234511045
    Estimated droppable tombstones: 0.035764466964587543
    Estimated droppable tombstones: 0.1894537114259597
    Estimated droppable tombstones: 0.13213090183839682
    Estimated droppable tombstones: 0.1208004932191389
    Estimated droppable tombstones: 0.07019800509981686
    Estimated droppable tombstones: 0.12196540423823726
    Estimated droppable tombstones: 0.11981627152867445
    Estimated droppable tombstones: 0.11780386543765468
    Estimated droppable tombstones: 0.17214256718348064

Heuristics is not a perfect solution and it may issue a compaction in cases where there are no expired tombstones to drop.
Should this happen, a tombstone compaction may begin needlessly, wasting CPU and I/O resources resulting in a compacted table which shows no change from the original SSTable.

Solution
^^^^^^^^

1. Prevent automatic tombstones compactions by increasing the value of `tombstone_compaction_interval` to some big value (it's integer value representing a time period in seconds).

For example this will change it to 4 days:

.. code-block:: shell

    ALTER table <your table name> with compaction = { 'class' : 'SizeTieredCompactionStrategy', 'tombstone_compaction_interval' : 345600 };


2. Make sure `unchecked_tombstone_compaction` is set to false. If it is set to true, `tombstone_threshold` will be ignored,
and compaction will be run whenever `tombstone_compaction_interval` has elapsed.

.. code-block:: shell

    ALTER table <your table name> with compaction = { 'class' : 'SizeTieredCompactionStrategy', 'unchecked_tombstone_compaction' : false };

.. include:: /troubleshooting/_common/ts-return.rst
