71 lines
3.1 KiB
ReStructuredText
71 lines
3.1 KiB
ReStructuredText
There Are A Lot Of Single SSTable Compactions That Don't Compact Anything
|
|
=========================================================================
|
|
|
|
This guide describes what to do if you start having many tombstones compactions that don't compact anything.
|
|
|
|
Phenomena
|
|
^^^^^^^^^
|
|
|
|
ScyllaDB's CPU utilization is unexpectedly high and we see too many compactions while there are not many
|
|
WRITE/UPDATE/DELETE/TTLed operations and you see many messages like this in the syslog:
|
|
|
|
.. code-block:: shell
|
|
|
|
compaction - Compacted 1 SSTables to ... (~100% of original) ...
|
|
|
|
Problem
|
|
^^^^^^^
|
|
|
|
ScyllaDB SSTables can have expired or soon to be expired tombstones in them and there is a need to clean them up eventually.
|
|
Tombstones can be generated by DELETE operations, TTL data, insertion of null fields or usage of collections.
|
|
Compactions will get rid of expired tombstones, but if there are no compactions currently happening, ScyllaDB may apply
|
|
heuristics to force compactions on a lone table that has a certain ratio of expired tombstones.
|
|
|
|
To validate that the SSTable being compacted indeed has tombstones:
|
|
|
|
.. code-block:: shell
|
|
|
|
sstablemetadata /var/lib/scylla/data/some_ks/some_cf-UUID/some_ks-ka-*-Data.db | grep "Estimated droppable tombstones:"
|
|
|
|
The output may be something like this:
|
|
|
|
.. code-block:: shell
|
|
|
|
Estimated droppable tombstones: 0.08672144669324656
|
|
Estimated droppable tombstones: 0.08935147496746765
|
|
Estimated droppable tombstones: 0.11850671228977766
|
|
Estimated droppable tombstones: 0.08511697234511045
|
|
Estimated droppable tombstones: 0.035764466964587543
|
|
Estimated droppable tombstones: 0.1894537114259597
|
|
Estimated droppable tombstones: 0.13213090183839682
|
|
Estimated droppable tombstones: 0.1208004932191389
|
|
Estimated droppable tombstones: 0.07019800509981686
|
|
Estimated droppable tombstones: 0.12196540423823726
|
|
Estimated droppable tombstones: 0.11981627152867445
|
|
Estimated droppable tombstones: 0.11780386543765468
|
|
Estimated droppable tombstones: 0.17214256718348064
|
|
|
|
Heuristics is not a perfect solution and it may issue a compaction in cases where there are no expired tombstones to drop.
|
|
Should this happen, a tombstone compaction may begin needlessly, wasting CPU and I/O resources resulting in a compacted table which shows no change from the original SSTable.
|
|
|
|
Solution
|
|
^^^^^^^^
|
|
|
|
1. Prevent automatic tombstones compactions by increasing the value of `tombstone_compaction_interval` to some big value (it's integer value representing a time period in seconds).
|
|
|
|
For example this will change it to 4 days:
|
|
|
|
.. code-block:: shell
|
|
|
|
ALTER table <your table name> with compaction = { 'class' : 'SizeTieredCompactionStrategy', 'tombstone_compaction_interval' : 345600 };
|
|
|
|
|
|
2. Make sure `unchecked_tombstone_compaction` is set to false. If it is set to true, `tombstone_threshold` will be ignored,
|
|
and compaction will be run whenever `tombstone_compaction_interval` has elapsed.
|
|
|
|
.. code-block:: shell
|
|
|
|
ALTER table <your table name> with compaction = { 'class' : 'SizeTieredCompactionStrategy', 'unchecked_tombstone_compaction' : false };
|
|
|
|
.. include:: /troubleshooting/_common/ts-return.rst
|