mirror of
https://github.com/scylladb/scylladb.git
synced 2026-04-24 10:30:38 +00:00
a69d98c3d0
" Cleanup compaction works by rewriting all sstables that need clean up, one at a time. This approach can cause bad write amplification because the output data is being made incrementally available for regular compaction. Cleanup is a long operation on large data sets, and while it's happening, new data can be written to buckets, triggering regular compaction. Cleanup fighting for resources with regular compaction is a known problem. With cleanup adding one file at a time to buckets, regular may require multiple rounds to compact the data in a given bucket B, producing bad writeamp. To fix this problem, cleanup will be made bucket aware. As each compaction strategy has its own definition of bucket, strategies will implement their own method to retrieve cleanup jobs. The method will be implemented such that all files in a bucket B will be cleaned up together, and on completion, they'll be made available for regular at once. For STCS / ICS, a bucket is a size tier. For TWCS, a bucket is a window. For LCS, a bucket is a level. In this way, writeamp problem is fixed as regular won't have to perform multiple rounds to compact the data in a given bucket. Additionally, cleanup will now be able to deduplicate data and will become way more efficient at garbage collecting expired data. The space requirement shouldn't be an issue, as compacting an entire bucket happens during regular compaction anyway. With leveled strategy, compacting an entire level is also not a problem because files in a level L don't overlap and therefore incremental compaction is employed to limit the space requirement. By the time being, only STCS cleanup was made bucket aware. The others will be using a default method, where one file is cleaned up at a time. Making cleanup of other strategies bucket aware is relatively easy now and will be done soon. Refs #10097. " * 'cleanup-compaction-revamp/v3' of https://github.com/raphaelsc/scylla: test: sstable_compaction_test: Add test for strategy cleanup method compaction: STCS: Implement cleanup strategy compaction_manager: Wire cleanup task into the strategy cleanup method compaction_strategy: Allow strategies to define their own cleanup strategy compaction: Introduce compaction_descriptor::sstables_size compaction: Move decision of garbage collection from strategy to task type