"Overly large metadata can hog memory which especially hurts in setups
with bad disk/memory ratio. To ease the pain compress the in-memory
compression-info.
The compression is implemented based on Avi's idea which is to group n
offsets together into segments, where each segment stores a base
absolute offset into the file, the other offsets in the segments being
relative offsets (and thus of reduced size). Also offsets are allocated
only just enough bits to store their maximum value. The offsets are thus
packed in a buffer like so:
arrrarrrarrr...
where n is 4, a is an absolute offset and r are offsets relative to a.
This of course means that stored offsets will not be aligned, not even
on a byte boundary, but the size reduction pretty convincing.
In addition, segments are stored in buckets, where each bucket has its
own base offset. In addition, segments in a buckets are optimized to
address as large of a chunk of the data as possible for a given chunk
size."
Ref #1946.
* 'bdenes/compress-compression-v3' of https://github.com/denesb/scylla:
Add unit test for compress::offsets
Optimise the storage of compression chunk offsets
Add script to precompute segmented compression parameters
ef85cf1cb3