mirror of
https://github.com/versity/scoutfs.git
synced 2026-09-05 23:56:58 +00:00
Zach Brown
70c7178e6a
We want to be able to read a region of items from a segment by searching for the key that starts the item. In the first version of the segment format we find a key by performing a binary search across an array of offsets that point to the items. Unfortunately the current format requires that we know the number of items before we start writing. With thousands of items per segment it's a little bonkers to ask compaction to walk through all the items twice. Worse still, we didn't want the item offset array entries to span pages so they're rounded up to a power of two after having seqs and offsets and lengths. This makes them surprisingly large and sometimes they can consume up to 60% (!) of a segment. We know that we're inserting in sort order so it's very easy to build an index as we insert. Skip lists give us a nice simple way to ensure o(log n) lookups with only an average of two links per node. CPU use is greatly reduced by removing a full redundant item walk and we know use up almost all of the space in segments. There's still little gaps at the ends of blocks as item's still won't cross block boundaries. Most of this change is safely mechanical. The big difference is in how the compaction loop is built. It used to count the items before hand. It would never try to append when out of segments and writing would stop after the exact number of items. Now it discovers its out of items by allocating and trying to append and finding that there's no more work to do. It required rethinking the loop exit and segment allocation and stopping conditions. Signed-off-by: Zach Brown <zab@versity.com>