Kefu Chai
a6152cb87b
sstables: do not include unused headers
...
these unused includes were identified by clangd. see
https://clangd.llvm.org/guides/include-cleaner#unused-include-warning
for more details on the "Unused include" warning.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com >
Closes scylladb/scylladb#16666
2024-01-09 11:45:44 +02:00
Yaniv Kaul
c658bdb150
Typos: fix typos in comments
...
Fixes some typos as found by codespell run on the code.
In this commit, I was hoping to fix only comments, not user-visible alerts, output, etc.
Follow-up commits will take care of them.
Refs: https://github.com/scylladb/scylladb/issues/16255
Signed-off-by: Yaniv Kaul <yaniv.kaul@scylladb.com >
2023-12-02 22:37:22 +02:00
Avi Kivity
fcb8d040e8
treewide: use Software Package Data Exchange (SPDX) license identifiers
...
Instead of lengthy blurbs, switch to single-line, machine-readable
standardized (https://spdx.dev ) license identifiers. The Linux kernel
switched long ago, so there is strong precedent.
Three cases are handled: AGPL-only, Apache-only, and dual licensed.
For the latter case, I chose (AGPL-3.0-or-later and Apache-2.0),
reasoning that our changes are extensive enough to apply our license.
The changes we applied mechanically with a script, except to
licenses/README.md.
Closes #9937
2022-01-18 12:15:18 +01:00
Avi Kivity
a55b434a2b
treewide: extent copyright statements to present day
2021-06-06 19:18:49 +03:00
Avi Kivity
fa8d0fe4d0
Revert "Revert "Revert "Revert "Merge "Compress in-memory compression-info" from Botond""""
...
This reverts commit 238877a0c6
2017-08-28 16:14:13 +03:00
Avi Kivity
238877a0c6
Revert "Revert "Revert "Merge "Compress in-memory compression-info" from Botond"""
...
This reverts commit 9d27455744
2017-08-27 13:32:37 +03:00
Avi Kivity
9d27455744
Revert "Revert "Merge "Compress in-memory compression-info" from Botond""
...
This reverts commit 9656fd79a0
2017-08-24 13:37:35 +03:00
Tomasz Grabiec
9656fd79a0
Revert "Merge "Compress in-memory compression-info" from Botond"
...
This reverts commit ef85cf1cb3de011ece52
2017-08-24 11:44:14 +02:00
Botond Dénes
028c7a0888
Optimise the storage of compression chunk offsets
...
To reduce the memory footprint of compression-info, n offsets are
grouped together into segments, where each segment stores a base
absolute offset into the file, the other offsets in the segments being
relative offsets (and thus of reduced size). Also offsets are
allocated only just enough bits to store their maximum value. The
offsets are thus packed in a buffer like so:
arrrarrrarrr...
where n is 4, a is an absolute offset and r are offsets relative to a.
The optimal value of n can be calculated for a given file_size (f) and
chunk_size (c), by finding the minima of the following function:
f(n) = (f/c)/n * (log2(f) + (n - 1)*log2((n-1)*(c + 64)))
This is done in an empirical way, using a script (see below).
Furthermore segments are stored in buckets, where each bucket has its
own base offset. Each bucket therefore can address an equal chunk of the
file and furthermore each segment in a bucket can address an equal
sub-chunk of this area.
The value of a given offset i is thus:
bucket_base_offset_for(i) + segment_base_offset_for(i) + offset(i)
To account for the bucketed storage we calculate a local_f, which is
optimized so that a bucketful of segmented offsets can address the
largest possible chunk of f. As value of this local_f only depends on
the bucket_size (b) and c the value of n can be made independent of f
and therefore only depend on one dynamic value, c. This makes life much
simpler as we don't need to know the size of the file up-front, we can
just append buckets to the storage on demand, while the required storage
is still less than a third [1] of the original storage requirements
(std::deque<uint64>).
The table with the minima(f(n)) for different f and c values is
pre-computed by gen_segmented_compress_params.py and
stored in sstables/segmented_compress_params.hh. This script also
creates a table with the best values of local_f for the given
bucket_size. At runtime we only select the best params based on c.
[1] This was calculated for c=4K and b=4K
2017-08-21 17:06:12 +03:00
