mirrors/scylladb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

db/config: Change default SSTable compressor to LZ4WithDictsCompressor

Browse Source 
`sstable_compression_user_table_options` allows configuring a
node-global SSTable compression algorithm for user tables via
scylla.yaml. The current default is `LZ4Compressor` (inherited from
Cassandra).

Make `LZ4WithDictsCompressor` the new default. Metrics from real datasets
in the field have shown significant improvements in compression ratios.

If the dictionary compression feature is not enabled in the cluster
(e.g., during an upgrade), fall back to the `LZ4Compressor`. Once the
feature is enabled, flip the default back to the dictionary compressor
using with a listener callback.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
This commit is contained in:
Nikos Dragazis
2025-10-23 20:47:32 +03:00
parent 96e727d7b9
commit 2fc812a1b9
2 changed files with 40 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
db/config.cc
View File

@@ -1315,11 +1315,11 @@ db::config::config(std::shared_ptr<db::extensions> exts)
, enable_sstables_mc_format(this, "enable_sstables_mc_format", value_status::Unused, true, "Enable SSTables 'mc' format to be used as the default file format. Deprecated, please use \"sstable_format\" instead.")
, enable_sstables_md_format(this, "enable_sstables_md_format", value_status::Unused, true, "Enable SSTables 'md' format to be used as the default file format. Deprecated, please use \"sstable_format\" instead.")
, sstable_format(this, "sstable_format", liveness::LiveUpdate, value_status::Used, "me", "Default sstable file format", {"md", "me", "ms"})
, sstable_compression_user_table_options(this, "sstable_compression_user_table_options", value_status::Used, compression_parameters{},
, sstable_compression_user_table_options(this, "sstable_compression_user_table_options", value_status::Used, compression_parameters{compression_parameters::algorithm::lz4_with_dicts},
"Server-global user table compression options. If enabled, all user tables"
"will be compressed using the provided options, unless overridden"
"by compression options in the table schema. The available options are:\n"
"* sstable_compression: The compression algorithm to use. Supported values: LZ4Compressor (default), LZ4WithDictsCompressor, SnappyCompressor, DeflateCompressor, ZstdCompressor, ZstdWithDictsCompressor, '' (empty string; disables compression).\n"
"* sstable_compression: The compression algorithm to use. Supported values: LZ4Compressor, LZ4WithDictsCompressor (default), SnappyCompressor, DeflateCompressor, ZstdCompressor, ZstdWithDictsCompressor, '' (empty string; disables compression).\n"
"* chunk_length_in_kb: (Default: 4) The size of chunks to compress in kilobytes. Allowed values are powers of two between 1 and 128.\n"
"* crc_check_chance: (Default: 1.0) Not implemented (option value is ignored).\n"
"* compression_level: (Default: 3) Compression level for ZstdCompressor and ZstdWithDictsCompressor. Higher levels provide better compression ratios at the cost of speed. Allowed values are integers between 1 and 22.")

39
main.cc
View File

@@ -2221,8 +2221,45 @@ To start the scylla server proper, simply invoke as: scylla server (or just scyl
// Semantic validation of sstable compression parameters from config.
// Adding here (i.e., after `join_cluster`) to ensure that the
// required SSTABLE_COMPRESSION_DICTS cluster feature has been negotiated.
//
// Also, if the dictionary compression feature is not enabled, use
// LZ4Compressor as the default algorithm instead of LZ4WithDictsCompressor.
const auto& dicts_feature_enabled = feature_service.local().sstable_compression_dicts;
auto& sstable_compression_options = cfg->sstable_compression_user_table_options;
gms::feature::listener_registration reg_listener;
if (!sstable_compression_options.is_set() && !dicts_feature_enabled) {
if (sstable_compression_options().get_algorithm() != compression_parameters::algorithm::lz4_with_dicts) {
on_internal_error(startlog, "expected LZ4WithDictsCompressor as default algorithm for sstable_compression_user_table_options.");
}
startlog.info("SSTABLE_COMPRESSION_DICTS feature is disabled. Overriding default SSTable compression to use LZ4Compressor instead of LZ4WithDictsCompressor.");
compression_parameters original_params{sstable_compression_options().get_options()};
auto params = sstable_compression_options().get_options();
params[compression_parameters::SSTABLE_COMPRESSION] = sstring(compression_parameters::algorithm_to_name(compression_parameters::algorithm::lz4));
smp::invoke_on_all([&sstable_compression_options, params = std::move(params)] {
if (!sstable_compression_options.is_set()) { // guard check; in case we ever make the option live updateable
sstable_compression_options(compression_parameters{params}, utils::config_file::config_source::None);
}
}).get();
// Register a callback to update the default compression algorithm when the feature is enabled.
// Precondition:
// The callback must run inside seastar::async context:
// - If the listener fires immediately, we are running inside seastar::async already.
// - If the listener is deferred, `feature_service::enable()` runs it inside seastar::async.
reg_listener = feature_service.local().sstable_compression_dicts.when_enabled([&sstable_compression_options, params = std::move(original_params)] {
startlog.info("SSTABLE_COMPRESSION_DICTS feature is now enabled. Overriding default SSTable compression to use LZ4WithDictsCompressor.");
smp::invoke_on_all([&sstable_compression_options, params = std::move(params)] {
if (!sstable_compression_options.is_set()) { // guard check; in case we ever make the option live updateable
sstable_compression_options(params, utils::config_file::config_source::None);
}
}).get();
});
}
try {
const auto& dicts_feature_enabled = feature_service.local().sstable_compression_dicts;
cfg->sstable_compression_user_table_options().validate(
compression_parameters::dicts_feature_enabled(bool(dicts_feature_enabled)));
} catch (const std::exception& e) {