scylla-precompiled-header defines the PCH (stdafx.hh) with PRIVATE
visibility, so targets linking to it do not inherit the PCH.
scylla-main was missing the PCH entirely, causing files like
sstables_loader.cc to fail with 'no member read_entire_stream' since
that symbol comes from <seastar/util/short_streams.hh> which is
included in stdafx.hh.
PR #29901 worked around this by adding the missing #include directly,
but the real fix is to propagate the PCH to scylla-main — matching
the configure.py behavior where every source file is compiled with
-include-pch stdafx.hh.pch.
Add target_precompile_headers(scylla-main REUSE_FROM
scylla-precompiled-header) so that all sources in scylla-main benefit
from the precompiled header.
Refs: https://github.com/scylladb/scylladb/pull/29901
Drop local formatter for seastar::http::reply, which should have
been added to Seastar in the first place, and now conflicts. Also
drop local formatters for types that are aliases for Seastar types
which have gained formatters.
Disable recently-gained TLS use of OpenSSL instead of gnutls. We
don't need it, and it causes link errors with LTO.
Fix incorrect skipping in encrypted_file_test, which computed
the remaining stream length but did not account for already
consumed size_to_compare.
Change utils::gcp::storage::client::object_data_source::skip()
to match new Seastar behavior (rejecting skip-past-eof with an
exception). This is needed since 30f1075544 switched the test's
data source to a Seastar implementation. It is also more correct -
if we're asked to skip n bytes but the stream doesn't have n bytes,
this is a protocol violation.
Contains test fix from Pavel, exposed by [1]:
test: Handle premature EOF in test_gcp_storage_skip_read
The test intentionally uses file_size larger than the actual object to
exercise EOF behavior. When input_stream::skip() is called after EOF,
it throws std::runtime_error("premature end of stream"). Catch this
specific exception from both streams, verify they agree, and exit the
loop gracefully.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[1] cbd1e17d2f, included in this Seastar submodule update
* seastar 4d268e0e...485a62b2 (50):
> reactor: open_directory(): honor bypass_fsync
> http: Add formatters for http::request and http::reply
> Merge 'Assorted set of io-tester cleanups' from Pavel Emelyanov
io_tester: Remove unused and internal-only accessor
io_tester: Move think-time machinery into thinker_state
io_tester: Move _file to io_class_data
io_tester: Replace class_data::_start member with a local variable
io_tester: Move _alignment from class_data to io_class_data
io_tester: Remove buffer allocation from top-level request issuing
io_tester: Cleanup context::stop() invocation
io_tester: Allocate write buffer once to fill a file
io_tester: Declare quantiles arrays as static constexpr
io_tester: Drop class_data::type_str()
io_tester: Replace != "" comparisons with .empty()
io_tester: Replace gen_class_data() if/else chain with a switch
io_tester: Deduplicate vectorized I/O classes
> io_tester: fix crash from missing metric during startup
> net: tls: adjust openssl integration to new module support
> http/client: Count and export integrated queue length
> Merge 'Introduce pipe_data_source_impl and pipe_data_sink_impl' from Pavel Emelyanov
fstream: add pipe_data_source_impl and pipe_data_sink_impl
pollable_fd: add write_some/write_all backed by writev
pollable_fd: rename write_some/write_all(iovec) to send_some/send_all
> reactor: Make pollable_fd_state helper methods private
> module: extend seastar.cppm with comprehensive public API exports
> Merge 'Add exhaustive input_stream invariant test + fixes' from Pavel Emelyanov
tests: add exhaustive input_stream read/skip invariant test
iostream: make skip() reject premature end of stream with exception
> Merge 'Allow runtime selectability of GnuTLS or OpenSSL' from Noah Watkins
net/tls: avoid potential read-past-buffer
net/tls: move credential methods to generic tls layer
net/tls: rename credentials_impl::dh_params to set_dh_params
test/tls: enable openssl tls unit test
test/tls: fix CA cert generation to use v3_ca extensions
github: disable parallel test execution in alpine workflow
crypto: support compiling seastar without gnutls
net/tcp: use crypto provider for md5 calculation
tls: fix test_peer_certificate_chain_handling for OpenSSL
net/tls: fix test for self-signed server cert opoenssl compat
net/tls: disable priority strings test for openssl provider
core/crypto: expose crypto backend name for introspection
test/tls: remove gnutls version guard
net/tls: add openssl tls backend
http: use backend agnostic tls error code
net/tls: make error codes configurable by each tls backend
net/tls: move reloadable_credentials to generic tls layer
net/tls: move build_certificate to generic tls layer
net/tls: move apply_to() to generic tls layer
net/tls: move credential methods to generic tls layer
net/tls: add OpenSSL-specific methods to public API with no-op defaults
net/tls: introduce dh_params and credentials abstraction layer
net/tls: add credentials_impl abstract base class
net/tls: dispatch tls::error_category() through crypto_provider
net/tls: dispatch wrap_client/wrap_server through crypto_provider
net/tls: add tls_backend interface to crypto_provider
net/tls: move public tls API methods to generic tls layer
net/tls: move formatting utilities to generic tls layer
net/tls: move credentials_builder blob methods to generic tls layer
net/tls: move dh_params::from_file to generic tls layer
net/tls: move abstract_credentials file methods to generic tls layer
net/tls: move tls_socket_impl to generic tls layer
net/tls: move server_session to general tls layer
net/tls: move tls_connected_socket_impl to generic tls layer
net/tls: move net::get_impl to generic tls layer
net/tls: move session_ref to generic tls layer
net/tls: add session_impl abstract interface for tls pluggability
net/tls: rename tls.cc to be gnutls specific
crypto: introduce crypto provider abstraction
http: remove unused include
> tls: test_send_two_large
> rpc: include exception type for remote errors
> GHA: increase timeout to 60 minutes
> apps/httpd: replace deprecated reply::done() with write_body()
> missing header(s)
> net: Fix missing throw for runtime_error in create_native_net_device
> tests/io_queue: account for token bucket refill granularity in bandwidth checks
> Merge 'iovec: fix iovec_trim_front infinite loop on zero-length iovecs' from Travis Downs
tests: add regression tests for zero-length iovec handling
iovec: fix iovec_trim_front infinite loop on zero-length iovecs
> util/process: graduate process management API from experimental
> cooking: don't register ready.txt as a build output
> sstring: make make_sstring not static
> Add SparkyLinux to debian list in install-dependencies.sh
> http: allow control over default response headers
> Merge 'chunked_fifo: make cached chunk retention configurable' from Brandon Allard
tests/perf: add chunked_fifo microbenchmarks
chunked_fifo: set the default free chunk retention to 0
chunked_fifo: make free chunk retention configurable
> Merge 'reactor_backend: fix pollable_fd_state_completion reuse in io_uring' from Kefu Chai
tests: add regression test for pollable_fd_state_completion reuse
reactor_backend: use reset() in AIO and epoll poll paths
reactor_backend: fix pollable_fd_state_completion reuse after co_await in io_uring
> Merge 'coroutine: Generator cleanups' from Kefu Chai
coroutine/generator: extract schedule_or_resume helper
coroutine/generator: remove unused next_awaiter classes
coroutine/generator: remove write-only _started field
coroutine/generator: assert on unreachable path in buffered await_resume
coroutine/generator: add elements_of tag and #include <ranges>
coroutine/generator: add empty() to bounded_container concept
> cmake: bump minimum Boost version to 1.79.0
> seastar_test: remove unnecessary headers
> cmake: bump minimum GnuTLS version to 3.7.4
> Merge 'reactor: add get_all_io_queues() method' from Travis Downs
tests: add unit test for reactor::get_all_io_queues()
reactor: add get_all_io_queues() method
reactor: move get_io_queue and try_get_io_queue to .cc file
> http: deprecate reply::done(), remove _response_line dead field
> core: Deprecate scattered_message
> ci: add workflow dispatch to tests workflow
> perf_tests: exit non-zero when -t pattern matches no tests
> Replace duplicate SEGV_MAPERR check in sigsegv_action() with SEGV_ACCERR.
> perf_tests: add total runtime to json output
> Merge 'Relax large allocation error originating from json_list_template' from Robert Bindar
implement move assignment operator for json_list_template
json_list_template copy assignment operator reserves capacity upfront
> perf_tests: add --no-perf-counters option
> Merge 'Fix to_human_readable_value() ability to work with large values' from Pavel Emelyanov
memory: Add compile-time test for value-to-human-readable conversion
memory: Extend list of suffixes to have peta-s
memory: Fix off-by-one in suffix calculation
memory: Mark to_human_readable_value() and others constexpr
> http: Improve writing of response_line() into the output
> Merge 'websocket: add template parameter for text/binary frame mode and implement client-side WebSocket' from wangyuwei
websocket: add template parameter for text/binary frame mode
websocket: impl client side websocket function
> file: Fix checks for file being read-only
> reactor: Make do_dump_task_queue a task_queue method
> Merge 'Implement fully mixed mode for output_stream-s' from Pavel Emelyanov
tests/output_stream: sample type patterns in sanitizer builds
tests/output_stream: extend invariant test to cover mixed write modes
iostream: allow unrestricted mixing of buffered and zero-copy writes
tests/output_stream: remove obsolete ad-hoc splitting tests
tests/output_stream: add invariant-based splitting tests
iostream: rename output_stream::_size to ::_buffer_size
> reactor_backend: replace virtual bool methods with const bool_class members
> resource: Avoid copying CPU vector to break it into groups
> perf_tests: increase overhead column precision to 3 decimal places
> Merge 'Move reactor::fdatasync() into posix_file_impl' from Pavel Emelyanov
reactor: Deprecate fdatasync() method
file: Do fdatasync() right in the posix_file_impl::flush()
file: Propagate aio_fdatasync to posix_file_impl
reactor: Move reactor::fdatasync() code to file.cc
reactor,file: Make full use of file_open_options::durable bit
file: Add file_open_options::durable boolean
file: Account io_stats::fsyncs in posix_file_impl::flush()
reactor: Move _fsyncs counter onto io_stats
> http: Remove connection::write_body()
Closesscylladb/scylladb#29553
Move the download_sstable and get_sstables_for_tablet static functions
from sstables_loader into a new file to make them reusable by the
tablet-aware restore code.
The supergroup replaces streaming (a.k.a. maintenance as well) group, inherits 200 shares from it and consists of four sub-groups (all have equal shares of 200 withing the new supergroup)
* maintenance_compaction. This group configures `compaction_manager::maintenance_sg()` group. User-triggered compaction runs in it
* backup. This group configures `snapshot_ctl::config::backup_sched_group`. Native backup activity runs there
* maintenance. It's a new "visible" name, everything that was called "maintenance" in the code ran in "streaming" group. Now it will run in "maintenance". The activities include those that don't communicate over RPC (see below why)
* `tablet_allocator::balance_tablets()`
* `sstables_manager::components_reclaim_reload_fiber()`
* `tablet_storage_group_manager::merge_completion_fiber()`
* metrics exporting http server altogether
* streaming. This is purely existing streaming group that just moves under the new supergroup. Everything else that was run there, continues doing so, including
* hints sender
* all view building related components (update generator, builder, workers)
* repair
* stream_manager
* messaging service (except for verb handlers that switch groups)
* join_cluster() activity
* REST API
* ... something else I forgot
The `--maintenance_io_throughput_mb_per_sec` option is introduced. It controls the IO throughput limit applied to the maintenance supergroup. If not set, the `--stream_io_throughput_mb_per_sec` option is used to preserve backward compatibility.
All new sched groups inherit `request_class::maintenance` (however, "backup" seem not to make any requests yet).
Moving more activities from "streaming" into "maintenance" (or its own group) is possible, but one will need to take care of RPC group switching. The thing is that when a client makes an RPC call, the server may switch to one of pre-negotiated scheduling groups. Verbs for existing activities that run in "streaming" group are routed through RPC index that negotiates "streaming" group on the server side. If any of that client code moves to some other group, server will still run the handlers in "streaming" which is not quite expected. That's one of the main reasons why only the selected fibers were moved to their own "maintenance" group. Similar for backup -- this code doesn't use RPC, so it can be moved. Restoring code uses load-and-stream and corresponding RPCs, so it cannot be just moved into its own new group.
Fixes SCYLLADB-351
New feature, not backporting
Closesscylladb/scylladb#28542
* github.com:scylladb/scylladb:
code: Add maintenance/maintenance group
backup: Add maintenance/backup group
compaction: Add maintenance/maintenance_compaction group
main: Introduce maintenance supergroup
main: Move all maintenance sched group into streaming one
database: Use local variable for current_scheduling_group
code: Live-update IO throughputs from main
Match configure.py's Boost handling:
- Add BOOST_ALL_DYN_LINK when using shared Boost libraries.
- Strip per-component defines (BOOST_UNIT_TEST_FRAMEWORK_DYN_LINK,
BOOST_REGEX_DYN_LINK, etc.) that CMake's Boost package config
adds on imported targets. configure.py only uses the umbrella
BOOST_ALL_DYN_LINK define.
Place add_compile_definitions(SEASTAR_TESTING_MAIN) after both
add_subdirectory(seastar) and add_subdirectory(abseil) are processed.
This matches configure.py's global define without leaking into
seastar's subdirectory build (which would cause a duplicate main
symbol in seastar_testing).
Remove the now-redundant per-test SEASTAR_TESTING_MAIN compile
definition from test/CMakeLists.txt.
Match configure.py line 2192: abseil gets sanitizer flags with
-fno-sanitize=vptr to exclude vptr checks which are incompatible
with abseil's usage of type-punning patterns.
- Set BUILD_SHARED_LIBS based on build type to match configure.py's
build_seastar_shared_libs: Debug and Dev build Seastar as a shared
library, all other modes build it static.
- Add sanitizer link options on the seastar target for Coverage
mode. Seastar's CMake only activates sanitizer targets for
Debug/Sanitize configs, but Coverage mode needs them too since
configure.py's seastar_libs_coverage carries -fsanitize flags.
- Disable CMake's automatic -fcolor-diagnostics injection for
Clang+Ninja (CMake 3.24+), matching configure.py which does not
add any color diagnostics flags.
- Add SEASTAR_NO_EXCEPTION_HACK and XXH_PRIVATE_API as global
defines (previously SEASTAR_NO_EXCEPTION_HACK was only on the
seastar target as PRIVATE; it needs to be project-wide).
- Add -fpch-validate-input-files-content to check precompiled
header content when timestamps don't match.
And move some activities from streaming group into it, namely
- tablet_allocator background group
- sstables_manager-s components reclaimer
- tablet storage group manager merge completion fiber
- prometheus
All other activity that was in streaming group remains there, but can be
moved to this group (or to new maintenance subgroup) later.
All but prometheus are patched here, prometheus still uses the
maintenance_sched_group variable in main.cc, so it transparently
moves into new group
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The snapshot_ctl::backup_task_impl runs in configured scheduling group.
Now it's streaming one. This patch introduces the maintenance/backup
group and re-configures backup task with it.
The group gets its --backup_io_throughput_mb_per_sec option that
controls bandwidth limit for this sub-group only.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Compaction manager tells compaction_sched_group from
maintenance_compaction_sched_group. The latter, however, is set to be
"streaming" group. This patch adds real maintenance_compaction group
under the maintenance supergroup and makes compaction manager use it.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Add precompiled header support to CMakeLists.txt and configure.py -
it improves compilation time by approximately 10%.
New header `stdafx.hh` is added, don't include it manually -
the compiler will include it for you. The header contains includes from
external libraries used by Scylla - seastar, standard library,
linux headers and zlib.
The feature is enabled by default, use CMake option `Scylla_USE_PRECOMPILED_HEADER`
or configure.py --disable-precompiled-header to disable.
The feature should be disabled, when trying to check headers - otherwise
you might get false negatives on missing includes from seastar / abseil and so on.
Note: following configuration needs to be added to ccache.conf:
sloppiness = pch_defines,time_macros,include_file_mtime,include_file_ctime
Closesscylladb/scylladb#26617
On clang 21.1.4 (Fedora 43) the abseil compilation started to fail with `builtin XXX is deprecated use YYY instead`. Suppress this for abseil compilation only
Closesscylladb/scylladb#27098
This commit:
- Increases the number of allowed scheduling groups to allow the
creation of `sl:driver`.
- Adds the `DRIVER_SERVICE_LEVEL` feature, which prevents creating
`sl:driver` until all nodes have increased the number of
scheduling groups.
- Starts using `get_create_driver_service_level_mutations`
to unconditionally create `sl:driver` on
`raft_initialize_discovery_leader`. The purpose of this code
path is ensuring existence of `sl:driver` in new system and tests.
- Starts using `migrate_to_driver_service_level` to create `sl:driver`
if it is not already present. The creation of `sl:driver` is
managed by `topology_coordinator`, similar to other system keyspace
updates, such as the `view_builder` migration. The purpose of this
code path is handling upgrades.
- Modifies related tests to pass after `sl:driver` is added.
Later in this patch series, `sl:driver` will be used by
`transport/server` to handle selected traffic, such as the driver's
schema and topology fetches.
Refs: scylladb/scylladb#24411
The PR #26154 dropped the `-fvisibility=hidden` compiler flag and
replaced it with `-fvisibility-inlines-hidden` as the former caused
issues in how the `noncopyable_function::operator bool` method executed
leading to incorrect return values. Apply the same fix to cmake.
Fixes#26391Closesscylladb/scylladb#26431
The querier object is a confusing one. Based on its name it should be in the query/ module and it is already in the query namespace. The query namespace is used for symbols which span the coordinator and replica, or that are mostly coordinator side. The querier is mainly in this namespace due to its similar name and because at the time it was introduced, namespace replica didn't exist yet. But this is a mistake which confuses people.
The querier is actually a completely replica-side logic, implementing the caching of the readers on the replica. Move it to the replica module and namespace to make this more clear.
Code cleanup, no backport.
Closesscylladb/scylladb#26280
* github.com:scylladb/scylladb:
replica: move querier code to replica namespace
root,replica: mv querier to replica/
We would like to have an additional service level
available for users of the Vector Store service,
which would allow us to de/prioritize vector
operations as needed. To allow that, we increase
the number of scheduling groups from 19 to 20
and adjust the related test accordingly.
Closesscylladb/scylladb#26316
Moved files:
- generic_server.hh
- generic_server.cc
- protocol_server.hh
Fixes: #22112
This is a cleanup, no need to backport
Closesscylladb/scylladb#25090
Seastar API level 8 changes a function type from std::function to
noncopyable_function. Apply those changes in tree and update the build
configuration.
Closesscylladb/scylladb#26006
The querier object is a confusing one. Based on its name it should be in
the query/ module and it is already in the query namespace. But this is
actually a completely replica-side logic, implementing the caching of
the readers on the replica. Move it to the replica module to make this
more clear.
It belongs there, it is a completely replica-side thing. Also take the
opportunity to rename it to multishard_query.{hh,cc}, it is not just
mutation anymore (data query is also implemented).
As requested in #22104, moved the files and fixed other includes and build system.
Moved files:
- combine.hh
- collection_mutation.hh
- collection_mutation.cc
- converting_mutation_partition_applier.hh
- converting_mutation_partition_applier.cc
- counters.hh
- counters.cc
- timestamp.hh
Fixes: #22104
This is a cleanup, no need to backport
Closesscylladb/scylladb#25085
Vector search related implementation moved to a new module vector_search.
As the vector search functionality is going to be extended, it is better to keep it in a separate module.
The DNS resolution logic and its background task are moved out of the `vector_store_client` and into a new, dedicated class `vector_search::dns`.
This refactoring is the first step towards supporting DNS hostnames that resolve to multiple IP addresses.
References: VECTOR-187
No backport needed as this is refactoring.
Closesscylladb/scylladb#26052
* github.com:scylladb/scylladb:
vector_store_client_test: Verify DNS is not refreshed when disabled
vector_store_client: Extract DNS logic into a dedicated class
vector_search: Apply clang-format
vector_store_client: Move to vector_search module
Vector search related implementation moved to a new module vector_search.
As the vector search functionality is going to be extended, it is
better to keep it in a separate module.
This commit:
- Increases the number of allowed scheduling groups to allow the
creation of `sl:driver`.
- Adds the `DRIVER_SERVICE_LEVEL` feature, which prevents creating
`sl:driver` until all nodes have increased the number of
scheduling groups.
- Starts using `get_create_driver_service_level_mutations`
to unconditionally create `sl:driver` on
`raft_initialize_discovery_leader`. The purpose of this code
path is ensuring existence of `sl:driver` in new system and tests.
- Starts using `migrate_to_driver_service_level` to create `sl:driver`
if it is not already present. The creation of `sl:driver` is
managed by `topology_coordinator`, similar to other system keyspace
updates, such as the `view_builder` migration. The purpose of this
code path is handling upgrades.
- Modifies related tests to pass after `sl:driver` is added.
Later in this patch series, `sl:driver` will be used by
`transport/server` to handle selected traffic, such as the driver's
schema and topology fetches.
Refs: scylladb/scylladb#24411
As requested in #22120, moved the files and fixed other includes and build system.
Moved files:
- query.cc
- query-request.hh
- query-result.hh
- query-result-reader.hh
- query-result-set.cc
- query-result-set.hh
- query-result-writer.hh
- query_id.hh
- query_result_merger.hh
Fixes: #22120
This is a cleanup, no need to backport
Closesscylladb/scylladb#25105
Add precompiled header support to CMakeLists.txt and configure.py -
it improves compilation time by approximately 10%.
New header `stdafx.hh` is added, don't include it manually -
the compiler will include it for you. The header contains includes from
external libraries used by Scylla - seastar, standard library,
linux headers and zlib.
The feature is enabled by default, use CMake option `Scylla_USE_PRECOMPILED_HEADER`
or configure.py --disable-precompiled-header to disable.
The feature should be disabled, when trying to check headers - otherwise
you might get false negatives on missing includes from seastar / abseil and so on.
Note: following configuration needs to be added to ccache.conf:
sloppiness = pch_defines,time_macros
Closes#25182
Fixes#22106
Moves the shared compress components to sstables, and rename to
match class type.
Adjust includes, removing redundant/unneeded ones where possible.
Closesscylladb/scylladb#25103
As requested in #22102, #22103 and #22105 moved the files and fixed other includes and build system.
Moved files:
- clustering_bounds_comparator.hh
- keys.cc
- keys.hh
- clustering_interval_set.hh
- clustering_key_filter.hh
- clustering_ranges_walker.hh
- compound_compat.hh
- compound.hh
- full_position.hh
Fixes: #22102Fixes: #22103Fixes: #22105Closesscylladb/scylladb#25082
We use patchelf to rewrite the dynamic loader (known as the interpreter)
of the binaries we ship, so we can point to our shipped dynamic loader,
which is compatible with our binaries, rather than rely on the distribution's
dynamic loader, which is likely to be incompatible.
Upstream patchelf losing compatibity [1] with Linux 5.17 and below.
This change was also picked up by Fedora 42, so we cannot update the
toolchain to that distribution until we have an alternative.
Here we add a minimal patchelf alternative. It was mostly written by
Claude. It is minimal in that it only supports --set-interpreter and
--print-interpreter, and works well enough for our needs. We still use
the original patchelf for --remove-rpath; this reduces our maintenance
needs.
[1] 43b75fbc9f
[2] 4b015255d1Closesscylladb/scylladb#24695
Add a fiber responsible for periodic re-training of compression dictionaries
(for tables which opted into dict-aware compression).
As of this patch, it works like this:
every `$tick_period` (15 minutes), if we are the current Raft leader,
we check for dict-aware tables which have no dict, or a dict older
than `$retrain_period`.
For those tables, if they have enough data (>1GiB) for a training,
we train a new dict and check if it's significantly better
than the current one (provides ratio smaller than 95% of current ratio),
and if so, we update the dict.
Unlike all other implementations of `compressor`, `zstd_processor`
has its own special object file and its own special
late binding mechanism (via the `class_registry`).
It doesn't need either.
Let's squash it into `compress.cc`. Keeping `zstd_processor` a separate "module"
would require adding even more headers and source files later in the
series (when adding dictionaries), and there's no benefit in being
so granular. All `compressor` logic can be in `compress.cc` and it will
still be small enough.
This commit also gets rid of the pointless `class_registry` late binding
mechanism and just constructs the `zstd_processor` in
`compressor::create()` with a regular constructor call.
ExternalProject automatically creates BINARY_DIR for Seastar, but generator
expressions are not supported in this setting. This caused CMake to create
an unused "build/$<CONFIG>/seastar" directory.
Instead, define a dedicated variable matching configure.py's naming and use
it in supported options like BUILD_COMMAND. This:
- Creates build files in the standard "Seastar-prefix/src/Seastar-build"
directory instead of "build/$<CONFIG>/seastar". see
https://cmake.org/cmake/help/latest/module/ExternalProject.html#directory-options
- Makes it clearer that the variable should match configure.py settings
No functional changes to the Seastar build process - purely a cleanup to
reduce confusion when inspecting the build directory.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#22437
This change introduces a new audit subsystem that allows tracking and logging of database operations for security and compliance purposes. Key features include:
- Configurable audit logging to either syslog or a dedicated system table (audit.audit_log)
- Selective auditing based on:
- Operation categories (QUERY, DML, DDL, DCL, AUTH, ADMIN)
- Specific keyspaces
- Specific tables
- New configuration options:
- audit: Controls audit destination (none/syslog/table)
- audit_categories: Comma-separated list of operation categories to audit
- audit_tables: Specific tables to audit
- audit_keyspaces: Specific keyspaces to audit
- audit_unix_socket_path: Path for syslog socket
- audit_syslog_write_buffer_size: Buffer size for syslog writes
The audit logs capture details including:
- Operation timestamp
- Node and client IP addresses
- Operation category and query
- Username
- Success/failure status
- Affected keyspace and table names
Fixes https://github.com/scylladb/scylla-enterprise/issues/5016#issuecomment-2558464631
EAR - encryption at rest. Allows on-disk file encryption of sstables and commitlog data.
Introduces OpenSSL based file level encrypted storage, managed via a set of providers
ranging from local files to cloud KMS providers.
For a more comprehensive explanation, see the included docs (or if possible, original
source tree).
Manual bulk merge of EAR feature from enterprise repo to main scylla repo.
Breaks some features apart, but main EAR is still a humongous commit, because to separate this
I would have to mess with code incrementally, adding time and risk.
This PR includes the local file gen tool, tests and also p11 validation.
Note: CI will not execute the full tests unless master CI is set to provide the same environment
as the enterprise one. Not sure about the status of this ATM.
Note: Includes code to compile against cryptsoft kmipc SDK, but not the SDK. If you happen to
check out this tree in the scylla folder and configure, it will be linked against and KMIP functionality
will be enabled, otherwise not.
Closesscylladb/scylladb#22233
* github.com:scylladb/scylladb:
docs: Add EAR docs
main/build: Add p11-kit and initialize
tools: Add local-file-key-generator tool
tests: Add EAR tests
tmpdir: shorten test tempdir path
EAR: port the ear feature from enterprise
cql_test_env: Add optional query timeout
schema/migration_manager: Add schema validate
sstables: add get_shared_components accessor
config/config_file: Add exports and definitions of config_type_for<>
This PR extends authentication with 2 mechanisms:
- a new role_manager subclass, which allows managing users via
LDAP server,
- a new authenticator, which delegates plaintext authentication
to a running saslauthd daemon.
The features have been ported from the enterprise repository
with their test.py tests and the documentation as part of
changing license to source available.
Fixes: scylladb/scylla-enterprise#5000Fixes: scylladb/scylla-enterprise#5001Closesscylladb/scylladb#22030
Bulk transfer of EAR functionality. Includes all providers etc.
Could maybe break up into smaller blocks, but once it gets down to
the core of it, would require messing with code instead of just moving.
So this is it.
Note: KMIP support is disabled unless you happen to have the kmipc
SDK in your scylla dir.
Adds optional encryption of sstables and commitlog, using block
level file encryption. Provides key sourcing from various sources,
such as local files or popular KMS systems.
Replace the reader concurrency semaphores for user reads and view
updates with the newly introduced reader concurrency semaphore group,
which assigns a semaphore for each service level.
Each group is statically assigned to some pool of memory on startup and
dynamically distribute this memory between the semaphores, relative to
the number of shares of the corresponding scheduling group.
The intent of having a separate reader concurrency semaphore for each
scheduling group is to prevent priority inversion issues due to reads
with different priorities waiting on the same semaphore, as well as make
memory allocation more fair between service levels due to the adjusted
number of shares.
