We adjust the documentation to include the new
VECTOR_SEARCH_INDEXING permission and its usage
and also to reflect the changes in the maximal
amount of service levels.
We were recently surprised (in pull request #25797) to "discover" that
Scylla does not allow granting SELECT permissions on individual
materialized views. Instead, all materialized views of a base table
are readable if the base table is readable.
In this patch we document this fact, and also add a test to verify
that it is indeed true. As usual for cqlpy tests, this test can also
be run on Cassandra - and it passes showing that Cassandra also
implemented it the same way (which isn't surprising, given that we
probably copied our initial implementation from them).
The test demonstrates that neither Scylla nor Cassandra prints an error
when attempting to GRANT permissions on a specific materialized view -
but this GRANT is simply ignored. This is not ideal, but it is the
existing behavior in both and it's not important now to change it.
Additionally, because pull request #25797 made CDC-log permissions behave
the same as materialized views - i.e., you need to make the base table
readable to allow reading from the CDC log, this patch also documents
this fact and adds a test for it also.
Fixes#25800Closesscylladb/scylladb#25827
Add a new section for key rotation, offering separate instructions per
key provider, organized in tabs.
The gist:
* Local Key Provider - Rotation requires creating a new key file per
node. It's a manual procedure.
* Replicated Key Provider - Rotation is not supported.
* KMIP Key Provider - Rotation is transparent to Scylla, but it requires
manually revoking the key in the server.
* {KMS,GCP} Key Provider - Rotation is transparent to Scylla and can be
automated in the server.
* Azure Key Provider - Rotation is automatically supported by Scylla by
keeping track of the key version along with the encrypted data. The
rotation needs to be done at the Key Vault server, and can be
automated.
Explain that, even after rotation, old keys may be still in use due to
caching, and that old SSTables will remain encrypted with the old key
until the next compaction. Provide instructions in case they prefer not
to wait.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
- Mention all types of system data that fall under system encryption.
- Add "Before you Begin" section with requirements per key provider.
The requirements are the same as in user encryption.
- Mention explicitly that the Replicated Key Provider cannot be used for
system encryption.
- Provide separate instructions for each key provider. Explain all the
configuration options.
- Provide an extra example for the Local Key Provider with a
``system_key_directory`` and ``key_name``.
- Highlight the code blocks as YAML. Make their indentation consistent
with the rest of the doc (2 spaces).
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
- Split the various scenarios into sub-sections, not just examples.
- Amend the example for changing cipher algorithm and key length. The
algorithm used in the example was the same.
- Point out that disabling encryption through the table schema is not
possible if a node has default encryption configured.
- Amend the `nodetool upgradesstables` command. The
`--include-all-sstables` is necessary.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
- Add a short intro.
- Add an early note about the fact that options from
``scylla_encryption_options`` cannot be mixed with options from
``user_info_encryption``.
- Add a new "Allow Per-Table Encryption" subsection to document the
``allow_per_table_encryption`` option.
- Move the top-level procedure into a new "Encrypt a New Table"
subsection to differentiate it from the "Update Encryption Properties
of Existing Tables"".
- Add tabs for provider-dependent steps in "Before you Begin" and
"Procedure".
- Amend "bytes" to "bits" (for the key length).
- Add examples for the replicated, KMIP, GCP, and Azure key providers.
Use consistent keyspace and table names in all examples.
- Remove step for upgrading SSTables. The table is new - no SSTables
exist yet.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
- Provide separate requirements and instructions for each key provider,
organized in tabs.
- Mention explicitly that the Replicated Key Provider cannot be used for
default encryption.
- Fix indentation for code blocks in examples (2 spaces).
- For KMS, GCP, and Azure, add the `master_key` option in the list of
options and remove the relevant example (not so common).
- Add steps for rolling restart.
- Amend "bytes" to "bits" (for the key length).
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
- Mark the `master_key` as required. Technically, it's not, since it can
be specified in the schema encryption options, but:
- It's better to keep it simple. The common case is to have a default
value that occasionally needs to be overridden.
- No functionality is lost.
- It is mentioned as required for AWS and GCP.
- Add a note about credential resolution.
- Make some minor formatting changes to be consistent with the AWS and
GCP sections.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
- Add list of requirements (KMS Key, credentials, permissions).
- Add a reference to "Create Encryption Keys" section.
- Amend description for `master_key`.
- Add one example per credential type.
- Explain how credentials are resolved if not explicitly specified in
the configuration.
- Fix indentation of "restart" command.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
- Add a list of requirements (KMS key, credentials, permissions).
- Add a reference to "Create Encryption Keys" section.
- Add one example per credential type.
- Explain how credentials are resolved from the environment, or the
AWS credentials file.
- Fix indentation of "restart" command.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
- Uncomment the code block to match the other hosts.
- Remove the ``certficate_revocation_list`` option; it's not supported.
- Amend the default values for ``key_cache_expiry`` and
``key_cache_refresh``.
- Add an example with mutual TLS authentication.
- Fix indentation of "restart" command.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
- Provide separate instructions for each key provider, organized in tabs.
Move the existing instructions with the key generator script under the
"Local Key Provider" tab. Point to the cloud provider's documentation
for AWS, GCP, and Azure keys. List the required attributes for KMIP
keys. List the required keys for the Replicated Key Provider.
- In the example for the key generator script, use the same algorithm
and key strength for both the secret key and the system key, since
this is the recommended case.
- Reorder the usage list of arguments for the key generator script.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
- Use monospace font for key provider factories.
- Add a sub-section for every key provider. Explain how they operate at
a high level and highlight any possible limitations.
- Remove version availability notes. The version 2019.1.3 is old and
unsupported.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
Turn an earlier reference to "algorithm descriptor" into a hyperlink.
Use monospace font in the table header for "cipher_algorithm" and
"secret_key_strength"; these are verbatim identifiers in "scylla.yaml"
and "scylla_encryption_options". Same for their supported values.
Restrict the Blowfish key size to 128 bits, due to
<https://github.com/scylladb/scylla-enterprise/issues/4848>.
Add notes on ECB vs. CBC, and on Blowfish's 64-bit block size. Emphasize
our recommendation more.
Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
- Referring to system info encryption vs. user info encryption as distinct
"encryption key types" is confusing. The behavior of encryption is
similar in both cases, only the sets of data that are subject to
encryption differ. Rename the section to "Data Classes for
Encryption".
- Introduce the two highest-level "scylla.yaml" stanzas,
"system_info_encryption" and "user_info_encryption". Subsequently, we'll
expand on their (common!) contents later.
- Remove the comment that, for the Local Key Provider, a keystore can be
created either manually or automatically. This is stated / repeated
elsewhere in the document.
- Remove the unused anchor "_Replicated".
- The notes on the Replicated Key Provider both lack nuance, and are
ill-placed, here. Remove those notes. Add a dedicated description for
Replicated later, elsewhere. Do mention
"system_replicated_keys.encrypted_keys" here in passing, as a system
table with sensitive contents.
- The short listing of key providers is ill-placed here. We have an entire
section dedicated to those. Furthermore, the various key providers apply
to system info encryption, too.
- Explain the two levels of configuration for SSTables of user tables.
- Move the note about preserving keys for restoring backups to Key
Providers | About Local Key Storage, at least temporarily. When keys are
stored on a key management server (KMIP, GCP, AWS, Azure), then
backing those up is its own admin task / responsibility.
Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
Clarify some table vs. SSTable differences.
Spell out the SSTable metadata ("Scylla.db") component. Spell out commit
log metadata files. Explain that encryption settings are "snapshotted"
into those meta-files.
Highlight that encryption config may vary per table *and* per node. (For
example, a local file key provider under the same pathname on each node,
referenced by the table's "scylla_encryption_options" in the schema, may
provide different keys for different nodes.)
Introduce "algorithm descriptor" and "key provider" as generic concepts.
Touch up the grammar / vocabulary slightly.
Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
- Remove the KMIP password from the list of system level data.
Encrypting this would require the `configuration_encryptor`, which has
been removed as part of the effort to decommission all our java tools.
- Provide an exhaustive list of system tables being encrypted.
- "Table level granularity" is redundant; either "table level" or "table
granularity" should suffice. Pick the latter.
- Distinguish "block cipher" from "mode of operation" more precisely.
Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
Wrap long lines at 80 chars. Seastar coding style suggests 160 chars,
but 80 chars is more comfortable for side-by-side PR diffs on GitHub.
Exclude arg lists and code blocks. Set the limit at 160 chars for arg
lists to avoid too much wrapping that would hurt readability. Do not
wrap code blocks at all.
Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
Extend the EaR ops guide to incorporate the new Azure Key Provider.
Document its options and provide instructions on how to configure it.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
This PR extends the KMS host to support temporary AWS security credentials provided externally via the Scylla configuration file, environment variables, or the AWS credentials file.
The KMS host already supports:
* Temporary credentials obtained automatically from the EC2 instance metadata service or via IAM role assumption.
* Long-term credentials provided externally via configuration, environment, or the AWS credentials file.
This PR is about temporary credentials that are external, i.e., not generated by Scylla. Such credentials may be issued, for example, through identity federation (e.g., Okta + gimme-aws-creds).
External temporary credentials are useful for short-lived tasks like local development, debugging corrupted SSTables with `scylla-sstable`, or other local testing scenarios. These credentials are temporary and cannot be refreshed automatically, so this method is not intended for production use.
Documentation has been updated to mention these additional credential sources.
Fixes#22470.
New feature, no backport is needed.
Closesscylladb/scylladb#22465
* github.com:scylladb/scylladb:
doc: Expose new `aws_session_token` option for KMS hosts
kms_host: Support authn with temporary security credentials
encryption_config: Mention environment in credential sources for KMS
Several audit test issues caused test failures, and in the result, almost all of audit syslog tests were marked with xfail.
This patch series enables the syslog audit tests, that should finally pass after the following fixes are introduced:
- bring back commas to audit syslog (scylladb#24410 fix)
- synchronize audit syslog server
- fix parsing of syslog messages
- generate unique uuid for each line in syslog audit
- allow audit logging from multiple nodes
Fixes: scylladb/scylladb#24410
Test improvements, no backport required.
Closesscylladb/scylladb#24553
* github.com:scylladb/scylladb:
test: audit: use automatic comparators in AuditEntry
test: audit: enable syslog audit tests
test: audit: sort new audit entries before comparing with expected ones
test: audit: check audit logging from multiple nodes
test: audit: generate unique uuid for each line in syslog audit
test: audit: fix parsing of syslog messages
test: audit: synchronize audit syslog server
docs: audit: update syslog audit format to the current one
audit: bring back commas to audit syslog
This commit migrates the Software Bill Of Materials (SBOM) page
added to the Enterprise docs with https://github.com/scylladb/scylla-enterprise/pull/5067.
The only difference is the link to the SBOM files - it was Enterprise SBOM in the Enterprise docs,
while here is a link to the ScyllaDB SBOM.
It's a follow-up of migration to Source Avalable and should be backported
to all Source Available versions - 2025.1 and later.
Fixes https://github.com/scylladb/scylladb/issues/24730Closesscylladb/scylladb#24735
The documentation of the syslog audit format was not updated when
scylladb#23099 and earlier audit log changes were introduced.
This commit includes the missing update.
This commit removes the information about FIPS out of the '.. only:: enterprise' directive.
As a result, the information will now show in the doc in the ScyllaDB repo
(previously, the directive included the note in the Entrprise docs only).
Refs https://github.com/scylladb/scylla-enterprise/issues/5020Closesscylladb/scylladb#22374
Fixes#21993
Removes configuration_encryptor mention from docs.
The tool itself (java) is not included in the main branch
java tools, thus need not remove from there. Only the words.
Closesscylladb/scylladb#22427
Adds detailed documentation covering the new audit subsystem:
- Add new audit.md design document explaining:
- Core concepts and design decisions
- CQL extensions for audit management
- Implementation details and trigger evaluation
- Prior art references from other databases
- Add user-facing documentation:
- New auditing.rst guide with configuration and usage details
- Integration with security documentation index
- Updates to cluster management procedures
- Updates to security checklist
The documentation covers all aspects of the audit system including:
- Configuration options and storage backends (syslog/table)
- Audit categories (DCL/DDL/AUTH/DML/QUERY/ADMIN)
- Permission model and security considerations
- Failure handling and logging
- Example configurations and output formats
This ensures users have complete guidance for setting up and using
the new audit capabilities.
As part of #18750, we added a CQL statement CREATE ROLE WITH SALTED HASH that prevented hashing a password when creating a role, effectively leading to inserting a hash given by the user directly into the database. In #21350, we noticed that Cassandra had implemented a CQL statement of similar semantics but different syntax. We decided to rename Scylla's statement to be compatible with Cassandra. Unfortunately, we didn't notice one more difference between what we had in Scylla and what was part of Cassandra.
Scylla's statement was originally supposed to only be used when restoring the schema and the user needn't have to be aware of its existence at all: the database produced a sequence of CQL statements that the user saved to a file and when a need to restore the schema arose, they would execute the contents of the file. That's why that although we documented the feature, it was only done in the necessary places. Those that weren't related to the backup & restore procedure were deliberately skipped.
Cassandra, on the other hand, added the statement for a different purpose (for details, see the relevant issue) and it was supposed to be used by the user by design. The statement is also documented as such.
Since we want to preserve compatibility with Cassandra, we document the statement and its semantics in the user documentation, explicitly implying that it can be used by the user.
We also add a test verifying that logging in works correctly.
Fixesscylladb/scylladb#21691
Backport: not needed. The relevant code didn't make it to 6.2 or any previous version of OSS.
Closesscylladb/scylladb#21752
* github.com:scylladb/scylladb:
docs: Update documentation on CREATE ROLE WITH HASHED PASSWORD
test/boost: Add test for creating roles with hashed passwords
Fixes https://github.com/scylladb/scylla-enterprise/issues/5016#issuecomment-2558464631
EAR - encryption at rest. Allows on-disk file encryption of sstables and commitlog data.
Introduces OpenSSL based file level encrypted storage, managed via a set of providers
ranging from local files to cloud KMS providers.
For a more comprehensive explanation, see the included docs (or if possible, original
source tree).
Manual bulk merge of EAR feature from enterprise repo to main scylla repo.
Breaks some features apart, but main EAR is still a humongous commit, because to separate this
I would have to mess with code incrementally, adding time and risk.
This PR includes the local file gen tool, tests and also p11 validation.
Note: CI will not execute the full tests unless master CI is set to provide the same environment
as the enterprise one. Not sure about the status of this ATM.
Note: Includes code to compile against cryptsoft kmipc SDK, but not the SDK. If you happen to
check out this tree in the scylla folder and configure, it will be linked against and KMIP functionality
will be enabled, otherwise not.
Closesscylladb/scylladb#22233
* github.com:scylladb/scylladb:
docs: Add EAR docs
main/build: Add p11-kit and initialize
tools: Add local-file-key-generator tool
tests: Add EAR tests
tmpdir: shorten test tempdir path
EAR: port the ear feature from enterprise
cql_test_env: Add optional query timeout
schema/migration_manager: Add schema validate
sstables: add get_shared_components accessor
config/config_file: Add exports and definitions of config_type_for<>
This PR extends authentication with 2 mechanisms:
- a new role_manager subclass, which allows managing users via
LDAP server,
- a new authenticator, which delegates plaintext authentication
to a running saslauthd daemon.
The features have been ported from the enterprise repository
with their test.py tests and the documentation as part of
changing license to source available.
Fixes: scylladb/scylla-enterprise#5000Fixes: scylladb/scylla-enterprise#5001Closesscylladb/scylladb#22030
remove the "ScyllaDB Enterprise" labels in document. because
there is no need to differentiate ScyllaDB Enterprise from its OSS
variant, let's stop adding the "ScyllaDB Enterprise" labels to
enterprise-only features. this helps to reduce the confusion.
as we are still in the process of porting the enterprise features
to this repo, this change does not fixscylladb/scylladb#22175.
we will review the document again when completing the migration.
we also take this opportunity to stop referencing "Enterprise" in
the changed paragraph.
Refs scylladb/scylladb#22175
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#22177
As part of #18750, we added a CQL statement CREATE ROLE WITH SALTED HASH
that prevented hashing a password when creating a role, effectively leading
to inserting a hash given by the user directly into the database. In #21350,
we noticed that Cassandra had implemented a CQL statement of similar semantics
but different syntax. We decided to rename Scylla's statement to be compatible
with Cassandra. Unfortunately, we didn't notice one more difference between
what we had in Scylla and what was part of Cassandra.
Scylla's statement was originally supposed to only be used when restoring
the schema and the user needn't have to be aware of its existence at all:
the database produced a sequence of CQL statements that the user saved to
a file and when a need to restore the schema arose, they would execute
the contents of the file. That's why that although we documented the feature,
it was only done in the necessary places. Those that weren't related to
the backup & restore procedure were deliberately skipped.
Cassandra, on the other hand, added the statement for a different purpose
(for details, see the relevant issue) and it was supposed to be used by
the user by design. The statement is also documented as such.
Since we want to preserve compatibility with Cassandra, we document
the statement and its semantics in the user documentation, explicitly
implying that it can be used by the user.
Fixesscylladb/scylladb#21691
Fixes#18903
Adds a "transitional" internode encryption mode, under which all _outgoing_ RPC connections will use TLS, but we will still accept any incoming non-tls connection.
This allows an operator to perform a move to TLS RPC without cluster downtime:
1. For each server, add certificate etc options to server_encryption_options + internode_encryption=none + set ssl_storage_port + restart (rolling)
2. For each server, set internode_encryption=transitional + RR
3. For each server, set internode_encryption=all + RR
Closesscylladb/scylladb#18939
* github.com:scylladb/scylladb:
test::topology: Add test for TLS upgrade and downgrade of internode encryption
docs: Add internode_encryption=transitional documentation
messaging_service: Add "transitional" internode encryptipn mode
messaging_service: Create TLS connector even if internode_enc=none when certs set
This commit removes the 5.4-to-6.0 upgrade guide and all references to it.
It mainly removes references to the Enable Consistent Topology Updates page,
which was added as enabling the feature was optional.
In rare cases, when a reference to that page is necessary,
the internal link is replaced with an external link to version 6.0.
Especially the Handling Cluster Membership Change Failures page was modified
for troubleshooting purposes rather than removed.
This doesn't apply for auth-v2 as we improved data placement and
removed cassandra quirk which was setting different CL for some
default superuser involved operations.
Fixes#18773Closesscylladb/scylladb#18785
This commit includes updates related to replacing system_auth with system_auth_v2.
- The keyspace name system_auth is renamed to system_auth_v2.
- The procedures are updated to account for system_auth_v2.
- No longer required system_auth RF changes are removed from procedures.
- The information is added that if the consistent topology updates feature
was not enabled upon upgrade from 5.4, there are limitations or additional
steps to do (depending on the procedure).
The files with that kind of information are to be found in _common folders
and included as needed.
- The upgrade guide has been updated to reflect system_auth_v2 and related impacts.
Closesscylladb/scylladb#18077
The goal is to make the available defaults safe for future use, as they
are often taken from existing config files or documentation verbatim.
Referenced issue: #14290Closesscylladb/scylladb#15947
This commit adds the information that ScyllaDB Enterprise
supports FIPS-compliant systems in versions
2023.1.1 and later.
The information is excluded from OSS docs with
the "only" directive, because the support was not
added in OSS.
This commit must be backported to branch-5.2 so that
it appears on version 2023.1 in the Enterprise docs.
Closes#15415
