Commit Graph

377 Commits

Author SHA1 Message Date
Botond Dénes
9d08a380db Merge 'Fix getendpoints command for compound keys containing ':'' from Taras Veretilnyk
Before, the `nodetool getendpoints` expected the key as one string separated by : (for example 1:val:ue). This caused errors if any part of the key had a colon because it was unclear whether a colon was a separator or part of the key.

This change adds a new API endpoint, `/storage_service/natural_endpoints/v2/{keyspace}`, which accepts composite partition keys as multiple key_component query parameters (e.g., ?key_component=1&key_component=val:ue). The `nodetool getendpoints` command was updated to support a new `--key-components` option, allowing users to pass key components as an array. The client and test infrastructure were extended to support multiple values for a query parameter, and tests were added to verify correct behavior with composite keys.

The previous method of passing partition keys as colon-separated strings is preserved for backward compatibility.

Backport is not required, since this change relies on recent Seastar updates

Fixes #16596

Closes scylladb/scylladb#26169

* github.com:scylladb/scylladb:
  docs: document --key-components option for getendpoints
  test/nodetool/test_getendpoints: add coverage for --key-components param in getendpoints
  nodetool: Introduce new option --key-components to specify compound partition keys as array
  rest_api/test_storage_service: add v2 natural_endpoints test for composite key with multiple components
  api/storage_service: add GET 'natural_endpoints' v2 to support composite keys with ':'
  rest_api_mock: support duplicate query parameters
  test/rest_api: support multiple query values per key in RestApiSession.send()
  nodetool: add support of new seastar query_parameters_type to scylla_rest_client
2025-10-02 09:04:40 +03:00
Taras Veretilnyk
6d8224b726 docs: document --key-components option for getendpoints 2025-10-01 15:53:25 +02:00
Avi Kivity
5b40d4d52b Merge 'root,replica: mv multishard_mutation_query -> replica/multishard_query' from Botond Dénes
The code in `multishard_mutation_query.cc` implements the replica-side of range scans and as such it belongs in the replica module. Take the opportunity to also rename it to `multishard_query`, the code implements both data and mutation queries for a long time now.

Code cleanup, no backport required.

Closes scylladb/scylladb#26279

* github.com:scylladb/scylladb:
  test/boost: rename multishard_mutation_query_test to multishard_query_test
  replica/multishard_query: move code into namespace replica
  replica/multishard_query.cc: update logger name
  docs/paged-queries.md: update references to readers
  root,replica: move multishard_mutation_query to replica/
2025-09-28 20:24:46 +03:00
Botond Dénes
34cc7aafae tools/scylla-sstable: introduce the upgrade command
An offline, scylla-sstable variant of nodetool upgradesstables command.
Applies latest (or selected) sstable version and latest schema.

Closes scylladb/scylladb#26109
2025-09-27 16:53:14 +03:00
Botond Dénes
ed50a307db replica/multishard_query.cc: update logger name
To reflect the new file name.
2025-09-26 11:15:38 +03:00
Botond Dénes
edaf67edcb tools/scylla-sstable: remove writetime-histogram command
This command was written for an investigation and was used exactly once.
This would have been a perfect candidate for the (also rarely used)
scylla-sstable script command, but it didn't exist yet.
Drop this command from the tool, such super-specific commands should be
written as sstable-scripts nowadays, which is what we will do if we ever
need this again.

Closes scylladb/scylladb#26062
2025-09-18 12:05:54 +03:00
Nadav Har'El
3c969e2122 cql: document and test permissions on materialized views and CDC
We were recently surprised (in pull request #25797) to "discover" that
Scylla does not allow granting SELECT permissions on individual
materialized views. Instead, all materialized views of a base table
are readable if the base table is readable.

In this patch we document this fact, and also add a test to verify
that it is indeed true. As usual for cqlpy tests, this test can also
be run on Cassandra - and it passes showing that Cassandra also
implemented it the same way (which isn't surprising, given that we
probably copied our initial implementation from them).

The test demonstrates that neither Scylla nor Cassandra prints an error
when attempting to GRANT permissions on a specific materialized view -
but this GRANT is simply ignored. This is not ideal, but it is the
existing behavior in both and it's not important now to change it.

Additionally, because pull request #25797 made CDC-log permissions behave
the same as materialized views - i.e., you need to make the base table
readable to allow reading from the CDC log, this patch also documents
this fact and adds a test for it also.

Fixes #25800

Closes scylladb/scylladb#25827
2025-09-18 07:41:35 +03:00
Botond Dénes
839056b648 docs/operating-scylla: scylla-sstable.rst: update write docs
scylla-sstable write (and scrub) moved to UUID generations in
514f59d157, but said patch forgot to
update the docs. This is fixed here.

Closes scylladb/scylladb#25965
2025-09-18 07:39:50 +03:00
Botond Dénes
a8d22a66fa Merge 'Improve Encryption at Rest documentation' from Nikos Dragazis
This PR introduces a major rewrite of the EaR document. The initial motivation for this PR was to fully cover all our supported key providers with working examples, and to add instructions for key rotation. However, many other improvements were made along the way.

Main changes in this PR:
* Add a high-level description for every key provider. Mention limitations.
* Better organize existing provider-specific instructions by placing them into clearly separated, tabbed sections.
* Add instructions for the replicated key provider. Mention explicitly that it cannot be used as default option for user or system encryption, and that it does not support key rotation.
* Add more examples for KMS and GCP to cover all credential types.
* Document missing configuration options.
* Add a new section for key rotation.

Notes:
* Some of the patches in this series have been cherry-picked from Laszlo's wip branch.
* This PR is expected to conflict with the Azure Key Vault PR, which should be merged first. (https://github.com/scylladb/scylladb/pull/23920/)
* Support for KMIP system keys in the Replicated Key Provider is currently broken. (https://github.com/scylladb/scylladb/issues/24443)

Fixes scylladb/scylla-enterprise#3535.
Refs scylladb/scylla-enterprise#3183.

Only doc changes. No backport is needed.

Closes scylladb/scylladb#24558

* github.com:scylladb/scylladb:
  encryption-at-rest.rst: add "Rotate Encryption Keys" section
  encryption-at-rest.rst: rewrite "Encrypt System Resources" section
  encryption-at-rest.rst: rewrite "Update Encryption Properties of Existing Tables" section
  encryption-at-rest.rst: rewrite "Encrypt a Single Table" section
  encryption-at-rest.rst: rewrite "Encrypt Tables" section
  encryption-at-rest.rst: update "Set the Azure Host" section
  encryption-at-rest.rst: update "Set the GCP Host" section
  encryption-at-rest.rst: update "Set the KMS Host" section
  encryption-at-rest.rst: update "Set the KMIP Host" section
  encryption-at-rest.rst: rewrite "Create Encryption Keys" section
  encryption-at-rest.rst: rewrite "Key Providers" section
  encryption-at-rest.rst: hoist and update "Cipher Algorithm Descriptors"
  encryption-at-rest.rst: rewrite/replace section "Encryption Key Types"
  encryption-at-rest.rst: About: describe high-level operation more precisely
  encryption-at-rest.rst: improve wording / formatting in About intro
  encryption-at-rest.rst: users (plural) typo fix
  encryption-at-rest.rst: rewrap
  encryption-at-rest.rst: strip trailing whitespace
2025-09-17 17:25:25 +03:00
Asias He
54162a026f scylla-nodetool: Add --incremental-mode option to cluster repair
The `--incremental-mode` option specifies the incremental repair mode.
Can be 'disabled', 'regular', or 'full'.

'regular': The incremental repair logic is enabled. Unrepaired sstables
will be included for repair.  Repaired sstables will be skipped. The
incremental repair states will be updated after repair.

'full': The incremental repair logic is enabled. Both repaired and
unrepaired sstables will be included for repair. The incremental repair
states will be updated after repair.

'disabled': The incremental repair logic is disabled completely. The
incremental repair states, e.g., repaired_at in sstables and
sstables_repaired_at in the system.tablets table, will not be updated
after repair.

When the option is not provided, it defaults to regular.

Fixes #25931

Closes scylladb/scylladb#25969
2025-09-16 10:23:22 +03:00
Dawid Mędrek
18cb748268 docs/snitch: Document default DC and rack
The existing article is already extensive and covers pretty much
all of the details useful to the user. However, the document
lacked minute information like the default names of the DC and rack
in case of SimpleSnitch or it didn't explicitly specify the behavior
of RackInferringSnitch (though arguably the existing example was more
than sufficient).

Fixes scylladb/scylladb#23528

Closes scylladb/scylladb#25700
2025-09-15 11:47:22 +02:00
Ran Regev
6eca083137 remove redis documentation
As part of removing redis from Scylla source tree.
This commit removes all related documentation.
Following commit remove the code itself.

Signed-off-by: Ran Regev <ran.regev@scylladb.com>
2025-08-20 17:53:23 +03:00
Dawid Mędrek
6a71461e53 treewide: Fix spelling errors
The errors were spotted by our GitHub Actions.

Closes scylladb/scylladb#24822
2025-08-19 13:07:43 +03:00
Pavel Emelyanov
0616407be5 Merge 'rest_api: add endpoint which drops all quarantined sstables' from Taras Veretilnyk
Added a new POST endpoint `/storage_service/drop_quarantined_sstables` to the REST API.
This endpoint allows dropping all quarantined SSTables either globally or
for a specific keyspace and tables.
Optional query parameters `keyspace` and `tables` (comma-separated table names) can be
provided to limit the scope of the operation.

Fixes scylladb/scylladb#19061

Backport is not required, it is new functionality

Closes scylladb/scylladb#25063

* github.com:scylladb/scylladb:
  docs: Add documentation for the nodetool dropquarantinedsstables command
  nodetool: add command for dropping quarantine sstables
  rest_api: add endpoint which drops all quarantined sstables
2025-08-06 11:55:15 +03:00
Taras Veretilnyk
bcb90c42e4 docs: Sort commands list in nodetool.rst
Fixes scylladb/scylladb#25330

Closes scylladb/scylladb#25331
2025-08-06 11:20:53 +03:00
Nikos Dragazis
b186c48a65 encryption-at-rest.rst: add "Rotate Encryption Keys" section
Add a new section for key rotation, offering separate instructions per
key provider, organized in tabs.

The gist:

* Local Key Provider - Rotation requires creating a new key file per
  node. It's a manual procedure.

* Replicated Key Provider - Rotation is not supported.

* KMIP Key Provider - Rotation is transparent to Scylla, but it requires
  manually revoking the key in the server.

* {KMS,GCP} Key Provider - Rotation is transparent to Scylla and can be
  automated in the server.

* Azure Key Provider - Rotation is automatically supported by Scylla by
  keeping track of the key version along with the encrypted data. The
  rotation needs to be done at the Key Vault server, and can be
  automated.

Explain that, even after rotation, old keys may be still in use due to
caching, and that old SSTables will remain encrypted with the old key
until the next compaction. Provide instructions in case they prefer not
to wait.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:46 +03:00
Nikos Dragazis
3abacaa465 encryption-at-rest.rst: rewrite "Encrypt System Resources" section
- Mention all types of system data that fall under system encryption.

- Add "Before you Begin" section with requirements per key provider.
  The requirements are the same as in user encryption.

- Mention explicitly that the Replicated Key Provider cannot be used for
  system encryption.

- Provide separate instructions for each key provider. Explain all the
  configuration options.

- Provide an extra example for the Local Key Provider with a
  ``system_key_directory`` and ``key_name``.

- Highlight the code blocks as YAML. Make their indentation consistent
  with the rest of the doc (2 spaces).

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:46 +03:00
Nikos Dragazis
c59f71b399 encryption-at-rest.rst: rewrite "Update Encryption Properties of Existing Tables" section
- Split the various scenarios into sub-sections, not just examples.

- Amend the example for changing cipher algorithm and key length. The
  algorithm used in the example was the same.

- Point out that disabling encryption through the table schema is not
  possible if a node has default encryption configured.

- Amend the `nodetool upgradesstables` command. The
  `--include-all-sstables` is necessary.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:46 +03:00
Nikos Dragazis
22f941b325 encryption-at-rest.rst: rewrite "Encrypt a Single Table" section
- Add a short intro.

- Add an early note about the fact that options from
  ``scylla_encryption_options`` cannot be mixed with options from
  ``user_info_encryption``.

- Add a new "Allow Per-Table Encryption" subsection to document the
  ``allow_per_table_encryption`` option.

- Move the top-level procedure into a new "Encrypt a New Table"
  subsection to differentiate it from the "Update Encryption Properties
  of Existing Tables"".

- Add tabs for provider-dependent steps in "Before you Begin" and
  "Procedure".

- Amend "bytes" to "bits" (for the key length).

- Add examples for the replicated, KMIP, GCP, and Azure key providers.
  Use consistent keyspace and table names in all examples.

- Remove step for upgrading SSTables. The table is new - no SSTables
  exist yet.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:46 +03:00
Nikos Dragazis
bd83f3e672 encryption-at-rest.rst: rewrite "Encrypt Tables" section
- Provide separate requirements and instructions for each key provider,
  organized in tabs.
- Mention explicitly that the Replicated Key Provider cannot be used for
  default encryption.
- Fix indentation for code blocks in examples (2 spaces).
- For KMS, GCP, and Azure, add the `master_key` option in the list of
  options and remove the relevant example (not so common).
- Add steps for rolling restart.
- Amend "bytes" to "bits" (for the key length).

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:46 +03:00
Nikos Dragazis
fb030b11c3 encryption-at-rest.rst: update "Set the Azure Host" section
- Mark the `master_key` as required. Technically, it's not, since it can
  be specified in the schema encryption options, but:
  - It's better to keep it simple. The common case is to have a default
    value that occasionally needs to be overridden.
  - No functionality is lost.
  - It is mentioned as required for AWS and GCP.
- Add a note about credential resolution.
- Make some minor formatting changes to be consistent with the AWS and
  GCP sections.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:45 +03:00
Nikos Dragazis
e25b283c8d encryption-at-rest.rst: update "Set the GCP Host" section
- Add list of requirements (KMS Key, credentials, permissions).
- Add a reference to "Create Encryption Keys" section.
- Amend description for `master_key`.
- Add one example per credential type.
- Explain how credentials are resolved if not explicitly specified in
  the configuration.
- Fix indentation of "restart" command.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:45 +03:00
Nikos Dragazis
d9242ba47f encryption-at-rest.rst: update "Set the KMS Host" section
- Add a list of requirements (KMS key, credentials, permissions).
- Add a reference to "Create Encryption Keys" section.
- Add one example per credential type.
- Explain how credentials are resolved from the environment, or the
  AWS credentials file.
- Fix indentation of "restart" command.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:45 +03:00
Nikos Dragazis
cf9301c573 encryption-at-rest.rst: update "Set the KMIP Host" section
- Uncomment the code block to match the other hosts.
- Remove the ``certficate_revocation_list`` option; it's not supported.
- Amend the default values for ``key_cache_expiry`` and
  ``key_cache_refresh``.
- Add an example with mutual TLS authentication.
- Fix indentation of "restart" command.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:45 +03:00
Nikos Dragazis
b777dd267d encryption-at-rest.rst: rewrite "Create Encryption Keys" section
- Provide separate instructions for each key provider, organized in tabs.
  Move the existing instructions with the key generator script under the
  "Local Key Provider" tab. Point to the cloud provider's documentation
  for AWS, GCP, and Azure keys. List the required attributes for KMIP
  keys. List the required keys for the Replicated Key Provider.

- In the example for the key generator script, use the same algorithm
  and key strength for both the secret key and the system key, since
  this is the recommended case.

- Reorder the usage list of arguments for the key generator script.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:45 +03:00
Nikos Dragazis
60df275197 encryption-at-rest.rst: rewrite "Key Providers" section
- Use monospace font for key provider factories.

- Add a sub-section for every key provider. Explain how they operate at
  a high level and highlight any possible limitations.

- Remove version availability notes. The version 2019.1.3 is old and
  unsupported.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:45 +03:00
Nikos Dragazis
3c2f4ed1e7 encryption-at-rest.rst: hoist and update "Cipher Algorithm Descriptors"
Turn an earlier reference to "algorithm descriptor" into a hyperlink.

Use monospace font in the table header for "cipher_algorithm" and
"secret_key_strength"; these are verbatim identifiers in "scylla.yaml"
and "scylla_encryption_options". Same for their supported values.

Restrict the Blowfish key size to 128 bits, due to
<https://github.com/scylladb/scylla-enterprise/issues/4848>.

Add notes on ECB vs. CBC, and on Blowfish's 64-bit block size. Emphasize
our recommendation more.

Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:45 +03:00
Laszlo Ersek
f07125cfea encryption-at-rest.rst: rewrite/replace section "Encryption Key Types"
- Referring to system info encryption vs. user info encryption as distinct
  "encryption key types" is confusing. The behavior of encryption is
  similar in both cases, only the sets of data that are subject to
  encryption differ. Rename the section to "Data Classes for
  Encryption".

- Introduce the two highest-level "scylla.yaml" stanzas,
  "system_info_encryption" and "user_info_encryption". Subsequently, we'll
  expand on their (common!) contents later.

- Remove the comment that, for the Local Key Provider, a keystore can be
  created either manually or automatically. This is stated / repeated
  elsewhere in the document.

- Remove the unused anchor "_Replicated".

- The notes on the Replicated Key Provider both lack nuance, and are
  ill-placed, here. Remove those notes. Add a dedicated description for
  Replicated later, elsewhere. Do mention
  "system_replicated_keys.encrypted_keys" here in passing, as a system
  table with sensitive contents.

- The short listing of key providers is ill-placed here. We have an entire
  section dedicated to those. Furthermore, the various key providers apply
  to system info encryption, too.

- Explain the two levels of configuration for SSTables of user tables.

- Move the note about preserving keys for restoring backups to Key
  Providers | About Local Key Storage, at least temporarily. When keys are
  stored on a key management server (KMIP, GCP, AWS, Azure), then
  backing those up is its own admin task / responsibility.

Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:45 +03:00
Laszlo Ersek
268f5b1564 encryption-at-rest.rst: About: describe high-level operation more precisely
Clarify some table vs. SSTable differences.

Spell out the SSTable metadata ("Scylla.db") component. Spell out commit
log metadata files. Explain that encryption settings are "snapshotted"
into those meta-files.

Highlight that encryption config may vary per table *and* per node. (For
example, a local file key provider under the same pathname on each node,
referenced by the table's "scylla_encryption_options" in the schema, may
provide different keys for different nodes.)

Introduce "algorithm descriptor" and "key provider" as generic concepts.

Touch up the grammar / vocabulary slightly.

Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:45 +03:00
Laszlo Ersek
8717102ae5 encryption-at-rest.rst: improve wording / formatting in About intro
- Remove the KMIP password from the list of system level data.
  Encrypting this would require the `configuration_encryptor`, which has
  been removed as part of the effort to decommission all our java tools.

- Provide an exhaustive list of system tables being encrypted.

- "Table level granularity" is redundant; either "table level" or "table
  granularity" should suffice. Pick the latter.

- Distinguish "block cipher" from "mode of operation" more precisely.

Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:45 +03:00
Laszlo Ersek
b45d7417ef encryption-at-rest.rst: users (plural) typo fix
scylladb presumably stores data for multiple users.

Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
2025-08-01 17:27:45 +03:00
Laszlo Ersek
68dfa41e69 encryption-at-rest.rst: rewrap
Wrap long lines at 80 chars. Seastar coding style suggests 160 chars,
but 80 chars is more comfortable for side-by-side PR diffs on GitHub.
Exclude arg lists and code blocks. Set the limit at 160 chars for arg
lists to avoid too much wrapping that would hurt readability. Do not
wrap code blocks at all.

Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-08-01 17:27:45 +03:00
Laszlo Ersek
54ad1fe35f encryption-at-rest.rst: strip trailing whitespace
Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
2025-08-01 17:27:45 +03:00
Taras Veretilnyk
15e3980693 docs: Add documentation for the nodetool dropquarantinedsstables command
Fixes scylladb/scylladb#19061
2025-08-01 11:46:33 +02:00
Taras Veretilnyk
6b6622e07a docs: fix typo in command name enbleautocompaction -> enableautocompaction
Renamed the file and updated all references from 'enbleautocompaction' to the correct 'enableautocompaction'.

Fixes scylladb/scylladb#25172

Closes scylladb/scylladb#25175
2025-07-28 12:49:26 +03:00
Botond Dénes
837424f7bb Merge 'Add Azure Key Provider for Encryption at Rest' from Nikos Dragazis
This PR introduces a new Key Provider to support Azure Key Vault as a Key Management System (KMS) for Encryption at Rest. The core design principle is the same as in the AWS and GCP key providers - an externally provided Vault key that is used to protect local data encryption keys (a process known as "key wrapping").

In more detail, this patch series consists of:
* Multiple Azure credential sources, offering a variety of authentication options (Service Principals, Managed Identities, environment variables, Azure CLI).
* The Azure host - the Key Vault endpoint bridge.
* The Azure Key Provider - the interface for the Azure host.
* Unit tests using real Azure resources (credentials and Vault keys).
* Log filtering logic to not expose sensitive data in the logs (plaintext keys, credentials, access tokens).

This is part of the overall effort to support Azure deployments.

Testing done:
* Unit tests.
* Manual test on an Azure VM with a Managed Identity.
* Manual test with credentials from Azure CLI.
* Manual test of `--azure-hosts` cmdline option.
* Manual test of log filtering.

Remaining items:
- [x] Create necessary Azure resources for CI.
- [x] Merge pipeline changes (https://github.com/scylladb/scylla-pkg/pull/5201).

Closes https://github.com/scylladb/scylla-enterprise/issues/1077.

New feature. No backport is needed.

Closes scylladb/scylladb#23920

* github.com:scylladb/scylladb:
  docs: Document the Azure Key Provider
  test: Add tests for Azure Key Provider
  pylib: Add mock server for Azure Key Vault
  encryption: Define and enable Azure Key Provider
  encryption: azure: Delegate hosts to shard 0
  encryption: Add Azure host cache
  encryption: Add config options for Azure hosts
  encryption: azure: Add override options
  encryption: azure: Add retries for transient errors
  encryption: azure: Implement init()
  encryption: azure: Implement get_key_by_id()
  encryption: azure: Add id-based key cache
  encryption: azure: Implement get_or_create_key()
  encryption: azure: Add credentials in Azure host
  encryption: azure: Add attribute-based key cache
  encryption: azure: Add skeleton for Azure host
  encryption: Templatize get_{kmip,kms,gcp}_host()
  encryption: gcp: Fix typo in docstring
  utils: azure: Get access token with default credentials
  utils: azure: Get access token from Azure CLI
  utils: azure: Get access token from IMDS
  utils: azure: Get access token with SP certificate
  utils: azure: Get access token with SP secret
  utils: rest: Add interface for request/response redaction logic
  utils: azure: Declare all Azure credential types
  utils: azure: Define interface for Azure credentials
  utils: Introduce base64url_{encode,decode}
2025-07-25 10:45:32 +03:00
Ran Regev
3d82b9485e docs: update nodetool restore documentation for --sstables-file-list
Fixes: #25128
A leftover from #25077

Closes scylladb/scylladb#25129
2025-07-22 14:43:35 +02:00
Nikos Dragazis
88554b7c7a docs: Document the Azure Key Provider
Extend the EaR ops guide to incorporate the new Azure Key Provider.
Document its options and provide instructions on how to configure it.

Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-07-16 23:06:11 +03:00
Botond Dénes
1f9f43d267 Merge 'kms_host: Support external temporary security credentials' from Nikos Dragazis
This PR extends the KMS host to support temporary AWS security credentials provided externally via the Scylla configuration file, environment variables, or the AWS credentials file.

The KMS host already supports:
* Temporary credentials obtained automatically from the EC2 instance metadata service or via IAM role assumption.
* Long-term credentials provided externally via configuration, environment, or the AWS credentials file.

This PR is about temporary credentials that are external, i.e., not generated by Scylla. Such credentials may be issued, for example, through identity federation (e.g., Okta + gimme-aws-creds).

External temporary credentials are useful for short-lived tasks like local development, debugging corrupted SSTables with `scylla-sstable`, or other local testing scenarios. These credentials are temporary and cannot be refreshed automatically, so this method is not intended for production use.

Documentation has been updated to mention these additional credential sources.

Fixes #22470.

New feature, no backport is needed.

Closes scylladb/scylladb#22465

* github.com:scylladb/scylladb:
  doc: Expose new `aws_session_token` option for KMS hosts
  kms_host: Support authn with temporary security credentials
  encryption_config: Mention environment in credential sources for KMS
2025-07-15 06:45:39 +03:00
Piotr Dulikowski
ea35302617 Merge 'test: audit: enable syslog audit tests' from Andrzej Jackowski
Several audit test issues caused test failures, and in the result, almost all of audit syslog tests were marked with xfail.
This patch series enables the syslog audit tests, that should finally pass after the following fixes are introduced:
 - bring back commas to audit syslog (scylladb#24410 fix)
 - synchronize audit syslog server
 - fix parsing of syslog messages
 - generate unique uuid for each line in syslog audit
 - allow audit logging from multiple nodes

Fixes: scylladb/scylladb#24410

Test improvements, no backport required.

Closes scylladb/scylladb#24553

* github.com:scylladb/scylladb:
  test: audit: use automatic comparators in AuditEntry
  test: audit: enable syslog audit tests
  test: audit: sort new audit entries before comparing with expected ones
  test: audit: check audit logging from multiple nodes
  test: audit: generate unique uuid for each line in syslog audit
  test: audit: fix parsing of syslog messages
  test: audit: synchronize audit syslog server
  docs: audit: update syslog audit format to the current one
  audit: bring back commas to audit syslog
2025-07-07 12:45:44 +02:00
Nikos Dragazis
fbc9ead182 doc: Expose new aws_session_token option for KMS hosts
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
2025-07-02 12:04:40 +03:00
Anna Stuchlik
9234e5a4b0 doc: add the SBOM page and the download link
This commit migrates the Software Bill Of Materials (SBOM) page
added to the Enterprise docs with https://github.com/scylladb/scylla-enterprise/pull/5067.

The only difference is the link to the SBOM files - it was Enterprise SBOM in the Enterprise docs,
while here is a link to the ScyllaDB SBOM.

It's a follow-up of migration to Source Avalable and should be backported
to all Source Available versions - 2025.1 and later.

Fixes https://github.com/scylladb/scylladb/issues/24730

Closes scylladb/scylladb#24735
2025-07-01 11:33:19 +03:00
Andrzej Jackowski
fcd88e1e54 docs: audit: update syslog audit format to the current one
The documentation of the syslog audit format was not updated when
scylladb#23099 and earlier audit log changes were introduced.
This commit includes the missing update.
2025-06-30 09:19:25 +02:00
Anna Stuchlik
7537f5f260 doc: fix the headings in the Admin Guide
This commit fixes incorrect headings in the Admin Guide and the files
that are included in that guide.

The purpose is to properly organize the content and improve the search,
as well as prevent potential build problems caused by a poor heading organization.

Fixes https://github.com/scylladb/scylladb/issues/24441

Closes scylladb/scylladb#24700
2025-06-27 20:07:09 +03:00
Botond Dénes
e715a150b9 tools/scylla-nodetool: backup: add --move-files parameter
Allow opting in for backup to move the files instead of copying them.

Fixes: https://github.com/scylladb/scylladb/issues/24372

Closes scylladb/scylladb#24503
2025-06-27 16:21:39 +03:00
Patryk Jędrzejczak
203ea5d8f9 docs: rely on the Raft-based topology being enabled
In 2025.2, we don't force enabling the Raft-based topology in the code,
but we stated in the upgrade guides that it's a mandatory step of the
upgrade to 2025.1. We also remind users to enable the Raft-based
topology in the upgrade guides to 2025.2. Hence, we can rely in the
the documentation on the Raft-based topology being enabled. If it is
still disabled, we can just send the user to the upgrade guides. Hence:
- we remove all documentation related to enabling the Raft-based
  topology, enabling the Raft-based schema (enabled Raft-based topology
  implies enabled Raft-based schema), and the gossip-based topology,
- we can replace the documentation of the old manual recovery procedure
  with the documentation of the new manual recovery procedure (done in
  the previous commit).
2025-06-26 14:17:54 +02:00
Robert Bindar
ca1a9c8d01 Add support for nodetool refresh --skip-reshape
This patch adds the new option in nodetool, patches the
load_new_ss_tables REST request with a new parameter and
skips the reshape step in refresh if this flag is passed.

Signed-off-by: Robert Bindar <robert.bindar@scylladb.com>

Closes scylladb/scylladb#24409
Fixes: #24365
2025-06-10 12:52:13 +03:00
Robert Bindar
c570941692 Add nodetool refresh --scope option
This change adds the --scope option to nodetool refresh.
Like in the case of nodetool restore, you can pass either of:
* node - On the local node.
* rack - On the local rack.
* dc - In the datacenter (DC) where the local node lives.
* all (default) - Everywhere across the cluster.
as scope.

The feature is based on the existing load_and_stream paths, so it
requires passing --load-and-stream to the refresh command.
Also, it is not compatible with the --primary-replica-only option.

Signed-off-by: Robert Bindar <robert.bindar@scylladb.com>

Closes scylladb/scylladb#23861
2025-05-29 16:12:09 +03:00
Botond Dénes
485df63fd5 Merge 'Extend compaction_history table with additional compaction statistics' from Łukasz Paszkowski
Currently, the `system.compaction_history` table miss information like the type of compaction (cleanup, major, resharding, etc), the sstable generations involved (in and out), shard's id the compaction was triggered on and statistics on purged tombstones to be collected during compaction.

The series extends the table with the following columns:

-  "compaction_type" (text)
- "shard_id" (int)
- "sstables_in" (list<sstableinfo_type>)
- "sstables_out" (list<sstableinfo_type>)
- "total_tombstone_purge_attempt" (long)
- "total_tombstone_purge_failure_due_to_overlapping_with_memtable" (long)
- "total_tombstone_purge_failure_due_to_overlapping_with_uncompacting_sstable" (long)

with a user defined type `sstableinfo_type` that holds the information about sstable file

- generation (uuid)
- origin (text)
- size (long)

Additional statistics stored in the compaction_history have been incorporated in the API  `/compaction_manager/compaction_history` and the `nodetool compactionhistory` command.

No backport is required. It extends the existing compaction history output.

Fixes https://github.com/scylladb/scylladb/issues/3791

Closes scylladb/scylladb#21288

* github.com:scylladb/scylladb:
  nodetool: Refactor of compactionhistory_operation
  nodetool: Add more stats into compactionhistory output
  api/compaction_manager: Extend compaction_history api
  compaction: Collect tombstone purge stats during compaction
  compacting_reader: Extend to accept tombstone purge statistics
  mutation_compactor: Collect tombstone purge attempts
  compaction_garbage_collector: Extend return type of max_purgeable_fn
  compaction: Extend compaction_result to collect more information
  system_keyspace: Upgrade compaction_history table
  system_keyspace: Create UDT: sstableinfo_type
  system_keyspace: Extract compaction_history struct
  system_keyspace: Squeeze update_compaction_history parameters
  compaction/compaction_manager: update_history accepts compaction_result as rvalue
2025-05-27 14:12:13 +03:00
Anna Stuchlik
d303edbc39 doc: remove copyright from Cassandra Stress
This commit removes the Apache copyright note from the Cassandra Stress page.

It's a follow up to https://github.com/scylladb/scylladb/pull/21723, which missed
that update (see https://github.com/scylladb/scylladb/pull/21723#discussion_r1944357143).

Cassandra Stress is a separate tool with separate repo with the docs, so the copyright
information on the page is incorrect.

Fixes https://github.com/scylladb/scylladb/issues/23240

Closes scylladb/scylladb#24219
2025-05-26 09:35:30 +02:00