The GCP host searches for application default credentials in known
locations within the user's home directory using
`seastar::file_exists()`. However, this function does not perform tilde
expansion in pathnames.
Replace tildes with the home directory from the HOME environment
variable.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
Fixes#22106
Moves the shared compress components to sstables, and rename to
match class type.
Adjust includes, removing redundant/unneeded ones where possible.
Closesscylladb/scylladb#25103
This PR introduces a new Key Provider to support Azure Key Vault as a Key Management System (KMS) for Encryption at Rest. The core design principle is the same as in the AWS and GCP key providers - an externally provided Vault key that is used to protect local data encryption keys (a process known as "key wrapping").
In more detail, this patch series consists of:
* Multiple Azure credential sources, offering a variety of authentication options (Service Principals, Managed Identities, environment variables, Azure CLI).
* The Azure host - the Key Vault endpoint bridge.
* The Azure Key Provider - the interface for the Azure host.
* Unit tests using real Azure resources (credentials and Vault keys).
* Log filtering logic to not expose sensitive data in the logs (plaintext keys, credentials, access tokens).
This is part of the overall effort to support Azure deployments.
Testing done:
* Unit tests.
* Manual test on an Azure VM with a Managed Identity.
* Manual test with credentials from Azure CLI.
* Manual test of `--azure-hosts` cmdline option.
* Manual test of log filtering.
Remaining items:
- [x] Create necessary Azure resources for CI.
- [x] Merge pipeline changes (https://github.com/scylladb/scylla-pkg/pull/5201).
Closes https://github.com/scylladb/scylla-enterprise/issues/1077.
New feature. No backport is needed.
Closesscylladb/scylladb#23920
* github.com:scylladb/scylladb:
docs: Document the Azure Key Provider
test: Add tests for Azure Key Provider
pylib: Add mock server for Azure Key Vault
encryption: Define and enable Azure Key Provider
encryption: azure: Delegate hosts to shard 0
encryption: Add Azure host cache
encryption: Add config options for Azure hosts
encryption: azure: Add override options
encryption: azure: Add retries for transient errors
encryption: azure: Implement init()
encryption: azure: Implement get_key_by_id()
encryption: azure: Add id-based key cache
encryption: azure: Implement get_or_create_key()
encryption: azure: Add credentials in Azure host
encryption: azure: Add attribute-based key cache
encryption: azure: Add skeleton for Azure host
encryption: Templatize get_{kmip,kms,gcp}_host()
encryption: gcp: Fix typo in docstring
utils: azure: Get access token with default credentials
utils: azure: Get access token from Azure CLI
utils: azure: Get access token from IMDS
utils: azure: Get access token with SP certificate
utils: azure: Get access token with SP secret
utils: rest: Add interface for request/response redaction logic
utils: azure: Declare all Azure credential types
utils: azure: Define interface for Azure credentials
utils: Introduce base64url_{encode,decode}
Do not use default, instead list all fall-through components
explicitly, so if we add a new one, the developer doing that
will be forced to consider what to do here.
Eliminate the `default` case from the switch in
`encryption_file_io_extension::wrap_sink`, and explicitly
handle all `component_type` values within the switch statement.
fixes: https://github.com/scylladb/scylladb/issues/23724Closesscylladb/scylladb#24987
As requested in #22114, moved the files and fixed other includes and build system.
Moved files:
- interval.hh
- Map_difference.hh
Fixes: #22114
This is a cleanup, no need to backport
Closesscylladb/scylladb#25095
Define the Azure Key Provider to connect the core EaR business logic
with the Azure-based Key Management implementation (Azure host).
Introduce "AzureKeyProviderFactory" as a new `key_provider` value in the
configuration.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
As in the AWS and GCP hosts, make all Azure hosts delegate their traffic
to shard 0 to avoid creating too many data encryption keys and API
calls to Key Vault.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
The encryption context maintains a cache per host type per thread.
Add a cache for the Azure host as well. Initialize the cache with Azure
hosts from the configuration, while registering the extensions for
encryption.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
Extend `get_or_create_key()` to accept host options that override the
config options. This will be used to pass encryption options from the
table schema. Currently, only the master key can be overridden.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
Inject a few fast retries to quickly recover from short-lived transient
errors. If a request is unauthorized, retry with no delay, since it may
be caused by expired tokens.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
Implement the `azure_host::init()` API that performs the async
initialization of the host.
Since the Azure host has no state that needs to be initialized, just
verify that we have access to the Vault key. This will cause the system
to fail earlier if not properly configured (e.g., the key does not
exist, the credentials have insufficient permissions, etc.).
Do not run any verification steps if no master key is configured in
`scylla.yaml`. The master key can be specified later or overridden
through the encryption options in table schema.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
Implement the `azure_host::get_key_by_id()` API, which retrieves a data
encryption key from a key ID.
Use a loading cache to reduce the API calls to Key Vault. When the cache
needs to refresh or reload a key, extract the ciphertext from the key ID
and unwrap it with the Vault key that is also encoded in the key ID.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
Add a cache to store data encryption keys based on their IDs. This will
be plugged into `get_key_by_id()` in a later patch to avoid unwrapping
keys that have been encountered recently, thereby reducing the API calls
to Key Vault.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
Implement the `azure_host::get_or_create_key()` API, which returns a
data encryption key for a given algorithm descriptor (cipher algorithm
and key length).
Use a loading cache to reduce the API calls to Key Vault. When the cache
needs to refresh or reload a key, always create a new one and wrap it
with the Vault key.
For the REST API calls to Key Vault, use an ephemeral HTTP client and
configure it to not wait for the server's response when terminating a
TLS connection. Although the TLS protocol requires clients to wait on
the server's response to a close_notify alert, the Key Vault service
ignores this, causing the client to block for 10 seconds (hardcoded)
before timing out.
Use the following identifier for each key:
<vault name>/<key name>/<key version>:<base64 encoded ciphertext of data encryption key>
The key version is required to support Vault key rotations.
Finally, define an exception for Vault errors.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
The Azure host needs credentials to communicate with Key Vault.
First search for credentials in the host options, and then fall back to
default credentials if the former are non-existent or incomplete.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
Add a cache to store data encryption keys based on their attributes
(cipher algorithm + key length). This will be plugged into
`get_or_create_key()` in a later patch to reuse the same keys in
multiple requests, thereby reducing the API calls to Key Vault.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
The Azure host manages cryptographic keys using Azure Key Vault.
This patch only defines the API.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
Add `make_data_or_index_source` to the storages to utilize new S3 based data source which should improve restore performance
* Introduce the `encrypted_data_source` class that wraps an existing data source to read and decrypt data on the fly using block encryption. Also add unit tests to verify correct decryption behavior.
* Add `make_data_or_index_source` to the `storage` interface, implement it for `filesystem_storage` storage which just creates `data_source` from a file and for the `s3_storage` create a (maybe) decrypting source from s3 make_download_source. This change should solve performance improvement for reading large objects from S3 and should not affect anything for the `filesystem_storage`
No backport needed since it enhances functionality which has not been released yet
fixes: https://github.com/scylladb/scylladb/issues/22458Closesscylladb/scylladb#23695
* github.com:scylladb/scylladb:
sstables: Start using `make_data_or_index_source` in `sstable`
sstables: refactor readers and sources to use coroutines
sstables: coroutinize futurized readers
sstables: add `make_data_or_index_source` to the `storage`
encryption: refactor key retrieval
encryption: add `encrypted_data_source` class
Fixes#24873
In KMIP host, do release of a connection (socket) due to our connection pool for the host being full, we currently don't close the connection properly, only rely on destructors.
This just makes sure `release` closes the connection if it neither retains or caches it.
Also, when running with the PyKMIP fixture, we tested the port being reachable using a normal socket. This makes python SSL generate errors -> log noise that look like actual errors.
Change the test setup to use a proper TLS connection + proper shutdown to avoid the noise logs.
This also adds a fixture helper for processes, and moves EAR test to use it (and by extension, seastar::experimental::process) instead of boost::process, removing a nasty non-seastarish dependency.
Closesscylladb/scylladb#24874
* github.com:scylladb/scylladb:
encryption_test: Make PyKMIP run under seastar::experimental::process
test/lib: Add wrapper helper for test process fixtures
kmip_host: Close connections properly if dropped by pool being full
encryption_at_rest_test: Do port check using TLS
Fixes#24873
Note: this happens like never. But if we, in KMIP host, do release
of a connection (socket) due to our connection pool for the host being
full, we currently don't close the connection properly, only rely on
destructors.
While not very serious, this would lead to possible TLS errors in the
KMIP host used, which should be avoided if possible.
Fix is simple, just make release close the connection if it neither retains
nor caches it.
Add `make_data_or_index_source` to the `storage` interface, implement it
for `filesystem_storage` storage which just creates `data_source` from a
file and for the `s3_storage` create a (maybe) decrypting source from s3
make_download_source.
This change should solve performance improvement for reading large objects
from S3 and should not affect anything for the `filesystem_storage`.
Introduce the `encrypted_data_source` class that wraps an existing data
source to read and decrypt data on the fly using block encryption. Also add
unit tests to verify correct decryption behavior.
NOTE: The wrapped source MUST read from offset 0, `encrypted_data_source` assumes it is
Co-authored-by: Calle Wilund <calle@scylladb.com>
There are two types of AWS security credentials:
* long-term credentials (access key id + secret access key)
* temporary credentials (access key id + secret access key + session token)
The KMS host can obtain these credentials from multiple sources:
* IMDS (config option `aws_use_ec2_credentials`)
* STS, by assuming an IAM role (config option `aws_assume_role_arn`)
* Scylla config (options `aws_access_key_id`, `aws_secret_access_key`)
* Env vars (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
* AWS credentials file (~/.aws/credentials)
First two sources return temporary credentials. The rest return
long-term credentials.
Extend the KMS host to support temporary credentials from the other
three sources as well. Introduce the config option `aws_session_token`,
and parse the same-named env var and config option from the credentials
file. Also, support `aws_security_token` as an alias, for backwards
compatibility.
This patch facilitates local debugging of corrupted SSTables, as well as
testing, using temporary credentials obtained from STS through other
authentication means (e.g., Okta + gimme-aws-creds).
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
The help string for the `--kms-hosts` command-line option mentions only
the AWS credentials file as a fall-back search path, in case no explicit
credentials are given.
Extend the help string to mention the environment as well. Make it clear
that the environment has higher precedence than the credentials file.
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
utils::loading cache has a timer that can, if we're unlucky, be runnnig
while the encryption context/extensions referencing the various host
objects containing them are destroyed in the case of unit testing.
Add a stop phase in encryption context shutdown closing the caches.
Fixed#24296
While the HTTP client used for REST calls in AWS/GCP KMS integration (EAR)
is not general enough to be called a HTTP client as such, it is general
enough to be called a REST client (limited to stateless, single-op REST
calls).
Other code, like general auth integrations (hello Azure) and similar
could reuse this to lessen code duplication.
This patch simply moves the httpclient class from encryption to "rest"
namespace, and explicitly "limits" it to such usage. Making an alias
in encryption to avoid touching more files than needed.
Closesscylladb/scylladb#24297
Fixes: #23970
use correct string literals:
KMIP_TAG_CRYPTOGRAPHIC_LENGTH_STR --> KMIP_TAGSTR_CRYPTOGRAPHIC_LENGTH
KMIP_TAG_CRYPTOGRAPHIC_USAGE_MASK_STR --> KMIP_TAGSTR_CRYPTOGRAPHIC_USAGE_MASK
From https://github.com/scylladb/scylladb/issues/23970 description of the
problem (emphasizes are mine):
When transparent data encryption at rest is enabled with KMIP as a key
provider, the observation is that before creating a new key, Scylla tries
to locate an existing key with provided specifications (key algorithm &
length), with the intention to re-use existing key, **but the attributes
sent in the request have minor spelling mistakes** which are rejected by
the KMIP server key provider, and hence scylla assumes that a key with
these specifications doesn't exist, and creates a new key in the KMIP
server. The issue here is that for every new table, ScyllaDB will create
a key in the KMIP server, which could clutter the KMS, and make key
lifecycle management difficult for DBAs.
Closesscylladb/scylladb#24057
Nowadays the whole class is public, but it's not in fact such.
Remove the SUDDENLY unused private _flush_pos member to please the
compiler.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Closesscylladb/scylladb#23677
Refs #22925
Adds some wrapping and helpers for the kind of REST operations we
expect to perform.
Some things like stream formatting is redundant visavi seastar,
but on that level we only have \r\n encoded writing to
output_stream and similar, which is less useful for things like
logging.
Adds a sibling type to encrypted file, a data_sink, that
will write a data stream in the same block format as a file
object would. Including end padding.
For making encrypted data sink writing less cumbersome.
schema_extension allows making invisible changes to system_schema
that evade upgrade rollback tests. They appear in system_schema
as an encoded blob which reduces serviceability, as they cannot
be read.
Deprecate it and point users to adding explicit columns in scylla_tables.
We could probably make use of the data structure, after we teach it
to encode its payload into proper named and typed columns instead of
using IDL.
Closesscylladb/scylladb#23151
Replace the abbreviated term "padd" with either "padding" or "pad" throughout
the encryption module. While "padd" was originally chosen to align with other
variable names ("type" and "mode"), using standard terminology improves code
readability and resolves codespell warnings.
Additionally, refactor relevant code to use C++ structured bindings for cleaner
implementation.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#23251
This commit eliminates unused boost header includes from the tree.
Removing these unnecessary includes reduces dependencies on the
external Boost.Adapters library, leading to faster compile times
and a slightly cleaner codebase.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#22997
Refs #22628
Mark problems parsing response (partial message, network error without exception etc
- hello testing), as "malformed_response_error", and promote this as well as
general "service_error" to recoverable exceptions (don't isolate node on error).
This to better handle intermittent network issues as well as making error-testing
more deterministic.
Adds exception handler + cleanup for the case where we have a
bad config/env vars (hint minio) or similar, such that we fail
with exception during setting up the EAR context.
In a normal startup, this is ok. We will report the exception,
and the do a exit(1).
In tests however, we don't and active context will instead be
freed quite proper, in which case we need to call stop to ensure
we don't crash on shared pointer destruction on wrong shard.
Doing so will hide the real issue from whomever runs the test.
This commit eliminates unused boost header includes from the tree.
Removing these unnecessary includes reduces dependencies on the
external Boost.Adapters library, leading to faster compile times
and a slightly cleaner codebase.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#22857
Fixes#22401
In the fix for scylladb/scylla-enterprise#892, the extraction and check for sstable component encryption mask was copied
to a subroutine for description purposes, but a very important 1 << <value> shift was somehow
left on the floor.
Without this, the check for whether we actually contain a component encrypted can be wholly
broken for some components.
Closesscylladb/scylladb#22398
Fixes#22236
If reading a file and not stopping on block bounds returned by `size()`, we could
allow reading from (_file_size+1-15) (block boundary) and try to decrypt this
buffer (last one).
Check on last block in `transform` would wrap around size due to us being >=
file size (l).
Simplest example:
Actual data size: 4095
Physical file size: 4095 + key block size (typically 16)
Read from 4096: -> 15 bytes (padding) -> transform return _file_size - read offset
-> wraparound -> rather larger number than we expected
(not to mention the data in question is junk/zero).
Just do an early bounds check and return zero if we're past the actual data limit.
v2:
* Moved check to a min expression instead
* Added lengthy comment
* Added unit test
v3:
* Fixed read_dma_bulk handling of short, unaligned read
* Added test for unaligned read
v4:
* Added another unaligned test case
"sie" is the short for "system info encryption". it is a wrapper around
a `opts` map so we can get the individual option by providing a default
value via an `optional<>` return value. but "sie" could be difficult to
understand without more context. and it is used like a function -- we
get the individual option using its operator().
so, in order to improve the readability, in this change, we rename it
to "get_opt".
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
