Add skip_reason_plugin.py — a framework-agnostic pytest plugin that
provides typed skip markers (skip_bug, skip_not_implemented, skip_slow,
skip_env) so that the reason a test is skipped is machine-readable in
JUnit XML and Allure reports. Bare untyped pytest.mark.skip now
triggers a warning (to become an error after full migration). Runtime
skips via skip() are also enriched by parsing the [type] prefix from
the skip message.
The plugin is a class (SkipReasonPlugin) that receives the concrete
SkipType enum and an optional report_callback from conftest.py, keeping
it decoupled from allure and project-specific types.
Extract SkipType enum and convenience runtime skip wrappers (skip_bug,
skip_env, etc.) into test/pylib/skip_types.py so callers only need a
single import instead of importing both SkipType and skip() separately.
conftest.py imports SkipType from the new module and registers the
plugin instance unconditionally (for all test runners).
New files:
- test/pylib/skip_reason_plugin.py: core plugin — typed marker
processing, bare-skip warnings, JUnit/Allure report enrichment
(including runtime skip() parsing via _parse_skip_type helper)
- test/pylib/skip_types.py: SkipType enum and convenience wrappers
(skip_bug, skip_not_implemented, skip_slow, skip_env)
- test/pylib_test/test_skip_reason_plugin.py: 17 pytester-based
test functions (51 cases across 3 build modes) covering markers,
warnings, reports, callbacks, and skip_mode interaction
Infrastructure changes:
- test/conftest.py: import SkipType from skip_types, register
SkipReasonPlugin with allure report callback
- test/pylib/runner.py: set SKIP_TYPE_KEY/SKIP_REASON_KEY stash keys
for skip_mode so the report hook can enrich JUnit/Allure with
skip_type=mode without longrepr parsing
- test/pytest.ini: register typed marker definitions (required for
--strict-markers even when plugin is not loaded)
Migrated test files (representative samples):
- test/cluster/test_tablet_repair_scheduler.py:
skip -> skip_bug (#26844), skip -> skip_not_implemented
- test/cqlpy/.../timestamp_test.py: skip -> skip_slow
- test/cluster/dtest/schema_management_test.py: skip -> skip_not_implemented
- test/cluster/test_change_replication_factor_1_to_0.py: skip -> skip_bug (#20282)
- test/alternator/conftest.py: skip -> skip_env
- test/alternator/test_https.py: use skip_env() wrapper
Fixes SCYLLADB-79
Closesscylladb/scylladb#29235
This patch series implements `object_storage_base::clone`, which was previously a stub that aborted at runtime. Clone creates a copy of an sstable under a new generation and is used during compaction.
The implementation uses server-side object copies (S3 CopyObject / GCS Objects: rewrite) and mirrors the filesystem clone semantics: TemporaryTOC is written first to mark the operation as in-progress, component objects are copied, and TemporaryTOC is removed to commit (unless the caller requested the destination be left unsealed).
The first two patches fix pre-existing bugs in the underlying storage clients that were exposed by the new clone code path:
- GCS `copy_object` used the wrong HTTP method (PUT instead of POST) and sent an invalid empty request body.
- S3 `copy_object` silently ignored the abort_source parameter.
1. **gcp_client: fix copy_object request method and body** — Fix two bugs in the GCS rewrite API call.
2. **s3_client: pass through abort_source in copy_object** — Stop ignoring the abort_source parameter.
3. **object_storage: add copy_object to object_storage_client** — New interface method with S3 and GCS implementations.
4. **storage: add make_object_name overload with generation** — Helper for building destination object names with a different generation.
5. **storage: make delete_object const** — Needed by the const clone method.
6. **storage: implement object_storage_base::clone** — The actual clone implementation plus a copy_object wrapper.
7. **test/boost: enable sstable clone tests for S3 and GCS** — Re-enable the previously skipped tests.
A test similar to `sstable_clone_leaving_unsealed_dest_sstable` was added to properly test the sealed/unsealed states for object storage. Works for both S3 and GCS.
Fixes: https://scylladb.atlassian.net/browse/SCYLLADB-1045
Prerequisite: https://github.com/scylladb/scylladb/pull/28790
No need to backport since this code targets future feature
Closesscylladb/scylladb#29166
* github.com:scylladb/scylladb:
compaction_test: enable sstable clone tests for S3 and GCS
storage: implement object_storage_base::clone
storage: make delete_object const in object_storage_base
storage: add make_object_name overload with generation
sstables: add get_format() accessor to sstable
object_storage: add copy_object to object_storage_client
s3_client: pass through abort_source in copy_object
gcp_client: fix copy_object request method and body
The vector-search feature introduced the somewhat confusing feature of
enabling CDC without explicitly enabling CDC: When a vector index is
enabled on a table, CDC is "enabled" for it even if the user didn't
ask to enable CDC.
For this, write-path code began to use a new cdc_enabled() function
instead of checking schema.cdc_options.enabled() directly. This
cdc_enabled() function checks if either this enabled() is true, or
has_vector_index() is true.
Unfortunately, LWT writes continued to use cdc_options.enabled() instead
of the new cdc_enabled(). This means that if a vector index is used and
a vector is written using an LWT write, the new value is not indexed.
This patch fixes this bug. It also adds a regression test that fails
before this patch and passes afterwards - the new test verifies that
when a table has a vector index (but no explicit CDC enabled), the CDC
log is updated both after regular writes and after successful LWT writes.
This patch was also tested in the context of the upcoming vector-search-
for-Alternator pull request, which has a test reproducing this bug
(Alternator uses LWT frequently, so this is very important there).
It will also be tested by the vector-store test suite ("validator").
Fixes SCYLLADB-1342
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closesscylladb/scylladb#29300
In the unregistered-ID branch, ldap_msgfree() was called on a result
already owned by an RAII ldap_msg_ptr, causing a double-free on scope
exit. Remove the redundant manual free.
Fixes: SCYLLADB-1344
Backport: 2026.1, 2025.4, 2025.1 - it's a memory corruption, with a one-line fix, so better backport it everywhere.
Closesscylladb/scylladb#29302
* github.com:scylladb/scylladb:
test: ldap: add regression test for double-free on unregistered message ID
ldap: fix double-free of LDAPMessage in poll_results()
Now that object_storage_base::clone is implemented,
remove the early-return skips and re-enable the
sstable_clone_leaving_unsealed_dest_sstable tests for
both S3 and GCS storage backends.
Implement the clone method for object_storage_base, which creates
a copy of an sstable with a new generation using server-side object
copies. Also add a const copy_object convenience wrapper, similar
to the existing put_object and delete_object wrappers.
A dedicated test for the new object storage clone path will be
added in the following commit. The preexisting local-filesystem
clone is already covered by the sstable_clone_leaving_unsealed_dest_sstable
test.
Add a make_object_name overload that accepts a target
generation parameter for constructing object names with
a generation different from the source sstable's own.
Refactor the original make_object_name to delegate to
the new overload, eliminating code duplication.
This is needed by clone to build destination object
names for the new generation.
Add a public get_format() accessor for the _format member, following
the same pattern as the existing get_version(). This allows storage
implementations to access the sstable format without reaching into
private members, and is needed by the upcoming object_storage_base::clone
to construct entry_descriptor for the sstables registry.
Add a copy_object method to the object_storage_client
interface for server-side object copies, with
implementations for both S3 and GCS wrappers.
The S3 wrapper delegates to s3::client::copy_object.
The GCS wrapper delegates to gcp::storage::client's
cross-bucket copy_object overload.
This is a prerequisite for implementing sstable clone
on object storage.
The abort_source parameter in s3::client::copy_object
was ignored — the function accepted it but always passed
nullptr to the underlying copy_s3_object. Forward it
properly so callers can cancel in-progress copies.
The GCP copy_object (rewrite API) had two bugs:
1. The request body was an empty string, but the GCP
rewrite endpoint always parses it as JSON metadata.
An empty string is not valid JSON, resulting in
400 "Metadata in the request couldn't decode".
Fix: send "{}" (empty JSON object) as the body.
2. The HTTP method was PUT, but the GCP Objects: rewrite
API requires POST per the documentation.
Fix: use POST.
Test coverage in a follow-up patch
When `BatchWriteItem` operates on multiple items sharing the same partition key in `always_use_lwt` write isolation mode, all CDC log entries are emitted under a single timestamp. The previous `get_records` parsing algorithm in `alternator/streams.cc` assumed that all CDC log entries sharing the same timestamp correspond to a single DynamoDB item change. As a result, it would incorrectly squash multiple distinct item changes into a single Streams record — producing wrong event data (e.g., one INSERT instead of four, with mismatched key/attribute values).
Note: the bug is specific to `always_use_lwt` mode because only in LWT mode does the entire batch share a single timestamp. In non-LWT modes, each item in the batch receives a separate timestamp, so the entries naturally stay separate.
**Commit 1: alternator: add BatchWriteItem Streams test**
- Adds new tests `test_streams_batchwrite_no_clustering_deletes_non_existing_items` and `test_streams_batchwrite_no_clustering_deletes_existing_items` that cover the corner cases of batch-deleting a existing and non-existing item in a table without a clustering key. CDC tables without clustering keys are handled differently, and this path was previously untested for delete operations.
- Adds a new test `test_streams_batchwrite_into_the_same_partition_will_report_wrong_stream_data`, that is a simple way to trigger a bug.
- Adds a new test `test_streams_batchwrite_into_the_same_partition_deletes_existing_items`, that validates various combinations of puts and deletes in a single BatchWrite against the same partition.
- Adds a new `test_table_ss_new_and_old_images_write_isolation_always` fixture and extends `create_table_ss` to accept `additional_tags`, enabling tests with a specific write isolation mode.
**Commit 2: alternator: fix BatchWriteItem squashed Streams entries**
The core fix rewrites the CDC log entry parsing in `get_records` to distinguish items by their clustering key:
- Introduces `managed_bytes_ptr_hash` and `managed_bytes_ptr_equal` helper structs for pointer-based hash map lookups on `managed_bytes`.
- Replaces the single `record`/`dynamodb` pair with a `std::unordered_map<const managed_bytes*, Record, ...>` (`records_map`) keyed by the base table's clustering key value from each CDC log row. For tables without a clustering key, all entries map to a single sentinel key.
- Adds a validation that Alternator tables have at most one clustering key column (as required by the DynamoDB data model).
- On end-of-record (`eor`), flushes all accumulated per-clustering-key records into the output, each with a unique `eventID` (the `event_id` format now includes an index suffix).
- Adjusts the limit check: since a single CDC timestamp bucket can now produce multiple output records, the limit may be slightly exceeded to avoid breaking mid-batch.
Fixes#28439
Fixes: SCYLLADB-540
Closesscylladb/scylladb#28452
* github.com:scylladb/scylladb:
alternator/test: explain why 'always' write isolation mode is used in tests
alternator/test: add scylla_only to always write isolation fixture
alternator: fix BatchWriteItem squashed Streams entries
alternator: add BatchWriteItem test (failing)
Queries against local vector indexes were failing with the error:
```ANN ordering by vector requires the column to be indexed using 'vector_index'```
This was a regression introduced by 15788c3734, which incorrectly
assumed the first column in the targets list is always the vector column.
For local vector indexes, the first column is the partition key, causing
the failure.
Previously, serialization logic for the target index option was shared
between vector and secondary indexes. This is no longer viable due to
the introduction of local vector indexes and vector indexes with filtering
columns, which have different target format.
This commit introduces a dedicated JSON-based serialization format for
vector index targets, identifying the target column (tc), filtering
columns (fc), and partition key columns (pk). This ensures unambiguous
serialization and deserialization for all vector index types.
This change is backward compatible for regular vector indexes. However,
it breaks compatibility for local vector indexes and vector indexes with
filtering columns created in version 2026.1.0. To mitigate this, usage
of these specific index types will be blocked in the 2026.1.0 release
by failing ANN queries against them in vector-store service.
Fixes: SCYLLADB-895
Backport to 2026.1 is required as this issue occurs also on this branch.
Closesscylladb/scylladb#28862
* github.com:scylladb/scylladb:
index: fix DESC INDEX for vector index
vector_search: test: refactor boilerplate setup
vector_search: fix SELECT on local vector index
index: test: vector index target option serialization test
index: test: secondary index target option serialization test
To create `process_staging` view building tasks, we firstly need to
collect informations about them on shard0, create necessary mutations,
commit them to group0 and move staging sstables objects to their
original shards.
But there is a possible race after committing the group0 command
and before moving the staging sstables to their shards.
Between those two events, the coordinator may schedule freshly created
tasks and dispatch them to the worker but the worker won't have the
sstables objects because they weren't moved yet.
This patch fixes the race by holding `_staging_sstables_mutex` locks
from necessary shards when executing `create_staging_sstable_tasks()`.
With this, even if the task will be scheduled and dispatched quickly,
the worker will wait with executing it until the sstables objects are
moved and the locks are released.
Fixes SCYLLADB-816
Add a test that verifies filesystem_storage::clone preserves the sstable
state: an sstable in staging is cloned to a new generation, the clone is
re-loaded from the staging directory, and its state is asserted to still
be staging.
The change proves that https://scylladb.atlassian.net/browse/SCYLLADB-1205
is invalid, and can be closed.
* No functional change and no backport needed
Closesscylladb/scylladb#29209
* github.com:scylladb/scylladb:
test: add test_sstable_clone_preserves_staging_state
test: derive sstable state from directory in test_env::make_sstable
sstables: log debug message in filesystem_storage::clone
`data_value::to_parsable_string()` crashes with a null pointer dereference when called on a `null` data_value. Return `"null"` instead.
Added tests after the fix. Manually checked that tests fail without the fix.
Fixes SCYLLADB-1350
This is a fix that prevents format crash. No known occurrence in production, but backport is desirable.
Closesscylladb/scylladb#29262
* github.com:scylladb/scylladb:
test: boost: test null data value to_parsable_string
cql3: fix null handling in data_value formatting
Extend test_basic to run with both RF=1 and RF=3 to verify that
object storage works correctly with multiple replicas. The test now
starts one server per replica (each on its own rack), flushes all
nodes, validates tablet replica counts for RF>1, and restarts all
servers before verifying data is still readable.
Fixes: SCYLLADB-546
Closesscylladb/scylladb#28583
As noted in issue #5027 and issue #29138, Alternator's support for
ReturnConsumedCapacity is lacking in a two areas:
1. While ReturnConsumedCapacity is supported for most relevant
operations, it's not supported in two operations: Query and Scan.
2. While ReturnConsumedCapacity=TOTAL is supported, INDEXES is not
supported at all.
This patch adds extensive tests for all these cases. All these tests
pass on DynamoDB but fail on Alternator, so are marked with "xfail".
The tests for ReturnConsumedCapacity=INDEXES are deliberately split
into two: First, we test the case where the table has no indexes, so
INDEXES is almost the same as TOTAL and should be very easy to
implement. A second test checks the cases where there are indexes,
and different operations increment the capacity of the base table
and/or indexes differently - it will require significantly more work
to make the second test pass.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closesscylladb/scylladb#29188
When a table has a vector index, cdc::cdc_enabled() returns true because
vector index writes are implemented via the CDC augmentation path. However,
register_cdc_operation_result_tracker() was checking only
cdc_options().enabled(), which is false for tables that have a vector index
but not traditional CDC.
As a result, the operation_result_tracker was never attached to write
response handlers for vector-indexed tables. This tracker was added in
commit 1b92cbe, and its job is to update metrics of CDC operations,
and since vector search really does use CDC under the hood, these
metrics could be useful when diagnosing problems.
Fix by using cdc::cdc_enabled() instead of cdc_options().enabled(), which
covers both traditional CDC and vector-indexed tables.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closesscylladb/scylladb#29343
Clang 22 verifies [[nodiscard]] for co_await,
causing compilation failures where return values of expected<> were
silently discarded.
These call sites were discarding the return value of client::request()
and vector_store_client::ann(), both of which return expected<> types
marked [[nodiscard]]. Rather than suppressing the warning with (void)
casts, properly check the return values using the established test
patterns: BOOST_CHECK(result) where the call is expected to succeed,
and BOOST_CHECK(!result) where the call is expected to fail.
Closesscylladb/scylladb#29297
This issue adds the upgrade guide for all patch releases within 2026.x major release.
In addition, it fixes the link to Upgrade Policy in the 2025.x-to-2026.1 upgrade guide.
Fixes SCYLLADB-1247
Closesscylladb/scylladb#29307
This PR introduces the vnodes-to-tablets migration procedure, which enables converting an existing vnode-based keyspace to tablets.
The migration is implemented as a manual, operator-driven process executed in several stages. The core idea is to first create tablet maps with the same token boundaries and replica hosts as the vnodes, and then incrementally convert the storage of each node to the tablets layout. At a high level, the procedure is the following:
1. Create tablet maps for all tables in the keyspace.
2. Sequentially upgrade all nodes from vnodes to tablets:
1. Mark a node for upgrade in the topology state.
2. Restart the node. During startup, while the node is offline, it reshards the SSTables on vnode boundaries and switches to a tablet ERM.
3. Wait for the node to return online before proceeding to the next node.
4. Finalize the migration:
1. Update the keyspace schema to mark it as tablet-based.
2. Clear the group0 state related to the migration.
From the client's perspective, the migration is online; the cluster can still serve requests on that keyspace, although performance may be temporarily degraded.
During the migration, some nodes use vnode ERMs while others use tablet ERMs. Cluster-level algorithms such as load balancing will treat the keyspace's tables as vnode-based. Once migration is finalized, the keyspace is permanently switched to tablets and cannot be reverted back to vnodes. However, a rollback procedure is available before finalization.
The patch series consists of:
* Load balancer adjustments to ignore tablets belonging to a migrating keyspace.
* A new vnode-based resharding mode, where SSTables are segregated on vnode boundaries rather than with the static sharder.
* A new per-node `intended_storage_mode` column in `system.topology`. Represents migration intent (whether migration should occur on restart) and direction.
* Four new REST endpoints for driving the migration (start, node upgrade/downgrade, finalize, status), along with `nodetool` wrappers. The finalization is implemented as a global topology request.
* Wiring of the migration process into the startup logic: the `distributed_loader` determines a migrating table's ERM flavor from the `intended_storage_mode` and the ERM flavor determines the `table_populator`'s resharding mode. Token metadata changes have been adjusted to preserve the ERM flavor.
* Cluster tests for the migration process.
Fixes SCYLLADB-722.
Fixes SCYLLADB-723.
Fixes SCYLLADB-725.
Fixes SCYLLADB-779.
Fixes SCYLLADB-948.
New feature, no backport is needed.
Closesscylladb/scylladb#29065
* github.com:scylladb/scylladb:
docs: Add ops guide for vnodes-to-tablets migration
test: cluster: Add test for migration of multiple keyspaces
test: cluster: Add test for error conditions
test: cluster: Add vnodes->tablets migration test (rollback)
test: cluster: Add vnodes->tablets migration test (1 table, 3 nodes)
test: cluster: Add vnodes->tablets migration test (1 table, 1 node)
scylla-nodetool: Add migrate-to-tablets subcommand
api: Add REST endpoint for vnode-to-tablet migration status
api: Add REST endpoint for migration finalization
topology_coordinator: Add `finalize_migration` request
database: Construct migrating tables with tablet ERMs
api: Add REST endpoint for upgrading nodes to tablets
api: Add REST endpoint for starting vnodes-to-tablets migration
topology_state_machine: Add intended_storage_mode to system.topology
distributed_loader: Wire vnode-based resharding into table populator
replica: Pick any compaction group for resharding
compaction: resharding_compaction: add vnodes_resharding option
storage_service: Preserve ERM flavor of migrating tables
tablet_allocator: Exclude migrating tables from load balancing
feature_service: Add vnodes_to_tablets_migrations feature
The estimate() function in the size_estimates virtual reader only
considered sstables local to the shard that happened to own the
keyspace's partition key token. Since sstables are distributed across
shards, this caused partition count estimates to be approximately
1/smp_count of the actual value.
This bug has been present since the virtual reader was introduced in
225648780d.
Use db.container().map_reduce0() to aggregate sstable estimates
across all shards. Each shard contributes its local count and
estimated_histogram, which are then merged to produce the correct
total.
Also fix the `test_partitions_estimate_full_overlap` test which becomes
flaky (xpassing ~1% of runs) because autocompaction could merge the
two overlapping sstables before the size estimate was read. Wrap the
test body in nodetool.no_autocompaction_context to prevent this race.
Fixes https://scylladb.atlassian.net/browse/SCYLLADB-1179
Refs https://github.com/scylladb/scylladb/issues/9083Closesscylladb/scylladb#29286
As discussed with @ScyllaPiotr in
https://github.com/scylladb/scylladb/pull/29232, the doc about to be
removed is just:
> Looking at history, I think this audit.md is a design doc: scylladb/scylla-enterprise@87a5c19, for which the feature has been implemented differently, eventually, and was created around the time when design docs, apparently, where stored within the repository itself. So for me it's some trash (sorry for strong language) that can be safely removed.
Closesscylladb/scylladb#29316
Python warns that the sequence "\(" is an invalid escape and
might be rejected in the future. Protect against that by using
a raw string.
Closesscylladb/scylladb#29334
Convert auth_test.cc to coroutines for improved readability. Each test is converted in its own commit. Some
are trivial.
Indentation is left broken in some commits to reduce the diff, then fixed up in the last commit.
Code cleanup, so no backport.
Closesscylladb/scylladb#29336
* github.com:scylladb/scylladb:
auth_test: fix whitespace
auth_test: coroutinize test_try_describe_schema_with_internals_and_passwords_as_anonymous_user
auth_test: coroutinize test_try_login_after_creating_roles_with_hashed_password
auth_test: coroutinize test_create_roles_with_hashed_password_and_log_in
auth_test: coroutinize test_try_create_role_with_hashed_password_as_anonymous_user
auth_test: coroutinize test_try_to_create_role_with_password_and_hashed_password
auth_test: coroutinize test_try_to_create_role_with_hashed_password_and_password
auth_test: coroutinize test_alter_with_workload_type
auth_test: coroutinize test_alter_with_timeouts
auth_test: coroutinize role_permissions_table_is_protected
auth_test: coroutinize role_members_table_is_protected
auth_test: coroutinize roles_table_is_protected
auth_test: coroutinize test_password_authenticator_operations
auth_test: coroutinize test_password_authenticator_attributes
auth_test: coroutinize test_default_authenticator
When an SSTable was encrypted with a KMS host that is not present in
scylla.yaml, the error thrown was:
std::invalid_argument (No such host: <host-name>)
This message is very obscure in general, and especially confusing when
encountered while using the scylla-sstable tool: it gives no indication
that the SSTable is encrypted, that a KMS host lookup is involved, or
what the user needs to do to fix the problem.
Replace it with a message that names the missing host and points
directly to the relevant scylla.yaml section:
Encryption host "<host-name>" is not defined in scylla.yaml.
Make sure it is listed under the "kmip_hosts" section.
The wording is intentionally kept neutral (not framed as an SSTable tool
problem) because the same code path is exercised by production ScyllaDB
when a node's configuration no longer contains a host referenced by an
existing data file (e.g. after a config rollback or when restoring data
from a different cluster). The production use-case takes precedence, but
the message is equally actionable from the tool.
Closesscylladb/scylladb#29228
start - end will result in negative length, rejected by the python
runtime. Use the correct end - start to calculate length.
Closesscylladb/scylladb#29249
data_dictionary::database was converted to replica::database in two
places, just to call find_keyspace(), then call
get_replication_strategy() on the returned keyspace. This is not
necessary, data_dictionary::database already has find_keyspace() and the
returned data_dictionary::keyspace also has get_replication_strategy().
This patch removes a small layering violation but more importantly, it
is necessary for the sstable tool to be able to load schemas from disk,
when said schema has tombstone_gc props.
Closesscylladb/scylladb#29279
This series updates the storage abstraction and extends the compaction tests to support object‑storage backends (S3 and GCS), while tightening several parts of the test environment.
The changes include:
- New exists/object_exists helpers across storage backends and clock fixes in the S3 client to make signature generation stable under test conditions.
- A new get_storage_for_tests accessor and adjustments to the test environment to avoid premature teardown of the sstable registry.
- Refactoring of compaction tests to remove direct sstable access, ensure proper schema setup, and avoid use of moved‑from objects.
- Extraction of test_env‑based logic into reusable functions and addition of S3/GCS variants of the compaction tests.
Not all tests were converted to be backend‑agnostic yet, and a few require further investigation before they can run cleanly against S3/GCS backends. These will be addressed in follow‑up work.
Fixes: https://scylladb.atlassian.net/browse/SCYLLADB-704 however, followup is needed
No backport needed since this change targeting future feature
Closesscylladb/scylladb#28790
* github.com:scylladb/scylladb:
compaction_test: fix formatting after previous patches
compaction_test: add S3/GCS variations to tests
compaction_test: extract test_env-based tests into functions
compaction_test: replace file_exists with storage::exists
compaction_test: initialize tables with schema via make_table_for_tests
compaction_test: use sstable APIs to manipulate component files
compaction_test: fix use-after-move issue
sstable_utils: add `get_storage` and `open_file` helpers
test_env: delay unplugging sstable registry
storage: add `exists` method to storage abstraction
s3_client: use lowres_system_clock for aws_sigv4
s3_client: add `object_exists` helper
gcs_client: add `object_exists` helper
View building worker was breaking semaphores without holding their locks.
This lead to races like SCYLLADB-844 and SCYLLADB-543,
where a new batch was started after `view_building_worker::state` was cleared in the `drain()` process.
This patch fix the race by:
- taking a lock of the mutex before breaking it
- distinguishing between `state::clear()`(can happen multiple times) and `state::drain()`(can be called only once during shutdown)
- asserting that the state is not doing any new work after it was drained
Fixes SCYLLADB-844
Fixes SCYLLADB-543
This PR should be backported to all versions containing view building coordinator (2025.4 and newer).
Closesscylladb/scylladb#29303
* github.com:scylladb/scylladb:
view_building_worker: extract starting a new batch to state's method
view_building_worker: distinguish between state's `clear()` and `drain()`
view_building_worker: lock mutexes before breaking them in `drain()`
view_building_worker: execute drain() once
The fix for SCYLLADB-1373 (b4f652b7c1) changed get_session() to use
the default timeout=30 for the retry loop in patient_*_cql_connection
(previously timeout=0.1). This correctly allowed retrying transient
NoHostAvailable errors during node startup, but introduced a new
flakiness in test_login and other auth tests.
The failure chain:
1. test_login connects with bad credentials (e.g. user="doesntexist")
2. get_session() calls patient_exclusive_cql_connection(), which calls
retry_till_success() with bypassed_exception=NoHostAvailable
3. The first attempt correctly fails: the server rejects the credentials
with AuthenticationFailed, wrapped in NoHostAvailable
4. retry_till_success() catches NoHostAvailable indiscriminately and
retries, not distinguishing between transient errors (node not ready)
and permanent errors (bad credentials)
5. A subsequent retry attempt times out (connect_timeout=5), producing
OperationTimedOut wrapped in NoHostAvailable
6. After 30 seconds, the last NoHostAvailable is raised -- now wrapping
OperationTimedOut instead of the original AuthenticationFailed
7. The assertion `isinstance(..., AuthenticationFailed)` fails
With the old timeout=0.1, the deadline was already exceeded after the
first attempt, so the original AuthenticationFailed propagated.
Fix: Add a `should_retry` predicate parameter to retry_till_success()
and use it in patient_cql_connection() and
patient_exclusive_cql_connection() to immediately re-raise
NoHostAvailable when it wraps AuthenticationFailed. Retrying
authentication failures is never useful since the credentials won't
change between attempts.
Fixes: SCYLLADB-1382
Closesscylladb/scylladb#29348
Following the previous commit, a new batch cannot be started if the
state was already drained.
This commit also adds a check that only one batch is running at a time.
While both of this methods do the same (abort current batch, clear
data), we can clear the state multiple times during view_building_worker
lifetime (for instance when processing base table is changed) but
`view_building_worker::state::drain()` should be called only once and
after this no other work on the state should be done.
Not doing this may lead to races like SCYLLADB-844.
If some consumer is holding a lock of a mutex and `drain()`
is just braking the mutex without locking it beforehand,
then the consumer may process its code which should be aborted.
An example of the race is SCYLLADB-844, where `work_on_tasks()` is
holding `_state._mutex` while it is broken by `drain()`.
This causes a new batch is started after the `_state` is cleared.
get_session() was passing timeout=0.1 to patient_exclusive_cql_connection
and patient_cql_connection, leaving only 0.1 seconds for the retry loop
in retry_till_success(). Since each connection attempt can take up to 5
seconds (connect_timeout=5), the retry loop effectively got only one
attempt with no chance to retry on transient NoHostAvailable errors.
Use the default timeout=30 seconds, consistent with all other callers.
Fixes: SCYLLADB-1373
Closesscylladb/scylladb#29332
