Gleb Natapov cbf4e27137 raft_topology: log sub-step progress in local_topology_barrier ...

When a node processes a barrier_and_drain topology command, it performs
two potentially long-running operations inside local_topology_barrier():
waiting for stale token metadata versions to be released
(stale_versions_in_use) and draining closing sessions
(drain_closing_sessions). Either of these can hang indefinitely -- for
example, stale_versions_in_use blocks until all references to previous
token metadata versions are released, which depends on in-flight
requests completing.

Previously, the only logging was a single 'done' message at the end,
making it impossible to determine which sub-step was blocking when a
barrier_and_drain RPC appeared stuck on a node. In a recent CI failure,
a node never responded to barrier_and_drain during a removenode
operation, and the logs showed the RPC was received but nothing about
what it was waiting on internally.

Add info-level logging before each blocking sub-step, including the
topology version for correlation. This allows diagnosing hangs by
showing whether the node is stuck waiting for stale metadata versions,
stuck draining sessions, or never reached these steps at all.

(cherry picked from commit e88ce09372)

2026-05-10 12:12:13 +03:00

..

auth_cluster

Merge 'service_levels: mark v2 migration complete on empty legacy table' from Alex Dathskovsky

2026-04-06 17:51:34 +03:00

dtest

test/alternator: stop concurrent-requests test when workers hit limit

2026-04-24 18:03:10 +03:00

lwt

tests(lwt): new test for LWT testing during tablet resize

2025-11-04 12:47:24 +01:00

mv

test/mv/test_mv_staging: wait for cql after restart

2026-04-30 08:58:09 +03:00

object_store

streaming: fix loop break condition in tablet_sstable_streamer::stream

2025-11-25 11:42:34 +02:00

random_failures

db/view: Require rf_rack_valid_keyspaces when creating view

2025-10-06 13:19:54 +00:00

tasks

Merge 'service: tasks: return successful status if a table was dropped' from Aleksandra Martyniuk

2026-04-24 18:06:09 +03:00

__init__.py

…

conftest.py

pylib: extract upgrade helpers from test_sstable_compression_dictionaries_upgrade.py

2025-09-15 12:34:45 +02:00

suite.yaml

test: dtest: limits_test.py: make the tests work

2025-10-01 22:40:29 +02:00

test_aggregation.py

test/pylib: servers_add: add auto_rack_dc parameter

2025-03-30 19:23:40 +03:00

test_alternator.py

raft_topology: log sub-step progress in local_topology_barrier

2026-05-10 12:12:13 +03:00

test_automatic_cleanup.py

test/pylib: servers_add: add auto_rack_dc parameter

2025-03-30 19:23:40 +03:00

test_bad_initial_token.py

test: cluster: add test_bad_initial_token

2025-04-25 12:25:15 +02:00

test_batchlog_manager.py

test: extend test_batchlog_replay_failure_during_repair

2025-12-22 14:45:08 +01:00

test_blocked_bootstrap.py

…

test_boot_after_ip_change.py

db/config: add tablets_mode_for_new_keyspaces option

2025-03-24 14:54:45 +02:00

test_boot_nodes.py

test: Add test_boot_nodes.py

2025-07-10 10:56:53 +08:00

test_bootstrap_with_quick_group0_join.py

raft_group0: join_group0: fix join hang when node joins group 0 before post_server_start

2026-04-09 15:53:43 +02:00

test_bti_index.py

test/cluster/test_bti_index.py: avoid a race with CQL tracing

2025-10-20 10:32:58 +03:00

test_cdc_generation_clearing.py

test_cdc_generation_clearing: wait for generations to propagate

2025-06-09 12:59:04 +02:00

test_cdc_generation_data.py

raft_group0: split shutdown into abort_and_drain and destroy

2025-07-25 17:16:14 +02:00

test_cdc_generation_publishing.py

test_cdc_generation_publishing: fix to read monotonically

2025-05-30 08:35:56 +02:00

test_cdc_with_alter.py

test: test concurrent writes with column drop with cdc preimage

2025-11-16 09:29:27 +01:00

test_cdc_with_tablets.py

test: cdc: extend cdc with tablets tests

2025-10-30 02:44:47 +00:00

test_change_ip.py

test/cluster: Adjust simple tests to RF-rack-validity

2025-05-10 16:30:18 +02:00

test_change_replication_factor_1_to_0.py

test: cluster: deflake consistency checks after decommission

2025-09-09 19:01:12 +02:00

test_change_rpc_address.py

test/cluster: Adjust simple tests to RF-rack-validity

2025-05-10 16:30:18 +02:00

test_cluster_features.py

test/pylib: servers_add: add auto_rack_dc parameter

2025-03-30 19:23:40 +03:00

test_commitlog_segment_data_resurrection.py

…

test_commitlog.py

…

test_compaction_backpressure.py

compaction_manager: fix maybe_wait_for_sstable_count_reduction() hanging forever

2026-02-27 01:39:05 +02:00

test_concurrent_schema.py

test/pylib: servers_add: add auto_rack_dc parameter

2025-03-30 19:23:40 +03:00

test_config_live_updates.py

test: add test for live updates of generic server config

2025-06-23 17:56:26 +02:00

test_config.py

…

test_conflicting_keys_read_repair.py

test/cluster: Adjust simple tests to RF-rack-validity

2025-05-10 16:30:18 +02:00

test_coordinator_queue_management.py

test.py: rework log_browsing for dtest migration

2025-05-19 11:50:55 +00:00

test_crash_coordinator_before_streaming.py

…

test_create_table_during_node_shutdown.py

migration_listener: fix deadlock in nested notifications

2026-02-18 12:47:30 +02:00

test_data_resurrection_after_cleanup.py

test: cluster: deflake consistency checks after decommission

2025-09-09 19:01:12 +02:00

test_data_resurrection_in_memtable.py

test/cluster/test_data_resurrection_in_memtable.py: use explicit CL

2026-04-30 21:46:28 +03:00

test_decommission_kill_then_replace.py

topology coordinator: complete pending operation for a replaced node

2026-01-19 09:42:20 +02:00

test_decommission.py

test: cluster: deflake consistency checks after decommission

2025-09-09 19:01:12 +02:00

test_deprecating_cluster_features.py

…

test_describe.py

cql3: Represent create_statement using managed_string

2025-07-01 12:58:02 +02:00

test_different_group0_ids.py

test.py: rewrite the wait_for_first_completed

2025-10-22 18:12:52 +02:00

test_encryption.py

db/config: add tablets_mode_for_new_keyspaces option

2025-03-24 14:54:45 +02:00

test_error_becoming_voter.py

…

test_fencing.py

test_fencing: add test_lwt_fencing_upgrade

2025-09-15 12:34:45 +02:00

test_global_ignore_nodes.py

test/cluster: Adjust simple tests to RF-rack-validity

2025-05-10 16:30:18 +02:00

test_gossip_boot.py

db/config: add tablets_mode_for_new_keyspaces option

2025-03-24 14:54:45 +02:00

test_gossiper_empty_self_id_on_shadow_round.py

gossiper: fix empty initial local node state

2025-09-08 11:38:31 +02:00

test_gossiper_orphan_remover.py

test: test_gossiper_orphan_remover: get host ID of the bootstrapping node before it crashes

2026-01-29 11:27:37 +01:00

test_gossiper_race.py

gossiper: check for a race condition in do_apply_state_locally

2025-09-08 11:38:30 +02:00

test_gossiper.py

test/pylib: servers_add: add auto_rack_dc parameter

2025-03-30 19:23:40 +03:00

test_group0_schema_versioning.py

test: test_group0_schema_versioning: wait for schema sync in system.local

2026-01-19 16:35:49 +01:00

test_hints.py

test/cluster: fix flaky test_hints_consistency_during_replace

2026-04-25 18:45:02 +03:00

test_incremental_repair.py

test/cluster/test_incremental_repair: fix flaky do_tablet_incremental_repair_and_ops

2026-04-25 18:43:47 +03:00

test_initial_token.py

test/cluster/conftest: cluster_con: provide default values for port and use_ssl

2025-08-22 09:51:24 +03:00

test_ip_mappings.py

test: test_full_shutdown_during_replace: retry replace after the replacing node is removed from gossip

2026-03-10 16:48:05 +01:00

test_keyspace_rf.py

test/cqlpy: add keyspace creation default replication factor tests

2025-08-28 01:42:34 +02:00

test_left_node_notification.py

raft topology: Notify that a node was removed only once

2025-12-30 11:17:41 +01:00

test_long_join.py

test: improve async execution in test_long_join

2025-09-08 17:14:37 +02:00

test_long_query_timeout_erm.py

test.py: rewrite the wait_for_first_completed

2025-10-22 18:12:52 +02:00

test_lwt_semaphore.py

…

test_maintenance_mode.py

test: test_maintenance_mode: enable maintenance mode properly

2026-02-03 11:33:53 +01:00

test_major_compaction.py

compaction: fix use after free when strategy is altered during compaction

2025-10-21 00:59:33 +00:00

test_metadata_id.py

test/cluster/conftest: cluster_con: provide default values for port and use_ssl

2025-08-22 09:51:24 +03:00

test_multidc.py

Merge 'test: cluster: Deflake test_startup_with_keyspaces_violating_rf_rack_valid_keyspaces' from Dawid Mędrek

2026-05-06 10:10:47 +02:00

test_mutation_schema_change.py

test/pylib: servers_add: add auto_rack_dc parameter

2025-03-30 19:23:40 +03:00

test_mv.py

tombstone_gc: don't use 'repair' mode for colocated tables

2025-11-26 08:36:52 +01:00

test_no_dc_rack_change.py

test: cluster: introduce test_no_dc_rack_change

2025-04-17 16:22:58 +02:00

test_no_removed_node_event_on_ip_change.py

test/cluster/conftest: cluster_con: provide default values for port and use_ssl

2025-08-22 09:51:24 +03:00

test_node_isolation.py

tiering (test.py): introduce tiering labels

2025-08-04 15:38:16 +03:00

test_node_ops_metrics.py

test/pylib/rest_client: fix ScyllaMetrics filtering

2025-08-10 10:16:00 +02:00

test_node_shutdown_waits_for_pending_requests.py

…

test_nodetool.py

…

test_not_enough_token_owners.py

test: cluster: Fix NoHostAvailable error in test_not_enough_token_owners

2026-01-09 19:10:11 +01:00

test_prepare_race.py

Merge 'cql3: pin prepared cache entry in prepare() to avoid invalid weak handle race' from Alex Dathskovsky

2026-04-20 12:59:53 +02:00

test_query_rebounce.py

…

test_raft_cluster_features.py

test/pylib: servers_add: add auto_rack_dc parameter

2025-03-30 19:23:40 +03:00

test_raft_fix_broken_snapshot.py

db/config: add tablets_mode_for_new_keyspaces option

2025-03-24 14:54:45 +02:00

test_raft_ignore_nodes.py

…

test_raft_no_quorum.py

test: test_raft_no_quorum: decrease group0_raft_op_timeout_in_ms after quorum loss

2026-03-18 10:15:34 +01:00

test_raft_recovery_basic.py

db/config: add tablets_mode_for_new_keyspaces option

2025-03-24 14:54:45 +02:00

test_raft_recovery_during_join.py

test: test_raft_recovery_during_join: get host ID of the bootstrapping node before it crashes

2026-01-22 18:19:36 +01:00

test_raft_recovery_entry_loss.py

test: test_raft_recovery_entry_loss: fix the typo in the test case name

2025-10-17 10:27:33 +00:00

test_raft_recovery_majority_loss.py

db/config: add tablets_mode_for_new_keyspaces option

2025-03-24 14:54:45 +02:00

test_raft_recovery_stuck.py

test: test_raft_recovery_stuck: ensure mutual visibility before using driver

2025-11-20 10:36:54 +02:00

test_raft_recovery_user_data.py

test: deflake driver reconnections in the recovery procedure tests

2025-09-22 17:21:06 +02:00

test_raft_snapshot_request.py

…

test_raft_snapshot_truncation.py

…

test_raft_voters.py

test/cluster/conftest: cluster_con: provide default values for port and use_ssl

2025-08-22 09:51:24 +03:00

test_random_tables.py

test/pylib: servers_add: add auto_rack_dc parameter

2025-03-30 19:23:40 +03:00

test_read_repair.py

test/cluster/test_read_repair: write 100 rows in trace test

2025-06-27 16:23:08 +03:00

test_refresh.py

Add nodetool refresh --scope option

2025-05-29 16:12:09 +03:00

test_remove_alive_node.py

…

test_remove_rpc_client_with_pending_requests.py

test/cluster: Adjust simple tests to RF-rack-validity

2025-05-10 16:30:18 +02:00

test_repair.py

repair: Allow min max range to be updated for repair history

2026-01-08 16:39:10 +02:00

test_replace_alive_node.py

test/pylib: servers_add: add auto_rack_dc parameter

2025-03-30 19:23:40 +03:00

test_replace_ignore_nodes.py

db/config: add tablets_mode_for_new_keyspaces option

2025-03-24 14:54:45 +02:00

test_replace_with_encryption.py

…

test_replace_with_same_ip_twice.py

…

test_replace.py

test.py: rework log_browsing for dtest migration

2025-05-19 11:50:55 +00:00

test_restart_cluster.py

…

test_resurrection.py

test: port of test and reproducer for resurrection during file based streaming

2025-03-30 13:39:40 +03:00

test_reversed_queries_during_simulated_upgrade_process.py

test/cluster: Adjust simple tests to RF-rack-validity

2025-05-10 16:30:18 +02:00

test_rpc_compression.py

test/cluster: Adjust simple tests to RF-rack-validity

2025-05-10 16:30:18 +02:00

test_select_from_mutation_fragments.py

test/cluster: Adjust simple tests to RF-rack-validity

2025-05-10 16:30:18 +02:00

test_shutdown_hang.py

…

test_snapshot.py

test: add type creation to test_snapshot

2025-07-10 10:46:55 +02:00

test_sstable_cleanup_stop.py

test/cluster: fix flaky test_cleanup_stop by using asyncio.sleep

2026-04-07 14:22:27 +03:00

test_sstable_compression_config.py

schema: Add initializer for compression defaults

2026-01-28 12:42:10 +02:00

test_sstable_compression_dictionaries_autotrain.py

test: fix configuration of test_autoretrain_dict

2026-02-26 09:55:57 +02:00

test_sstable_compression_dictionaries_basic.py

db/config: Deprecate sstable_compression_dictionaries_allow_in_ddl

2025-11-04 15:40:46 +02:00

test_sstable_compression_dictionaries_upgrade.py

test: add a missing reconnect_driver in test_sstable_compression_dictionaries_upgrade.py

2026-04-16 10:56:59 +03:00

test_sstable_set.py

test: Verify partitioned set store split and unsplit correctly

2025-04-29 15:47:33 -03:00

test_start_bootstrapped_with_invalid_seed.py

test: disable test_start_bootstrapped_with_invalid_seed

2026-01-15 17:01:30 +02:00

test_streaming_deadlock.py

test: limit test_streaming_deadlock_removenode concurrency

2025-09-19 12:50:20 +03:00

test_table_desc_read_barrier.py

…

test_table_drop.py

sstables_loader: prevent use-after-free on table drop during streaming

2026-04-24 10:33:51 +02:00

test_tablet_repair_scheduler.py

repair: Add tablet repair progress report support

2026-01-19 09:39:13 +02:00

test_tablet_stats.py

topology_coordinator: Make tablet_load_stats_refresh_interval configurable

2025-07-31 14:31:55 +03:00

test_tablets2.py

test: add test and reproducer for load_stats refresh exception

2026-02-02 16:35:08 +01:00

test_tablets_colocation.py

test: fix test flakiness in test_colocated_tables_gc_mode

2025-12-19 17:32:12 +01:00

test_tablets_cql.py

test/cluster: Adjust simple tests to RF-rack-validity

2025-05-10 16:30:18 +02:00

test_tablets_intranode.py

…

test_tablets_lwt.py

raft_topology: log sub-step progress in local_topology_barrier

2026-05-10 12:12:13 +03:00

test_tablets_merge.py

test_tablets_merge: test_tablet_split_merge_with_many_tables: reduce number of tables in debug mode

2025-09-29 15:30:13 +03:00

test_tablets_migration.py

test: test_restart_leaving_replica_during_cleanup: reconnect driver after restart

2026-02-18 12:43:31 +02:00

test_tablets_removenode.py

test/cluster: Disable rf_rack_valid_keyspaces in problematic tests

2025-05-10 16:30:49 +02:00

test_tablets.py

test: Verify that repair doesn't block disabling of tablet load balancing

2026-02-02 21:26:18 +01:00

test_tls.py

test/cluster: Adjust simple tests to RF-rack-validity

2025-05-10 16:30:18 +02:00

test_tombstone_gc.py

test: test group0 tombstone GC in the Raft-based recovery procedure

2025-10-22 17:13:34 +00:00

test_topology_failure_recovery.py

db/config: add tablets_mode_for_new_keyspaces option

2025-03-24 14:54:45 +02:00

test_topology_ops_encrypted.py

test: cluster: deflake consistency checks after decommission

2025-09-09 19:01:12 +02:00

test_topology_ops.py

test: cluster: deflake consistency checks after decommission

2025-09-09 19:01:12 +02:00

test_topology_recovery_basic.py

test.py: apply the nightly label on test_topology_recovery_basic

2025-09-01 14:16:29 +02:00

test_topology_recovery_majority_loss.py

test/cluster/conftest: cluster_con: provide default values for port and use_ssl

2025-08-22 09:51:24 +03:00

test_topology_rejoin.py

test/pylib: servers_add: add auto_rack_dc parameter

2025-03-30 19:23:40 +03:00

test_topology_remove_decom.py

raft_topology: Modify the conditional logic in remove node operation to enhance concurrency for raft enabled clusters.

2025-09-17 15:23:32 +05:30

test_topology_remove_garbage_group0.py

test: test_remove_garbage_group0_members: wait for token ring and group0 consistency before removenode

2026-03-24 16:09:02 +01:00

test_topology_schema.py

test/pylib: servers_add: add auto_rack_dc parameter

2025-03-30 19:23:40 +03:00

test_topology_smp.py

test/cluster: Adjust simple tests to RF-rack-validity

2025-05-10 16:30:18 +02:00

test_topology_upgrade_not_stuck_after_recent_removal.py

db/config: add tablets_mode_for_new_keyspaces option

2025-03-24 14:54:45 +02:00

test_topology_upgrade_stuck.py

test.py: rewrite the wait_for_first_completed

2025-10-22 18:12:52 +02:00

test_topology_upgrade.py

db/config: add tablets_mode_for_new_keyspaces option

2025-03-24 14:54:45 +02:00

test_truncate_concurrent_writes.py

truncate: add test for truncate with concurrent writes

2025-08-05 13:54:14 +02:00

test_truncate_with_drop.py

system_keyspace: Prune dropped tables from truncation on start/drop

2025-09-03 07:25:34 +03:00

test_truncate_with_tablets.py

Merge 'table: don't create new split compaction groups if main compaction group is disabled' from Ferenc Szili

2026-04-29 18:46:43 +02:00

test_unfinished_writes_during_shutdown.py

storage_service: Cancel all write requests on storage_proxy shutdown

2025-07-22 15:03:30 +02:00

test_vector_store.py

index: allow vector indexes without rf_rack_valid_keyspces

2025-12-05 20:13:02 +01:00

test_view_build_status.py

test/cluster: add view build status tests

2025-08-27 10:23:04 +02:00

test_view_building_coordinator.py

test/cluster/test_view_building_coordinator: fix flakiness in test_file_streaming

2026-01-08 16:42:08 +02:00

test_write_query_during_cql_server_shutdown.py

generic_server: Two-step connection shutdown.

2025-07-28 10:08:06 +02:00

test_writes_to_previous_cdc_generations.py

…

test_zero_token_nodes_multidc.py

test: test_zero_token_nodes_multidc: properly handle reads with CL=LOCAL_ONE

2026-01-22 18:22:05 +01:00

test_zero_token_nodes_no_replication.py

test/cluster/conftest: cluster_con: provide default values for port and use_ssl

2025-08-22 09:51:24 +03:00

test_zero_token_nodes_topology_ops.py

test/cluster/test_zero_token_nodes_topology_ops: Adjust to RF-rack-validity

2025-05-10 16:30:34 +02:00

util.py

test: use DROP KEYSPACE IF EXISTS in new_test_keyspace cleanup

2026-04-25 18:44:39 +03:00