mirror of
https://github.com/scylladb/scylladb.git
synced 2026-05-01 21:55:50 +00:00
e414b2b0b9
Six cluster test files override failure_detector_timeout_in_ms to 2000ms
for faster failure detection. In debug and sanitize builds, this causes
flaky node join failures. The following log analysis shows how.
The coordinator (server 614, IP 127.2.115.3) accepts the joining node
(server 615, host_id 53b01f0b, IP 127.2.115.2) into group0:
20:10:57,049 [shard 0] raft_group0 - server 614 entered
'join group0' transition state for 53b01f0b
The joining node begins receiving the raft snapshot 100ms later:
20:10:57,150 [shard 0] raft_group0 - transfer snapshot from 9fa48539
It then spends ~280ms applying schema changes -- creating 6 keyspaces
and 12+ tables from the snapshot:
20:10:57,511 [shard 0] migration_manager - Creating keyspace
system_auth_v2
...
20:10:57,788 [shard 0] migration_manager - Creating
system_auth_v2.role_members
Meanwhile, the coordinator's failure detector pings the joining node.
Under debug+ASan load the RPC call times out after ~4.6 seconds:
20:11:01,643 [shard 0] direct_failure_detector - unexpected exception
when pinging 53b01f0b: seastar::rpc::timeout_error
(rpc call timed out)
25ms later, the coordinator marks the joining node DOWN and removes it:
20:11:01,668 [shard 0] raft_group0 - failure_detector_loop:
Mark node 53b01f0b as DOWN
20:11:01,717 [shard 0] raft_group0 - bootstrap: failed to accept
53b01f0b
The joining node was still retrying the snapshot transfer at that point:
20:11:01,745 [shard 0] raft_group0 - transfer snapshot from 9fa48539
It then receives the ban notification and aborts:
20:11:01,844 [shard 0] raft_group0 - received notification of being
banned from the cluster
Replace the hardcoded 2000ms with the failure_detector_timeout fixture
from conftest.py, which scales by MODES_TIMEOUT_FACTOR: 3x for
debug/sanitize (6000ms), 2x for dev (4000ms), 1x for release (2000ms).
Test measurements (before -> after fix):
debug mode:
test_replace_with_same_ip_twice 24.02s -> 25.02s
test_banned_node_notification 217.22s -> 221.72s
test_kill_coordinator_during_op 116.11s -> 127.13s
test_node_failure_during_tablet_migration
[streaming-source] 183.25s -> 192.69s
test_replace (4 tests) skipped in debug (skip_in_debug)
test_raft_replace_ignore_nodes skipped in debug (run_in_dev only)
dev mode:
test_replace_different_ip 10.51s -> 11.50s
test_replace_different_ip_using_host_id 10.01s -> 12.01s
test_replace_reuse_ip 10.51s -> 12.03s
test_replace_reuse_ip_using_host_id 13.01s -> 12.01s
test_raft_replace_ignore_nodes 19.52s -> 19.52s
33 lines
1.2 KiB
Python
33 lines
1.2 KiB
Python
#
|
|
# Copyright (C) 2024-present ScyllaDB
|
|
#
|
|
# SPDX-License-Identifier: LicenseRef-ScyllaDB-Source-Available-1.1
|
|
#
|
|
from test.pylib.scylla_cluster import ReplaceConfig
|
|
from test.pylib.manager_client import ManagerClient
|
|
from test.pylib.internal_types import ServerInfo
|
|
import pytest
|
|
import logging
|
|
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_replace_with_same_ip_twice(manager: ManagerClient, failure_detector_timeout) -> None:
|
|
logger.info("starting a cluster with two nodes")
|
|
servers = await manager.servers_add(3, config={'failure_detector_timeout_in_ms': failure_detector_timeout})
|
|
logger.info(f"cluster started {servers}")
|
|
|
|
async def replace_with_same_ip(s: ServerInfo) -> ServerInfo:
|
|
logger.info(f"stopping server {s.server_id}")
|
|
await manager.server_stop_gracefully(s.server_id)
|
|
|
|
logger.info(f"replacing server {s.server_id} with same ip")
|
|
replace_cfg = ReplaceConfig(replaced_id = s.server_id, reuse_ip_addr = True, use_host_id = False)
|
|
return await manager.server_add(replace_cfg)
|
|
|
|
test_server = servers[1]
|
|
test_server = await replace_with_same_ip(test_server)
|
|
await replace_with_same_ip(test_server)
|