mirrors/scylladb
1
0
Fork 0
mirror of https://github.com/scylladb/scylladb.git synced 2026-05-21 23:32:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

storage_service: Cancel all write requests on storage_proxy shutdown

Browse Source 
During a graceful node shutdown, RPC listeners are stopped in `storage_service::drain_on_shutdown`
as one of the first steps. However, even after RPCs are shut down, some write handlers in
`storage_proxy` may still be waiting for background writes to complete. These handlers retain the ERM.
Since the RPC subsystem is no longer active, replies cannot be received, and if any RPC commands are
concurrently executing `barrier_and_drain`, they may get stuck waiting for those writes. This can block
the messaging server shutdown and delay the entire shutdown process until the write timeout occurs.

This change introduces the cancellation of all outstanding write handlers in `storage_proxy`
during shutdown to prevent unnecessary delays.

Fixes scylladb/scylladb#23665
This commit is contained in:
Sergey Zolotukhin
2025-06-27 19:12:55 +02:00
parent bc934827bc
commit e0dc73f52a
3 changed files with 12 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
service/storage_proxy.cc
View File

@@ -6955,7 +6955,7 @@ future<> storage_proxy::drain_on_shutdown() {
//NOTE: the thread is spawned here because there are delicate lifetime issues to consider
// and writing them down with plain futures is error-prone.
return async([this] {
cancel_write_handlers([] (const abstract_write_response_handler&) { return true; });
cancel_all_write_response_handlers().get();
_hints_resource_manager.stop().get();
});
}
@@ -6979,4 +6979,13 @@ future<utils::chunked_vector<dht::token_range_endpoints>> storage_proxy::describ
return locator::describe_ring(_db.local(), _remote->gossiper(), keyspace, include_only_local_dc);
}
future<> storage_proxy::cancel_all_write_response_handlers() {
while (!_response_handlers.empty()) {
_response_handlers.begin()->second->timeout_cb();
if (!_response_handlers.empty()) {
co_await maybe_yield();
}
}
}
}

2
service/storage_proxy.hh
View File

@@ -521,6 +521,8 @@ public:
bool is_me(gms::inet_address addr) const noexcept;
bool is_me(const locator::effective_replication_map& erm, locator::host_id id) const noexcept;
future<> cancel_all_write_response_handlers();
private:
bool only_me(const locator::effective_replication_map& erm, const host_id_vector_replica_set& replicas) const noexcept;

1
test/cluster/test_unfinished_writes_during_shutdown.py
View File

@@ -24,7 +24,6 @@ from test.cluster.conftest import skip_mode
logger = logging.getLogger(__name__)
@pytest.mark.asyncio
@pytest.mark.xfail(reason="#23665")
@skip_mode('release', 'error injections are not supported in release mode')
async def test_unfinished_writes_during_shutdown(request: pytest.FixtureRequest, manager: ManagerClient) -> None:
""" Test a simultaneous topology change and write query during shutdown, which may cause the node to get stuck (see https://github.com/scylladb/scylladb/issues/23665).