mirror of
https://github.com/scylladb/scylladb.git
synced 2026-04-20 16:40:35 +00:00
e8a01ef8c6
The view building coordinator manages the process by sending RPC requests to all nodes in the cluster, instructing them what to do. If processing that message fails, the coordinator decides if it wants to retry it or (temporarily) abandon the work. An example of the latter scenario could be if one of the target nodes dies and any attempts to communicate with it would fail. Unfortunately, the current approach to it is not perfect and may result in a storm of warnings, effectively clogging the logs. As an example, take a look at scylladb/scylladb#26686: the gossiper failed to mark one of the dead nodes as DOWN fast enough, and it resulted in a warning storm. To prevent situations like that, we implement a form of backoff. If processing an RPC message fails, we postpone finishing the task for a second. That should reduce the number of messages in the logs and avoid retries that are likely to fail as well. We provide a reproducer test. Fixes scylladb/scylladb#26686 Backport: impact on the user. We should backport it to 2025.4. - (cherry picked from commit4a5b1ab40a) - (cherry picked from commitacd9120181) - (cherry picked from commit393f1ca6e6) Parent PR: #26729 Closes scylladb/scylladb#27027 * github.com:scylladb/scylladb: tet/cluster/mv: Clean up test_backoff_when_node_fails_task_rpc db/view/view_building_coordinator: Rate limit logging failed RPC db/view: Add backoff when RPC fails