ebdfa10c8f
The test waited for two "Finished tablet repair" log messages on the
coordinator, expecting one per tablet. But there are two log sources
that emit messages matching this pattern:
repair module (repair/repair.cc:2329):
"Finished tablet repair for table=..."
topology coordinator (topology_coordinator.cc:2083):
"Finished tablet repair host=..."
When the coordinator is also a repair replica (always the case with
RF=3 and 3 nodes), both messages appear in the coordinator log for the
same tablet within 1ms of each other. The test consumed both, thinking
both tablets were done, while the second tablet repair was still running.
From the CI failure logs:
04:08:09.658 Found: repair[...]: Finished tablet repair for table=...
global_tablet_id=e42fd650-3542-11f1-9756-85403784a622:0
04:08:09.660 Found: raft_topology - Finished tablet repair host=...
tablet=e42fd650-3542-11f1-9756-85403784a622:0
Both messages are for tablet :0. Tablet :1 repair had not finished yet.
The test then wrote keys 20-29 while the second tablet repair was still
in progress. That repair flushed the memtable (via
prepare_sstables_for_incremental_repair), including keys 20-29 in the
repair scan, and mark_sstable_as_repaired set repaired_at=2 on the
resulting sstable. This caused the assertion failure on servers[0]:
"should not have post-repair keys in repaired sstables, got:
{20, 21, 22, 23, 24, 25, 26, 27, 28, 29}"
Fix by matching "Finished tablet repair host=" which is unique to the
topology coordinator message and avoids the ambiguity.
Also fix an incorrect comment that said being_repaired=null when at that
point in the test being_repaired is still set to the session_id (the
delay_end_repair_update injection prevents end_repair from running).
Fixes: SCYLLADB-1478
Closes scylladb/scylladb#29444
Scylla in-source tests.
For details on how to run the tests, see docs/dev/testing.md
Shared C++ utils, libraries are in lib/, for Python - pylib/
alternator - Python tests which connect to a single server and use the DynamoDB API unit, boost, raft - unit tests in C++ cqlpy - Python tests which connect to a single server and use CQL topology* - tests that set up clusters and add/remove nodes cql - approval tests that use CQL and pre-recorded output rest_api - tests for Scylla REST API Port 9000 scylla-gdb - tests for scylla-gdb.py helper script nodetool - tests for C++ implementation of nodetool
If you can use an existing folder, consider adding your test to it. New folders should be used for new large categories/subsystems, or when the test environment is significantly different from some existing suite, e.g. you plan to start scylladb with different configuration, and you intend to add many tests and would like them to reuse an existing Scylla cluster (clusters can be reused for tests within the same folder).
To add a new folder, create a new directory, and then
copy & edit its suite.ini.