test: add read barrier to test_balance_empty_tablets

The test creates a single node cluster, then creates 3 tables which
remain empty. Then it adds another node with half the disk capacity of
the first one, and then it waits for the balancer to migrate tablets to
the newly added node by calling the quiesce topology API. The number of
tablets on the smaller node should be exactly half the number of tablets
on the larger node.

After waiting for quiesce topology, we could have a situation where we
query the number of tablets from the node which still hasn't processed
the last tablet migrations and updated system.tablets.

This patch adds a read barrier so that both nodes see the same tablets
metadata before we query the number of tablets.

Fixes: SCYLLADB-603

Closes scylladb/scylladb#28598

test/cluster/test_size_based_load_balancing.py

@@ -4,6 +4,7 @@
 # SPDX-License-Identifier: LicenseRef-ScyllaDB-Source-Available-1.0
 #
 from test.pylib.manager_client import ManagerClient
+from test.pylib.rest_client import read_barrier
 from test.cluster.util import new_test_keyspace
 from collections import defaultdict
 import pytest
@@ -54,6 +55,9 @@ async def test_balance_empty_tablets(manager: ManagerClient):
 
         await manager.api.quiesce_topology(servers[0].ip_addr)
 
+        # Ensure all nodes see the same data in system.tablets
+        await asyncio.gather(*[read_barrier(manager.api, s.ip_addr) for s in servers])
+
         replicas_per_node = defaultdict(int)
         tablets_per_shard = {}
         for row in await cql.run_async('SELECT * FROM system.tablets'):