a3360ee385
Symptom: the rest_api_mock subprocess exits with status 1 during fixture
setup, e.g.:
subprocess.CalledProcessError: Command '[..., 'rest_api_mock.py',
'127.29.88.1', '34093']' returned non-zero exit status 1
Root cause: aiohttp's TCPSite.start() raises OSError(EADDRINUSE) and the
process exits 1. The bind fails because of how the (ip, port) pair is
chosen across modules within one test.py process:
* Each test module leases a 127.x.y.z IP from the host registry. The
registry recycles released IPs, so the same IP is shared across
modules sequentially.
* The original code picked the port via random.randint(10000, 65535).
A previous module on the same IP could have left that port in
TIME_WAIT (or worse, still actively in use) when a later module
happened to pick the same port.
SCYLLADB-1275 (PR 29314) tried to fix this by binding a probe socket to
(ip, 0) to obtain an OS-assigned free port, closing the probe, then
launching the mock server which would bind to that port. Two issues
remained:
1. TOCTOU: between probe close and mock-server bind, any other process
on the host could grab the just-freed port.
2. TIME_WAIT could still bite if the host registry recycled an IP and
the OS reused the same port number for the probe.
Fix: drop port discovery entirely. Use a fixed port (12345, matching the
unshare-namespace path already in this fixture) on the unique IP from
the host registry. Because IPs are unique per test module within one
test.py process, the (ip, 12345) pair is unique to each module, so no
port-collision dance is needed.
reuse_address=True on TCPSite handles the residual TIME_WAIT case when
the host registry recycles an IP within the same test.py process and
the previous mock server's socket has not finished TIME_WAIT yet.
reuse_port=True is dropped, as it was only useful while attempting to
have multiple processes share a single port.
This mirrors the design used in test/cqlpy/run.py: pick a unique IP,
keep the port fixed.
Fixes: SCYLLADB-1718
Closes scylladb/scylladb#29656
Front-end tests for nodetool
The tests in this directory exercise the nodetool client itself, mocking the API backend. This allows for testing all combinations of all supported options, and still keeping the tests quick.
The tests can be run against both the scylla-native nodetool (default), or the inherited, C*-based nodetool.
Run all tests against the scylla-native nodetool:
pytest --nodetool=scylla .
You can specify the path to the scylla binary with the --nodetool-path option. By default the tests will pick up the ScyllaDB executable, that is appropriate for the --mode option (defaults to dev).
Run all tests against the C* nodetool:
pytest --nodetool=cassandra .
Again, you can specify the path to the nodetool binary with --nodetool-path option. By default, <scylladb.git>/tools/java/bin/nodetool will be used.
When running the test against the java-nodeotol, you can specify the path to JMX with --jmx-path option. By default, <scylladb.git>/tools/jmx/scripts/scylla-jmx will be used.
If you add new tests, make sure to run all tess against both nodetool implementations, to avoid regressions. Note that CI/promotion will only run the tests against the scylla-native nodetool.