Enable debug logging for "exception" logger inside protocol exception tests.
The exceptions will be logged, and it will be possible to see which ones
occured if a protocol exceptions test fails.
Refs #27272
Refs #27325
The initial problem:
Some of the tests in test_protocol_exceptions.py started failing. The failure is
on the condition that no more than `cpp_exception_threshold` happened.
Test logic:
These tests assert that specific code paths do not throw an exception anymore.
Initial implementation ran a code path once, and asserted there were 0 exceptions.
Sometimes an exception or several can occur, not directly related to the code paths
the tests check, but those would fail the tests.
The solution was to run the tests multiple times. If there is a regression, there
would be at least as many exceptions thrown as there are test runs. If there is no
regression, a few exceptions might happen, up to 10 per 100 test runs.
I have arbitrarily chosen `run_count = 100` and `cpp_exception_threshold = 10` values.
Note that the exceptions are counted per shard, not per code path.
The new problem:
The occassional exceptions thrown by some parts of the server now throw a bit more
than before. Based on the logs linked on the issues, it is usually 12.
There are possibly multiple ways to resolve the issue. I have considered logging
exceptions and parsing them. I would have to filter exception logs only for wanted
exceptions. However, if a new, different exception is introduced, it might not be
counted.
Another approach is to just increase the threshold a bit. The issue of throwing
more exceptions than before in some other server modules should be addressed by
a set of tests for that module, just like these tests check protocol exceptions,
not caring who used protocol check code paths.
For those reasons, the solution implemented here is to increase `cpp_exception_threshold`
to `20`. It will not make the tests unreliable, because, as mentioned, if there is a
regression, there would be at least `run_count` exceptions per `run_count` test runs
(1 exception per single test run).
Still, to make "background exceptions" occurence a bit more normalized, `run_count` too
is doubled, from `100` to `200`. At the first glance this looks like nothing is changed,
but actually doubling both run count and exception threshold here implies that the
exception burst does not scale as much as run count, it is just that the "jitter" is
bigger than the old threshold.
Fixes#27247Fixes#27325
Add protocol exception tests that check errors and exceptions.
`test_process_startup_invalid_string_map`: `STARTUP` (0x01) with declared
map count, but missing entries - `read_string_map` out-of-range.
`test_process_query_internal_malformed_query`: `QUERY` (0x07) long string
declared larger than available bytes - `read_long_string_view`.
`test_process_query_internal_fail_read_options`: `QUERY` (0x07) with
`PAGE_SIZE` flag, but truncated page_size - `read_options` path.
`test_process_prepare_malformed_query`: `PREPARE` (0x09) long string declared
larger than available bytes - `read_long_string_view` in prepare.
`test_process_execute_internal_malformed_cache_key`: `EXECUTE` (0x0A) cache key
short bytes declared larger than provided bytes - `read_short_bytes`.
`test_process_register_malformed_string_list`: `REGISTER` (0x0B) string list
with truncated element - `read_string_list`/`read_string`.
Each test asserts an `ERROR` frame is returned and `protocol_error` metrics
increase, without causing C++ exceptions.
Refs: #24567
Frame building is repetitive and increases verbosity, reducing code readability.
This patch solves it by extracting common functionality of frame building into
`_build_frame`. Also, helpers `_send_frame` and `_recv_frame` are introduced.
While `_recv_frame` is not really useful, it goes well in pair with `_send_frame`.
Refs: #24567
The code that measures errors and exceptions in `test_protocol_exceptions.py` tests
is repetitive. This patch refactors common functionality in a separate `_test_impl`
function, improving readability.
Refs: #24567
Add `test_unknown_compression_algorithm` test case to `test_protocol_exceptions.py` test suite.
This change improves test coverage for zero throws protocol exception handling.
Refs: #24567
Tested code paths should not throw exceptions. `scylla_reactor_cpp_exceptions`
metric is used. This is a global metric. To address potential test flakiness,
each test runs multiple times:
- `run_count = 100`
- `cpp_exception_threshold = 10`
If a change in the code introduced an exception, expectation is that the number
of registered exceptions will be > `cpp_exception_threshold` in `run_count` runs.
In which case the test fails.
Add a helper to fetch scylla_transport_cql_errors_total{type="protocol_error"} counter
from Scylla's metrics endpoint. These metrics are used to track protocol error
count before and after each test.
Add cql_with_protocol context manager utility for session creation with parameterized
protocol_version value. This is used for testing connection establishment with
different protocol versions, and proper disposal of successfully established sessions.
The tests cover two failure scenarios:
- Protocol version mismatch in test_protocol_version_mismatch which tests both supported
and unsupported protocol version
- Malformed frames via raw socket in _protocol_error_impl, used by several test functions,
and also test_no_protocol_exceptions test to assert that the error counters never decrease
during test execution, catching unintended metric resets
Refs: #24567
