fed2c6ba83
data_consume_rows_context_m has a _column_value buffer it uses to read key and column values into, preparing for parsing and consuming them. This buffer is reset (released) in a few different cases: * When using it for key - after consuming its content * When using it for column value - when a colum has no value However, the buffer is not released when used for a column value and the column is consumed. This means that if a large column is read from the sstable, this buffer can potentially linger and keep consuming memory until either one of the other release scenarios is hit, or the reader is destroyed. Add a third release scenario, releasing the buffer after the row end was consumed. This allows the buffer to be re-used between columns of the same row, at the same time ensuring that a large buffer will not linger. This patch can almost halve the memory consumption of reads in certain circumstances. Point in case: the test test_reader_concurrency_semaphore_memory_limit_engages starts to fail after this fix, because the read doesn't trigger the OOM limit anymore and needs doubling of the concurrency to keep passing. This issue was found in a dtest (`test_ics_refresh_with_big_sstable_files`), which writes some large cells of up to 7MiB. After reading the row containing this large cell, the reader holds on to the 7MiB buffer causing the semaphore's OOM protection to kick in down the line. Fixes: https://github.com/scylladb/scylladb/issues/21160 Closes scylladb/scylladb#21132
Scylla in-source tests.
For details on how to run the tests, see docs/dev/testing.md
Shared C++ utils, libraries are in lib/, for Python - pylib/
alternator - Python tests which connect to a single server and use the DynamoDB API unit, boost, raft - unit tests in C++ cqlpy - Python tests which connect to a single server and use CQL topology* - tests that set up clusters and add/remove nodes cql - approval tests that use CQL and pre-recorded output rest_api - tests for Scylla REST API Port 9000 scylla-gdb - tests for scylla-gdb.py helper script nodetool - tests for C++ implementation of nodetool
If you can use an existing folder, consider adding your test to it. New folders should be used for new large categories/subsystems, or when the test environment is significantly different from some existing suite, e.g. you plan to start scylladb with different configuration, and you intend to add many tests and would like them to reuse an existing Scylla cluster (clusters can be reused for tests within the same folder).
To add a new folder, create a new directory, and then
copy & edit its suite.ini.