f55dc71c3f
With commitsed7d352e7dandbb1867c7c7, we now have input streams for both compressed and uncompressed SSTables that provide seamless checksum and digest checking. The code for these was based on `validate_checksums()`, which implements its own validation logic over raw streams. This has led to some duplicate code. This PR deduplicates the uncompressed case by modifying `validate_checksums()` to use a checksummed input stream instead of a raw stream. The same cannot be done for compressed SSTables though. The reason is that `validate_checksums()` needs to examine the whole data file, even if an invalid chunk is encountered. In the checksummed case we support that by offloading the error handling logic from the data source via a function parameter. In the compressed data source we cannot do that because it needs to return decompressed data and decompression may fail if the data are invalid. This PR also enables `validate_checksums()` to partially verify SSTables with just the per-chunk checksums if the digest is missing. In more detail, this PR consists of: * Port of some integrity checks from `do_validate_uncompressed()` to the checksummed data source. It should now be able to detect corruption due to truncated or appended chunks (expected number of chunks is retrieved from the CRC component). * Introduction of `error_handler` parameter in checksummed data source and `data_stream()`. * Refactoring of `validate_checksums()`. The JSON response of `sstable validate-checksums` was also modified to report a missing digest. * Tests for `validate_checksums()` against SSTables with truncated data, appended data, invalid digests, or no digest. Refs #19058. This PR is a hybrid of cleanup and feature. No backport is needed. Closes scylladb/scylladb#20933 * github.com:scylladb/scylladb: tools/scylla-sstable: Rename valid_checksums -> valid test: Check validate_checksums() with missing digest sstables: Allow validate_checksums() to report missing digests sstables: Refactor validate_checksums() to use checksummed data stream sstables: Add error_handler parameter to data_stream() sstables: Add error handler in checksummed data source sstables: Check for excessive chunks in checksummed data source sstables: Check for premature EOF in checksummed data source test: test_validate_checksums: Check SSTable with invalid digest test: test_validate_checksums: Check SSTable with appended data test: test_validate_checksums: Complement test for truncated SSTable
Scylla in-source tests.
For details on how to run the tests, see docs/dev/testing.md
Shared C++ utils, libraries are in lib/, for Python - pylib/
alternator - Python tests which connect to a single server and use the DynamoDB API unit, boost, raft - unit tests in C++ cqlpy - Python tests which connect to a single server and use CQL topology* - tests that set up clusters and add/remove nodes cql - approval tests that use CQL and pre-recorded output rest_api - tests for Scylla REST API Port 9000 scylla-gdb - tests for scylla-gdb.py helper script nodetool - tests for C++ implementation of nodetool
If you can use an existing folder, consider adding your test to it. New folders should be used for new large categories/subsystems, or when the test environment is significantly different from some existing suite, e.g. you plan to start scylladb with different configuration, and you intend to add many tests and would like them to reuse an existing Scylla cluster (clusters can be reused for tests within the same folder).
To add a new folder, create a new directory, and then
copy & edit its suite.ini.