mirror of
https://github.com/scylladb/scylladb.git
synced 2026-06-02 21:17:01 +00:00
17eaf552f0
" Grab the lowest hanging fruits. This patch-set makes three important changes: * Consume the memory for I/O operations on tracked files, *before* they are forwarded to the underlying file. * Track memory consumed by buffers created for parsing in `continuous_data_consumer`. As this is the basis for the data, index and promoted index parsers, all three are covered now in this regard. * Track the index file. The remaining, not-so-low handing fruits in order of gain/cost(performance) ratio: * Track in-memory index lists. * Track in-memory promoted index blocks. * Track reader buffer memory. Note that this ordering might change based on the workload and other environmental factors. Also included in this series is an infrastructure refactoring to make tracking memory easier and involve including lighter headers, as well as a manual test designed to allow testing and experimenting with the effects of changes to the accuracy of the tracking of reader memory consumption. Refs: #4176 Refs: #2778 Tests: unit(dev), manual(sstable_scan_footprint_test) The latter was run as: build/dev/test/manual/sstable_scan_footprint_test -c1 -m2G --reads=4000 --read-concurrency=1 --logger-log-level test=trace --collect-stats --stats-period-ms=20 This will trickle reads until the semaphore blocks, then wait until the wait queue drains before sending new reads. This way we are not testing the effectiveness of the pre-admission estimation (which is terribly optimistic) and instead check that with slowly ramping up read load the semaphore will block on memory preventing OOM. This now runs to completion without a single `std::bad_alloc`. The read concurrency semaphore allows between 15-30 reads, and is always blocked on memory. " * 'more-accurate-reader-resource-tracking/v1' of ssh://github.com/denesb/scylla: test/manual/sstable_scan_footprint_test: improve memory consumption diagnostics tests/manual/sstable_scan_footprint_test: use the semaphore to determine read rate tests/manual: Add test measuring memory demand of concurrent sstable reads index_reader: make the index file tracked sstables/continuous_data_consumer: track buffers used for parsing reader_concurrency_semaphore: tracking_file_impl: consume memory speculatively reader_concurrency_semaphore: bye reader_resource_tracker treewide: replace reader_resource_tracer with reader_permit reader_permit: expose make_tracked_temporary_buffer() reader_permit: introduce make_tracked_file() reader_permit: introduce memory_units reader_concurrency_semaphore: mv reader_resources and reader_permit to reader_permit.hh reader_concurrency_semaphore: reader_permit: make it a value type reader_concurrency_semaphore: s/resources/reader_resources/ reader_concurrency_semaphore::reader_permit: move methods out-of-line