mirror of
https://github.com/scylladb/scylladb.git
synced 2026-04-23 10:00:35 +00:00
aab6b0ee27
" This is the first part of the first step of switching Scylla. It covers converting cells to the new serialisation format. The actual structure of the cells doesn't differ much from the original one with a notable exception of the fact that large values are now fragmented and linearisation needs to be explicit. Counters and collections still partially rely on their old, custom serialisation code and their handling is not optimial (although not significantly worse than it used to be). The new in-memory representation allows objects to be of varying size and makes it possible to provide deserialisation context so that we don't need to keep in each instance of an IMR type all the information needed to interpret it. The structure of IMR types is described in C++ using some metaprogramming with the hopes of making it much easier to modify the serialisation format that it would be in case of open-coded serialisation functions. Moreover, IMR types can own memory thanks to a limited support for destructors and movers (the latter are not exactly the same thing as C++ move constructors hence a different name). This makes it (relatively) to ensure that there is an upper bound on the size of all allocations. For now the only thing that is converted to the IMR are atomic_cells and collections which means that the reduction in the memory footprint is not as big as it can be, but introducing the IMR is a big step on its own and also paves the way towards complete elimination of unbounded memory allocations. The first part of this patchset contains miscellaneous preparatory changes to various parts of the Scylla codebase. They are followed by introduction of the IMR infrastructure. Then structure of cells is defined and all helper functions are implemented. Next are several treewide patches that mostly deal with propagating type information to the cell-related operations. Finally, atomic_cell and collections are switched to used the new IMR-based cell implementation. The IMR is described in much more detail in imr/IMR.md added in "imr: add IMR documentation". Refs #2031. Refs #2409. perf_simple_query -c4, medians of 30 results: ./perf_base ./perf_imr diff read 308790.08 309775.35 0.3% write 402127.32 417729.18 3.9% The same with 1 byte values: ./perf_base1 ./perf_imr1 diff read 314107.26 314648.96 0.2% write 463801.40 433255.96 -6.6% The memory footprint is reduced, but that is partially due to removal of small buffer optimisation (whether it will be restored depends on the exact mesurements of the performance impact). Generally, this series was not expected to make a huge difference as this would require converting whole rows to the IMR. Memory footprint: Before: mutation footprint: - in cache: 1264 - in memtable: 986 After: mutation footprint: - in cache: 1104 - in memtable: 866 Tests: unit (release, debug) " * tag 'imr-cells/v3' of https://github.com/pdziepak/scylla: (37 commits) tests/mutation: add test for changing column type atomic_cell: switch to new IMR-based cell reperesentation atomic_cell: explicitly state when atomic_cell is a collection member treewide: require type for creating collection_mutation_view treewide: require type for comparing cells atomic_cell: introduce fragmented buffer value interface treewide: require type to compute cell memory usage treewide: require type to copy atomic_cell treewide: require type info for copying atomic_cell_or_collection treewide: require type for creating atomic_cell atomic_cell: require column_definition for creating atomic_cell views tests: test imr representation of cells types: provide information for IMR data: introduce cell data: introduce type_info imr/utils: add imr object holder imr: introduce concepts imr: add helper for allocating objects imr: allow creating lsa migrators for IMR objects imr: introduce placeholders ...