buffer_size() exposes the collective size of the external memory
consumed by the mutattion-fragments in the flat reader's buffer. This
provides a basis to build basic memory accounting on. Altought this is
not the entire memory consumption of any given reader it is the most
volatile component and usually by far the largest one too.
"
This series switches granularity of memory-pressure-induced eviction in cache
from a partition to a row.
Since 9b21a9b cache can store partial partitions with row granularity but they
were still evicted as a unit. This is problematic for the following reasons:
- more is evicted than necessary, which decreases cache efficiency. In the
worst case, whole cache gets evicted at once
- evicting large amounts of memory (large partitions) at once may impact
latency badly
Fixes#2576.
See the documentation added in patch titled "doc: Document row cache eviction"
for details on how eviction works.
Open issues to be fixed incrementally:
- range tombstones are not evictable
- cache update still has partition granularity, which
causes bad latency on memtable flush with large partitions
"
* tag 'tgrabiec/row-level-eviction-v3' of github.com:scylladb/seastar-dev: (43 commits)
doc: Document row cache eviction
tests: cache: Add tests for row-level eviction
tests: cache: Check that data is evictable after schema change
tests: cache: Move definitions to the top
tests: perf_cache_eviction: Switch eviction counter to row granularity
tests: row_cache_alloc_stress: Avoid quadratic behavior
cache: Introduce unlink_from_lru()
cache: Add row-level stats about cache update from memtable
mvcc: Propagate information if insertion happened from ensure_entry_if_complete()
cache: Track number of rows and row invalidations
cache: Evict with row granularity
cache: Track static row insertions separately from regular rows
tests: mvcc: Use apply_to_incomplete() to create versions
tests: mvcc: Fix test_apply_to_incomplete()
tests: cache: Do not depend on particular granularity of eviction
tests: cache: Make sure readers touch rows in test_eviction()
mvcc: Store complete rows in each version in evictable entries
mvcc: Introduce partition_snapshot_row_cursor::ensure_entry_in_latest()
tests: cache: Invoke partial eviction in test_concurrent_reads_and_eviction
cache: Ensure all evictable partition_versions have a dummy after all rows
...
Will be used in row_cache_alloc_stress to unlink partitions which we
don't want to get evicted, instead of reapeatedly calling touch() on
them after each subsequent population. After switching to row-level
LRU, doing so greatly increases run time of the test due to quadratic
behavior.
Instead of evicting whole partitions, evicts whole rows.
As part of this, invalidation of partition entries was changed to not
evict from snapshots right away, but unlink them and let them be
evicted by the reclaimer.
We will need to propagate a cache_tracker reference to evict(). Instead
of evicting from destructor, do so before cache_entry gets unlinked
from the tree. Entries which are not linked, don't need to be
explicitly evicted.
Commit 6ccd317 introduced a bug in partition_entry::evict() where a
partition entry may be partially evicted if there are non-evictable
snapshots in it. Partially evicting some of the versions may violate
consistency of a snapshot which includes evicted versions. For one,
continuity flags are interpreted realtive to the merged view, not
within a version, so evicting from some of the versions may mark
reanges as continuous when before they were discontinuous. Also, range
tombtsones of the snapshot are taken from all versions, so we can't
partially evict some of them without marking all affected ranges as
discontinuous.
The fix is to revert back to full eviciton, and avoid moving
non-evictable snapshots to cache. When moving whole partition entry to
cache, we first create a neutral empty partition entry and then merge
the memtable entry into it just like we would if the entry already
existed.
Fixes#3215.
Tests: unit (release)
Message-Id: <1518710592-21925-2-git-send-email-tgrabiec@scylladb.com>
We have had a quota of partitions to process in clear_gently /
update_cache, so that we don't overwork. However, with those things now
being in their own task group there is no harm in allowing it to run
until we reach a natural preemption point.
While we are at it, clear_gently did not check for need_preempt()
before, so this patch fixes it.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
We have moved clear_gently from using a seastar::thread's scheduling_group to
using the CPU scheduler's. However, update_cache was forgotten.
This patch fixes that and gets rid of the old group just in case.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
In the last patch, we enabled per-request timeouts, we enable timeouts
in fill_buffer. There are many places, though, in which we
fast_forward_to before we fill_buffer, so in order to make that
effective we need to propagate the timeouts to fast_forward_to as well.
In the same way as fill_buffer, we make the argument optional wherever
possible in the high level callers, making them mandatory in the
implementations.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
As part of the work to enable per-request timeouts, we enable timeouts
in fill_buffer.
The argument is made optional at the main classes, but mandatory in all
the ::impl versions. This way we'll make sure we didn't forget anything.
At this point we're still mostly passing that information around and
don't have any entity that will act on those timeouts. In the next patch
we will wire that up.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
compiler: gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)
Problems introduced in f6a461c7a4
and 37b19ae6ba, respectively.
They both fail to compile due to use of method in lambda without
explicit mention of this. Some of failure is fixed by not using
auto in lambda parameter.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20171218222144.12297-1-raphaelsc@scylladb.com>
and add read_context::enter_flat_partition. This will
temporarily coexist with read_context::enter_partition
but after everything in cache is migrated to flat reader
the new method will replace old one.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
