Try to avoid recompilations by reducing inclusions of system_keyspace.hh
in other header files.
Tests: unit (release)
* tag 'system_keyspace.hh/v1' of https://github.com/avikivity/scylla:
storage_service: remove system_keyspace.hh include
locator: de-inline reconnectable_snitch_helper
locator: de-inline production_snitch_base
cql3: remove #include of system_keyspace.hh
De-inlining allows us to remove some dependencies, and those functions
are too complex to inline anyway.
A few always-throwing functions get the [[noreturn]] attribute to
avoid damaging code generation.
* seastar 08e02dc...42159d4 (9):
> memory: avoid unconditional calls to __tls_init
> io_tester: bring back information about think time
> Merge "Avoid continuations in I/O Scheduler path" from Glauber
> Merge "Extend io_tester to support CPU loads" from Glauber
> tutorial: fix undue complication in semaphore get_units() example
> Tutorial: in HTML target, inline code snippets shouldn't be gray
> tutorial: add build target for split HTML file
> tutorial: mention seastar::thread as option for object lifetime management
> tutorial: document new seastar::future::wait()
* dist/ami/files/scylla-ami 3aa87a7...5170011 (3):
> scylla_install_ami: install enhanced networking NIC drivers
> scylla_install_ami: set kernel-ml as default kernel
> scylla_install_ami: fix NIC down with enhanced networking on new base AMI
already called in update_info_for_opened_data() which is called by
open_data(); no need for clustering components to be set early
either.
found it when auditing the code.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20180310225213.26017-1-raphaelsc@scylladb.com>
unsigned type was incorrectly used for keeping track of min and max
timestamp, so a negative number would be treated as a very high
number that would *incorrectly* end up as max timestamp in sstable
metadata.
Fixes#3000.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20180308162217.18963-1-raphaelsc@scylladb.com>
"
Refs #2692Fixes#3246
The current restricting algorithm [1] restricts the active-reader queue
based on the memory consumption of the existing active readers. When
this memory consumption is above the limit new readers are not admitted.
The inactive reader queue on the other hand has a fixed length.
This caused performance regressions on two workloads:
* read-only: since the inactive-reader queue length is severly limited
(compared to the previous situation) reads will timeout at loads
comfortably handled before.
* mixed: since the memory consumption happens only at admission time
(already created active readers are not limited) memory consumption
growed significantly causing problems when compactions kicked in.
The solution is to reintroduce the old limit of 100 active concurrent
user-reads while still keeping the memory-based limit as well. For
workloads that don't consume a lot of memory or on large boxes with lots
of memory the count-based limit will be reached which is reverting to the
old well-known behaviour. For memory-hungry workloads or on small boxes
with little memory the memory based-limit will kick in sooner avoiding
memory overconsumption.
[1] introduced by bdbbfe9390
"
* 'restricted-reader-dual-limit/v3' of https://github.com/denesb/scylla:
Modify unit tests so that they test the dual-limits
Use the reader_concurrency_semaphore to limit reader concurrency
Add reader_concurrency_semaphore
Add reader_resource_tracker param to mutation_source
mv reader_resource_tracker.hh -> reader_concurrency_semaphore.hh
This semaphore implements the new dual, count and memory based active
reader limiting. As purely memory-based limiting proved to cause
problems on big boxes admitting a large number of readers (more than any
disk could handle) the previous count-based limit is reintroduced in
addition to the existing memory-based limit.
When creating new readers first the count-based limit is checked. If
that clears the memory limit is checked before admitting the reader.
reader_conccurency_semaphore wraps the two semaphores that implement
these limits and enforces the correct order of limit checking.
This class also completely replaces the restricted_reader_config struct,
it encapsulates all data and related functinality of the latter, making
client code simpler.
Soon, reader_resource_tracker will only be constructible after the
reader has been admitted. This means that the resource tracker cannot be
preconstructed and just captured by the lambda stored in the mutation
source and instead has to be passed in along the other parameters.
In preparation to reader_concurrency_semaphore being added to the file.
The reader_resource_tracker is really only a helper class for
reader_concurrency_semaphore so the latter is better suited to provide
the name of the file.
"
This series switches granularity of memory-pressure-induced eviction in cache
from a partition to a row.
Since 9b21a9b cache can store partial partitions with row granularity but they
were still evicted as a unit. This is problematic for the following reasons:
- more is evicted than necessary, which decreases cache efficiency. In the
worst case, whole cache gets evicted at once
- evicting large amounts of memory (large partitions) at once may impact
latency badly
Fixes#2576.
See the documentation added in patch titled "doc: Document row cache eviction"
for details on how eviction works.
Open issues to be fixed incrementally:
- range tombstones are not evictable
- cache update still has partition granularity, which
causes bad latency on memtable flush with large partitions
"
* tag 'tgrabiec/row-level-eviction-v3' of github.com:scylladb/seastar-dev: (43 commits)
doc: Document row cache eviction
tests: cache: Add tests for row-level eviction
tests: cache: Check that data is evictable after schema change
tests: cache: Move definitions to the top
tests: perf_cache_eviction: Switch eviction counter to row granularity
tests: row_cache_alloc_stress: Avoid quadratic behavior
cache: Introduce unlink_from_lru()
cache: Add row-level stats about cache update from memtable
mvcc: Propagate information if insertion happened from ensure_entry_if_complete()
cache: Track number of rows and row invalidations
cache: Evict with row granularity
cache: Track static row insertions separately from regular rows
tests: mvcc: Use apply_to_incomplete() to create versions
tests: mvcc: Fix test_apply_to_incomplete()
tests: cache: Do not depend on particular granularity of eviction
tests: cache: Make sure readers touch rows in test_eviction()
mvcc: Store complete rows in each version in evictable entries
mvcc: Introduce partition_snapshot_row_cursor::ensure_entry_in_latest()
tests: cache: Invoke partial eviction in test_concurrent_reads_and_eviction
cache: Ensure all evictable partition_versions have a dummy after all rows
...
Partitions corresponding to keys have 40k rows. With row-level
eviction touching them inside the loop became a serious performance
issue, because touch() now needs to walk over all rows.
Will be used in row_cache_alloc_stress to unlink partitions which we
don't want to get evicted, instead of reapeatedly calling touch() on
them after each subsequent population. After switching to row-level
LRU, doing so greatly increases run time of the test due to quadratic
behavior.
The address and keyspace should be swapped.
Before:
range_streamer - Bootstrap with ks3 for keyspace=127.0.0.1 succeeded,
took 56 seconds
After:
range_streamer - Bootstrap with 127.0.0.1 for keyspace=ks3 succeeded,
took 56 seconds
Message-Id: <5c49646f1fbe45e3a1e7545b8470e04b166922c4.1520416042.git.asias@scylladb.com>
Instead of evicting whole partitions, evicts whole rows.
As part of this, invalidation of partition entries was changed to not
evict from snapshots right away, but unlink them and let them be
evicted by the reclaimer.
For row-level eviction we need to ensure that each version has
complete rows so that eviction from older versions doesn't affect the
value of the row in newer snapshots.
This is achieved by copying the row from an older version before
applying the increment in the new version.
Only affects evictable entries, memtables are not affected.
Every evictable version will have a dummy entry at the end so that it can be
tracked in the LRU.
It is also needed to allow old versions to stay around (with
tombstones and static rows) after all rows are evicted. Such versions
must be fully discontinuous, and we need some entry to mark that.
We will need to propagate a cache_tracker reference to evict(). Instead
of evicting from destructor, do so before cache_entry gets unlinked
from the tree. Entries which are not linked, don't need to be
explicitly evicted.
