This patch changes the migration path for table updates such that the
base table mutations are sent and applied atomically with the view
schema mutations.
This ensures that after schema merging, we have a consistent mapping
of base table versions to view table versions, which will be used in
later patches.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
"This introduces an API which allows forward navigation in a stream of mutation
fragments. It allows one to consume only a subset of the stream by iteratively
specifying sub-ranges from which fragments should be returned.
API outline:
When in forwarding mode, the stream does not return all fragments right away,
but only those belonging to the current range. Initially current range only
covers the static row. The stream can be forwarded, even before reaching end-
of-stream for current range, to a later range with fast_forward_to().
Forwarding doesn't change initial restrictions of the stream, it can only be
used to skip over data.
Monotonicity of positions is preserved by forwarding. That is fragments
emitted after forwarding will have greater positions than any fragments
emitted before forwarding.
For any range, all range tombstones relevant for that range which are present
in the original stream will be emitted. Range tombstones emitted before
forwarding which overlap with the new range are not necessarily re-emitted.
When not in forwarding mode, the stream acts as if the current range was equal
to the full range. This implies that fast_forward_to() cannot be
used.
Whether stream is in forwarding mode or not is specified when the stream
is created, typically via mutation_source interface.
What's left for later series:
Optimization by providing specialized implementations. This series implements
forwarding support in all mutation sources via generic wrapper which simply
drops fragments."
* tag 'tgrabiec/clustering-fast-forward-to-v2' of github.com:scylladb/seastar-dev:
tests: mutation_source_tests: Verify monotonicty of positions
tests: random_mutation_generator: Spread the keys more
tests: mutation_source_test: Make blobs more easily distinguishable
tests: streamed_mutation: Test that merged stream passes mutation source tests
tests: mutation_source_test: Add tests for forwarding of streamed_mutation
tests: streamed_mutation_assertions: Add methods for navigating the stream
tests: Add range generators to random_mutation_generator
partition_slice_builder: Add with_ranges()
query: Introduce full_clustering_range
streamed_mutation: Add non-owning variant of mutation_from_streamed_mutation()
db: Enable creating forwardable readers via mutation_source
mutation_source: Document liveness requirements
mutation_source: Cleanup
db: Replace virtual_reader_type with mutation_source_opt
partition_version: Refactor make_partition_snapshot_reader() overloads
database: Fix mutation_source created by as_mutation_source() to not ignore trace_state_ptr
memtable: Accept all mutation_source parameters
streamed_mutation: Implement fast_forward_to() in stream merger
streamed_mutation: Add generic implementation of forwardable streamed_mutation
streamed_mutation: Add fast_forward_to() API
position_in_partition: Introduce position_range
position_in_partition: Introduce position constructor for right after the static row
streamed_mutation: Make cast to view non-explicit
streamed_mutation: Make schema() getter non-copying
The test started to fail sporadically on jenkins after
7a00dd6985 due to quiesce() timing out. It's not
clear though if this is a regression because before the series such timeouts
would not cause test failure if the future resulves eventually, timeout was
only logged.
I was not able to reproduce it on my setup nor on jenkins, so let's add more
debugging output and trigger a coredump next time the test fails.
Message-Id: <1487089576-27147-1-git-send-email-tgrabiec@scylladb.com>
Tests using random mutation generator should be provided with bot
counter and non-counter mutations to ensure that both cases are
sufficiently covered. However, mixed schemas (with both counter and
non-counter columns) are not allowed so the RMG has to be explicitly
told whether to use counter or non-counter schema.
Test case test_reading_with_different_schemas uses randomly generated
pairs of mutations and tries to upgrade one to the schema of the other.
However, there are cases when one schema cannot be upgraded to another,
for example, counter and non-counter schemas.
This helps achieve more repeatable runs that can then be compared via the
Linux perf tool. The option overrides duration-based testing and runs the
test for a specific number of iterations.
Message-Id: <20170204172937.8462-1-avi@scylladb.com>
"Before, the logic for releasing writes blocked on dirty worked like this:
1) When region group size changes and it is not under pressure and there
are some requests blocked, then schedule request releasing task
2) request releasing task, if no pressure, runs one request and if there are
still blocked requests, schedules next request releasing task
If requests don't change the size of the region group, then either some request
executes or there is a request releasing task scheduled. The amount of scheduled
tasks is at most 1, there is a single releasing thread.
However, if requests themselves would change the size of the group, then each
such change would schedule yet another request releasing thread, growing the task
queue size by one.
The group size can also change when memory is reclaimed from the groups (e.g.
when contains sparse segments). Compaction may start many request releasing
threads due to group size updates.
Such behavior is detrimental for performance and stability if there are a lot
of blocked requests. This can happen on 1.5 even with modest concurrency
because timed out requests stay in the queue. This is less likely on 1.6 where
they are dropped from the queue.
The releasing of tasks may start to dominate over other processes in the
system. When the amount of scheduled tasks reaches 1000, polling stops and
server becomes unresponsive until all of the released requests are done, which
is either when they start to block on dirty memory again or run out of blocked
requests. It may take a while to reach pressure condition after memtable flush
if it brings virtual dirty much below the threshold, which is currently the
case for workloads with overwrites producing sparse regions.
I saw this happening in a write workload from issue #2021 where the number of
request releasing threads grew into thousands.
Fix by ensuring there is at most one request releasing thread at a time. There
will be one releasing fiber per region group which is woken up when pressure is
lifted. It executes blocked requests until pressure occurs."
* tag 'tgrabiec/lsa-single-threaded-releasing-v2' of github.com:cloudius-systems/seastar-dev:
tests: lsa: Add test for reclaimer starting and stopping
tests: lsa: Add request releasing stress test
lsa: Avoid avalanche releasing of requests
lsa: Move definitions to .cc
lsa: Simplify hard pressure notification management
lsa: Do not start or stop reclaiming on hard pressure
tests: lsa: Adjust to take into account that reclaimers are run synchronously
lsa: Document and annotate reclaimer notification callbacks
tests: lsa: Use with_timeout() in quiesce()
