The test started to fail sporadically on jenkins after
7a00dd6985 due to quiesce() timing out. It's not
clear though if this is a regression because before the series such timeouts
would not cause test failure if the future resulves eventually, timeout was
only logged.
I was not able to reproduce it on my setup nor on jenkins, so let's add more
debugging output and trigger a coredump next time the test fails.
Message-Id: <1487089576-27147-1-git-send-email-tgrabiec@scylladb.com>
Tests using random mutation generator should be provided with bot
counter and non-counter mutations to ensure that both cases are
sufficiently covered. However, mixed schemas (with both counter and
non-counter columns) are not allowed so the RMG has to be explicitly
told whether to use counter or non-counter schema.
Test case test_reading_with_different_schemas uses randomly generated
pairs of mutations and tries to upgrade one to the schema of the other.
However, there are cases when one schema cannot be upgraded to another,
for example, counter and non-counter schemas.
This helps achieve more repeatable runs that can then be compared via the
Linux perf tool. The option overrides duration-based testing and runs the
test for a specific number of iterations.
Message-Id: <20170204172937.8462-1-avi@scylladb.com>
"Before, the logic for releasing writes blocked on dirty worked like this:
1) When region group size changes and it is not under pressure and there
are some requests blocked, then schedule request releasing task
2) request releasing task, if no pressure, runs one request and if there are
still blocked requests, schedules next request releasing task
If requests don't change the size of the region group, then either some request
executes or there is a request releasing task scheduled. The amount of scheduled
tasks is at most 1, there is a single releasing thread.
However, if requests themselves would change the size of the group, then each
such change would schedule yet another request releasing thread, growing the task
queue size by one.
The group size can also change when memory is reclaimed from the groups (e.g.
when contains sparse segments). Compaction may start many request releasing
threads due to group size updates.
Such behavior is detrimental for performance and stability if there are a lot
of blocked requests. This can happen on 1.5 even with modest concurrency
because timed out requests stay in the queue. This is less likely on 1.6 where
they are dropped from the queue.
The releasing of tasks may start to dominate over other processes in the
system. When the amount of scheduled tasks reaches 1000, polling stops and
server becomes unresponsive until all of the released requests are done, which
is either when they start to block on dirty memory again or run out of blocked
requests. It may take a while to reach pressure condition after memtable flush
if it brings virtual dirty much below the threshold, which is currently the
case for workloads with overwrites producing sparse regions.
I saw this happening in a write workload from issue #2021 where the number of
request releasing threads grew into thousands.
Fix by ensuring there is at most one request releasing thread at a time. There
will be one releasing fiber per region group which is woken up when pressure is
lifted. It executes blocked requests until pressure occurs."
* tag 'tgrabiec/lsa-single-threaded-releasing-v2' of github.com:cloudius-systems/seastar-dev:
tests: lsa: Add test for reclaimer starting and stopping
tests: lsa: Add request releasing stress test
lsa: Avoid avalanche releasing of requests
lsa: Move definitions to .cc
lsa: Simplify hard pressure notification management
lsa: Do not start or stop reclaiming on hard pressure
tests: lsa: Adjust to take into account that reclaimers are run synchronously
lsa: Document and annotate reclaimer notification callbacks
tests: lsa: Use with_timeout() in quiesce()
Currently the test does not wait for cache update
to finish before carrying on with the checks.
This makes the test nondeterministic and purely wrong
because checks expect update to be finished.
This patch changes the test to wait for update to finish.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <2a99bba24b1628466d3495332b48ef3ccdb43c26.1485862389.git.piotr@scylladb.com>
Currently, the code is using bytes_opt and bytes_view_opt to represent
CQL values, which can hold a value or null. In preparation for
supporting a third state, unset value introduced in CQL v4, introduce
new raw_value and raw_value_view types and use them instead.
The new types are based on boost::variant<> and are capable of holding
null, unset values, and blobs that represent a value.
This reverts commit d61002cc33.
Introduced a regression in row_cache_alloc_stress.
The problem is that reclaim_from_evictable() evicts way too much after
the refactor due to the stop condition not taking into account how
much data was evicted so far and only looking at occupancy of the
minimal segment. This may lead to eviction of the whole region.
Add a boolean to short circuit the read path on empty range
hoping for some speedup.
tested in read write with cs using:
cl=QUORUM duration=1m -mode native cql3 -rate threads=700 -node localhost
Will do some additional benchmark.
Fixes#1056
Signed-off-by: Benoît Canet <benoit@scylladb.com>
Message-Id: <20170118194451.16836-1-benoit@scylladb.com>
Currently eviction is performed until occupancy of the whole region
drops below the 85% threshold. This may take a while if region had
high occupancy and is large. We could improve the situation by only
evicting until occupancy of the sparsest segment drops below the
threshold, as is done by this change.
I tested this using a c-s read workload in which the condition
triggers in the cache region, with 1G per shard:
lsa-timing - Reclamation cycle took 12.934 us.
lsa-timing - Reclamation cycle took 47.771 us.
lsa-timing - Reclamation cycle took 125.946 us.
lsa-timing - Reclamation cycle took 144356 us.
lsa-timing - Reclamation cycle took 655.765 us.
lsa-timing - Reclamation cycle took 693.418 us.
lsa-timing - Reclamation cycle took 509.869 us.
lsa-timing - Reclamation cycle took 1139.15 us.
The 144ms pause is when large eviction is necessary.
The change improves worst case latency. Reclamation time statistics
over 30 second period after cache fills up, in microseconds:
Before:
avg = 1524.283148
stdev = 11021.021118
min = 12.934000
max = 144356.000000
sum = 257603.852000
samples = 169
After:
avg = 1317.362414
stdev = 1913.542802
min = 263.935000
max = 19244.600000
sum = 175209.201000
samples = 133
Refs #1634.
Message-Id: <1484730859-11969-1-git-send-email-tgrabiec@scylladb.com>
Since ce083308a1
"random_mutation_generator: Generate RTs by default" random mutation
generator produces range tombstones. However, so far the tests were run
with all features disabled (because of incomplete initialization of all
services) which meant that RANGE_TOMBSTONE feature was not enabled and
the code couldn't handle range tombstones that weren't just prefixes.
This patch solves the problem by forcing all features to be enabled when
tests are run.
Message-Id: <20170116103324.22956-1-pdziepak@scylladb.com>
This patch changes the random_mutation_generator so it generates range
tombstones by default. This fixes a bug where reversibly applying
range tombstones wasn't being tested.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20170110164822.28747-1-duarte@scylladb.com>
"Intended to reduce memory usage when resharding by sharing sstable
components among shards. File descriptors are also shared from now
on, meaning that a much smaller number of file descriptors will be
used during resharding.
Fixes #1951."
branch 'excessive_memory_usage_v4' of github.com:raphaelsc/scylla
* 'excessive_memory_usage_v4' of github.com:raphaelsc/scylla:
db: avoid excessive memory usage during resharding
checked_file_impl: add support to dup
sstables: group sstable components that can be shared among shards
sstables: rename sstable member
After resharding, sstables may be owned by all shards, which
means that file descriptors and memory usage for metadata will
increase by a factor equal to number of shards. That can easily
lead to OOM.
SSTable components are immutable, so they can be stored in one
shard and shared with others that need it. We use the following
formula to decide which shard will open the sstable and share
it with the others: (generation % smp::count), which is the
inverse of how we calculate generation for new sstables.
So if no resharding is performed, everything is shard-local.
With this approach, resource usage due to loaded sstables will
be evenly distributed among shards.
For this approach to work, we now only populate keyspaces from
shard 0. It's now the sole responsible for iterating through
column family dirs. In addition, most of population functions
are now free and take distributed database object as parameter.
Fixes#1951.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
"This patch series adds support for CQL 3.3.1. The changes to CQL are listed
here:
https://github.com/apache/cassandra/blob/cassandra-2.2/doc/cql3/CQL.textile#changes
The following CQL features are already supported by Scylla:
- TRUNCATE TABLE alias
- Double-dollar string literals
- Aggregate functions: MIN, MAX, SUM, and AVG
This series adds the following CQL features:
- New data types: tinyint, smallint, date, and time
- CQL binary protocol v4 (required by the new data types)
- Advertise Cassandra 2.2.8 version from Scylla so that drivers correctly
detect the presence of CQL 3.3.1
The following CQL features are not supported by Scylla:
- Role-based access control (issue #1941)
- JSON data type
- User-defined functions (UDFs)
- User-defined aggregates (UDAs)
The following CQL binary protocol v4 changes are not implemented by this
series:
- Read_failure and Write_failure error codes are not implemented.
They error codes not used by the smart drivers but as they are
propagated to application code, we eventually need to wire them up
to our storage proxy implementation.
- Function_failure error code is only used by user-defined functions
and the fromJson function, which are not implemented by Scylla.
Fixes #1284."
* 'penberg/cql-3.3.1/v5' of github.com:cloudius-systems/seastar-dev:
version: Bump Cassandra version to 2.2.8
db/schema_tables: Add schema_functions and schema_aggregates tables
tests/type_tests: TIME type test cases
tests/cql_query_test: TIME type test cases
cql3: TIME data type support
tests/type_tests: DATE type test cases
tests/cql_query_test: DATE type test cases
cql3: DATE type support
date.h: 64-bit year and days representation
licenses: Add utils/date.h license
utils/date.h: Import date and time library sources
tests/type_tests: TINYINT and SMALLINT type test cases
tests/cql_query_test: TINYINT and SMALLINT type test cases
cql3: TINYINT and SMALLINT data type support
types: Fix integer_type_impl::parse_int() for bytes
We intend to share immutable sstable components among shards to
reduce excessive memory usage when resharding shared sstables.
This change is about grouping those components into a structure,
and using foreign ptr to make sure that the structure will be
deleted by whichever shard created it.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Rename _components to _recognized_components because _components
will be used to name a field with shareable components.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
"Reduce the size of mutation_partition by implementing intrusive set using
bi::rbtree_algorithms directly and using tree nodes optimized for size.
This will reduce the size of mutation_partition by:
24 bytes + <number of cql rows> * 8 bytes
This should have a positive impact on performance because mutation_partitions
are stored both in memtable and cache.
Fixes #742."
* 'haaawk/742' of github.com:cloudius-systems/seastar-dev:
intrusive_set: rename size() to calculate_size()
Make intrusive_set_external_comparator::_value_traits static
Implement intrusive set using rbtree_algorithms
mutation_partition: make apply_reversibly_intrusive_set nongeneric
mutation_partition: take schema in find_row and clustered_row
mutation_partition: Extract intrusive set logic to a class.
mutation_partition: Replace value_comp with key_comp calls
This hopefully will make it more apparent that
the time complexity of this method is O(N) not O(1).
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
The test case is interested in the upper boundary of 32-bit integer
because we already test the lower boundary in assertions below. The old
test passed, of course, but it wasn't very interesting.
Message-Id: <1483522773-6008-1-git-send-email-penberg@scylladb.com>
"This patchset contains fixes for the changes introduced in "Query result
size limiting". It also improves handling of short data reads.
I order to minimise chances of digest mismatch during data queries replicas
that were asked just to return a digest also keep track of the size of the
data (in the IDL representation) so that they would stop at the same point
nodes doing full data queries would. Moreover, data queries are not
affected by per-shard memory limit and the coordinator sends individual
result size limits to replicas in order not to depend on hardcoded values.
It is still possible to get digest mismatches if the IDL changes (e.g. a
new field is added), but, hopefully, that won't be a serious problem."
* 'pdziepak/short-read-fixes/v4' of github.com:cloudius-systems/seastar-dev:
query: introduce result_memory_accounter::foreign_state
storage_proxy: fix short reads in parallel range queries
storage_proxy: pass maximum result size to replicas
mutation_partition: use result limiter for digest reads
query: make result_memory_limiter constants available for linker
result_memory_limiter: add accounter for digest reads
idl: allow writers to use any output stream
result_memory_limiter: split new_read() to new_{data, mutation}_read()
idl: is_short_read() was added in 1.6
mutation_partition: honour allowed_short_read for static rows
storage_proxy: fix _is_short_read computation
storage_proxy: disallow short reads if got no live rows
storage_proxy: don't stop after result with no live rows
This reverts commit aa392810ff, reversing
changes made to a24ff47c637e6a5fd158099b8a65f1191fc2d023; it uses
boost::intrusive::detail directly, which it must not, and doesn't compile on
all boost versions as a consequence.
"Reduce the size of mutation_partition by implementing intrusive set using
bi::rbtree_algorithms directly and using tree nodes optimized for size.
This will reduce the size of mutation_partition by:
24 bytes + <number of cql rows> * 8 bytes
This should have a positive impact on performance because mutation_partitions
are stored both in memtable and cache.
Fixes #742."
* 'haaawk/742' of github.com:cloudius-systems/seastar-dev:
intrusive_set: rename size() to calculate_size()
Make intrusive_set_external_comparator::_value_traits static
Implement intrusive set using rbtree_algorithms
mutation_partition: make apply_reversibly_intrusive_set nongeneric
mutation_partition: take schema in find_row and clustered_row
mutation_partition: Extract intrusive set logic to a class.
mutation_partition: Replace value_comp with key_comp calls
