If index was used to skip to the next partition (because the current
partition wasn't consumed in full) and reader's partition range ends
before the data file ends, we did not detect that we're out of range
before returning a streamed_mutation. Fix by checking _context.eof()
before doing that.
Refs #2733.
* seastar 0083ee8...85ca12d (1):
> Merge "Run-time logging configuration" from Jesse
Includes patch from Jesse:
"Switch to Seastar for logging option handling
In addition to updating the abstraction layer for Seastar logging in `log.hh`,
the configuration system (`db/config.{hh,cc}`) has been updated in two ways:
- The string-map type for Boost.program_options is now defined in Seastar.
- A configuration value can be marked as `UsedFromSeastar`. This is like `Used`,
except the option is expected to be defined in the Boost.Program_options
description for Seastar. If the option is not defined in Seastar, or it is
defined with a different type, then a run-time exception is thrown early in
Scylla's initialization. This is necessary because logging options which are
now defined in Seastar were previously defined in Scylla and support for these
options in the YAML file cannot be dropped. In order to be able to verify that
options marked `UsedFromSeastar` are actually defined in Seastar, the
interface for adding options to `db::config` has changed from taking a
`boost::program_options::options_description_easy_init` (which is handle into
a `boost::program_options::options_description` which only allows adding
options) to taking a `boost::program_options::options_description`
directly (which also allows querying existing options).
Scylla also fully defers to Seastar's support for run-time logging
configuration."
Signed-off-by: Jesse Haber-Kucharsky <jhaberku@scylladb.com>
Message-Id: <ef26cffb91bef1ae95d508187a6dd861a6c4fc84.1503344007.git.jhaberku@scylladb.com>
"We have recently found two problems with the drop_column_family code
that needs addressing. The first is that exceptions in truncate() may
lead to stop() being skipped, which can cause Scylla to crash.
The other is that a truncate() issued before drop_column_family may get
the chance to execute only after the column family is already dropped
and also crash (That is issue 2726).
The second problem is the classic problem of asynchronous execution on
an object that may terminate, which we have been traditionally solving
with a gate. We add a gate to the column family that will be closed
during CF stop(), and we will require all asychronous operations to
enter it.
The immediate fix is for truncate(), where we have seen a real, concrete
problem. But it would be good to audit other code paths to make sure
that they are sane.
The most obvious ones, flush, compaction and sstable deletion are
already sane, since they are waited on explicitly during stop()."
Fixes#2726.
* 'issue-2726-v2-master' of github.com:glommer/scylla:
database: add gate for generic async operations to column family
database: make sure that column family is always stopped when dropped
run_with_compaction_disabled(), which is called by truncate, has a
pretty large defer point in remove(). When the code gets to finally
execute, we can't guarantee that the column family will still be alive.
That is true in particular if we issued a drop table command following
truncate: by the time truncate gets to resume, the CF will be gone.
Before the column family is dropped, it will always call its stop()
method, which means we have an opportunity to do some waiting there. We
already wait for flushes and current compactions to end.
Traditionally, we have been solving similar problems by adding a gate
that will catch asynchronous operations and making sure that potentially
asynchronous operations will enter the gate before executing. Let's do
the same thing here. We will close() the gate during stop().
Fixes#2726
Signed-off-by: Glauber Costa <glauber@scylladb.com>
truncate can throw exceptions. If it does, cf->stop() will never be
called because it is contained in a .then clause instead of finally.
One of the things that truncate does - in a finally block of its own -
is initiate a final compaction. If it returns an exception nobody will
wait for that compaction to finish (since cf->stop() is the one doing
that) and we'll crash.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
"Preserve the networking configuration mode during the upgrade by generating the /etc/scylla.d/perftune.yaml
file and using it."
Fixes#2725.
* 'dist_respect_cpuset_conf-v3' of https://github.com/vladzcloudius/scylla:
scylla_prepare: respect the cpuset.conf when configuring the networking
scylla_cpuset_setup: rm perftune.yaml
scylla_cpuset_setup: add a missing "include" of scylla_lib.sh
Choose the networking configuration mode according to the current contents of /etc/scylla.d/cpuset.conf.
If it doesn't exist - use the default mode.
If it exists - use the mode that has been used for generation of the CPU set.
Store the configuration into the /etc/scylla.d/perftune.yaml
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
scylla_setup resets our configuration and perftune.yaml is a part of it.
perftune.yaml is generated based on the contents of cpuset.conf therefore we should reset
these together.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
The scylla_cpuset_setup uses a verify_args() function that is defined in the scylla_lib.sh.
Fixes#2716
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
* seastar e96881a...b9f1eb7 (9):
> httpd: indentation patch
> httpd: handle exception when shutting down
> stall-detector: Allow backtrace throttling to be configured
> stall-detector: Fix messages about suppresssion not appearing
> scripts: posix_net_conf.sh: allow passing a perftune.py configuration file as a parameter
> scripts: perftune.py: add the possibility to pass the parameters in a configuration file and print the YAML file with the current configuration
> scripts: perftune.py: actually use the number of Rx queues when comparing to the number of CPU threads
> core: make current_backtrace() noexcept
> memory: add large allocation detector stubs for default allocator
Fixes#2723.
* tag 'asias/repair_issue_2723_v1' of github.com:cloudius-systems/seastar-dev:
repair: Do not allow repair until node is in NORMAL status
gossip: Add is_normal helper
The following backtrace was reported by user when running repair and keeping restarting the node at the same time.
#0 0x00007eff077281d7 in raise () from /lib64/libc.so.6
#1 0x00007eff07729a08 in abort () from /lib64/libc.so.6
#2 0x00007eff07721146 in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007eff077211f2 in __assert_fail () from /lib64/libc.so.6
#4 0x00000000010ef2c2 in locator::token_metadata::first_token_index (this=0x641000214e98, start=...) at locator/token_metadata.cc:133
#5 0x00000000010ef2d9 in locator::token_metadata::first_token (this=0x641000214e98, start=...) at locator/token_metadata.cc:143
#6 0x00000000010e329d in locator::abstract_replication_strategy::get_natural_endpoints (this=0x641000494000, search_token=...)
at locator/abstract_replication_strategy.cc:66
#7 0x0000000001481186 in get_neighbors (hosts=std::vector of length 0, capacity 0, data_centers=std::vector of length 0, capacity 0,
range=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ksname=..., db=...) at repair/repair.cc:196
#8 repair_range<nonwrapping_range<dht::token> > (range=..., ri=...) at repair/repair.cc:781
#9 <lambda(auto:99&)>::<lambda(auto:100&&)>::<lambda(auto:101&)>::<lambda()>::operator() (__closure=0x7efec07f7460) at repair/repair.cc:1005
#10 futurize<future<bool_class<stop_iteration_tag> > >::apply<repair_ranges(repair_info)::<lambda(auto:99&)>::
It is reproduced with
1) while true; do curl -X POST --header "Content-Type: application/json" --header "Accept: application/json" "http://127.0.0.1:10000/storage_service/repair_async/ks3"; done
2) start node 127.0.0.1, stop node 127.0.0.1 in a loop
The problem is, during boot up, the token_metadata is not replicated to all shards until
the node goes into NORMAL status.
To fix, check until node is in NORMAL status before allowing repair.
Fixes#2723
The number of keysapce and column family metrics reported is
proportional to the number of shards times the number of keysapce/column
families.
This can cause a performance issue both on the reporting system and on
the collecting system.
This patch adds a configuration flag (set to false by default) to enable
or disable those metrics.
Fixes#2701
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <20170821113843.1036-1-amnon@scylladb.com>
"Overly large metadata can hog memory which especially hurts in setups
with bad disk/memory ratio. To ease the pain compress the in-memory
compression-info.
The compression is implemented based on Avi's idea which is to group n
offsets together into segments, where each segment stores a base
absolute offset into the file, the other offsets in the segments being
relative offsets (and thus of reduced size). Also offsets are allocated
only just enough bits to store their maximum value. The offsets are thus
packed in a buffer like so:
arrrarrrarrr...
where n is 4, a is an absolute offset and r are offsets relative to a.
This of course means that stored offsets will not be aligned, not even
on a byte boundary, but the size reduction pretty convincing.
In addition, segments are stored in buckets, where each bucket has its
own base offset. In addition, segments in a buckets are optimized to
address as large of a chunk of the data as possible for a given chunk
size."
Ref #1946.
* 'bdenes/compress-compression-v3' of https://github.com/denesb/scylla:
Add unit test for compress::offsets
Optimise the storage of compression chunk offsets
Add script to precompute segmented compression parameters
To reduce the memory footprint of compression-info, n offsets are
grouped together into segments, where each segment stores a base
absolute offset into the file, the other offsets in the segments being
relative offsets (and thus of reduced size). Also offsets are
allocated only just enough bits to store their maximum value. The
offsets are thus packed in a buffer like so:
arrrarrrarrr...
where n is 4, a is an absolute offset and r are offsets relative to a.
The optimal value of n can be calculated for a given file_size (f) and
chunk_size (c), by finding the minima of the following function:
f(n) = (f/c)/n * (log2(f) + (n - 1)*log2((n-1)*(c + 64)))
This is done in an empirical way, using a script (see below).
Furthermore segments are stored in buckets, where each bucket has its
own base offset. Each bucket therefore can address an equal chunk of the
file and furthermore each segment in a bucket can address an equal
sub-chunk of this area.
The value of a given offset i is thus:
bucket_base_offset_for(i) + segment_base_offset_for(i) + offset(i)
To account for the bucketed storage we calculate a local_f, which is
optimized so that a bucketful of segmented offsets can address the
largest possible chunk of f. As value of this local_f only depends on
the bucket_size (b) and c the value of n can be made independent of f
and therefore only depend on one dynamic value, c. This makes life much
simpler as we don't need to know the size of the file up-front, we can
just append buckets to the storage on demand, while the required storage
is still less than a third [1] of the original storage requirements
(std::deque<uint64>).
The table with the minima(f(n)) for different f and c values is
pre-computed by gen_segmented_compress_params.py and
stored in sstables/segmented_compress_params.hh. This script also
creates a table with the best values of local_f for the given
bucket_size. At runtime we only select the best params based on c.
[1] This was calculated for c=4K and b=4K
Some (most?) users don't read logs or release notes, so they won't notice
that the ByteOrdered and Random partitioners were deprecated in 2.0. Make
them notice by refusing to start with a deprecated partitioner, unless a
switch is explicitly enabled.
Message-Id: <20170820073424.8331-1-avi@scylladb.com>
Limiting summary entry generation to at most one summary entry
per 64k of index data can lead to large index pages, with thousands
of index entries per summary entry. These are slow to parse, and there
is no real gain from the limit, since we already enforce a size
limit on the summary.
Remove the limit and allow summary entry generation based solely on
spanned data size.
Fixes#2711.
Message-Id: <20170819184255.14181-1-avi@scylladb.com>
split_range_to_single_shard() returns a vector of size 4096, with
each element (a partition_range) of size 100. The total of 400k can
cause defragmentation if memory is fragmented.
Fix by using a deque.
Fixes#2707.
Message-Id: <20170819141017.28287-1-avi@scylladb.com>
The script generates sstables/segmented_compress_params.hh which
contains a list with the optimal number of grouped offsets for
different data and chunk sizes as well as a list with the best
nominal data sizes for different chunk sizes, given a bucket size.
Data sizes are in the range of [2**4,2**50] and chunks in the
range of [2**4, 2**30]. Data sizes that are not used with the current
bucket_size are ommited.
See next commit for details of how the calculated values are used.
Large allocations can require cache evictions to be satisfied, and can
therefore induce long latencies. Enable the seastar large allocation
warning so we can hunt them down and fix them.
Message-Id: <20170819135212.25230-1-avi@scylladb.com>
Two reasons for this change:
1) every compaction should be multiplexed to manager which in turn
will make decision when to schedule. improvements on it will
immediately benefit every existing compaction type.
2) active tasks metric will now track ongoing reshard jobs.
Fixes#2671.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20170817224334.6402-1-raphaelsc@scylladb.com>
"Now that range queries go through the normal digest path, we rely on
query::result::calculate_counts() to count the amount of partitions
and rows returned.
This series optimizes it, in case it is needed, and also changes the
result message to include the partition and row counts, avoiding the
calculation altogether."
* 'calculate-counts/v3' of github.com:duarten/scylla:
query-result: Send row and partition count over the wire
query::result: Optimize calculate_counts()
These two admin related packages will be packaged under the "app-admin"
category and not the "dev-db" one.
This fixes the detection path of the packages for scylla_setup.
Signed-off-by: Alexys Jacob <ultrabug@gentoo.org>
Message-Id: <20170817094756.21550-1-ultrabug@gentoo.org>
Scylla's configure.py contains stuff we copied from Seastar's
configure.py, but is no longer used. Let's get rid of some of it.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20170813150842.12603-1-nyh@scylladb.com>
