Commit 9eb8ea8b11 installed
scylla_blocktune.py as part of preparing the rpm, but forgot
to add it to the installed file list, breaking the rpm build.
Fix by listing the file in the %files section.
Message-Id: <20180506202807.5719-1-avi@scylladb.com>
"
SSTables 3.x (format 'm') use CRC32 instead of Adler32 for calculating
checksums. This patchset introduces support for CRC32 along with Adler32
in checksummed_file_writer to be used for SSTables written in 'mc'
format.
Structures and helpers introduced for CRC32 will be later used for
calculating checksums for compressed files as well (not a part of this
patchset).
Tests: unit {release}
"
* 'projects/sstables-30/write-digest-crc/v3' of https://github.com/argenet/scylla:
tests: Add test covering checksumming SSTables 3.0 with CRC32.
sstables: Support CRC32 checksum for SSTables 3.x.
sstables: Move adler32 routines under the scope of a class.
sstables: Move checksum utils into separate header.
sstables: Remove unused 'checksum_file' flag from checksummed_file_writer.
"
The native protocol server generates mant reactor tasks that
can be easily eliminated. I measured a read workload with 100%
cache hit rate, seeing the number of tasks per request drop
from ~31 to ~27, and an increase of 3% in throughput.
"
* tag 'transport-optimize-1/v1' of https://github.com/avikivity/scylla:
transport: remove unused capture of flags variable
transport: merge response write and error handling continuations
transport: make write_repsonse() return void
transport: de-template a lambda
transport: merge memory-management and logging continuations
transport: remove gate continuation
transport: merge two response processing continuations
transport: simplify response processing continuation
transport: remove gratuitous continuation from process_request_one()
It just schedules the response, and returns immediately.
(I thought about calling it schedule_response(), but usually it will
write the response immediately, since waiting for network writes is
rare in a local network).
with_gate() generates a continuation if the protected function defers.
Avoid that by merging a gate::leave() call with another, preexisting,
continuation.
We have one coninuation transforming the result, and another shutting
down tracing. Since the first cannot defer, we can merge the two, reducing
the number of tasks processed by the reactor.
This patch fixes a bug where queries using a secondary index would, in
some cases, produce the same rows multiple times.
The problem was that the code begins by finding a list of primary keys
that match the search, and then work on the partitions containing them.
If multiple rows matched in the same partition, the partition was considered
multiple times, and the same rows were output multiple times.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20180510203141.17157-1-nyh@scylladb.com>
If we're given a single reader (can be common in a low-write-rate table,
where most of the data will be in a single large sstable, or in leveled
tables) then we can avoid the overhead of the combining reader by returning
the single input.
Tests: unit (release)
Message-Id: <20180513130333.15424-1-avi@scylladb.com>
"
This patchset implements separate timeouts for range queries, and lays
the foundations for separate timeouts for other query types.
While the feature in itself is worthy, the real motivation is to have
the timeouts decided by the caller, instead of storage_proxy. This in
turn is required to disentangle each layer behaving differently
depending on whether the query is internal or not; instead, the goal
is to have each caller declare its needs in terms of consistency level
and timeouts, and have the lower layers implement its requirements
instead of making their own decisions.
Fixes#3013.
Tests: unit (release)
"
* tag '3013/v1.1' of https://github.com/avikivity/scylla:
storage_proxy: remove default_query_timeout()
storage_proxy: don't use default timeouts
query_options: augment with timeout_config
thrift: configure thrift transport and handler with a timeout_config
transport: configure native transport with a timeout_config
cql3: define and populate timeout_config_selector
timeout_config: introduce timeout configuration
Before we accept running while not in developer mode, we verify that
the I/O Scheduler is properly configured. Up until now, that meant
verifying that --max-io-requests is properly set and that the number
of I/O Queues is enough to leave at least 4 requests per I/O Queue.
Systems that move to newer versions of Scylla may continue doing that,
so we need to be backwards compatible and keep testing for that.
However, newer systems will not set that option, but pass a YAML
property file (or string) instead. So we need to make sure that
either one of those is set.
If the property file is set, I am deciding here not to test for
number of I/O queues. scylla_io_setup will usually configure that
anyway, plus we plan on soon moving to all-shards-dispatch making
that less important.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <20180509163737.5907-1-glauber@scylladb.com>
There is an ongoing discussion in issue 2678 about the right time to
release permits. Right now we are releasing the permit after we flush
all data for the memtable plus the SSTables accompanying components -
plus flushing them, closing them, etc.
During all that time, we are increasing virtual dirty by adding more
data to the buffers but we are not able to decrease it-- until we
release the permit we can't start flushing the next memtable. This is
much more of a concern than I/O overlapping as described in the issue.
We have a hook in the SSTable write process that is (should be) called
as soon as data is written. We should move the permit release there.
We aren't, though, calling that as early as we could. The call to the
data written hook is writing after the Index is closed, summary is
sealed, etc.
This patch fixes that.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <20180508182746.28310-2-glauber@scylladb.com>
We have conflict between scylla-libgcc72/scylla-libstdc++72 and
scylla-libgcc73/scylla-libstdc++73, need to replace *72 package with
scylla-2.2 metapackage to prevent it.
Fixes#3373
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20180510081246.17928-1-syuu@scylladb.com>
"Previously, partition tombstone was not written for partitions with no
rows causing corrupted data files.
This is now fixed and covered with tests.
In addition, we now track partition tombstones while collecting encoding
statistics."
* 'projects/sstables-30/fix-partition-tombstone/v3' of https://github.com/argenet/scylla:
tests: Don't use deprecated schema constructor.
tests: Add tests to cover partitions consisting only of partition keys.
sstables: Make sure partition level tombstone is written for partitions with no rows.
memtable: Collect statistics from partition-level tombstone.
"
This is preparatory cleanup series with fixes/cleanup of miscellaneous
issues that I discovered while working on the stateful range-scans.
Since the stateful range-scans series, even without these patches, is a
20+ patches strong series I'd like to fast-track this, to ease reviewing
the former.
Most of the changes here are related to code-hygenie and effectiveness
and there is a patch that is correctness-related ("querier: check only
the end bound of ranges when matching them") and one that is related to
ease-of-use ("range: clean the deduced transformed type").
Note that altough these changes were made in the context of working on
the stateful range-scans they make sense on their own as well.
Tests: unit(release, debug)
"
* '1865/pre-range-scans-cleanup/v1' of https://github.com/denesb/scylla:
multishard_combining_reader: use optimized optional for the shard reader
Use dht::token_range alias for last/preferred replicas
storage_proxy::coordinator_query_result: merge constructors into one w/ default params
querier: check only the end bound of ranges when matching them
querier: take range and slice by value
querier: remove const params from make_compaction_state()
querier: make _range and _slice const
flat_multi_range_mutation_reader: optimize for non-plural range vectors
range: clean the deduced transformed type
"
Fixes#3420.
Tests: dtest (`auth_test.py`), unit (release)
"
* 'jhk/fix_3420/v2' of https://github.com/hakuch/scylla:
cql3: Include custom options in LIST ROLES
auth: Query custom options from the `authenticator`
auth: Add type alias for custom auth. options
"
These patches were extracted from much larger series that introduces new
in-memory representation of cells. They contain various enhanecments and
fixes that to a varying degree make sense on its own. Sending them
separately will hopefuly ease the review and merging proces of the whole
IMR effort.
Tests: unit(release).
"
* tag 'pre-imr/v1' of https://github.com/pdziepak/scylla:
tests/perf: add microbenchmarks for basic row operations
tests: simple_schema: add make_row_from_serialized_value()
row: add clear_hash()
types: move compare_unsigned() to bytes.hh
lsa: provide migrator with the object size
lsa: add free() that does not require object size
db/view/build_progress: avoid copying mutation fragment
mutation_partition: enable ADL for cell swap
types: make some collection_type_impl functions non-static
counters: drop revertability of apply()
mutable_view: add default constructor and const_iterator
tests/mutation_reader: do not apply mutations created on another shard
sstables: do not call atomic_cell::value() for dead cells
lsa: sanitize use of migrators
lsa: reuse registered migrator ids
lsa: make migrators table thread-local
The querier provides a `matches(const nonwrapping_range&)` member to
allow for checking whether a range matches that with which the querier
was originally created. The check for match is more lax than a strict
equality check as ranges are shrunk query progresses.
Because of this the above member only checked that one of the bounds of
the examined ranges matches. This is adequate as for this purpose
because, in the context of a single query, it is guaranteed that no
two read requests to the same replica will have overlapping range.
However Avi pointed out in a recent, related review, that this check can
be made a little more strict by requiring that the end-bounds of the
two ranges *always* matches, instead of allowing any of the bounds to
match.
Don't create a flat_multi_range_mutation_reader when the range vector
has 0 or 1 element. In the former case create an empty reader and in the
latter just create a reader with the mutation-source with the only range
in the vector.
wrapping_range and nonwrapping_range offer a transform() member function
which allows creating a new range by applying a transformer function to
the bounds of the current range. The type of bounds of the new range is
deduced from the return type for this transformer function. However the
return type is used as-is, with any CV or reference attached to it.
Since it doesn't make sense to create a range of references or a type
with CV qualifiers strip these off the deduced type.
An implementation of `authenticator` can support custom options for a
each role.
If, to make up an example, the authenticator supported the `region` key,
then a role would be created as follows:
CREATE ROLE jsmith WITH OPTIONS = { 'region': 'north_america' }
AND PASSWORD = 'super_secure';
LIST ROLES will now print this custom option map as an additional column
with the heading "options".
However, none of the implementations of `authenticator` in Scylla
currently support OPTIONS, so LIST ROLES will in practice, for now,
print the empty set:
role | super | login | options
-----------+-------+-------+---------
cassandra | True | True | {}
None of the `authenticator` implementations we have support custom
options, but we should support this operation to support the relevant
CQL statements.
simple_schema::make_row() is not very well suited for performance tests
of row and cell creation since it serialises the value. This patch
introduces a new function that performs only minimal actions.
