"Previously, partition tombstone was not written for partitions with no
rows causing corrupted data files.
This is now fixed and covered with tests.
In addition, we now track partition tombstones while collecting encoding
statistics."
* 'projects/sstables-30/fix-partition-tombstone/v3' of https://github.com/argenet/scylla:
tests: Don't use deprecated schema constructor.
tests: Add tests to cover partitions consisting only of partition keys.
sstables: Make sure partition level tombstone is written for partitions with no rows.
memtable: Collect statistics from partition-level tombstone.
"
This is preparatory cleanup series with fixes/cleanup of miscellaneous
issues that I discovered while working on the stateful range-scans.
Since the stateful range-scans series, even without these patches, is a
20+ patches strong series I'd like to fast-track this, to ease reviewing
the former.
Most of the changes here are related to code-hygenie and effectiveness
and there is a patch that is correctness-related ("querier: check only
the end bound of ranges when matching them") and one that is related to
ease-of-use ("range: clean the deduced transformed type").
Note that altough these changes were made in the context of working on
the stateful range-scans they make sense on their own as well.
Tests: unit(release, debug)
"
* '1865/pre-range-scans-cleanup/v1' of https://github.com/denesb/scylla:
multishard_combining_reader: use optimized optional for the shard reader
Use dht::token_range alias for last/preferred replicas
storage_proxy::coordinator_query_result: merge constructors into one w/ default params
querier: check only the end bound of ranges when matching them
querier: take range and slice by value
querier: remove const params from make_compaction_state()
querier: make _range and _slice const
flat_multi_range_mutation_reader: optimize for non-plural range vectors
range: clean the deduced transformed type
"
Fixes#3420.
Tests: dtest (`auth_test.py`), unit (release)
"
* 'jhk/fix_3420/v2' of https://github.com/hakuch/scylla:
cql3: Include custom options in LIST ROLES
auth: Query custom options from the `authenticator`
auth: Add type alias for custom auth. options
"
These patches were extracted from much larger series that introduces new
in-memory representation of cells. They contain various enhanecments and
fixes that to a varying degree make sense on its own. Sending them
separately will hopefuly ease the review and merging proces of the whole
IMR effort.
Tests: unit(release).
"
* tag 'pre-imr/v1' of https://github.com/pdziepak/scylla:
tests/perf: add microbenchmarks for basic row operations
tests: simple_schema: add make_row_from_serialized_value()
row: add clear_hash()
types: move compare_unsigned() to bytes.hh
lsa: provide migrator with the object size
lsa: add free() that does not require object size
db/view/build_progress: avoid copying mutation fragment
mutation_partition: enable ADL for cell swap
types: make some collection_type_impl functions non-static
counters: drop revertability of apply()
mutable_view: add default constructor and const_iterator
tests/mutation_reader: do not apply mutations created on another shard
sstables: do not call atomic_cell::value() for dead cells
lsa: sanitize use of migrators
lsa: reuse registered migrator ids
lsa: make migrators table thread-local
The querier provides a `matches(const nonwrapping_range&)` member to
allow for checking whether a range matches that with which the querier
was originally created. The check for match is more lax than a strict
equality check as ranges are shrunk query progresses.
Because of this the above member only checked that one of the bounds of
the examined ranges matches. This is adequate as for this purpose
because, in the context of a single query, it is guaranteed that no
two read requests to the same replica will have overlapping range.
However Avi pointed out in a recent, related review, that this check can
be made a little more strict by requiring that the end-bounds of the
two ranges *always* matches, instead of allowing any of the bounds to
match.
Don't create a flat_multi_range_mutation_reader when the range vector
has 0 or 1 element. In the former case create an empty reader and in the
latter just create a reader with the mutation-source with the only range
in the vector.
wrapping_range and nonwrapping_range offer a transform() member function
which allows creating a new range by applying a transformer function to
the bounds of the current range. The type of bounds of the new range is
deduced from the return type for this transformer function. However the
return type is used as-is, with any CV or reference attached to it.
Since it doesn't make sense to create a range of references or a type
with CV qualifiers strip these off the deduced type.
An implementation of `authenticator` can support custom options for a
each role.
If, to make up an example, the authenticator supported the `region` key,
then a role would be created as follows:
CREATE ROLE jsmith WITH OPTIONS = { 'region': 'north_america' }
AND PASSWORD = 'super_secure';
LIST ROLES will now print this custom option map as an additional column
with the heading "options".
However, none of the implementations of `authenticator` in Scylla
currently support OPTIONS, so LIST ROLES will in practice, for now,
print the empty set:
role | super | login | options
-----------+-------+-------+---------
cassandra | True | True | {}
None of the `authenticator` implementations we have support custom
options, but we should support this operation to support the relevant
CQL statements.
simple_schema::make_row() is not very well suited for performance tests
of row and cell creation since it serialises the value. This patch
introduces a new function that performs only minimal actions.
compare_unsigned() is a general utility function that compares two
bytes_view byte-by-byte. There is no need to include whole type.hh in
order to make it available.
While the migration function should have enough information to obtain
the object size itself, the LSA logic needs to compute it as well.
IMR is going to make calculating object sizes more expensive, so by
providing the infromation to the migrator we can avoid some needless
operations.
It is non-trivial to get the size of an IMR object. However, the
standard allocator doesn't really need it and LSA can compute it itself
by asking the migrator.
Calling fully qualified std::swap() prohibits the cell objects from
using their own swap implementations. This patch invokes std::swap in
the usual ADL-friendly way.
The switch to the new in-memory representation will require a larger
parts of the logic be aware of the type of the values they are dealing
with. In most cases it is not a significant burden for the users.
Scylla uses shared-nothing architecture and communication between the
shards is supposed to be very restricted. Applying to a memtable
mutations created on another shard is way to complex operation to be
allowed. Using frozen mutations is a much safer option.
Having migrators dynamically registered and deregistered opens a new
class of bugs. This patch adds some additional checks in the debug mode
with the hopes of catching any misuse early.
With the introduction of the new in-memory representation we will get
type- and schema-dependent migrators. Since there is no bound how many
times they can be created and destroyed it is better to be safe and
reuse registered migrator ids.
"
SSTable 3.0 format introduces serialization header which is used in reading SSTables in that format.
This patchset implements loading of this new component of Statistics.db.
Tests: units (release)
"
* 'haaawk/sstables3/load_serialization_header_v2' of ssh://github.com/scylladb/seastar-dev:
Load serialization_header from statistics
Add parse for disk_array_vint_size
Add helpers to read/parse vints
Add signed_vint::serialized_size_from_first_byte
Add sstable::get_serialization_header
Move random_access_reader to separate header
250ms is too high of a period for memtable controller. Since memtable
flushes are relatively efficient, specially in comparison to
compactions, if the shares are high we can flush a lot of data down with
the high shares - so in the next adjustment period our shares will be
minuscule and we won't flush much at all.
This leads to oscillating behavior that is mitigated by adjusting
faster.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <20180508182746.28310-3-glauber@scylladb.com>
"Fixes a bug in partition_snapshot::merge_partition_versions(), which would not
attempt merging if the snapshot is attached to the latest version (in which
case _version is nullptr and _entry is != nullptr). This would cause
partition_version objects to accumulate if there was an older snapshot and it
went away before the latest snapshot. Versions will be removed when the whole
entry goes away (flush or eviction).
May cause performance problems.
Fixes #3402."
* 'tgrabiec/fix-merge_partition_versions' of github.com:tgrabiec/scylla:
mvcc: Test version merging when snapshots go away
anchorless_list: Make ranges conform to SinglePassRange
anchorless_list: Drop deprecated use of std::iterator
mvcc: Fix partition_snapshot::merge_partition_versions() to not leave latest versions unmerged
The newer version of iotune, recently merged to Seastar, accepts
a new parameter that tells us where should we store the properties
about the disk.
We are already generating that properties file for the AMI case.
Let's also pass that parameter when calling iotune.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <20180507175757.9144-1-glauber@scylladb.com>
Starting from 2018.1 and 2.2 there was a change in the repository path.
It was made to support multiple product (like manager and place the
enterprise in a different path).
As a result, the regular expression that look for the repository fail.
This patch change the way the path is searched, both rpm and debian
varations are combined and both options of the repository path are
unified.
See scylladb/scylla-enterprise#527
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <20180429151926.20431-1-amnon@scylladb.com>
"
In SSTables 3.0, the base and increment fields have been swapped in
Bloom filters to reduce collisions (see CASSANDRA-8413). This affects
the resulting values written to Filter.db.
This patchset adds support for reading/writing Filter.db in the format
corresponding to the version of SSTables.
Tests: unit {release}
Filter.db files have been generated using Cassandra 3.11 with same data
as in unit tests and are validated to match those generated by Scylla.
"
* 'projects/sstables-30/write-filter/v1-2' of https://github.com/argenet/scylla:
Fix mistakes and typos in comments (minor clean-up)
Check Filter.db in SSTables 3.x write tests.
Support Bloom filter format used in SSTables 3.0.
Remove unused overload of i_filter::get_filter().
