"
This patchset adds proper support for sliced reads of partitions
containing range tombstones.
Given the SSTables 3.x repesentation of range tombstones by separate
start and end markers, we refer to the index for the information about
the currently opened range tombstone, if any, when skipping to the next
promoted index block.
Note that for this we have to take the promoted index block immediately
preceding the one we are jumping to.
Tests: unit {release}
"
* 'projects/sstables-30/range-tombstones-slicing/v3' of https://github.com/argenet/scylla:
tests: Test filtering and forwarding on a partition with interleaved rows and RTs.
tests: Add tests for reading wide partitions with range tombstones.
sstables: Support slicing for range tombstones.
sstables: Set/reset range tombstone start from end open marker.
sstables: Fix end_open_marker population in promoted index blocks.
sstables: Add need_skip() helper to data_consume_context.
sstables: For end_open_marker, return both position in partition and deletion time.
Change the validity timeout from 1s to 1h in order to avoid false alarms
on busy systems: for a short value there is a chance that
(loading_cache.size() == num_loaders) check is going to run after some elements
of the cache have already been evicted.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Message-Id: <20180904193026.7304-1-vladz@scylladb.com>
This ensures that row::external_memory_usage() is invariant to
insertion order of cells.
It should be so, so that accounting of a clustering_row, merged from
multiple MVCC versions by the partition_snapshot_flat_reader on behalf
of a memtable flush, doesn't give a greater result than what is used
by the memtable region. Overaccounting leads to assertion failure in
~flush_memory_accounter.
Fixes#3625 (hopefully).
Message-Id: <1535982513-19922-1-git-send-email-tgrabiec@scylladb.com>
test_streamed_mutation_forwarding_is_consistent_with_slicing already has
a REQUIRE() for the mutation read with the slicing reader. Add another
one for the forwarding reader. This makes it more consistent and also
helps finding problems with either the forwarding or slicing reader.
Instead of hiding what compaction method the querier uses (and only
expose it via rejecting 'can_be_used_for_page()`) make it very explicit
that these are really two different queriers. This allows using
different indexes for the two queriers in `querier_cache` and
eliminating the possibility of picking up a querier with the wrong
compaction method (read kind).
This also makes it possible to add new querier type(s) that suit the
multishard-query's needs without making a confusing mess of `querier` by
making it a union of all querying logic.
Splitting the queriers this way changes what happens when a lookup finds
a querier of the wrong kind (e.g. emit_only_live::yes for an
emit_only_live::no command). As opposed to dropping the found (but
wrong) querier the querier will now simply not be found by the lookup.
This is a result of using separate search indexes for the different
mutation kinds. This change should have no practical implications.
Splitting is done by making querier templated on `emit_only_live_rows`.
It doesn't make sense to duplicate the entire querier as the two share
99% of the code.
Requiring the caller of lookup() to pass in a `create_fun()` was not
such a good idea in hindsight. It leads to awkward call sites and even
more awkward code when trying to find out whether the lookup was
successfull or not.
Returning an optional gives calling code much more flexibility and makes
the code cleaner.
Extend `remote_reader_factory` interface so that it accepts all standard
mutation reader creation parameters. This allows factory lambdas to be
truly stateless, not having to capture any standard parameters that is
needed for creating the reader.
Standard parameters are those accepted by
`mutation_source::make_reader()`.
Additional tests for cases surrounding issue #3362, where base rows
disappear (or not) and view rows need to disappear (or not) as well.
These new tests focus on checking that view_updates::do_delete_old_entry()
is correct.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20180829131914.16042-2-nyh@scylladb.com>
"
When a view's partition key contains only columns from the base's partition
key (and not an additional one), the liveness - existance or disappearance -
of a view-table row is tied to the liveness of the base table row. And
that, in turn, depends not only on selected columns (base-table columns
SELECTed to also appear in the view) but also on unselected columns.
This means that we may need to keep a view row alive even without data,
just because some unselected column is alive in the base table. Before this
patch set we tried to build a single "row marker" in the view column which
tried to summarize the liveness information in all unselected columns.
But this proved unworkable, as explained in issue #3362 and as will be
demonstrated in unit tests at the end of this series.
Because we can't replace several unselected cells by one row marker, what
we do in this series is to add for each for the unselected cells a "virtual
cell" which contains the cell's liveness information (timestamp, deletion,
ttl) but not its value. For collections, we can't represent the entire
collection by one virtual cell, and rather need a collection of virtual
cells.
Fixes#3362
"
* 'virtual-cols-v3' of https://github.com/nyh/scylla:
Materialized Views: test that virtual columns are not visible
Materialized Views: unit test reproducing fixed issue #3362
Materialized Views: no need for elaborate row marker calculations
Materialized Views: add unselected columns as virtual columns
Materialized Views: fill virtual columns
Do not allow selecting a virtual column
schema: persist "view virtual" columns to a separate system table
schema: add "view virtual" flag to schema's column_definition
Add "empty" type name to CQL parser, but only for internal parsing
"
Fix loading_cache_test flakiness by retrying assertions.
Tests: unit(loading_cache_test(debug, release))
Fixes#3723
"
* 'loading-cache-test-flake/v4' of https://github.com/duarten/scylla:
tests/loading_cache_test: Unflake test_loading_cache_loading_reloading
tests/loading_cache_test: Use eventually() instead of open-coding it
tests/mutation_reader_test: Extract eventually_true() to eventually.hh
tests/cql_test_env: Lift eventually() to its own header file
Implement and test support for reading range tombstones in SSTables 3.
Does not yet support reads which are using slicing or fast forwarding.
From github.com/scylladb/seastar-dev.git haaawk/sstables3/tombstones_v11:
Piotr Jastrzebski (5):
sstables: Add consumer_m::consume_range_tombstone
sstables: Support null columns in ck
sstables: Support reading range_tombstones
sstables: Test reading range_tombstones
sstables: Add test for RT with non-full key
Vladimir Krivopalov (2):
sstables: Add operator<< overload for bound_kind_m.
keys: Add clustering_key_prefix::make_full helper.
The `loading_cache_test::test_loading_cache_loading_reloading` test
case is flaky, and fails in both debug and release mode. In an
over-provisioned environment, it's possible that when the reactor
runs, the timers for the `sleep()` and for reloading the
`loading_cache` are both expired, and continuations are scheduled with
an arbitrary order, causing the test to fail.
Fixes#3723
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
"
This series is a refactor of password management, motivated by a
combination of correctness bugs, improving testability, improving
clarity, and adding documentation.
Tests: unit (release)
"
* 'jhk/passwords_refactor/v2' of https://github.com/hakuch/scylla:
auth: Clean up implementation comments
auth: Remove unnecessary local variable
auth: Allow different random engines for salt
auth: Correct modulo bias in salt generation
auth: Extract random byte generation for salt
auth: Split out test for best supported scheme
auth: Rename function to use full words
auth: Add domain-specific exception for passwords
auth: Document passwords interface
auth: Move passsword stuff to its own namespace
auth: Identify password hashing errors correctly
auth: Add unit tests for password handling
auth: Move password handling to its own files
auth: Construct `std::random_device` instances once
This patchset adds support for skipping inside wide partitions using
index for sliced queries. This can significantly reduce disk I/O for
queries that only need to read a small amount of data from a wide
partition.
Other changes include general code clean-up and simplification.
* github.com/argenet/scylla.git tree/projects/sstables-30/skip_using_index/v6:
sstables: Support resetting data_consume_rows_context_m to
indexable_element::cell.
tests: Add tests to cover skipping with index through SSTables 3.x.
sstables: Support skipping inside wide partitions using index.
to_string: Add operator<< overload for std::optional.
sstables: Use std::optional instead of std::experimental::optional.
We see occasional bad_alloc failures in release mode; this is due
to the random mutation generator generating large mutations.
Reduce the mutation count to 300. I tested 100 runs and all passed,
so it reduces the false positive rate to < 1%.
This fix adds proper support for skipping inside wide partitions using
index for sliced reads. This significantly reduces disk I/O for filtered
queries.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
The constraint is no longer relevant, since Casandra removed
it in version 2.2. In addition the mechanism for handling this
case is already implemented and is identical in case of
clustering keys with single column EQ,= and IN relations.
(Cartesian product of singular ranges).
A unit test for this test case was added.
Fixes#1735
Tests:
1. Unit Tests.
2. Manual testing with the case described in the issue.
3. dtest: ql_additional_tests.py:TestCQL.composite_row_key_test
Signed-off-by: Eliran Sinvani <eliransin@scylladb.com>
Message-Id: <83b43fdc1ca0e0cc287f66f11816fc71b8bd2925.1534430405.git.eliransin@scylladb.com>
LIMIT should restrict the output result and not the query whose result
set is aggregated. when using aggregate the output is guarantied to
be only one row long. since LIMIT accepts only none negative numbers,
it has no effect and can be ignored.
Fixes#2028
Tests: The issue described Testcase , UnitTests.
Signed-off-by: Eliran Sinvani <eliransin@scylladb.com>
Message-Id: <6c235376c81f052020e2ed23d0a3d071b36d4415.1534416997.git.eliransin@scylladb.com>
In the previous patches, we added "virtual columns" to materialized views
to solve row liveness issues (issue #3362). Here we add a test that confirms
that although these virtual columns exist in the view, they should not be
visible to the user. They cannot be explicitly SELECTed from the view table,
and a "SELECT *" will skip them.
Refs #3362.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
This patch includes several tests reproducing issue #3362 - the effect
of unselected columns on view-table row liveness - and confirming
that it was fixed.
We found two example scenarios to demonstrate the bug. One scenario,
test_3362_with_ttls(), involves an unselected column with a TTL. The other,
test_3362_no_ttls() demonstrates the same bug without using TTL, and using
explicit updates and deletions instead. These two tests are heavily
commented, to explain what they test, and why.
In addition to these two basic tests, we also include similar tests
involving multiple items in a collection column, instead of multiple
separate columns, which demonstrate the same problem exists there (and
why, unfortunately, the "virtual columns" we add in that case need to
be collections too).
We also test that the virtual columns - and the problems they fix -
work not only on columns originally created with the view, but also
with unselected columns added later with ALTER TABLE on the base table.
Refs #3362.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
When the list of values in the IN list of a single column contains
duplicates, multiple executors are activated since the assumption
is that each value in the IN list corresponds to a different partition.
this results in the same row appearing in the result number times
corresponding to the duplication of the partition value.
Added queries for the in restriction unitest and fixed with a bad result check.
Fixes#2837
Tests: Queries as in the usecase from the GitHub issue in both forms ,
prepared and plain (using python driver),Unitest.
Signed-off-by: Eliran Sinvani <eliransin@scylladb.com>
Message-Id: <ad88b7218fa55466be7bc4303dc50326a3d59733.1534322238.git.eliransin@scylladb.com>
Change the test so that services are correctly teared down, by the
correct order (e.g., storage_service access the messaging_service when
stopping).
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20180814112111.8521-2-duarte@scylladb.com>
