'"The issue is triggered by compaction of sstables of level higher than 0.
The problem happens when interval map of partitioned sstable set stores
intervals such as follow:
[-9223362900961284625 : -3695961740249769322 ]
(-3695961740249769322 : -3695961103022958562 ]
When selector is called for first interval above, the exclusive lower
bound of the second interval is returned as next token, but the
inclusivess info is not returned.
So reader_selector was returning that there *were* new readers when
the current token was -3695961740249769322 because it was stored in
selector position field as inclusive, but it's actually exclusive.
This false positive was leading to infinite recursion in combined
reader because sstable set's incremental selector itself knew that
there were actually *no* new readers, and therefore *no* progress
could be made."
Fixes #2908.'
* 'high_level_compaction_infinite_recursion_fix_v4' of github.com:raphaelsc/scylla:
tests: test for infinite recursion bug when doing high-level compaction
Fix potential infinite recursion when combining mutations for leveled compaction
dht: make it easier to create ring_position_view from token
dht: introduce is_min/max for ring_position
(cherry picked from commit 375ed938b4)
Migrate all the places that used memtable::make_reader to use
memtable::make_flat_reader and remove memtable::make_reader.
* seastar-dev.git haaawk/remove_memtable_make_reader_v2_rebased:
Remove memtable::make_reader
Stop using memtable::make_reader in row_cache_stress_test
Stop using memtable::make_reader in row_cache_test
Stop using memtable::make_reader in mutation_test
Stop using memtable::make_reader in streamed_mutation_test
Stop using memtable::make_reader in memtable_snapshot_source.hh
Stop using memtable::make_reader in memtable::apply
Add consume_partitions(flat_mutation_reader& reader, Consumer consumer)
Add default parameter values in make_combined_reader
Migrate test_virtual_dirty_accounting_on_flush to flat reader
Migrate test_adding_a_column_during_reading_doesnt_affect_read_result
Simplify flat_reader_assertions& produces(const mutation& m)
Migrate test_partition_version_consistency_after_lsa_compaction_happens
flat_mutation_reader: Allow setting buffer capacity
Add next_mutation() to flat_mutation_reader_assertions
cf::for_all_partitions::iteration_state: don't store schema_ptr
read_mutation_from_flat_mutation_reader: don't take schema_ptr
Migrate test_fast_forward_to_after_memtable_is_flushed to flat reader
(cherry picked from commit b0a56a91c2)
'char' and int8_t ('unsigned char') are different types. 'bytes' base type
is int8_t - use the correct type for casting.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
(cherry picked from commit 22ca5d2596)
The following patches contain fixes for skipping to the next parititon
in multi_range_reader and completelty dissable support for fast
forwarding inside a single partition, which is not needed and would only
add unnecessary complexity.
* https://github.com/pdziepak/scylla.git fix-multi_range_reader/v1:
flat_multi_range_mutation_reader: disallow
streamed_mutation::forwarding
flat_multi_range_mutation_reader: clear buffer on next_partition()
tests/flat_multi_range_mutation_reader: test skipping to next
partition
(cherry picked from commit 71cc63dfa6)
end_bound was not updated in one of the cases in which end and
end_kind was changed, as a result later merging decision using
end_bound were incorrect. end_bound was using the new key, but the old
end_kind.
Fixes#3083.
Message-Id: <1513772083-5257-1-git-send-email-tgrabiec@scylladb.com>
(cherry picked from commit dfe48bbbc7)
"Fixes #3057."
* 'summary_recreation_fixes_v2' of github.com:raphaelsc/scylla:
tests: sstable summary recreation sanity test
sstables: make loading of sstable without summary to work again
sstables: fix summary generation with dynamic index sampling
(cherry picked from commit 11de20fc33)
"While aa8c2cbc16 'Merge "Migrate sstables
to flat_mutation_reader" from Piotr' has converted the low-level sstable
reader to the new flat_mutation_reader interface there were still
multiple readers related to sstables that required converting,
including:
- restricted reader
- filtering reader
- single partition sstable reader
This series completes their conversion to the flat stream interface."
* tag 'flat_mutation_reader-sstable-readers/v2' of https://github.com/pdziepak/scylla:
db: convert single_key_sstalbe_reader to flat streams
db: fully convert incremental_reader_selector to flat readers
db: make make_range_sstable_reader() return flat reader
db: make column_family::make_reader() return flat reader
db: make column_family::make_sstable_reader() return a flat reader
filtering_reader: switch to flat mutation fragment streams
filtering_reader: pass a const dht::decorated_key& to the callback
mutation_reader: drop make_restricted_reader()
db: use make_restricted_flat_reader
mutation_reader: convert restricted reader to flat streams
(cherry picked from commit 6cb3b29168)
"Fixes cache reader to not skip over data in some cases involving overlapping
range tombstones in different partition versions and discontinuous cache.
Introduced in 2.0
Fixes #3053."
* tag 'tgrabiec/fix-range-tombstone-slicing-v2' of github.com:scylladb/seastar-dev:
tests: row_cache: Add reproducer for issue #3053
tests: mvcc: Add test for partition_snapshot::range_tombstones()
mvcc: Optimize partition_snapshot::range_tombstones() for single version case
mvcc: Fix partition_snapshot::range_tombstones()
tests: random_mutation_generator: Do not emit dummy entries at clustering row positions
(cherry picked from commit 051cbbc9af)
"Didn't affect any release. Regression introduced in 301358e.
Fixes#3041"
* 'resharding_fix_v4' of github.com:raphaelsc/scylla:
tests: add sstable resharding test to test.py
tests: fix sstable resharding test
sstables: Fix resharding by not filtering out mutation that belongs to other shard
db: introduce make_range_sstable_reader
rename make_range_sstable_reader to make_local_shard_sstable_reader
db: extract sstable reader creation from incremental_reader_selector
db: reuse make_range_sstable_reader in make_sstable_reader
(cherry picked from commit d934ca55a7)
"Convert combined_mutation_reader into a flat_mutation_reader impl. For
now - in the name of incremental progress - all consumers are updated to
use the combined reader through the
mutation_reader_from_flat_mutation_reader adaptor. The combined reader also
uses all it's sub mutation_readers through the
flat_mutation_reader_from_mutation_reader adaptor."
* 'bdenes/flatten-combined-reader-v8' of https://github.com/denesb/scylla:
Add unit tests for the combined reader - selector interactions
Add flat_mutation_reader overload of make_combined_reader
Flatten the implementation of combined_mutation_reader
Add mutation_fragment_merger
mutation_fragment::apply(): handle partition start and end too
Add non-const overload of partition_start::partition_tombstone()
Make combined_mutation_reader a flat_mutation_reader
Move the mutation merging logic to combined_mutation_reader
Remove the unnecessary indirection of mutation_reader_merger::next()
Move the implementation of combined_mutation_reader into mutation_reader_merger
Remove unused mutation_and_reader::less_compare and operator<
(cherry picked from commit 046991b0b7)
"This series makes sstable tests use flat stream interface. The main
motivation is to allow eventual removal of mutation_reader and
streamed_mutation and ensuring that the conversion between the
interfaces doesn't hide any bugs that would be otherwise found."
* tag 'flat_mutation_reader-sstable-tests/v1' of https://github.com/pdziepak/scylla:
sstables: drop read_range_rows()
tests/mutation_reader: stop using read_range_rows()
incremental_reader_selector: do not use read_range_rows()
tests/sstable: stop using read_range_rows()
sstables: drop read_row()
tests/sstables: use read_row_flat() instead of read_row()
database: use read_row_flat() instead of read_row()
tests/sstable_mutation_test: get flat_mutation_readers from mutation sources
tests/sstables: make sstable_reader return flat_mutation_reader
sstable: drop read_row() overload accepting sstable::key
tests/sstable: stop using read_row() with sstable::key
tests/flat_mutation_reader_assertions: add has_monotonic_positions()
tests/flat_mutation_reader_assertions: add produces(Range)
tests/flat_mutation_reader_assertions: add produces(mutation)
tests/flat_mutation_reader_assertions: add produces(dht::decorated_key)
tests/flat_mutation_reader_assertions: add produces(mutation_fragment::kind)
tests/flat_mutation_reader_assertions: fix fast forwarding
(cherry picked from commit 601a03dda7)
Fix two issues with serializing non-compound range tombstones as
compound: convert a non-compound clustering element to compound and
actually advertise the issue to other nodes.
* git@github.com:duarten/scylla.git rt-compact-fixes/v1:
compound_compact: Allow rvalues in size()
sstables/sstables: Convert non-compound clustering element to compound
tests/sstable_mutation_test: Verify we can write/read non-correct RTs
service/storage_service: Export non-compound RT feature
(cherry picked from commit e9cce59b85)
This series converts memtable flush reader to the new flat mutation
readers. Just like the scanning reader, flush reader concatenates
multiple partition snapshot readers in order to provide a stream
of all partitions in the memtable.
* https://github.com/pdziepak/scylla.git flat_mutation_reader-memtable-flush/v1
tests/flat_mutation_reader_assertion: add produces_partition()
memtable: make make_flush_reader() return flat_mutation_reader
flat_mutation_reader: add optimised flat_mutation_reader_opt
memtable: switch flush reader implementation to flat streams
tests/memtable: add test for flush reader
(cherry picked from commit 04106b4c96)
These tests now require having the storage service initialize, which
is needed to decide whether correct non-compound range tombstones
should be emitted or not.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20171126152921.5199-1-duarte@scylladb.com>
(cherry picked from commit 922f095f22)
The following patches convert sstable writers to use flat mutation
readers instead of the legacy mutation_reader interface.
Writers were already using flat consumer interface and used
consume_flattened_in_thread(), so most of the work was limited to
providing an appropriate equivalent for flat mutation readers.
* https://github.com/pdziepak/scylla.git flat_mutation_reader-sstable-write/v1:
flat_mutation_reader: move consumer_adapter out of consume()
flat_mutation_reader: introduce consume_in_thread()
tests/flat_mutation_reader: test consume_in_thread()
sstables: switch write_components() to flat_mutation_reader
streamed_mutation: drop streamed_mutation_returning()
sstables: convert compaction to flat_mutation_reader
mutation_reader: drop consume_flattened_in_thread()
(cherry picked from commit 596ebaed1f)
This series mainly fixes issues with the serialization of promoted
index entries for non-compound schemas and with the serialization of
range tombstones, also for non-compound schemas.
We lift the correct cell name writing code into its own function,
and direct all users to it. We also ensure backward compatibility with
incorrectly generated promoted indexes and range tombstones.
Fixes#2995Fixes#2986Fixes#2979Fixes#2992Fixes#2993
* git@github.com:duarten/scylla.git promoted-index-serialization/v3:
sstables/sstables: Unify column name writers
sstables/sstables: Don't write index entry for a missing row maker
sstables/sstables: Reuse write_range_tombstone() for row tombstones
sstables/sstables: Lift index writing for row tombstones
sstables/sstables: Leverage index code upon range tombstone consume
sstables/sstables: Move out tombstone check in write_range_tombstone()
sstables/sstables: A schema with static columns is always compound
sstables/sstables: Lift column name writing logic
sstables/sstables: Use schema-aware write_column_name() for
collections
sstables/sstables: Use schema-aware write_column_name() for row marker
sstables/sstables: Use schema-aware write_column_name() for static row
sstables/sstables: Writing promoted index entry leverages
column_name_writer
sstables/sstables: Add supported feature list to sstables
sstables/sstables: Don't use incorrectly serialized promoted index
cql3/single_column_primary_key_restrictions: Implement is_inclusive()
cql3/delete_statement: Constrain range deletions for non-compound
schemas
tests/cql_query_test: Verify range deletion constraints
sstables/sstables: Correctly deserialize range tombstones
service/storage_service: Add feature for correct non-compound RTs
tests/sstable_*: Start the storage service for some cases
sstables/sstable_writer: Prepare to control range tombstone
serialization
sstables/sstables: Correctly serialize range tombstones
tests/sstable_assertions: Fix monotonicity check for promoted indexes
tests/sstable_assertions: Assert a promoted index is empty
tests/sstable_mutation_test: Verify promoted index serializes
correctly
tests/sstable_mutation_test: Verify promoted index repeats tombstones
tests/sstable_mutation_test: Ensure range tombstone serializes
correctly
tests/sstable_datafile_test: Add test for incorrect promoted index
tests/sstable_datafile_test: Verify reading of incorrect range
tombstones
sstables/sstable: Rename schema-oblivious write_column_name() function
sstables/sstables: No promoted index without clustering keys
tests/sstable_mutation_test: Verify promoted index is not generated
sstables/sstables: Optimize column name writing and indexing
compound_compat: Don't assume compoundness
(cherry picked from commit bd1efbc25c)
TTL of 1 second may cause the cell to expire right after we write it,
if the second component of current time changes right after it. Use
larger ttl to avoid spurious faliures due to this.
Message-Id: <1511463392-1451-1-git-send-email-tgrabiec@scylladb.com>
(cherry picked from commit 35e404b1a2)
* seastar-dev.git haaawk/flat_reader_remove_read_rows:
sstable_mutation_test: use read_rows_flat instead of read_rows
perf_sstable: use read_rows_flat instead of read_rows
Remove sstable::read_rows
(cherry picked from commit e9ffe36d65)
These patches convert queries (data, mutation and counter) to flat
mutation readers. All of them already use consume_flattened() to
consume a flat stream of data, so the only major missing thing
was adding support for reversed partitions to
flat_mutation_reader::consume().
* pdziepak flat_mutation_reader-queries/v3-rebased:
flat_mutation_reader: keep reference to decorated key valid
flat_muation_reader: support consuming reversed partitions
tests/flat_mutation_reader: add test for
flat_mutation_reader::consume()
mutation_partition: convert queries to flat_mutation_readers
tests/row_cache_stress_test: do not use consume_flattened()
mutation_reader: drop consume_flattened()
streamed_mutation: drop reverse_streamed_mutation()
(cherry picked from commit 6969a235f3)
This series reworks handling of range tombstones in reversed queries
so that they are applied to correct rows. Additionally, the concept
of flipped range tombstones is removed, since it only made it harder
to reason about the code.
Fixes#2982.
* https://github.com/pdziepak/scylla fix-reverse-query-range-tombstone/v2:
streamed_mutation: fix reversing range tombstones
range_tombstone: drop flip()
tests/cql_query_test: test range tombstones and reverse queries
tests/range_tombstone_list: add test for range_tombstone_accumulator
(cherry picked from commit cec5b0a5b8)
"This patch series addresses #2929. The objective is to eliminate global
state from the implementation and use of all access-control functionlity.
I've made every effort to make these patches logically independent and
incremental, but the final patch is big: this was necessary because
eliminating the global instances themselves is an atomic change."
* 'jhk/non_global_auth/v2' of https://github.com/hakuch/scylla:
auth: Switch to sharded service
tracing/trace_keyspace_helper: Use internal `client_state`
auth: Make the QP an explicit dependency
auth: Unify Java class name attributes
auth: Make life-time control more consistent
auth: Move metadata constants
auth: Don't expose internal constant
auth: Extract `permissions_cache`
utils/loading_cache: Include necessary dependency
auth: Fix static constant initialization
auth: Extract `delayed_tasks` from `auth.cc`
This change appears quite large, but is logically fairly simple.
Previously, the `auth` module was structured around global state in a
number of ways:
- There existed global instances for the authenticator and the
authorizer, which were accessed pervasively throughout the system
through `auth::authenticator::get()` and `auth::authorizer::get()`,
respectively. These instances needed to be initialized before they
could be used with `auth::authenticator::setup(sstring type_name)`
and `auth::authorizer::setup(sstring type_name)`.
- The implementation of the `auth::auth` functions and the authenticator
and authorizer depended on resources accessed globally through
`cql3::get_local_query_processor()` and
`service::get_local_migration_manager()`.
- CQL statements would check for access and manage users through static
functions in `auth::auth`. These functions would access the global
authenticator and authorizer instances and depended on the necessary
systems being started before they were used.
This change eliminates global state from all of these.
The specific changes are:
- Move out `allow_all_authenticator` and `allow_all_authorizer` into
their own files so that they're constructed like any other
authenticator or authorizer.
- Delete `auth.hh` and `auth.cc`. Constants and helper functions useful
for implementing functionality in the `auth` module have moved to
`common.hh`.
- Remove silent global dependency in
`auth::authenticated_user::is_super()` on the auth* service in favour
of a new function `auth::is_super_user()` with an explicit auth*
service argument.
- Remove global authenticator and authorizer instances, as well as the
`setup()` functions.
- Expose dependency on the auth* service in
`auth::authorizer::authorize()` and `auth::authorizer::list()`, which
is necessary to check for superuser status.
- Add an explicit `service::migration_manager` argument to the
authenticators and authorizers so they can announce metadata tables.
- The permissions cache now requires an auth* service reference instead
of just an authorizer since authorizing also requires this.
- The permissions cache configuration can now easily be created from the
DB configuration.
- Move the static functions in `auth::auth` to the new `auth::service`.
Where possible, previously static resources like the `delayed_tasks`
are now members.
- Validating `cql3::user_options` requires an authenticator, which was
previously accessed globally.
- Instances of the auth* service are accessed through `external`
instances of `client_state` instead of globally. This includes several
CQL statements including `alter_user_statement`,
`create_user_statement`, `drop_user_statement`, `grant_statement`,
`list_permissions_statement`, `permissions_altering_statement`, and
`revoke_statement`. For `internal` `client_state`, this is `nullptr`.
- Since the `cql_server` is responsible for instantiating connections
and each connection gets a new `client_state`, the `cql_server` is
instantiated with a reference to the auth* service.
- Similarly, the Thrift server is now also instantiated with a reference
to the auth* service.
- Since the storage service is responsible for instantiating and
starting the sharded servers, it is instantiated with the sharded
auth* service which it threads through. All relevant factory functions
have been updated.
- The storage service is still responsible for starting the auth*
service it has been provided, and shutting it down.
- The `cql_test_env` is now instantiated with an instance of the auth*
service, and can be accessed through a member function.
- All unit tests have been updated and pass.
Fixes#2929.
Using "Meyer's singletons" eliminate the problem of static constant
initialization order because static variables inside functions are
initialized only the first time control flow passes over their
declaration.
Fixes#2966.
The wrapper is no longer needed because
read_range_rows returns ::mutation_reader instead of
sstables::mutation_reader and the reader returned from
it keeps the pointer to shared_sstable that was used to
create the reader.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
This will make migration to flat_mutation_reader much
easier and sstables::mutation_reader is going away with
this migration anyway.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
"Otherwise, such strategies couldn't behave as expected when it needs to do STCS."
* 'respecting_stcs_options_v2' of github.com:raphaelsc/scylla:
tests: enable twcs test that relied on size-tiered properties
twcs: respect stcs options by forwarding them to stcs method
lcs: forward stcs options to respect them
stcs: make most_interesting_bucket respect size-tiered options
stcs: make most_interesting_bucket respect thresholds
compaction: make size_tiered_most_interesting_bucket static method of stcs class
stcs: introduce new ctor
stcs: make header self contained
stcs: inline function definition so as not to break one definition rule
Make sure loading_cache::stop() is always called where appropriate:
regardless whether the test failed or there was an exception during the test.
Otherwise a false-alarm use-after-free error may occur.
Fixes#2955
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Message-Id: <1510625736-3109-1-git-send-email-vladz@scylladb.com>
"Fixes #2944."
* tag 'tgrabiec/cache-exception-safety-fixes-v2' of github.com:scylladb/seastar-dev:
tests: row_cache: Add test for exception safety of multi-partition scans
tests: row_cache: Add test for exception safety of single-partition reads
tests: mutation_source_tests: Always print the seed
tests: Disable alloc failure injection in test assertions
tests: Avoid needless copies
row_cache: Fix exception safety of cache_entry::read()
row_cache: scanning_and_populating_reader: Fix exception unsafety causing read to skip data
row_cache: partition_range_cursor: Extract valid() and advance_to() from refresh()
cache_streamed_mutation: Add trace-level logging to cache_streamed_mutation
mvcc: Lift noexcept off partition_snapshot_row_weakref assignment/constructors
cache_streamed_mutation: Make advancing to the next range exception-safe
cache_streamed_mutation: Make add_clustering_row_to_buffer() exception-safe
cache_streamed_mutation: Make drain_tombstones() exception-safe
cache_streamed_mutation: Return void from start_reading_from_underlying()
cache_streamed_mutation: Document invariants related to exception-safety
streamed_mutation: Add reserve_one()
lsa: Guarantee invalidated references on allocating section retry
mvcc: partition_snapshot_row_cursor: Mark allocation points
BOOST_TEST_MESSAGE() is not logged by default, and for some tests we
don't want to enable that because it's too noisy. But we need to know
the seed to reproduce a failure, so we better to always print it.
