Previously the limit was erroneously applied per page
instead of being accumulated, which might have caused returning
too many rows. As of now, LIMIT is handled properly inside
restrictions filter.
Fixes#4100
"
This series adds generating view updates from sstables added through
/upload directory if their tables have accompanying materialized views.
Said sstables are left in /upload directory until updates are generated
from them and are treated just like staging sstables from /staging dir.
If there are no views for a given tables, sstables are simply moved
from /upload dir to datadir without any changes.
Tests: unit (release)
"
* 'add_handling_staging_sstables_to_upload_dir_5' of https://github.com/psarna/scylla:
all: rename view_update_from_staging_generator
distributed_loader: fix indentation
service: add generating view updates from uploaded sstables
init: pass view update generator to storage service
sstables: treat sstables in upload dir as needing view build
sstables,table: rename is_staging to requires_view_building
distributed_loader: use proper directory for opening SSTable
db,view: make throttling optional for view_update_generator
"
This series addresses the problem mentioned in issue 4032, which is a race
between creating a view and streaming sstables to a node. Before this patch
the following scenario is possible:
- sstable X arrives from a streaming session
- we decide that view updates won't be generated from an sstable X
by the view builder
- new view is created for the table that owns sstable X
- view builder doesn't generate updates from sstable X, even though the table
has accompanying views - which is an inconsistency
This race is fixed by making the view builder wait for all ongoing streams,
just like it does for reads and writes. It's implemented with a phaser.
Tests:
unit (release)
dtest(not merged yet: materialized_views_test.TestMaterializedViews.stream_from_repair_during_build_process_test)
"
* 'add_stream_phasing_2' of https://github.com/psarna/scylla:
repair: add stream phasing to row level repair
streaming: add phasing incoming streams
multishard_writer: add phaser operation parameter
view: wait for stream sessions to finish before view building
table: wait for pending streams on table::stop
database: add pending streams phaser
SSTables loaded to the system via /upload dir may sometimes be needed
to generate view updates from them (if their table has accompanying
views).
Fixes#4047
In some cases, sstables put in the upload dir should have view updates
generated from them. In order to avoid moving them across directories
(which then involves handling failure paths), upload dir will also be
treated as a valid directory where staging sstables reside.
Regular sstables that are not needed for view updates will be
immediately moved from upload/ dir as before.
Previous implementation assumes that each SSTable resides directly
in table::datadir directory, while what should actually be used
is directory path from SSTable descriptor.
This patch prevents a regression when adding staging sstables support
for upload/ dir.
Currently registering new view updates is throttled by a semaphore,
which makes sense during stream sessions in order to avoid overloading
the queue. Still, registration also occurs during initialization,
where it makes little sense to wait on a semaphore, since view update
generator might not have started at all yet.
"
Cleanup various cases related to updating of metatdata stats and encoding stats
updating in preparation for 64-bit gc_clock (#3353).
Fixes#4026Fixes#4033Fixes#4035Fixes#4041
Refs #3353
"
* 'projects/encoding-stats-fixes/v6' of https://github.com/bhalevy/scylla:
sstables: remove duplicated code in data_consume_rows_context CELL_VALUE_BYTES
sstables: mc: use api::timestamp_type in write_liveness_info
sstables: mc: sstable_write encoding_stats are const
mp_row_consumer_k_l::consume_deleted_cell rename ttl param to local_deletion_time
memtable: don't use encoding_stats epochs as default
memtable: mc: udpate min_ttl encoding stats for dead row marker
memtable: mc: add comment regarding updating encoding stats of collection tombstones
sstables: metadata_collector: add update tombstone stats
sstables: assert that delete_time is not live when updating stats
sstables: move update_deletion_time_stats to metadata collector
sstables: metadata_collector: introduce update_local_deletion_time_and_tombstone_histogram
sstables: mc: write_liveness_info and write_collection should update tombstone_histogram
sstables: update_local_deletion_time for row marker deletion_time and expiration
Presence checker is constructed and destroyed in the standard
allocator context, but the presence check was invoked in the LSA
context. If the presence checker allocates and caches some managed
objects, there will be alloc-dealloc mismatch.
That is the case with LeveledCompactionStrategy, which uses
incremental_selector.
Fix by invoking the presence check in the standard allocator context.
Fixes#4063.
Message-Id: <1547547700-16599-1-git-send-email-tgrabiec@scylladb.com>
In order to allow other services to wait for incoming streams
to finish, row level repair uses stream phasing when creating
new sstables from incoming data.
Fixes scylladb#4032
During streaming, there's a race between streamed sstables
and view creation, which might result in some tables not being
used to generate view updates, even though they should.
That happens when the decision about view update path for a table
is done before view creation, but after already receiving some sstables
via streaming. These will not be used in view building even though
they should.
Hence, a phaser is used to make the view builder wait for all ongoing
stream sessions for a table to finish before proceeding with build steps.
Refs #4032
"
This mini-series adds counters for the inactive reads registered in the
reader concurrency semaphore.
"
* 'reader-concurrency-semaphore-counters/v6' of https://github.com/denesb/scylla:
tests/querier_cache: use stats to get the no. of inactive reads
reader_concurrency_semaphore: add counters for inactive reads
The existence of LZ4_compress_default is a property of the lz4
library, not seastar.
With this patch scylla does its own configure check instead of
depending on the one done by seastar.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20190114013737.5395-1-espindola@scylladb.com>
"
Currently the logic is scattered between types.*, cql3_types.* and
sstables/mc/writer.cc.
This patchset places all the logic in types.* and makes sure we
correctly add "frozen<...>" and "FrozenType(...)" to the names of
tuples and UDTs.
Fixes#4087
Tests: unit(release)
"
* 'haaawk/4087_v1' of github.com:scylladb/seastar-dev:
Add comment explaining tuple type name creation
Add "FrozenType(...)" to UDT name only when it's frozen
Move "FrozenType(...)" addition to UDT name to user_type_impl
Add "frozen<...>" to tuple CQL name only when it's frozen
Move "frozen<...>" addition to tuple CQL name to tuple_type_impl
Merge make_cql3_tuple_type into tuple_type_impl::as_cql3_type
Add "frozen<...>" to UDT CQL name only when it's frozen
Move "frozen<...>" addition to UDT CQL name to user_type_impl
Why default to an artificial minimum when you can do better
with zero effort? Track the actual minima in the memtable instead.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Update min ttl with expired_liveness_ttl (although it's value of max int32
is not expected to affect the minimum).
Fixes#4041
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
When the row flag has_complex_deletion is set, some collection columns may have
deletion tombstones and some may not. we don't strictly need to update stats
will not affect the encoding_stats anyway.
Fixes#4035
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Now that continuous_data_consumer::position() is meaningful (since
36dd660), we can use our position in the stream to calculate offsets
instead of duplicating state machine in offset calculations.
The value of position() - data.size() always holds the current offset
in the stream.
Message-Id: <1547219234-21182-1-git-send-email-tgrabiec@scylladb.com>
Put package names one per line. This makes it easier to review changes,
and to backport changes to this file. No content changes.
Message-Id: <20190112091024.21878-1-avi@scylladb.com>
Do not allow write access to the sstable list via this accessor. Luckily
there are no violations, and now we enforce it.
Message-Id: <20190111151049.16953-1-avi@scylladb.com>
To keep format compatibiliti we never wrap tuple type name
into "org.apache.cassandra.db.marshal.FrozenType(...)".
Even when the tuple is frozen.
This patch adds a comment in tuple_type_impl::make_name that
explains the situation.
For more details see #4087
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
At the moment Scylla supports only frozen UDTs but
the code should be able to handle non-frozen UDTs as well.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
At the moment Scylla supports only frozen tuples but
the code should be able to handle non-frozen tuples as well.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
