"When mutation reader enters the partition using index,
streamed_mutation object is returned to the user before the row start
fragment is processed. In that case, when we process the row start, we
should ignore it and not call setup_for_partition() again. That may
override user's fast_forward_to() request."
* 'tgrabiec/fix-initial-fast-forward-to-for-single-key-sstable-readers' of github.com:scylladb/seastar-dev:
tests: mutation_source_test: Test forwarding in single-key readers
sstables: Remove unused code
sstables: mutation_reader: Fix setup_for_partition() being called twice in some cases
sstables: Fix verify_end_state() to tolerate ATOM_START_2 state
"This series adds private repository support to scylla-housekeeping"
* 'amnon/housekeeping_private_repo_v3' of github.com:cloudius-systems/seastar-dev:
scylla-housekeeping service: Support private repositories
scylla-housekeeping-upstart: Use repository id, when checking for version
scylla-housekeeping: support private repositories
From http://github.com/avikivity/scylla exponential-sharder/v3.
The sharder, which takes a range of tokens and splits it among shards, is
slow with large shard count and the default
murmur3_partitioner_ignore_msb_bits.
This patchset fixes excessive iteration in sstable sharding metadata writer and
nonsignular range scans.
Without this patchset, sealing a memtable takes > 60 ms on a 48-shard
system. With the patchset, it drops below the latency tracker threshold I
used (5 ms).
Nonsingular queries used exponential expansion of the token space to
avoid spending too much cpu time on near-empty tables, but the generation
of the search space was itself exponential. Switch to the exponential sharder
which has linear cost.
The "exponentiality" is not carried over from one range to another, because
we expect one or two ranges (two ranges result from a wrapped around thrift
token range).
Like the regular sharder, the exponential sharder divides a range into
subranges owned by individual ranges. Unlike the regular sharder, it
generates ever-increasing subranges, spanning more and more shards, and
eventually returns several subranges per shard. To avoid using
exponential cpu and memory, subranges belonging to a single shard are merged,
and a flag is set to indicate the subranges are not ordered wrt. each other.
Right now, next_token_for_shard() only allows iterating linearly in shard
order. Add the ability to select a specific shard to skip to (in case we're
only interested in a single shard), and to select larger ranges (so that
exponential increases are not implemented by iteration).
The partitioners now depend on smp::count to be initialized correctly,
but smp::count isn't available at static initialization time.
The scylla executable isn't affected because it calls set_global_partitioner()
after smp::count has been initialized.
Fix by deferring initialization to the first global_partitioner() call.
"This patch ensures we read the base table rows that an update
is modifying, in order to correctly calculate the set of
materialized view updates.
The read-before-write is performed on the shard applying the
update and attempts to do a precise read of the rows being modified,
which can be more than one in case of ranged deletions or a
batch update."
* 'materialized-views/read-existing/v2' of https://github.com/duarten/scylla:
database: Read existing base mutations
db/view: Calculate clustering ranges for MV read-before-write query
db/view: Replace entry if cells don't match
view_info: Store base regular col in the view's PK as column_id
compound_view_wrapper: Add tri_compare
bound_view: Build range bound from bound_view
clustering_bounds_comparator: Enable Range concept
range: Add lvalue version of transform()
tests: Add test case for nonwrapping_range::intersection()
nonwrapping_range: Add intersection() function
When generating updates for a materialized view we need to read the
existing base row, to be able to determine the primary key of the view
row the new base update will supplant, in case the view includes a
base non-primary key column in its own primary key. That old view row
will be tombstoned or updated, if it exists, depending on the difference
between the new base row and the existing one, if any.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Introduce the calculate_affected_clustering_ranges() function to
calculate the smallest subject of affected clustering ranges that we
need to query for.
The update_requires_read_before_write() function checks whether
a view is potentially affected by the base update.
The patch also cleans up the may_be_affected_by() function.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
If a base table regular columns is part of the view's pk, and if that
column changes, we should replace the entry, by deleting the row(s)
with the old value and inserting a new one.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch stores the base_non_pk_column_in_view column as column_id,
which is more convenient, and it also stores a two-level optional to
encode both lazy initialization and the absence of such a column.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
We introduce the bound_view::to_range_bound() function, which builds a
wrapping_range or nonwrapping_range bound from a bound_view.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
intersection() returns an optional range with the intersection of the
this range and the other, specified range.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
"Defines origin v3-format for system/schema tables, and use them for
schema storage/retrival.
Includes a legacy_schema_migrator implementation/port from origin. Note
that since we don't support features like triggers, functions and
aggregates, it will bail if encountering such a feature used.
Note also that this patch set does not convert the "hints" and
"backlog" tables, even though these have changed in v3 as well.
That will be a separate patch set.
Tested against dtests. Note that patches for dtest + ccm
will follow."
* 'calle/systemtables' of github.com:cloudius-systems/seastar-dev: (36 commits)
legacy_schema_migrator: Actually truncate legacy schema tables on finish
database: Extract "remove" from "drop_columnfamily"
v3 schema test fixes
thrift: Update CQL mapping of static CFs
schema_tables: Use v3 schema tables and formats
type_parser: Origin expects empty string -> bytes_type
cf_prop_defs: Add crc_check_chance as recognized (even if we don't use)
types_test: v3 style schemas enforce explicit "frozen" in tupes/ut:s
cql3_type: v3 to_string
cql_types: Introduce cql3_type::empty and associate with empty data_type
schema: rename column accessors to be in line with origin
schema: Add "is_static_compact_table"
schema_builder: Add helper to generate unique column names akin origin
schema: Add utility functions for static columns
schema: Use heterogeneous comparator for columns bounds
cql3_type_parser: Resolve from cql3 names/expressions
cql3_type: Add "prepare_interal" and "references_user_type"
cql3::cql3_type: Add prepare_internal path using only "local" holders
cql3_type: Add virtual destructor.
database/main: encapsulate system CF dir touching
...
* seastar 4a3118c...45b718b (7):
> tests: make connect_test use a random port
> log: Introduce log.info0
> configure.py: link to DPDK PMD drivers which are already built on build/dpdk and enabled by default on DPDK config
> Update fmt submodule
> perftune: fix perftune.py IndexError when NIC uses less IRQs than requested.
> build: Add more required build dependencies to the Dockerfile
> Prometheus: Reserve in protobuf object before iterating
"Tested with:
- test.py --mode relase
- debug/test-serialization
- c-s with both debug and relase compiled scylla with authentication enabled:
cassandra-stress write n=10000 no-warmup -rate threads=10 -mode native unprepared cql3 user='cassandra' password='cassandra'
Tested with:
- test.py --mode relase
- debug/test-serialization
- c-s with both debug and relase compiled scylla with authentication enabled:
cassandra-stress write n=10000 no-warmup -rate threads=10 -mode native unprepared cql3 user='cassandra' password='cassandra'"
* 'compress_tracing_session_id-v6' of github.com:cloudius-systems/seastar-dev:
cql_server::response: rework the tracing session ID insertion
utils::UUID: align the UUID serialization API with the similar API of other classes in the project
utils: serialization: unify the variety of serialize_XXX(...)
cql_server::response: rework the compress(...) method
cql_server::response: store the frame flags inside the class
"This series fixes a set of regressions introduced by
f7bc88734a, resulting in two failed
tests:
testDenseNonCompositeTable(org.apache.cassandra.cql3.validation.operations.CreateTest)
and
testStaticColumnsWithDistinct(org.apache.cassandra.cql3.validation.entities.StaticColumnsTest)"
* 'cql-fixes/v1' of github.com:duarten/scylla:
update_statement: Reject empty values for dense clustering key
modification_statement: Fix detection of clustering keys
cql3/restrictions/statement_restrictions: Consider statement type
cql3/statements/modification_statement: Extract statement_type
Insert the tracing session ID into the response body in the cql_server::response constructor.
Fixes#2356
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
The standard serialization API (e.g. in data_value) includes the following methods:
size_t serialized_size() const;
void serialize(bytes::iterator& it) const;
bytes serialize() const;
Align the utils::UUID API with the pattern above.
The only addition is that we are going to make an output iterator parameter of a second method above
a template so that we may serialize into different output sources.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Use the same templated implementation for all different serialize_XXX(...).
The chosen implementation is based on the std::copy_n(char*, size, OutputIterator),
which is heavily optimized and will be using memcpy/memmove where possible.
This patch also removes the not needed specializations that accept signed integer
values since we were casting them to unsigned value anyway.
The std::ostream based specifications are also removed since they are not used
anywhere except for a test-serialization.cc and adjusting the ostream to the iterator
is a single-liner.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Cleanup the compress(...) method interface:
- Encapsulate the technical details inside the method:
- Re-write the _body inside the method instead of returning it.
- Set the response::_flags inside the method.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
It makes a lot more sense to keep the flags mask inside the response and update it each time
the corresponding feature is set instead of holding the separate components like tracing state
pointer.
This patch adds this ability to set the flags.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
The code that removes each sstable runs in a thread. Parallel
removing of a lot of sstables may start a lot of threads each of which
is taking 128k for its stack. There is no much benefit in running
deletion in parallel anyway, so fix it by deleting sstables sequentially.
Fixes#2384
Message-Id: <20170516103018.GQ3874@scylladb.com>
When mutation reads enters the partition using index,
streamed_mutation object is returned to the user before the row start
fragment is processed. In that case, when we process the row start, we
should ignore it and not call setup_for_partition() again. That may
override user's fast_forward_to() request.
We would be in that state if consume_row_start() returns porceed::yes
and the stream ends after that. This can happen if slicing using
promoted index determined that there are no cells in the partition in
the range.
Newer setuptools parse_version() don't like dashed version strings,
so we should trim it to avoid false negative version_compare() checks.
Signed-off-by: Alexys Jacob <ultrabug@gentoo.org>
Message-Id: <20170511162646.22129-1-ultrabug@gentoo.org>
