The goal is to simplify flow-control where the order in which
variables are updated depends on their location in the source.
With functions, this is difficult.
The goal is to simplify flow-control where the order in which
variables are updated depends on their location in the source.
With functions, this is difficult.
The goal is to simplify flow-control where the order in which
variables are updated depends on their location in the source.
With functions, this is difficult.
The goal is to simplify flow-control where the order in which
variables are updated depends on their location in the source.
With functions, this is difficult.
The goal is to simplify flow-control where the order in which
variables are updated depends on their location in the source.
With functions, this is difficult.
This unifies the signature with possible_lhs_values(), paving the way
to deduplicating the two functions. We always have the schema and may as
well pass it.
All query plans that try to solve for the possible values a column
(or token, or column-tuple) can take have been converted to set
analyzed_column::solve_for. Recognize that by removing the
fallback path.
This removes the last possible_column_values() call that isn't bound
(using std::bind_front), and will allow moving it to prepare time.
By moving query_options to the end, we can use std::bind_front to
convert it from a build-time to a run-time function that depends
only on the query_options.
Multi-column restrictions (a, b) > (:v1, :v2) do not obey normal
comparison rules. For example, given
(a, b) > (5, 1) AND a <= 5
We see that (a, b) = (5, 2) satisfies the constraint, but if we tried
to solve for the interval
( (5, 1), (5) ]
We'd have to conclude that (5,1) <= (5).
It's possible to extend the CQL type system to support this, but
that would be a lot of work, and in fact the current code doesn't
depend on it (by solving these intersections in its own code path
(multi_column_range_accumulator_builder's prefix3cmp).
So, we just mark such solvers as non-comparable, and generate an
internal error if we try to compare them in make_conjunction.
In statement_restriction's constructor, we check that all the boolean factors
are relations. This means the code to handle a constant here is dead code.
Remove it; while it's good to handle it, it should be handled at the top
level, not in multi-column restriction processing.
range_from_raw_bound processes restrictions of the form
(a, b) > SCYLLA_CLUSTERING_BOUND(?, ?)
indicating that comparisons respect whether columns are reversed or not.
Iterate over expressions during the prepare phase only; generating
"builder" functions to be executed during the query phase.
The get_clustering_bounds() family works in terms of vectors of
clustering ranges (to support IN) and in fact the only caller converts
it to a vector. Converting it immediately simplifies later patching.
multi_column_range_accumulator analyzes an expression containing multi-column
restrictions of the form (a, b) > (?, ?) and simultaneously analyzes
them and solves for the set of intervals that satisfy those restrictions.
Split this into prepare-time phase (that generates "builders", functions
that operator on the accumulator), and a query phase that executes
the builders. Importantly, the expression visitor ends up on the prepare
phase, so it can be merged with other parts of the analysis.
Helper functions of the visitor are made static, since they need to
run during the query phase but the visitor only exists during the
prepare phase.
Lay the groundwork for analyzing multi column clustering bounds by
splitting the function into prepare-time and execute-time parts.
To start with, all of the work is done at query time, but later
patches will move bits into prepare time.
Doing this splits the multi-column processing code into a preparation
phase and an evaluation phase in a single call, making it easier to
further split prepare/evaluate.
Change _clustering_prefix_restrictions and _idx_tbl_ck_prefix
(the latter is the equivalent of the former, for indexed queries),
to use predicate instead of expressions. This lets us do
more of the work of solving restrictions during prepare time.
We only handle single-column restrictions here. Multi-column
restrictions use the existing path.
We introduce two helpers:
- value_set_to_singleton() converts a restriction solution to a singleton
when we know that's the only possible answer
- replace_column_def() overload for predicate, similar to the
existing overload for expressions
There is a wart in get_single_column_clustering_bounds(): we arrive at
his point with the two vectors possibly pointing at different
columns. Previously, possible_lhs_values() did this check while solving.
We now check for it here.
The predicate::on variant gets another member, for clustering key prefixes.
Since everything is still handled by the legacy paths, we mostly
error out.
To allow more work to be carried out during prepare time, wrap
the body in an std::function, which will be called at execution time.
Currently we actually do the work during execution time; but the
way is prepared.
value_for() is a general function that solves for values that
satisfy an expression set to TRUE. This goes against our goal to
prepare solvers for all the expressions we use. Fortunately, it's only
called with one expression, which comes from statement_restrictions, so
we can add an accessor that provides the expression from our own state.
Later, we'll be able to do prepare-time work on it.
During prepare time, build functions for use during execution time.
Currently, the wrappers are very shallow, and practically all the
work is done at execution time. But the stage is set for more peeling.
The index clustering ranges had on_internal_error()s if an index
was not used. They're converted to returning a null function. If
executed (which is never supposed to happen), it will throw
a bad_function_call.
For indexed queries, statement_restrictions calculates _view_schema,
which is passed via get_view_schema() to indexed_select_statement(),
which passes it right back to statement_restrictions via one of three
functions to calculate clustering ranges.
Avoid the back-and-forth and use the stored value. Using a different
value would be broken.
This change allows unifying the signatures of the four functions that
get clustering ranges.
Convert token range restrictions to the predicate format we
introduced earlier, where we have a function to solve for the token
range rather than running the analysis at runtime. Again the truth is
that the function will delegate to possible_partition_token_values()
which actually will do the analysis at runtime, but it's one step closer.
We add a new variant element for predicate::on, since it doesn't
fit the existing element (the token isn't a column).
The expression tree for partition keys is analyzed during runtime:
in partition_range_from_singles() (for example), we call find_binop
and get_subscripted_column() to understand the expression structure.
This analysis is problematic because it has to match the analysis
during prepare time; and they have to evolve in lock step.
Here, we move the analysis to the prepare stage. This is done
by augmenting the expression into a new predicate struct. It
contains the original expression (as a fallback for paths not yet
converted), as well as a solve_for function which contains
a function built at prepare time that embeds all the necessary analysis.
We introduce the `predicate` type which is an augmentation
of boolean expressions. In addition to the expression, we remember
what column the expression is on, and a function that computes
what values the column can take on that would make the expression
true.
The field that says what column the predicate is about is typed
as a variant since later on we will have predicates on non-columns
(the token, or a clustering prefix).
Note that currently the function engages in some run-time analysis of
its own, since it calls possible_lhs_values that itself does analysis,
but this is a step in the right direction.
An indexed SELECT of the from
SELECT ...
WHERE pk['sub'] = ?
is impossible because our indexes do not support frozen maps, and
partition key collections must be frozen. Stop collecting such constructs
for the purpose of determining the partition range. This reduces having
to deal with combinations of restrictions on the column and its entries
later on.
In case we start supporting indexes on frozen maps, leave an
on_internal_error to remind us.
_partition_range_restrictions are a vector of expressions, one per
partition key column, except that it can be empty if there is no
restriction on the partition that can be translated to a read command,
and if the restriction is on a token range, the first element only
is used.
Separate the three cases into distinct structs. After this, additional
work can be done utilizing the specialization.
statement_restrictions::get_partition_key_ranges() re-interprets
the expressions used to specify the partition key. This means that
the analysis phase (determining what those expressions are and how
they are to be used) and the execution phase (using them) are in separate
places. This makes it very hard to refactor while preserving correctness.
As a first step in unifying the two phases, we move the selection
of the strategy (using token, cartesian product, or single partition)
from execution to analysis, by making the if-tree return a function to
be executed at execution time, rather than running the if-tree itself
at execution time.
Prevent copying/moving, that can change the address, and instead enforce
using shared_ptr. Most of the code is already using shared_ptr, so the
changes aren't very large.
To forbid non-shared_ptr construction, the constructors are annotated
with a private_tag tag class.
In preparation for refactoring statement_restrictions, add a simple
and an exhaustive regression test, encoding the index selection
algorithm into the test. We cannot change the index selection algorithm
because then mixed-node clusters will alter the sorting key mid-query
(if paging takes place).
Because the exhaustive space has such a large stack frame, and
because Address Santizer bloats the stack frame, increase it
for debug builds.
Migrate 3 bare skip sites that appeared in upstream/master after the
initial migration:
- test/cluster/test_strong_consistency.py: 2 @pytest.mark.skip →
@pytest.mark.skip_bug (SCYLLADB-1056)
- test/cqlpy/conftest.py: pytest.skip() → skip_env() in
skip_on_scylla_vnodes fixture
Harden the skip_reason_plugin to reject bare @pytest.mark.skip at
collection time with pytest.UsageError instead of warnings.warn().
Add test/pylib_test/test_no_bare_skips.py with three guard tests:
- AST scan for bare pytest.skip() runtime calls
- Real pytest --collect-only against all Python test directories
Update 7 comments/docstrings across 5 files that still referenced
pytest.skip() to reference the typed skip_env() wrapper for
consistency with the migrated code.
Migrate 2 runtime pytest.skip() calls referencing known bugs to use
the typed skip_bug() wrapper from test.pylib.skip_types:
- test/alternator/test_ttl.py: Streams on tablets (#23838)
- test/scylla_gdb/test_task_commands.py: coroutine task not found (#22501)
Migrate runtime pytest.skip() calls across 34 files to use the typed
skip_env() wrapper from test.pylib.skip_types.
These sites skip at runtime because a required feature, config option,
library version, build mode, or runtime topology is not available.
Also fixes 'raise pytest.skip(...)' in test_audit.py — skip_env()
already raises internally, so the explicit raise was incorrect.
Each file gains one new import:
from test.pylib.skip_types import skip_env
Migrate 2 bare @pytest.mark.skip decorators (no reason string) to
@pytest.mark.skip_not_implemented with an explicit reason referencing
issue #3882 (COMPACT STORAGE not implemented).
Migrate 10 @pytest.mark.skip decorator sites to
@pytest.mark.skip_not_implemented across 5 test files where the
skip reason indicates a feature not yet implemented.
In Alternator's HTTP API, response headers can dominate bandwidth for
small payloads. The Server, Date, and Content-Type headers were sent on
every response but many clients never use them.
This patch introduces three Alternator config options:
- alternator_http_response_server_header,
- alternator_http_response_disable_date_header,
- alternator_http_response_disable_content_type_header,
which allow customizing or suppressing the respective HTTP response
headers. All three options support live update (no restart needed).
The Server header is no longer sent by default; the Date and
Content-Type defaults preserve the existing behavior.
The Server and Date header suppression uses Seastar's
set_server_header() and set_generate_date_header() APIs added in
https://github.com/scylladb/seastar/pull/3217. This patch also
fixes deprecation warnings from older Seastar HTTP APIs.
Tests are in test/alternator/test_http_headers.py.
Fixes https://scylladb.atlassian.net/browse/SCYLLADB-70Closesscylladb/scylladb#28288
DynamoDB Streams API can only convey a single parent per stream shard.
Tablet merges produce two parents, making them incompatible with
Alternator Streams. This series blocks tablet merges when streams are
active on a tablet table.
For CreateTable, a freshly created table has no pending merges, so
streams are enabled immediately with tablet merges blocked.
For UpdateTable on an existing table, stream enablement is deferred:
the user's intent is stored via `enable_requested`, tablet merges are
blocked (new merge decisions are suppressed and any active merge
decision is revoked), and the topology coordinator finalizes enablement
once no in-flight merges remain.
The topology coordinator is woken promptly on error injection release
and tablet split completion, reducing finalization latency from ~60s
to seconds.
`test_parent_children_merge` is marked xfail (merges are now blocked),
and downward (merge) steps are removed from `test_parent_filtering` and
`test_get_records_with_alternating_tablets_count`.
Not addressed here: using a topology request to preempt long-running
operations like repair (tracked in SCYLLADB-1304).
Refs SCYLLADB-461
Closesscylladb/scylladb#29224
* github.com:scylladb/scylladb:
topology: Wake coordinator promptly for stream enablement lifecycle
test/cluster: Test deferred stream enablement on tablet tables
alternator/streams: Block tablet merges when Alternator Streams are enabled
Currently, ListStreams paging works by looking in the list of tables
for ExclusiveStartStreamArn and starting there. But it's possible
that during the paging process, one of the tables got deleted and
ExclusiveStartStreamArn no longer points to an existing table. In
the current implementation this caused the paging to stop (think
it reached the end).
The solution is simple: ListStreams will now sort the list of tables
by name (it anyway needs to be sorted by something to be consistent
across pages), and will look with std::upper_bound for the first
table *after* the ExclusiveStartStreamArn - we don't need to find
that table name itself.
The patch also includes a test reproducing this bug. As usual, the
test passes on DynamoDB, fails on Alternator before this patch,
and passes with the patch.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
