* 'raft-misc-v3' of github.com:scylladb/scylla-dev:
raft: rename snapshot into snapshot_descriptor
raft: drop snapshot if is application failed
raft: fix local snapshot detection
raft: replication_test: store multiple snapshots in a state machine
raft: do not wait for entry to become stable before replicate it
"
There are 4 places out there that do the same steps parsing
"client_|server_encryption_options" and configuring the
seastar::tls::creds_builder with the values (messaging, redis,
alternator and transport).
Also to make redis and transport look slimmer main() cleans
the client_encryption_options by ... parsing it too.
This set introduces a (coroutinized) helper to configure the
creds_builder with map<string, string> and removes the options
beautification from main.
tests: unit(dev), dtest.internode_ssl_test(dev)
"
* 'br-generalize-tls-creds-builder-configuration' of https://github.com/xemul/scylla:
code: Generalize tls::credentials_builder configuration
transport, redis: Do not assume fixed encryption options
messaging: Move encryption options parsing to ms
main: Open-code internode encryption misconfig warning
main, config: Move options parsing helpers
The link to the docker build instructions was outdated - from the time
our docker build was based on a Redhat distribution. It no longer is,
it's now based on Ubuntu, and the link changed accordingly.
Fixes#9276.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210901083055.445438-1-nyh@scylladb.com>
...when using Segment/TotalSegment option.
The requirement is not specified in DynamoDB documents, but found
in DynamoDB Local:
{"__type":"com.amazon.coral.validate#ValidationException",
"message":"Exclusive start key must lie within the segment"}
Fixes#9272
Signed-off-by: Liu Lan <liulan_yewu@cmss.chinamobile.com>
Closes#9270
Return the pre- 6773563d3 behavior of demanding ALLOW FILTERING when partition slice is requested but on potentially unlimited number of partitions. Put it on a flag defaulting to "off" for now.
Fixes#7608; see comments there for justification.
Tests: unit (debug, dev), dtest (cql_additional_test, paging_test)
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
Closes#9126
* github.com:scylladb/scylla:
cql3: Demand ALLOW FILTERING for unlimited, sliced partitions
cql3: Track warnings in prepared_statement
test: Use ALLOW FILTERING more strictly
cql3: Add statement_restrictions::to_string
When a query requests a partition slice but doesn't limit the number
of partitions, require that it also says ALLOW FILTERING. Although
do_filter() isn't invoked for such queries, the performance can still
be unexpectedly slow, and we want to signal that to the user by
demanding they explicitly say ALLOW FILTERING.
Because we now reject queries that worked fine before, existing
applications can break. Therefore, the behavior is controlled by a
flag currently defaulting to off. We will default to "on" in the next
Scylla version.
Fixes#7608; see comments there for justification.
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
Introduce a general-purpose search and replace function to manipulate
expressions, and use it to simplify replace_column_def() and
replace_token().
Closes#9259
* github.com:scylladb/scylla:
cql3: expr: rewrite replace_token in terms of search_and_replace()
cql3: expr: rewrite replace_column_def in terms of search_and_replace()
cql3: expr: add general-purpose search-and-replace
"
When cloning throws in the middle it may leak some child
nodes triggering the respective assertion in node destructor.
Also there's a chance to mis-assert the linear node roll-back.
tests: unit(dev)
"
Fixes#9248
Backport: 4.5
* 'br-btree-clone-exceptions-2' of https://github.com/xemul/scylla:
btree: Add commens in .clone() and .clear()
btree, test: Test exception safety and non-leakness of btree::clone_from
btree, test: Test key copy constructor may throw
btree: Dont leak kids on clone roll-back
btree: Destroy, not drop, node on clone roll-back
There are two tricky places about corner leaves pointers
managements. Add comments describing the magic.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Use search_and_replace() to simplify replace_token(). Note the conversion
does not have 100% fidelity - the previous implementation throws on
some impossible subexpression types, and the new one passes them
through. It should be the caller's responsibility anyway, not a
side effect of replacing tokens, and since these subexpressions are
impossible there is no real effect on execution.
Note that this affects only TOKEN() calls on the partition key
columns in the right order. Other uses of the token function
(say with constants) won't be translated to the token subexpression
type. So something like
WHERE token(pk) = token(?)
would only see the left-hand side replaced, not the right-hand
side, even if it were an expression rather than a term.
We're won't introduce new expression types that are equivalent
to column_value, and search_and_replace() takes care of all
expressions that need to recurse, so we don't need std::visit()
for the search/replace lambda.
Add a recursive search-and-replace function on expressions. The
caller provides a search/replace function to operate on subexpressions,
returning nullopt if they want the default behavior of recursively
copying, or a new expression to terminate the search (in the current
subtree) and replace the current node with the returned expression.
To avoid endlessly specifying the subexpression types that get the
the common behavior (copying) since they don't contain any subexpressions,
we add a new concept LeafExpression to signify them.
Existing functions such as replace_token() can be reimplemented in
terms of search_and_replace, but that is left for later.
When failed-to-be-cloned node cleans itself it must also clear
all its child nodes. Plain destroy() doesn't do it, it only
frees the provided node.
fixes: #9248
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The node in this place is not yet attached to its parent, so
in btree::debug::yes (tests only) mode the node::drop()'s parent
checks will access null parent pointer.
However, in non-tesing runtime there's a chance that a linear
node fails to clone one of its keys and gets here. In this case
it will carry both leftmost and rightmost flags and the assertion
in drop will fire.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The (oddly-placed) document dist/docker/debian/README.md explains how a
developer can build a Scylla docker image using a self-built Scylla
executable.
While the document begins by saying that you can "build your own
Scylla in whatever build mode you prefer, e.g., dev.", the rest of the
instructions don't fit this example mode "dev" - the second command does
"ninja dist-deb" which builds *all* modes, while the third command
forgets to pass the mode at all (and therefore defaults to "release").
The forth command doesn't work at all, and became irrelevant during
a recent rewrite in commit e96ff3d.
This patch modifies the document to fix those problems.
It ends with an example of how to run the resulting docker image
(this is usually the purpose of building a docker image - to run it
and test it). I did this example using podman because I couldn't get
it to work in docker. Later we can hopefully add the corresponding
docker example.
Fixes#9263.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210829182608.355748-1-nyh@scylladb.com>
Our docker image accepts various command-line arguments and translates
them into Scylla arguments. For example, Alternator's getting-started
document has the following example:
```
docker run --name scylla -d -p 8000:8000 scylladb/scylla-nightly:latest
--alternator-port=8000 --alternator-write-isolation=always```
Recently, this stopped working and the extra arguments at the end were
just ignored.
It turns out that this is a regression caused by commit
e96ff3d82d that changed our docker image
creation process from Dockerfile to buildah. While the entry point
specified in Dockerfile was a string, the same string in buildah has
a strange meaning (an entry point which can't take arguments) and to
get the original meaning, the entry point needs to be a JSON array.
This is kind-of explained in https://github.com/containers/buildah/issues/732.
So changing the entry point from a string to a JSON array fixes the
regression, and we can again pass arguments to Scylla's docker image.
Fixes#9247.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210829180328.354109-1-nyh@scylladb.com>
We have in our Ninja build file various targets which ask to build just
a single build mode. For example, "ninja dev" builds everything in dev
mode - including Scylla, tests, and distribution artifacts - but
shouldn't build anything in other build modes (debug, release, etc.),
even if they were previously configured by configure.py.
However, we had a bug where these build-mode-specific targets
nevertheless compiled *all* configured modes, not just the requested
mode.
The bug was introduced in commit edd54a9463 -
targets "dist-server-compat" and "dist-unified-compat" were introduced,
but instead of having per-build-mode versions of these targets, only
one of each was introduced building all modes. When these new targets
were used in a couple of places in per-build-mode targets, it forced
these targets to build all modes instead of just the chosen one.
The solution is to split the dist-server-compat target into multiple
dist-server-compat-{mode}, and similarly split dist-unified-compat.
The unsplit target is also retained - for use in targets that really
want all build modes.
Fixes#9260.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210829123418.290333-1-nyh@scylladb.com>
The code assumes that the snapshot that was taken locally is never
applied. Currently logic to detect that is flawed. It relies on an
id of a most recently applied snapshot (where a locally taken snapshot
is considered to be applied as well). But if between snapshot creation
and the check another local snapshot is taken ids will not match.
The patch fixes this by propagating locality information together with
the snapshot itself.
State machine should be able to store more then one snapshot at a time
(one may be the currently used one and another is transferred from a
leader but not applied yet).
Since io_fiber persist entries before sending out messages even non
stable entries will become stable before observed by other nodes.
This patch also moves generation of append messages into get_outptut()
call because without the change we will lose batching since each
advance of last_idx will generate new append message.
This method has nothing to do with storage service and
is only needed to move feature service options from one
method to another. This can be done by the only caller
of it.
tests: unit(dev)
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Message-Id: <20210827133954.29535-1-xemul@scylladb.com>
Before this commit, the service_level_controller will notify
the subscribers on stale deletes, meaning, deletes of localy
non exixtent service_levels.
The code flow shouldn't ever get to such a state, but as long
as this condition is checked instead of being asserted it is
worthwhile to change the code to be safe.
Closes#9253
This series converts the `term::raw` objects the grammar produces to expressions.
For each grammar production, an expression type is either reused or created.
The term::raw methods are converted to free functions accepting expressions as
input (but not yet generating expressions as output).
There is some friction because the original code base had four different expression
domains: `term`, `term::raw`, `selectable`, and `selectable::raw`. They don't match
exactly, so in some cases we need to add additional state to distinguish between them.
There are also many run-time checks introduced (on_internal_error) since the union
of the domains is much larger than each individual domain.
The method used is to erect a bi-directional bridge between term::raw and expressions,
convert various term::raw subclasses one by one, and then remove the bridge and
term::raw.
Test: unit (dev).
Closes#9170
* github.com:scylladb/scylla:
cql3: expr: eliminate column_specification_or_tuple
cql3: expr: hide column_specification_or_tuple
cql3: term::raw: remove term::raw and scaffolding
cql3: grammar: collapse conversions between term::raw and expressions
cql3: relation: convert to_term() to experssions
cql3: functions: avoid intermediate conversions to term::raw
cql3: create_aggregate_statement: convert term::raw to expression
cql3: update_statement, insert_statement: convert term::raw to expression
cql3: select_statement: convert term::raw to expression
cql3: token_relation: convert term::raw to expressions
cql3: operation: convert term::raw to expression
cql3: multi_column_relation: convert term::raw to expressions
cql3: single_column_relation: convert term::raw to expressions
cql3: column_condition: convert term::raw to expressions
cql3: expr: don't convert subexpressions to term::raw during the prepare phase
cql3: attributes: convert to expressions
cql3: expr: introduce test_assignment_all()
cql3: expr: expose prepare_term, test_assignment in the expression domain
cql3: expr: provide a bridge between expressions and assignment_testable
cql3: expr, user types: convert user type literals to expressions
cql3: selection: make selectable.hh not include expr/expresion.hh
cql3: sets, user types: move user types raw functions around
cql3: expr, sets, maps: convert set and map literals to collection_constructor
cql3: sets, maps, expr: move set and map raw functions around
cql3: expr, lists: convert lists::literal to new collection_constructor
cql3: lists, expr: move list raw functions around
cql3: tuples, expr: convert tuples::literal to expr::tuple_constructor
cql3: expr, tuples: deinline and move tuple raw functions
cql3: expr, constants: convert constants::literal to untyped_constant
cql3: constants: move constants::literal implementation around
cql3: expr, abstract_marker: convert to expressions
cql3: column_condition: relax types around abstact_marker::in_raw
cql3: tuple markers: deinline and rearrange
cql3: abstract_marker, term_expr: rearrange raw abstract marker implementation
cql3: expr, constants: convert cql3::constants::null_literal to new cql3::expr::null
cql3: expr, constants: deinline null_literal
cql3: constants: extricate cql3::constants::null_literal::null_value from null_literal
cql3: term::raw, expr: convert type casts to expressions
cql3: type_cast: deinline some methods
cql3: expr: prepare expr::cast for unprepared types
cql3: expr, functions: move raw function calls to expressions
cql3: expr, term::raw: add conversions between the two types
cql3: expr, term::raw: add reverse bridge
cql3: term::raw, expr: add bridge between term::raw and expressions
cql3: eliminate multi_column_raw
cql3: term::raw, multi_column_raw: unify prepare() signatures
Whenever a node_head_ptr is assigned to nil_root, the _backref inside it is
overwritten. But since nil_root is shared between shards, this causes severe
cache line bouncing. (It was observed to reduce the total write throughput
of Scylla by 90% on a large NUMA machine).
This backreference is never read anyway, so fix this bug by not writing it.
Fixes#9252Closes#9246
column_specification_or_tuple is now used internally, wrapping and
a column_specification or a vector and immediately unwrapping in
the callee. The only exceptions are bind_variable and tuple_constructor,
which handles both cases.
Use the underlying types directly instead, and add dispatching
to prepare_term_multi_column() for the two cases it handles.
column_specification_or_tuple was introduced since some terms
were prepared using a single receiver e.g. (receiver = <term>) and
some using multiple receivers (e.g. (r1, r2) = <term>. Some
term types supported both.
To hide this complexity, the term->expr conversion used a single
interface for both variations (column_expression_or_tuple), but now
that we got rid of the term class and there are no virtual functions
any more, we can just use two separate functions for the two variants.
Internally we still use column_expression_or_tuple, it can be
removed later.
The grammar now talks to expression API:s solely, so it can be converted
internally to expressions too. Calls to as_term_raw() and as_expression()
are removed, and productions return expressions instead of term::raw:s.
Change term::raw in multi_column_relation to expressions. Because a single
raw class is used to represent multiple shapes (IN ? and IN (x, y, z)),
some of the expressions are optional, corresponding to nullables before the
conversion.
to_term() is not converted, since it's part of the larger relation
hierarchy.
Change term::raw in single_column_relation to expressions. Because a single
raw class is used to represent multiple shapes (IN ? and IN (x, y, z)),
some of the expressions are optional, corresponding to nullables before the
conversion.
to_term() is not converted, since it's part of the larger relation
hierarchy.
Change term::raw in column_condition::raw to expressions. Because a single
raw class is used to represent multiple shapes (IN ? and IN (x, y, z)),
some of the expressions are optional, corresponding to nullables before the
conversion.
to_term() is not converted, since it's part of the larger relation
hierarchy.
Now that we have the prepare machinery exposed as expression API:s
(not just term::raw) we can avoid conversions from expressions to
term::raw when preparing subexpressions.
Convert the three variables in attrbutes::raw to expressions. Since
those attributes are optional, use std::optional to indicate it
(since we can't rely on shared_ptr<term::raw> being null).
