Code that waited for all remote view updates was already there. This
commit modifies the conditions of this wait to take into account the
"synchronous mode" (enabled when db::SYNCHRONOUS_VIEW_UPDATES_TAG_KEY is
set).
This commit defines a new tag key (SYNCHRONOUS_VIEW_UPDATES_TAG_KEY) to
be used for marking "synchronous mode" views. This key is used in
`cf_prop_defs::apply_to_builder` if the properties contain
KW_SYNCHRONOUS_UPDATES.
Tags are a useful mechanism that could be used outside of alternator
namespace. My motivation to move tags_extension and other utilities to
db/tags/ was that I wanted to use them to mark "synchronous mode" views.
I have extracted `get_tags_of_table`, `find_tag` and `update_tags`
method to db/tags/utils.cc and moved alternator/tags_extension.hh to
db/tags/.
The signature of `get_tags_of_table` was changed from `const
std::map<sstring, sstring>&` to `const std::map<sstring, sstring>*`
Original behavior of this function was to throw an
`alternator::api_error` exception. This was undesirable, as it
introduced a dependency on the alternator module. I chose to change it
to return a potentially null value, and added a wrapper function to the
alternator module - `get_tags_of_table_or_throw` to keep the previous
throwing behavior.
This property can be used with CREATE MATERIALIZED VIEW and ALTER
MATERIALIZED VIEW statements. Setting it allows global views to enter
"synchronous mode". In this mode, all view updates are also applied
synchronously as if the view was local. This may reduce their
availability, but has the benefit of propagating a potential
inconsistency risk (in form of a write error) to the user, who can
respond to it appropriately (e.g. retry the write or fix the view
later).
Fix https://github.com/scylladb/scylla-docs/issues/4041
I've added the upgrade guides from 2022.x.y to 2022.x.z. They are based on the previous upgrade guides for patch releases.
Closes#11104
* github.com:scylladb/scylla:
doc: add the new upgrade guide to the toctree
doc: add the upgrage guides from 2022.x.y to 2022.x.z
Expiring entries are added when a message is received from an unknown
host. If the host is later added to the raft configuration they become
non expiring. After that they can only be removed when the host is
dropped from the configuration, but they should never become expiring
again.
Refs #10826
This patch avoids unncessary CACHE_HITRATES updates through gossip.
After this patch:
Publish CACHE_HITRATES in case:
- We haven't published it at all
- The diff is bigger than 1% and we haven't published in the last 5 seconds
- The diff is really big 10%
Note: A peer node can know the cache hitrate through read_data
read_mutation_data and read_digest RPC verbs which have cache_temperature in
the response. So there is no need to update CACHE_HITRATES through gossip in
high frequency.
We do the recalculation faster if the diff is bigger than 0.01. It is useful to
do the calculation even if we do not publish the CACHE_HITRATES though gossip,
since the recalculation will call the table->set_global_cache_hit_rate to set
the hitrate.
Fixes#5971Closes#11079
In issue #10966, a user noticed that Alternator writes may be reordered
(a later write to an item is ignored with the earlier write to the same
item "winning") if Scylla nodes do not have synchronized time and if
always_use_lwt write isolation mode is not used.
In this patch I add to docs/alternator/compatibility.md a section about
this issue, what causes it, and how to solve or at least mitigate it.
Fixes#10966
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closes#11094
The Scylla Alternator documentation is now part of the Scylla user documentation (previously it was dev documentation).
This PR updates the links to the Alternator documentation.
Closes#11089
* github.com:scylladb/scylla:
doc: update the link to Alternator for DynamoDB users
doc: fix the links to Alternator
This PR removes all code that used classes `restriction`, `restrictions` and their children.
There were two fields in `statement_restrictions` that needed to be dealt with: `_clustering_columns_restrictions` and `_nonprimary_key_restrictions`.
Each function was reimplemented to operate on the new expression representaiion and eventually these fields weren't needed anymore.
After that the restriction classes weren't used anymore and could be deleted as well.
Now all of the code responsible for analyzing WHERE clause and planning a query works on expressions.
Closes#11069
* github.com:scylladb/scylla:
cql3: Remove all remaining restrictions code
cql3: Move a function from restrictions class to the test
cql3: Remove initial_key_restrictions
cql3: expr: Remove convert_to_restriction
cql3: Remove _new from _new_nonprimary_key_restrictions
cql3: Remove _nonprimary_key_restrictions field
cql3: Reimplement uses of _nonprimary_key_restrictions using expression
cql3: Keep a map of single column nonprimary key restrictions
cql3: Remove _new from _new_clustering_columns_restrictions
cql3: Remove _clustering_columns_restrictions from statement_restrictions
cql3: Use a variable instead of dynamic cast
cql3: Use the new map of single column clustering restrictions
cql3: Keep a map of single column clustering key restrictions
cql3: Return an expression in get_clustering_columns_restrctions()
cql3: Reimplement _clustering_columns_restrictions->has_supporting_index()
cql3: Don't create single element conjunction
cql3: Add expr::index_supports_some_column
cql3: Reimplement has_unrestricted_components()
cql3: Reimplement _clustering_columns_restrictions->need_filtering()
cql3: Reimplement num_prefix_columns_that_need_not_be_filtered
cql3: Use the new clustering restrictions field instead of ->expression
cql3: Reimplement _clustering_columns_restrictions->size() using expressions
cql3: Reimplement _clustering_columns_restrictions->get_column_defs() using expressions
cql3: Reimplement _clustering_columns_restrictions->is_all_eq() using expressions
cql3: expr: Add has_only_eq_binops function
cql3: Reimplement _clustering_columns_restrictions->empty() using expressions
This PR is V2 of https://github.com/scylladb/scylla/pull/11065.
The scope of updates:
- Created a _/cql/_ folder.
- Moved all the CQL-related pages from _/getting-started/_ to _/cql/_ .
- Moved the _cql-extensions.md_ file from _/dev/_ to _/cql/_ .
- Removed the outdated files and references.
- Updated the links to the CQL-related pages.
Closes#11083
* github.com:scylladb/scylla:
doc: update the links following the content reorganization
doc: remove the outdated cql pages and delete them from the indexes
doc: add index.rst for the cql folder and add it to toctree
doc: move cql-extensions.md from the dev docs to the cql folder
doc: move the CQL pages from getting-started to cql
doc: add redirections for the CQL pages
* seastar 6d4a0cb7a3...1d4432ed28 (11):
> rpc: Ignore failed future in connection::send()
> install-dependencies: centos-{7,8}: use {DTS,GTS}-11 instead of {DTS,GTS}-9
> coroutine: change access specifier of seastar::task member
> Merge 'build: try to enable io_uring if it is not specified' from Kefu Chai
> build: find_package() only if necessary
> Merge "rpc: handle connection negotiation error during stream sink creation " from Gleb
> build: try to enable io_uring if it is not specified
> test: rpc: add test that inject error during stream connection negotiation.
> test: rpc: inject errors only on streaming connections
> test: rpc: allow specifying after what limit a connection start producing errors
> rpc: do not destroy stream connection without stopping in case of negotiation failure
Fixes#10943Closes#11082
query_result was the wrong place to put last position into. It is only
included in data-responses, but not on digest-responses. If we want to
support empty pages from replicas, both data and digest responses have
to include the last position. So hoist up the last position to the
parent structure: query::result. This is a breaking change inter-node
ABI wise, but it is fine: the current code wasn't released yet.
Closes#11072
Whether a server can vote in a Raft configuration is not part of the
address. `server_address` was used in many context where `can_vote` is
irrelevant.
Split the struct: `server_address` now contains only `id` and
`server_info` as it did before `can_vote` was introduced. Instead we
have a `config_member` struct that contains a `server_address` and the
`can_vote` field.
Also remove an "unsafe" constructor from `server_address` where `id` was
provided but `server_info` was not. The constructor was used for tests
where `server_info` is irrelevant, but it's important not to forget
about the info in production code. Replace the constructor with helper
functions which specify in comments that they are supposed to be used in
tests or in contexts where `info` doesn't matter (e.g. when checking
presence in an `unordered_set`, where the equality operator and hash
operate only on the `id`).
Closes#11047
* github.com:scylladb/scylla:
raft: fsm: fix `entry_size` calculation for config entries
raft: split `can_vote` field from `server_address` to separate struct
serializer_impl: generalize (de)serialization of `unordered_set`
to_string: generalize `operator<<` for `unordered_set`
The node operations using node_ops_cmd have the following procedure:
1) Send node_ops_cmd::replace_prepare to all nodes
2) Send node_ops_cmd::replace_heartbeat to all nodes
In a large cluster 1) might take a long time to finish, as a result when
the node starts to perform 2), the heartbeat timer on the peer nodes which
is 30s might have already timed out. This fails the whole node
opeartions.
We have patches to make 1) more efficient and faster.
https://github.com/scylladb/scylla/pull/10850https://github.com/scylladb/scylla/pull/10822
In addition to that, this patch increases the heartbeat timeout to reduce
the false positive of timeout.
Refs #10337
Refs #11078Closes#11081
The classes restriction, restrictions and its children
aren't used anywhere now and can be safely removed.
Some includes need to be modified for the code to compile.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
statement_restrictions_test uses a function that is defined
in multi_column_restriction.hh.
This file will be removed soon and for the test to still work
the function is moved to the test source.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
initial_key restrictions was a class used by statement_restrictions
to represent empty restrictions of different types and simplify
restriction merging logic. They are not used anymore and can
be removed.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
The _new prefix was used to distinguish the new field
from the old represenation.
Now the new field has fully replaced the old one
and _new can be removed from its name.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
All code that made use of _nonprimary_key_restrictions
has been modified to use _new_nonprimary_key_restrictions
instead.
The field can be removed.
Additionally the old code responsible for adding new restrictions
can be fully removed, everything is now done using add_restriction.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
All parts of the code that use _nonprimary_key_restrictions
are changed to use _new_nonprimary_key_restrictions instead.
I decided not to split this into multiple commits,
as there isn't a lot of changes and they are
analogous to the ones done before for partition
and clustering columns.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
Keep a map of extracted restrictions for each restricted nonprimar column.
This map will be useful, just like the ones for clustering and partition
columns.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
The _new was used to distinguish from the old field
during transition. Now the old field has been deleted
and the new one can take its place.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
Currently, all the mutations this test generates are applied on shard 0.
In rare cases, this may lead to the following crash, when the flushed
sstable doesn't contain any key that belongs to the current shard,
as seen in https://jenkins.scylladb.com/job/releng/job/Scylla-CI/1390/artifact/testlog/x86_64/dev/database_test.test_truncate_without_snapshot_during_writes.114.log
```
WARN 2022-07-17 17:41:36,630 [shard 0] sstable - create_sharding_metadata: range=[{-468459073612751032, pk{00046b657930}}, {-468459073612751032, pk{00046b657930}}] has no intersection with shard=0 first_key={key: pk{00046b657930}, token:-468459073612751032} last_key={key: pk{00046b657930}, token:-468459073612751032} ranges_single_shard=[] ranges_all_shards={{1, {[{-468459073612751032, pk{00046b657930}}, {-468459073612751032, pk{00046b657930}}]}}}
ERROR 2022-07-17 17:41:36,630 [shard 0] table - failed to write sstable /jenkins/workspace/releng/Scylla-CI/scylla/testlog/x86_64/dev/scylla-e2b694c7-db4f-4f9d-9940-9c6c21850888/ks/cf-8f74aba005de11ed92fa8661a0ed7890/me-2-big-Data.db: std::runtime_error (Failed to generate sharding metadata for /jenkins/workspace/releng/Scylla-CI/scylla/testlog/x86_64/dev/scylla-e2b694c7-db4f-4f9d-9940-9c6c21850888/ks/cf-8f74aba005de11ed92fa8661a0ed7890/me-2-big-Data.db)
ERROR 2022-07-17 17:41:36,631 [shard 0] table - Memtable flush failed due to: std::runtime_error (Failed to generate sharding metadata for /jenkins/workspace/releng/Scylla-CI/scylla/testlog/x86_64/dev/scylla-e2b694c7-db4f-4f9d-9940-9c6c21850888/ks/cf-8f74aba005de11ed92fa8661a0ed7890/me-2-big-Data.db). Aborting, at 0x329e28e 0x329e780 0x329ea88 0xf5bc69 0xf956b1 0x3196dc4 0x3198037 0x319742a 0x32be2e4 0x32bd8e1 0x32ba01c 0x317f97d /lib64/libpthread.so.0+0x92a4 /lib64/libc.so.6+0x100322
```
Instead, generate random keys and apply them on their
owning shard, and truncate all database shards.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Closes#11066
* github.com:scylladb/scylla:
database_test: test_truncate_without_snapshot_during_writes: apply mutation on the correct shard
table: try_flush_memtable_to_sstable: consume: close reader on error
All code using the _clustering_columns_restrictions field
has been modified to instead use _new_clustering_columns_restrictions
expression representation.
The old field can now be removed.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
There is a dynamic cast used to determine whether
clustering columns are restricted by a multi column
restriction.
Instead of doing that we can just use the _has_multi_column
variable.
It's also used a few lines higher, which means that
it should be already initialized.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
This PR extends #9209. It consists of 2 main points:
To enable parallelization of user-defined aggregates, reduction function was added to UDA definition. Reduction function is optional and it has to be scalar function that takes 2 arguments with type of UDA's state and returns UDA's state
All currently implemented native aggregates got their reducible counterpart, which return their state as final result, so it can be reduced with other result. Hence all native aggregates can now be distributed.
Local 3-node cluster made with current master. `node1` updated to this branch. Accessing node with `ccm <node-name> cqlsh`
I've tested belowed things from both old and new node:
- creating UDA with reduce function - not allowed
- selecting count(*) - distributed
- selecting other aggregate function - not distributed
Fixes: #10224Closes#10295
* github.com:scylladb/scylla:
test: add tests for parallelized aggregates
test: cql3: Add UDA REDUCEFUNC test
forward_service: enable multiple selection
forward_service: support UDA and native aggregate parallelization
cql3:functions: Add cql3::functions::functions::mock_get()
cql3: selection: detect parallelize reduction type
db,cql3: Move part of cql3's function into db
selection: detect if selectors factory contains only simple selectors
cql3: reducible aggregates
DB: Add `scylla_aggregates` system table
db,gms: Add SCYLLA_AGGREGATES schema features
CQL3: Add reduce function to UDA
gms: add UDA_NATIVE_PARALLELIZED_AGGREGATION feature
"
Same thing was done for compaction class some time ago, now
it's time for streaming to keep repair-generated IO in bounds.
This set mostly resembles the one for compaction IO class with
the exception that boot-time reshard/reshape currently runs in
streaming class, but that's nod great if the class is throttled,
so the set also moves boot-time IO into default IO class.
"
* 'br-streaming-class-throttling-2' of https://github.com/xemul/scylla:
distributed_loader: Populate keyspaces in default class
streaming: Maintain class bandwidth
streaming: Pass db::config& to manager constructor
config: Add stream_io_throughput_mb_per_sec option
sstables: Keep priority class on sstable_directory
Having this map is useful in a bunch of places.
To keep code simple it could be created from scratch each time,
but it's also used in do_filter, so this could actually
affect performance.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
get_clustering_columns_restrctions() used to return
a shared pointer to the clustering_restrictions class.
Now everything is being converted to expression,
so it should return an expression as well.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
Currently, all the mutations this test generates are applied on shard 0.
In rare cases, this may lead to the following crash, when the flushed
sstable doesn't contain any key that belongs to the current shard,
as seen in https://jenkins.scylladb.com/job/releng/job/Scylla-CI/1390/artifact/testlog/x86_64/dev/database_test.test_truncate_without_snapshot_during_writes.114.log
```
WARN 2022-07-17 17:41:36,630 [shard 0] sstable - create_sharding_metadata: range=[{-468459073612751032, pk{00046b657930}}, {-468459073612751032, pk{00046b657930}}] has no intersection with shard=0 first_key={key: pk{00046b657930}, token:-468459073612751032} last_key={key: pk{00046b657930}, token:-468459073612751032} ranges_single_shard=[] ranges_all_shards={{1, {[{-468459073612751032, pk{00046b657930}}, {-468459073612751032, pk{00046b657930}}]}}}
ERROR 2022-07-17 17:41:36,630 [shard 0] table - failed to write sstable /jenkins/workspace/releng/Scylla-CI/scylla/testlog/x86_64/dev/scylla-e2b694c7-db4f-4f9d-9940-9c6c21850888/ks/cf-8f74aba005de11ed92fa8661a0ed7890/me-2-big-Data.db: std::runtime_error (Failed to generate sharding metadata for /jenkins/workspace/releng/Scylla-CI/scylla/testlog/x86_64/dev/scylla-e2b694c7-db4f-4f9d-9940-9c6c21850888/ks/cf-8f74aba005de11ed92fa8661a0ed7890/me-2-big-Data.db)
ERROR 2022-07-17 17:41:36,631 [shard 0] table - Memtable flush failed due to: std::runtime_error (Failed to generate sharding metadata for /jenkins/workspace/releng/Scylla-CI/scylla/testlog/x86_64/dev/scylla-e2b694c7-db4f-4f9d-9940-9c6c21850888/ks/cf-8f74aba005de11ed92fa8661a0ed7890/me-2-big-Data.db). Aborting, at 0x329e28e 0x329e780 0x329ea88 0xf5bc69 0xf956b1 0x3196dc4 0x3198037 0x319742a 0x32be2e4 0x32bd8e1 0x32ba01c 0x317f97d /lib64/libpthread.so.0+0x92a4 /lib64/libc.so.6+0x100322
```
Instead, generate random keys and apply them on their
owning shard, and truncate all database shards.
Fixes#11076
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
In case the expression is empty and we want to merge it
with a new restriction we can just set the expression
to the new restriction.
Later this will make it easier to distinguish which case
of multi column restrictions are we dealing with.
IN and EQ can only have a single binary operator,
but slice might have two.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
Add a function that checks if there is an index
which supports one of the columns present in
the given expression.
This functionality will soon be needed for
clustering and nonprimary columns so it's
good to separate into a reusable function.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
If an exception is throws in `consume` before
write_memtable_to_sstable is called or if the latter fails,
we must close the reader passed to it.
Fixes#11075
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
This patch makes memtable_flush_static_shares liveupdateable
to avoid having to restart the cluster after updating
this config.
Signed-off-by: Igor Ribeiro Barbosa Duarte <igor.duarte@scylladb.com>
