This patch adds several more tests for Alternator's UpdateItem operation.
These tests verify a few simple cases that, surprisingly, never had test
coverage. The new tests pass (on both DynamoDB and Alternator) so did not
expose any bug.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closes#11025
If the compaction_descriptor returned by `time_window_compaction_strategy::get_sstables_for_compaction`
is marked with `has_only_fully_expired::yes` it should always be compacted
since `time_window_compaction_strategy::get_sstables_for_compaction` is not idempotent.
It sets `_last_expired_check` and if compaction is postponed and retried before
`expired_sstable_check_frequency` has passed, it will not look for those fully-expired
sstables again. Plus, compacting them is the cheapest possible as it does not require
reading anything, just deleting the input sstables, so there's no reason not postpone it.
Also, extend `max_ongoing_compaction_test` to test serialization of compaction jobs with the same weight.
Fixes#10989Closes#10990
* github.com:scylladb/scylla:
compaction_manager: always register descriptor with fully expired sstables for compaction
test: max_ongoing_compaction_test: test serialization of regular compaction with same weight
test: max_ongoing_compaction_test: reindent refactored code
test: max_ongoing_compaction_test: define compact_all_tables lambda
test: max_ongoing_compaction_test: refactor make_table_with_single_fully_expired_sstable
test: max_ongoing_compaction_test: reduce number of tables
This series adds the infrastructure needed for testing user permissions, like the ability to create temporary roles and CQL sessions which log in as different users, and a few initial test cases for granting and revoking permissions.
Closes#10998
* github.com:scylladb/scylla:
cql-pytest: add a case for granting/revoking data permissions
cql-pytest: add new_user and new_session utils
cql-pytest: speed up permissions refresh period for tests
Said parameter is a convenience so downstream consumers of the
mutation compactors don't have to check the `bool is_live` already
passed to them. This convenience however causes a template parameter and
additional logic for the compactor. As the most prominent of these
consumers (the query result builder) will soon have to switch to
`emit_only_live_rows::no` for other reasons anyway (it will want to count
tombstones), we take the opportunity to switch everybody to ::no. This
can be done with very little additional complexity to these consumers --
basically an additional if or two. With everybody using the `::no` variant
of the compactor, we can remove this template parameter and the logic
associated with it altogether.
Closes#10931
* github.com:scylladb/scylla:
multishard_mutation_query: remove now pointless compact_for_result_state typedef
mutation_compactor: remove only-live related logic
mutation_compactor: remove emit_only_live_rows template parameter
mutation_compactor: remove unused compact_mutation_state::parameters
querier: remove {data,mutation}_querier aliases
querier: remove now pointless emit_only_live_rows template parameter
tree: use emit_only_live_rows::no
querier: querier_cache: de-override insert() methods
This is part of support installing executables from PIP package,
now we support installing executable from PIP package but it will
install under /opt/scylladb/python3/bin.
To call these commands without speciying full path, we also need to install
symlink to /usr/bin.
To do this, we need new list which specifies command name for symlink.
Closes#10748
This series includes an assortment of loosely related improvements developed for a recent investigation. The changes include:
* Fix broken `std_deque` wrapper.
* Make `scylla smp-queues` fast.
* Teach `scylla smp-queues` to filter for both sender CPU (`--from`) and receiver CPU (`--to`) or both.
* Teach `scylla smp-queues` to make histogram over content of the queues -- i.e. the type of tasks in the smp queues.
* Teach `scylla smp-queues` to filter for tasks belonging to a certain scheduling group.
* Teach `scylla task_histogram` to include only tasks in the histogram.
* Teach `scylla task_histogram` to filter for tasks belonging to a certain scheduling group.
* Teach `scylla-fiber` to walk in both directions.
And some refactoring.
Fixes: https://github.com/scylladb/scylla/issues/7059Closes#11019
* github.com:scylladb/scylla:
docs/dev/debugging.md: update continuation chain traversal guide
scylla-gdb.py: scylla fiber: walk continuation chain in both directions
scylla-gdb.py: scylla fiber: allow passing analyzed pointers to _probe_pointer()
scylla-gdb.py: scylla fiber: hoist preparatory code out of _walk()
scylla-gdb.py: scylla task_histogram: add --scheduling-groups option
scylla-gdb.py: scylla task_histogram: add --filter-tasks option
scylla-gdb.py: scylla task_histogram: use histogram class
scylla-gdb.py: scylla-fiber: extract symbol matching logic
scylla-gdb.py: histogram: add limit feature
scylla-gdb.py: histogram: handle formatting errors
scylla-gdb.py: intrusive_slist: avoid infinite recursion in __len__()
scylla-gdb.py: scylla smp-queues: add --scheduling-group option
scylla-gdb.py: scylla smp-queues: add --content switch
scylla-gdb.py: smp-queue: add filtering capability
scylla-gdb.py: make scylla smp-queues fast
scylla-gdb.py: fix disagreement between std_deque len() and iter()
If the compaction_descriptor returned by time_window_compaction_strategy::get_sstables_for_compaction
is marked with has_only_fully_expired::yes it should always be compacted
since time_window_compaction_strategy::get_sstables_for_compaction is not idempotent.
It sets _last_expired_check and if compaction is postponed and retried before
expired_sstable_check_frequency has passed, it will not look for those fully-expired
sstables again. Plus, compacting them is the cheapest possible as it does not require
reading anything, just deleting the input sstables, so there's no reason not postpone it.
Fixes#10989
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
To test both expired and non-expired sstables scenarios
we need to pass this helper function the expected number
of sstables before compaction and after compaction.
When compaction a set of fully-expired sstables,
we expect none to remain, while when the set of sstables
is not fully expired, we'll expect 1 output sstable
after compaction.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
So we can use the lower-level build blocks to
test compaction serialization of both fully-expired
and non-fully-expired sstables scenarios in the following patches.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
The command wasn't tested fully,
and when tested, started failing scylla-gdb test.
Before Raft, the list of connections to print in a single-node setup
was always empty, so a mistake in the gdb script command 'scylla netw'
didn't lead to a test failure.
With raft, there is always an RPC connection to self after initial
bootstrap, and the test begins to print connections (and fail, because
there is a bug in the printing code).
Fix that bug.
Closes#11012
Now that we use emit_only_live_rows::no everywhere we can remove this
template parameters. Only the template parameter is removed, the
internal logic around it is left in place (will be removed in a next
patch), by hard-wiring `only_live()`.
emit_only_live_rows is a convenience so downstream consumers of the
mutation compactors don't have to check the `bool is_live` already
passed to them. This convenience however causes a template parameter and
additional logic for the compactor. As the most prominent of these
consumers (the query result builder) will soon have to switch to
emit_only_live_rows::no for other reasons anyway (it will want to count
tombstones), we take the opportunity to switch everybody to ::no. This
can be done with very little additional complexity to these consumer --
basically an additional if or two.
This prepares the ground for removing this template parameter and the
associate logic from the compactor.
Soon, the currently two distinct types of queriers will be merged, as
the template parameter differentiating them will be gone. This will make
using type based overload for insert() impossible, as 2 out of the 3
types will be the same. Use different names instead.
Parameterize _walk() with a method that does the actual walking. This is
a trivial change as it was already delegating all the walking logic to
_do_walk(). The latter is renamed to _walk_forward() and we add a new
method called _walk_backward() which implements walking the continuation
chain backwards (towards tasks waited on by the queried task).
The starting task is now printed at index #0, tasks waited on by the
starting task have negative indexes, tasks waiting on the starting task
have positive indexes (like before).
With this scylla fiber can be used to dump an entire fiber (barring any
difficulties detecting following more special tasks like threads).
We soon want to teach _walk() to walk in both directions. In preparation
to that, we extract all generic preparatory code that is related to the
starting task and combining arguments. This now resides in invoke(),
_walk() should only be concerned with traversing the continuation chain.
This PR introduces improvements to `expr::to_restriction` and prepares the validation part for restriction classes removal.
`expr::to_restriction` is currently used to take a restriction from the WHERE clause, prepare it, perform some validation checks and finally convert it to an instance of the restriction class.
Soon we will get rid of the restriction class.
In preparation for that `expr::to_restriction` is split into two independent parts:
* The part that prepares and validates a binary_operator
* The part that converts a binary_operator to restriction
Thanks to this split getting rid of restriction class will be painless, we will just stop using the second part.
`to_restriction.cc` is replaced by `restrictions.hh/cc`. In the future we can put all the restriction expressions code there to avoid clutter in `expression.hh/cc`.
This change made it much easier to fix#10631, so I did that as well.
Fixes: #10631Closes#10979
* github.com:scylladb/scylla:
cql-pytest: Test that IS NOT only accepts NULL
cql-pytest: Enable testInvalidCollectionNonEQRelation
cql3: Move single element IN restrictions handling
cql3: Check for disallowed operators early
cql3: Simplify adding restrictions
cql3: Reorganize to_restriction code
cql3: Fix IS NOT NULL check in to_restriction
cql3: Swap order of arguments in error message
bytes_ostream is an incremental builder for a discontiguous byte container.
managed_bytes is a non-incremental (size must be known up front) byte
container, that is also compatible with LSA. So far, conversion between
them involves copying. This is unfortunate, since query_result is generated
as a bytes_ostream, but is later converted to managed_bytes (today, this
is done in cql3::expr::get_non_pk_values() and
compound_view_wrapper::explode(). If the two types could be made compatible,
we could use managed_bytes_view instead of creating new objects and avoid
a copy. It's also nicer to have one less vocabulary type.
This patch makes bytes_ostream use managed_bytes' internal representation
(blob_storage instead of bytes_ostream::chunk) and provides a conversion
to managed_bytes. All bytes_ostream users are left in place, but the goal
is to make bytes_ostream a write-only type with the only observer a conversion
to managed_bytes.
It turns out to be relatively simple. The internal representations were
already similar. I made blob_storage::ref_type self-initializing to
reduce churn (good practice anyway) and added a private constructor
to managed_bytes for the conversion.
Note that bytes_ostream can only be used to construct a non-LSA managed_bytes,
but LSA uses of managed_bytes are very strictly controlled (the entry
points to memtable and cache) so that's not a problem.
A unit test is added.
Closes#10986
After acquiring the _compaction_state write lock,
select all sstables using get_candidates and register them
as compacting, then unlock the _compaction_state lock
to let regular compaction run in parallel.
Also, run major compaction in maintenance scheduling group.
We should separate the scheduling groups used for major compaction
from the the regular compaction scheduling group so that
the latter can be affected by the backlog tracker in case
backlog accumulates during a long running major compaction.
Fixes#10961Closes#10984
* github.com:scylladb/scylla:
compaction_manager: major_compaction_task: run in maintenance scheduling groupt
compaction_manager: allow regular compaction to run in parallel to major
The IS_NOT operator can only be used during materialized view creation
and it can only be used to express IS NOT NULL.
Trying to write something like IS NOT 42 should cause an error.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
Restrictions like
col IN (1)
get converted to
col = 1
as an optimization/simplification.
This used to be done in prepare_binary_operator,
but it fits way better inside of
validate_and_prepare_new_restriction.
When it was being done in prepare_binary_operator
the conversion happened before validation checks
and the error messages would describe an equality
restriction despite the user making an IN restriction.
Now the conversion happens after all validation
is finished, which ensures that all checks are
being done on the original expression.
Fixes: #10631
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
Move checking for disallowed operators
earlier in the code flow.
This is needed to pass some tests that
expect one error message instead of the other.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
The code that adds restrictions in statement_restrictions.cc
is unnecessarily convoluted.
The code to handle IS NOT NULL is actually repeated twice,
once in the constructor and once in add_is_not_restriction.
I missed this when I orignally modified this code.
There is no need to keep duplicate code, we can just
use the new add_is_not_restriction.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
expr::to_restriction is currently used to
take a restriction from the WHERE clause,
prepare it, perform some validation checks
and finally convert it to an instance of
the restriction class.
Soon we will get rid of the restriction class.
In preparation for that expr::to_restriction
is split into two independent parts:
* The part that prepares and validates a binary_operator
* The part that converts a binary_operator to restriction
Thanks to this split getting rid of restriction class
will be painless, we will just stop using the
second part.
This commit splits expr::to_restriction into two functions;
* validate_and_prepare_new_restriction
* convert_to_restriction
that handle each of those parts.
All helper validation methods in the anonymous namespace
are copied from the to_restriction.cc file.
to_restriction.cc isn't the best filename for the new functionality,
so it has been renamed to restrictions.hh/cc.
In the future all the code regarding restrictions could be
put there to reduce clutter in expression.hh/cc
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
expr::to_restriction performs a check to see if
the restriction is of form: `col IS NOT NULL`
There is a mistake in this check.
It uses is<null>(prepared_binop.rhs)
to determine if the right hand side of binary operator
is a null, but the binary operator is already prepared.
During preparation expr::null is converted to expr::constant
and that wouldn't be detected by this check.
The check has been changed to check for null constant instead
of expr::null.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
The error message displays two arguments in
a specific order, but the tests actually
expect them to be swapped.
Swap the arguments to match the expected
error messages in tests.
It wasn't detected earlier because the
check was never reached, but this will change
soon in the following commits.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
Causing the histogram to be made from the scheduling groups of the found
tasks. Allows for finding out which scheduling group dominates in-memory
tasks. This currently cannot be determined, scylla task-queues only
includes ready tasks.
Allowing to include only task objects in the histogram. Leads to
histograms with less noise but might exclude potentially important items
due to the filtering being inexact.
Both the content and the formatter method is caller-provided. Mistakes
are easy to come by. Instead of aborting the entire operation, just a
print an error if an item fails to format.
Said method currently uses a list() to iterate over all elements,
determining the length. Passing `self` to `list()` will however make
call `len()` first, causing infinite recursion.
When present on the command line, the histogram is created over the
content of the queues, rather than the number of items in them.
It is possible to filter in combination with --content. In particular it
can be used to see the content of a single queue when all three of
`--to`, `--from` and `--content` is present on the command line.
