For historic reasons, we have (in bytes.hh) a type sstring_view which is an alias for std::string_view - since the same standard type can hold a pointer into both a seastar::sstring and std::string.
This alias in unnecessary and misleading to new developers, who might be misled to believe it is assume it is somehow different from std::string_view - when it isn't.
This series removes all uses of sstring_view (changing them to use std::string_view), and in the last patch removes the alias itself. A few functions whose name referred to "sstring" but take a std::string_view were renamed.
The patches are fairly mechanical and trivial, with no functional changes intended. To ease the review the series was split to a few smaller patches that modify specific areas of the code.
Fixes#4062.
Closesscylladb/scylladb#21617
* github.com:scylladb/scylladb:
bytes: remove unused alias sstring_view
change remaining sstring_view to std::string_view
test: change sstring_view to std::string_view
cql3: change sstring_view to std::string_view
alternator: change sstring_view to std::string_view
type: change from_sstring() to from_string_view()
cross-tree: change to_sstring_view() to to_string_view()
Wean the mutation code (at least the headers) from boost ranges to std ranges,
in order to reduce the dependency load.
Cleanup, so no backport.
Closesscylladb/scylladb#21601
* github.com:scylladb/scylladb:
partition_snapshot_row_cursor.hh: switch from boost ranges to std ranges
mutation: mutation_partition_v2.hh: switch from boost ranges to std ranges
mutation: mutation_partition.hh: switch from boost ranges to std ranges
partition_snapshot_reader.hh: drop unused include boost/range/algorithm/heap_algorithm.hpp
Our "sstring_view" is an historic alias for the standard std::string_view.
The test/ directory used this old alias in a few of random places, let's
change them to use the standard type name.
Refs #4062.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
For historic reasons, we have (in bytes.hh) a type sstring_view which
is an alias for std::string_view - since the same standard type can hold
a pointer into both a seastar::sstring and std::string.
This alias in unnecessary and misleading to new developers (who might
assume it is somehow different from std::string_view). This patch doesn't
yet remove all occurances of sstring_view (the request in #4062), but
begins to do it by renaming one commonly-used function, to_sstring_view(bytes)
to to_string_view() and of course changes all its uses to the new name.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
tablet_repair_task_impl keeps a vector of tablet_repair_task_meta,
each of which keeps an effective_replication_map_ptr. So, after
the task completes, the token metadata version will not change for
task_ttl seconds.
Implement tablet_repair_task_impl::release_resources method that clears
tablet_repair_task_meta vector when the task finishes.
Set task_ttl to 1h in test_tablet_repair to check whether the test
won't time out.
Fixes: #21503.
Closesscylladb/scylladb#21504
This PR enables compaction tasks to verify the integrity of the input data through checksum and digest checks. The mechanism for integrity checking was introduced in previous PRs (#20207, #20720) as a built-in functionality of the input streams. This PR integrates this mechanism with compaction. The change applies to all compaction types and covers both compressed and uncompressed SSTables adhering to the 3.x format. If a compaction task reads only part of an SSTable, then only the per-chunk checksums are verified, not the digest.
The PR consists of:
* Changes to mx readers to support integrity checking. The kl readers, considered as compatibility-only, were left unchanged. Also, integrity checking on single-partition reversed reads (`data_consume_reversed_partition()`) remains unsupported by mx readers as this is not used in compaction.
* Changes to `sstable` and `sstable_set` APIs to allow toggling integrity checks for mx readers.
* Activation of integrity checking for all compaction types.
* Tests for all compaction types with corrupted SSTables.
Integrity checks come at a cost. For uncompressed SSTables, the cost is the loading of the CRC and Digest components from disk, and the calculation of checksums and digest from the actual data. For compressed SSTables, checksums are stored in-place and they are being checked already on all reads, so the only extra cost is the loading and calculation of the digest. The measurements show a ~5% regression in compaction performance for uncompressed SSTables, and a negligible regression for compressed SSTables.
Command: `perf-sstable --smp=1 --cpuset=1 --poll-mode --mode=compaction --iterations=1000 --partitions 10000 --sstables=1 --key_size=4096 --num_columns=15 --column_size={32, 1024, 3500, 7000, 14500}`
Uncompressed SSTables:
```
+--------------+-----------------------+----------------------+------------+
| SSTable Size | No Integrity (p/sec) | Integrity (p/sec) | Regression |
+--------------+-----------------------+----------------------+------------+
| 50 MiB | 65175.59 +- 80.82 | 61814.63 +- 72.88 | 5.16% |
| 200 MiB | 41795.10 +- 60.39 | 39686.28 +- 45.05 | 5.05% |
| 500 MiB | 21087.41 +- 30.72 | 20092.93 +- 25.05 | 4.72% |
| 1 GiB | 12781.64 +- 21.77 | 12233.94 +- 21.71 | 4.29% |
| 2 GiB | 6629.99 +- 9.40 | 6377.13 +- 8.28 | 3.81% |
+--------------+-----------------------+----------------------+------------+
```
Compressed SSTables:
```
+--------------+-----------------------+----------------------+------------+
| SSTable Size | No Integrity (p/sec) | Integrity (p/sec) | Regression |
+--------------+-----------------------+----------------------+------------+
| 50 MiB | 53975.05 +- 63.18 | 53825.93 +- 62.28 | 0.28% |
| 200 MiB | 28687.94 +- 26.58 | 28689.41 +- 26.91 | 0% |
| 500 MiB | 13865.35 +- 15.50 | 13790.41 +- 14.88 | 0.54% |
| 1 GiB | 7858.10 +- 7.71 | 7829.75 +- 9.66 | 0.36% |
| 2 GiB | 4023.11 +- 2.43 | 4010.54 +- 2.55 | 0.31% |
+--------------+-----------------------+----------------------+------------+
(p/sec = partitions/sec)
```
Refs #19071.
New feature, no backport is needed.
Closesscylladb/scylladb#21153
* github.com:scylladb/scylladb:
test: Add test for compaction with corrupted SSTables
compaction: Enable integrity checks for all compaction types
sstables: Add integrity option to factories for sstable_set readers
sstables: Add integrity option to sstable::make_reader()
sstables: Add integrity option to mx::make_reader()
sstables: Load checksums and digests in mx full-scan reader
sstables: Add integrity option to data_consume_single_partition()
sstables: Disengage integrity_check from sstable class
sstables: Allow data sources to disable digest check
data_consume_rows_context_m has a _column_value buffer it uses to read
key and column values into, preparing for parsing and consuming them.
This buffer is reset (released) in a few different cases:
* When using it for key - after consuming its content
* When using it for column value - when a colum has no value
However, the buffer is not released when used for a column value and the
column is consumed. This means that if a large column is read from the
sstable, this buffer can potentially linger and keep consuming memory
until either one of the other release scenarios is hit, or the reader is
destroyed.
Add a third release scenario, releasing the buffer after the row end was
consumed. This allows the buffer to be re-used between columns of the
same row, at the same time ensuring that a large buffer will not linger.
This patch can almost halve the memory consumption of reads in certain
circumstances. Point in case: the test
test_reader_concurrency_semaphore_memory_limit_engages starts to fail
after this fix, because the read doesn't trigger the OOM limit anymore
and needs doubling of the concurrency to keep passing.
This issue was found in a dtest
(`test_ics_refresh_with_big_sstable_files`), which writes some large
cells of up to 7MiB. After reading the row containing this large cell,
the reader holds on to the 7MiB buffer causing the semaphore's OOM
protection to kick in down the line.
Fixes: https://github.com/scylladb/scylladb/issues/21160Closesscylladb/scylladb#21132
The later includes the former and in addition to `seastar::format()`,
`print.hh` also provides helpers like `seastar::fprint()` and
`seastar::print()`, which are deprecated and not used by scylladb.
Previously, we include `seastar/core/print.hh` for using
`seastar::format()`. and in seastar 5b04939e, we extracted
`seastar::format()` into `seastar/core/format.hh`. this allows us
to include a much smaller header.
In this change, we just include `seastar/core/format.hh` in place of
`seastar/core/print.hh`.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#21574
just for better readability:
* chain comparison statement when appropriate
* do not use f-string when there are no place holders
* use list comprehension when initializing a set
* remove unused import statement
* move import statement of the standard library before
those which import the 3rd-party modules
* put two empty lines in-between top-level functions.
this is recommended by PEP8.
* remove the extraneous spaces around `=` in parameter
list.
* remove the extraneous spaces in a list like `[ 1, 2, 3 ]`
so it looks like `[1, 2, 3]`.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#21561
This patch moves (after straightforward translation) the test
"test_views_with_future_tombstone", a regression test for #5793,
from the C++ boost framework to the Python cqlpy framework.
The main motivation this move is the ease of debugging failures:
During the work on a patch for #20679 (eliminating read-before-write)
this test began to fail, and understanding where the C++ failed was
near impossible: the Boost test framework reports that the test failed,
but not in which line or why, and adding printouts to this huge source
file require a ridiculous amount of time for recompilation every time.
In contrast, the new pytest-based version shows exactly where the
error is, beautifully:
```
> assert [] == list(cql.execute(f'select * from {mv}'))
E assert [] == [Row(b=2, a=1, c=3, d=4, e=5)]
test_materialized_view.py:1614: AssertionError
```
It shows exactly which assertion failed, and exactly what were the
values that were compared. Beautiful and super helpful for debugging.
Beyond the ease of debugging, moving this (and later, other) test to
the cql-pytest framework has additional advantages:
1. The test was misplaced, in the cql_test source file, and it belongs
with materialized views tests so let's use this opportunity to move
it to the right place.
2. Can easily run the same test on multiple versions of Scylla, and
also on Cassandra. It's a good way to confirm the test is correct.
3. No need to recompile the test after every attempt to fix the bug.
The cql_query_test.cc is huge - over 6,000 lines - and takes over
a minute to compile after every attempt to fix a bug.
Refs #16134 (the issue asks to move all MV tests to cql-pytest)
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closesscylladb/scylladb#21552
Added test cases to reproduce issues with tablet migration involving views.
Refs #19149
Refs #21564
No backport needed as the PR adds only testcases.
Closesscylladb/scylladb#21566
* github.com:scylladb/scylladb:
topology_custom/test_tablets.py: add testcase for tablet migration of staged sstables
topology_custom/test_tablets.py: add testcase for tablet migration with unbuilt views
Tablet migration mixes staged and non staged sstables causing base view
inconsistencies in the pending replica. Added a testcase to reproduce
this issue.
Refs #19149.
Signed-off-by: Lakshmi Narayanan Sreethar <lakshmi.sreethar@scylladb.com>
When a tablet gets migrated right after view was created but before the
view builder registered the new view, the pending replica will not
register the sstables in the tablet for view building causing base view
inconsistencies. This commit adds a testcase to reproduce the issue.
Refs #21564
Signed-off-by: Lakshmi Narayanan Sreethar <lakshmi.sreethar@scylladb.com>
We recently added a "--release <version>" option to test/cql-pytest/run
to run a cql-pytest test against a released version of Scylla, downloaded
automatically from ScyllaDB's precompiled binary repository. This patch
adds the same capability also to test/alternator/run - allowing to run
a current test/alternator test on older releases of Scylla. The
implementation in this patch reuses the same implementation from the
cql-pytest patch.
Here is an example use case: the pull request #19941 claimed that
a certain bug fix was backported to release 6.0. Was it? Let's run
the test reproducing that bug on two releases:
test/alternator/run --release 6.0 test_streams.py::test_stream_list_tables
test/alternator/run --release 6.1 test_streams.py::test_stream_list_tables
It shows that the test passes on 6.1 (so the bug is fixed there) but the
test fails 6.0. It turns out that although the fix was backported to
branch-6.0, this happened shortly after 6.0.4 was released and no later
6.0 minor release came afterwards! So the bug wasn't actually fixed
on any official release of 6.0.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Closesscylladb/scylladb#21343
The test is only sending a subset of the running servers for the rolling
restart. The rolling restart is checking the visibility of the restarted
node agains the other nodes, but if that set is incomplete some of the
running servers might not have seen the restarted node yet.
Improved the manager client rolling restart method to consider all the
running nodes for checking the restarted node visibility.
Fixes: scylladb/scylladb#19959Closesscylladb/scylladb#21477
this change was created in the same spirit of aebb5329, which
included the fmt/iostream.h and iostream when appropriate so that
the tree can build with seastar submodule including e96932b0.
in the seastar change, we stopped including unused `fmt/ostream.h`
in a public header in seastar, so the parent projects relying on
the header to indirectly include fmt/ostream.h and iostream would
have to include these headers explicitly.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#21525
In the previous patch we enabled integrity checking on all compaction
types. This means that compaction jobs should now fail if they encounter
an SSTable with an invalid checksum or digest.
Add a test to verify this behavior. Test every compaction type with:
* compressed/uncompressed SSTables with invalid checksums
* compressed/uncompressed SSTables with invalid digests
Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>
Split compaction divides the partitions in an existing sstable into two groups and writes them into two new sstables, which replace the original one. The partition count from the original sstable is used as an estimate when writing the new ones, but this estimate is not accurate as the partitions are split between the two new sstables and each will contain only a portion of the original partition count. This also causes the bloom filters to be rebuilt at the end of compaction, as they were initially built with inaccurate estimates.
Fix this by using a better estimate for the output sstables, which is half the original partition count.
Fixes#20253
Improvement; No need to backport.
Closesscylladb/scylladb#20908
* github.com:scylladb/scylladb:
compaction: use better partition estimate for split compaction
compaction::table_state: implement `get_token_range_after_split()` wrapper
replica/table: implement `get_token_range_after_split()` wrappers
tablet_map: introduce `get_token_range_after_split()`
tablet_map: implement existing get_token_range() using the new variant
tablet_map: introduce `get_token_range()` variant
tablet_map: introduce `get_last_token()` variant
Add test to ensure backup tasks properly handle non-existent snapshots
by:
- Verifying backup task reports failure status
- Ensuring error is propagated through task status API
Previously untested edge case when backing up a snapshot that doesn't
exist in the test_backup.py tests.
Refs scylladb/scylladb#21381
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#21385
1. Add `retry_strategy` interface and default implementation for exponential back-off retry strategy.
2. Add new S3 related errors, also introduce additional errors to describe pure http errors that has no additional information in the body.
3. Add retries to the s3 client, all retries are coordinated by an instance of `retry_strategy`. In a case of error also parse response body in attempt to retrieve additional and more focused error information as suggested by AWS. See https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html. Introduce `aws_exception` to carry the original `aws_error`.
4. Discard whatever exception is thrown in `abort_upload` when aborting multipart upload since we don't care about cleanly aborting it since there are other means to clean up dangling parts, for example `rclone cleanup` or S3 bucket's Lifecycle Management Policy.
5. Add tests to cover retries, and retry exhaustion. Also add tests for jumbo upload.
6. Add the S3 proxy which is used to randomly inject retryable S3 errors to test the "retry" part of the S3 client. Switch the `s3_test` to use the S3 proxy. `s3_tests` set afloat `put_object` problem that was causing segmentation when retrying, fixed.
7. Extend the `s3_test` to use both `minio` and `proxy` configurations.
8. Add parameter to the proxy to seed the error injection randomization to make it replayable.
fixes: #20611fixes: #20613Closesscylladb/scylladb#21054
* github.com:scylladb/scylladb:
aws_errors: Make error messages more verbose.
test: Make the minio proxy randomization re-playable
test/boost/s3_test: add error injection scenarios to existing test suite
test: Switch `s3_test` to use proxy
test: Add more tests
client: Stop returning error on `DELETE` in multipart upload abortion
client: Fix sigsegv when retrying
client: Add retries
client: Adjust `map_s3_client_exception` to return exception instance
aws_errors: Change aws_error::parse to return std::optional<>
aws_errors: Add http errors mapping into aws_error
client: Add aws_exception mapping
aws_error: Add `aws_exeption` to carry original `aws_error`
aws_errors: Add new error codes
client: Introduce retry strategy
Add a buffer hint to the multishard reader. This is an internal hint, used by the multishard reader to provide a hint to the shard reader, on how much data exactly is needed by the multishard reader from the respective shard. This hint allows eliminating extraneous cross-shard round-trips and possible shard reader evict-recreate cycles. Building on this, repair sets its own row buffer size as the max buffer size on the multishard reader, ensuring that the row buffer is filled with the minimum amount of cross-shard round trips and minimal reader recreation.
To further eliminate unnecessary evictions, this PR also disables the multishard reader's read-ahead which is a mechanism that was designed to reduce latency for user-reads but it can be too aggressive for repair, causing unnecessary extra congestion on the already struggling streaming semaphores.
Refs: https://github.com/scylladb/scylladb/issues/18269
Fixes: https://github.com/scylladb/scylladb/issues/21113
The performance impact was measured with an SCT test, which creates a cluster of 3 nodes with 16 shards, then adds a 4th one with 12 shards.
Currently, it is the bootstrap time which is the worse in the case of mixed shard clusters, see below for the improvement measured during bootstrap:
| | master | buffer-hint | metric |
| ------------ | ------------- | ------------- | --------------------------------------------------- |
| evictions | 0.9M | 93.0K | scylla_database_paused_reads_permit_based_evictions |
| read (bytes) | 9.0T | 3.9T | scylla_reactor_aio_bytes_read |
| read (ops) | 88.0M | 33.5M | scylla_reactor_aio_reads |
| time | 56min | 20min | N/A |
This is a performance improvement, no backport required.
Closesscylladb/scylladb#20815
* github.com:scylladb/scylladb:
test/boost/mutation_reader_test: add test for multishard reader buffer hint
repair/row_level: disable read-ahead
db/config: introduce repair_multishard_reader_enable_read_ahead
readers/multishard: implement the read_ahead flag
replica/database: make_multishard_streaming_reader(): expose the read_ahead parameter
readers/multishard: add read_ahead parameter
repair/row_level: set max buffer size on multishard reader
replica/database: make_multishard_streaming_reader(): expose buffer_hint parameter
db/config: introduce enable_repair_multishard_reader_buffer_hint
readers/multishard: multishard_reader: pass hint to shard_reader
readers/multishard: shard_reader_v2::fill_reader_buffer(): respect the hint
readers/multishard: propagate fill_buffer_hint to shard_reader:fill_reader_buffer()
readers/multishard: shard_reader: extract buffer-fill into its own method
In scylladb/scylladb#19745, view_builder was migrated to group0 and since then it is dependant on group0_service.
Because of this, group0_service should be initialized/destroyed before/after view_builder.
This patch also adds error injection to `raft_server_with_timeouts::read_barrier`, which does 1s sleep before doing the read barrier. There is a new test which reproduces the use after free bug using the error injection.
Fixesscylladb/scylladb#20772scylladb/scylladb#19745 is present in 6.2, so this fix should be backported to it.
Closesscylladb/scylladb#21471
* github.com:scylladb/scylladb:
test/boost/secondary_index_test: add test for use after free
api/raft: use `get_server_with_timeouts().read_barrier()` in coroutines
main,cql_test_env: start group0_service before view_builder
Reproduces scylladb/scylladb#20772.
Add error injection to `raft_server_with_timeouts::read_barrier`,
which does 1s sleep before doing the read barrier.
Separate the configuration for enabling the tablets feature from the enablement of tablets when creating new keyspaces.
This change always enables the TABLETS cluster feature and the tablets logic respectively.
The `enable_tablets` config option just controls whether tablets are enabled or disabled by default for new keyspaces.
If `enable_tablets` is set to `true`, tablets can be disabled using `CREATE KEYSPACE WITH tablets = { 'enabled': false }` as it is today.
If `enable_tablets` is set to `false`, tablets can be enabled using `CREATE KEYSPACE WITH tablets = { 'enabled': true }`.
The motivation for this change is to simplify the user experience of using tablets by setting the default for new keyspaces to false amd allowing the user to simply opt-in by using tablets = {enabled: true }.
This is not pissible today.
The user has to enable tablets by default for all new keyspaces (that use the NetworkTopologyStrategy) and then actively opt-out to use vnodes.
* Not required to be backported to OSS versions. May be backported to specific enterprise versions
* This PR resubmits https://github.com/scylladb/scylladb/pull/20729 that was reverted in 73b1f66b70 due to https://github.com/scylladb/scylladb/issues/21159 which is now fixed
Closesscylladb/scylladb#21451
* github.com:scylladb/scylladb:
data_dictionary: keyspace_metadata::describe: print tablets enabled also when defaulted
tablets_test: test enable/disable tablets when creating a new keyspace
treewide: always allow tablets keyspaces
feature_service: prevent enabling both tablets and gossip topology changes
alternator: create_keyspace_metadata: enable tablets using feature_service
For performance reasons, mutation_partition_v2::maybe_drop(), and by extension
also mutation_partition_v2::apply_monotonically(mutation_partition_v2&&)
can evict empty row entries, and hence change the continuity of the merged
entry.
For checking that apply_to_incomplete respects continuity,
test_apply_to_incomplete_respects_continuity obtains the continuity of
the partition entry before and after apply_to_incomplete by calling
e.squashed().get_continuity(). But squashed() uses apply_monotonically(),
so in some circumstances the result of squashed() can have smaller
continuity than the argument of squashed(), which messes with the thing
that the test is trying to check, and causes spurious failures.
This patch changes the method of calculating the continuity set,
so that it matches the entry exactly, fixing the test failures.
Fixesscylladb/scylladb#13757Closesscylladb/scylladb#21459
Provide a seed to the proxy randomization, the idea that the `test.py` will initialize the seed from `/dev/urandom` and print the seed when starting, in case some tests failed the dev is supposed to re-play it locally with the same seed (if it didnt repro otherwise) using the `start_s3_proxy.py` and providing it with the aforementioned seed using `--rnd-seed` command line argument
Add variants of existing S3 tests that route through a proxy instead of connecting directly to MinIO. The proxy allows injecting errors to validate error handling and recovery mechanisms under failure conditions.
Switch `s3_test` to use the S3 proxy which is used to randomly inject retryable S3 errors to test the "retry" part of the S3 client.
Fix `put_object` to make it retryable
In scylladb/scylladb#19745, view_builder was migrated to group0
and since then it is dependent on group0_service.
Because of this, group0_service should be initialized/destroyed
before/after view_builder.
Fixesscylladb/scylladb#20772
Co-authored-by: Dawid Mędrek <dawid.medrek@scylladb.com>
Python and Python developers don't like directory names to include a minus sign, like "cql-pytest". In this patch we rename test/cql-pytest to test/cqlpy, and also change a few references in other code (e.g., code that used test/cql-pytest/run.py) and also references to this test suite in documentation and comments.
Arguably, the word "test" was always redundant in test/cql-pytest, and I want to leave the "py" in test/cqlpy to emphasize that it's Python-based tests, contrasting with test/cql which are CQL-request-only approval tests.
The second patch in the series fixes a small regression in the test/cqlpy/run script.
Fixes#20846
Test organization only, so backports not strictly necessary, but let's do them anyway because otherwise it will make any future backporting of tests in the cqlpy directory more messy than it needs to be.
Closesscylladb/scylladb#21446
* github.com:scylladb/scylladb:
test/cqlpy: fix "run" script without any parameters
test: rename "cql-pytest" to "cqlpy"
Test both configuration values for `enable_tablets`
and the possibility to explicitly enable or disable
tablets, respectively, when creating a keyspace using the
`tablets = {'enabled': true|false}` CREATE KEYSPACE option.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
With the tablets feature always enabled (Unless gossip toopology
changes are forced), the enable_tablets option now controls only
the default for newly created keyspaces.
Even when set to `false`, tablets are still enabled as a
feature and the user may explicitly enable tablets
using `CREATE KEYSPACE <name> WITH tablets = {'enabled': true}`
Note: best viewed with `git show -w`
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Tablets require raft consistent topology changes.
Therefore, document that they are incompatible in
the config help and prevent their usage in
`feature_config_from_db_config`
Fixesscylladb/scylladb#21075
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Continuing the previous patch, expose the just added read_ahead
parameter of make_multishard_combining>_reader_v2().
Set to read_ahead::yes by all callers, keeping the current default.
Expose the buffer hint functionality added by the previous commits, to
callers of make_multishard_streaming_reader(). All callers disable it
currently, it will be used in the next patch.
Fixes#20716
Adds optional abort_source to all s3 client operations. If provided, will propagate to actual HTTP client and allow for aborting actual net op.
Note: this uses an abort source per call, not a client-local one.
This is for two reasons:
1.) The usage pattern of the client object is to create it outside the eventual owning object (task) that hosts the relevant abort source
2.) It is quite possible to want to have different/no abort source for some operation usage.
Also adds forward usage of task abort_source in backup tasks upload s3 call, making it more readily abort-able.
Closesscylladb/scylladb#21431
* github.com:scylladb/scylladb:
backup_task: Use task abort source in s3 client call
s3::client: Make operations (individually) abortable
Fixes#20716
Propagates abort source in task object to actual network call,
thus making the upload workload more quickly abortable.
v2: Fix test to handle two versions after each other
A recent improvement to test/cqlpy/run to add the "--release" option
broke the ability to run this script it without *any* options (no test
name, etc.). This patch fixes this case.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Python and Python developers don't like directory names to include a
minus sign, like "cql-pytest". In this patch we rename test/cql-pytest
to test/cqlpy, and also change a few references in other code (e.g., code
that used test/cql-pytest/run.py) and also references to this test suite
in documentation and comments.
Arguably, the word "test" was always redundant in test/cql-pytest, and
I want to leave the "py" in test/cqlpy to emphasize that it's Python-based
tests, contrasting with test/cql which are CQL-request-only approval
tests.
Fixes#20846
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
test_tablets: add rack decommission test cases
Test scenarios where decommissioing a compelte rack
should succeed, and reproduce scylladb/scylladb#19475
where decommissioning a rack would fail since the
number of remaining racks is insufficient to satisfy
the replication factor, even though the number of nodes
is sufficient, enshrining this behavior.
Refs scylladb/scylladb#19475
* This PR adds unit tests and improves an error message. No backport required.
Closesscylladb/scylladb#20747
* github.com:scylladb/scylladb:
tablet_allocator: improve error message when unable to find replicas when draining
test_tablets: add rack decommission test cases
topology_experimental_raft/test_tablets: get_tablet_count_per_shard_for_host: move shards_count param to be last
test/pylib: ServerInfo: add datacenter and rack attributes
test: everywhere: drop unused imports of ServerInfo
now that we are allowed to use C++23. we now have the luxury of using
`std::ranges::all_of` and `std::ranges::any_of`
in this change, we replace `boost::algorithm::all_of` and `boost::algorithm::any_of` with
`std::ranges::all_of` and `std::ranges::any_of` respectively.
to reduce the dependency to boost for better maintainability, and leverage standard library features for better long-term support.
this change is part of our ongoing effort to modernize our codebase and reduce external dependencies where possible.
---
it's a cleanup, hence no need to backport.
Closesscylladb/scylladb#21411
* github.com:scylladb/scylladb:
treewide: s/boost::algorithm::any_of/std::ranges::any_of/
treewide: s/boost::algorithm::all_of/std::ranges::all_of/
In the past, DESC SCHEMA would produce create statements for both the base
and the log table. That was incorrect as the log table is automatically
created alongside the base one. That was solved in scylladb/scylladb@9ab57b1
(scylladb/scylladb#18467).
The mentioned changes implemented the following solution:
* DESC SCHEMA/KEYSPACE/TABLE would still print a create statement for the
CDC base table,
* DESC SCHEMA/KEYSPACE would start printing an alter statement for the
CDC log table. That statement would ensure that the restored log table
has the same parameters as the original one,
* DESC TABLE <base table> would behave as DESC SCHEMA/KEYSPACE, i.e.
it would print a create statement for the base table and an alter
statement for the log table,
* DESC TABLE <log table> would result in an error.
While that solution was good and behaved correctly in the context of
restoring the schema, it had one flaw: describe statement aren't only
used as a means for producing a backup; they also serve an informative
purpose to learn about the schema, e.g. to learn what parameters a specific
table uses. Because we didn't allow for describing CDC log tables, the user
couldn't look them up directly via a describe statement -- they had to
describe the base table for that.
Attempting to describe a log table ended with an error, e.g.:
```
$ DESC TABLE ks.t_scylla_cdc_log;
ks.t_scylla_cdc_log is a cdc log table and it cannot be described directly. Try `DESC TABLE ks.t` to describe cdc base table and it's log table.
```
In these changes, we allow for describing CDC log tables again. The
semantics of the first three bullets above remains unchanged, but
we impose new behavior for DESC TABLE <log table>:
* When the user executes DESC TABLE <log table>, a create statement
will be returned, treating the table as if it were a regular one,
* The create statement will be wrapped in CQL comment markers.
The rationale for the second bullet is that although we want to give the
user a means to look into the structure and options of a CDC log table,
the returned statement is not supposed to be ever executed by them. We
want to minimize the risk of that.
An example of the behavior after the change:
```
$ DESC TABLE ks.t_scylla_cdc_log;
/* Do NOT execute this statement! It's only for informational purposes.
A CDC log table is created automatically when the base is created.
CREATE TABLE ks.t_scylla_cdc_log (
"cdc$stream_id" blob,
"cdc$time" timeuuid,
"cdc$batch_seq_no" int,
"cdc$end_of_batch" boolean,
"cdc$operation" tinyint,
"cdc$ttl" bigint,
p int,
PRIMARY KEY ("cdc$stream_id", "cdc$time", "cdc$batch_seq_no")
) WITH CLUSTERING ORDER BY ("cdc$time" ASC, "cdc$batch_seq_no" ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'enabled': 'false', 'keys': 'NONE', 'rows_per_partition': 'NONE'}
AND comment = 'CDC log for ks.t'
AND compaction = {'class': 'TimeWindowCompactionStrategy', 'compaction_window_size': '60', 'compaction_window_unit': 'MINUTES', 'expired_sstable_check_frequency_seconds': '1800'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1
AND default_time_to_live = 0
AND gc_grace_seconds = 0
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND speculative_retry = '99.0PERCENTILE';
*/
```
We also extend the developer documentation regarding DESCRIBE
statements on CDC tables.
Fixesscylladb/scylladb#21235
Backport: these changes are an enhancement, so not needed.
Closesscylladb/scylladb#21228
* github.com:scylladb/scylladb:
docs/dev: Document semantics of describing CDC tables
cql3: Allow for describing CDC log tables
The overload was introduced by a8b14b0227 (utils: add timeout error
injection with lambda), but is only used by the test nowadays.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Closesscylladb/scylladb#21377
