The constraint on on cryptopp_hasher<>::impl is illegal, since it's not
on the base template. Convert it to a static_assert.
We could have moved it to the base template, but that would have undone
the work to push all the implementation details in .cc and reduce #include
load.
Found by clang.
While we need CryptoPP Update() functions not to throw, they aren't
marked noexcept.
Since there is no reason for them to throw, we'll just hope they
don't, and relax the requirement.
Found by clang. Apparently gcc didn't bother to check the constraint
here.
"
Based on https://github.com/scylladb/scylla/issues/7199,
it looks like storage_service::set_tables_autocompaction
may be called on shards other than 0.
Use run_with_api_lock to both serialize the action
and to check _initialized on shard 0.
Fixes#7199
Test: unit(dev), compaction_test:TestCompaction_with_SizeTieredCompactionStrategy.disable_autocompaction_nodetool_test
"
* tag 'set_tables_autocompaction-v1' of github.com:bhalevy/scylla:
storage_service: set_tables_autocompaction: fixup indentation
storage_service: set_tables_autocompaction: run_with_api_lock
storage_service: set_tables_autocompaction: use do_with to hold on to args
storage_service: set_tables_autocompaction: log message in info level
We need to skip internet access on offline installation.
To do this we need following changes:
- prevent running yum/apt for each script
- set default "NO" for scripts it requires package installation
- set default "NO" for scripts it requires internet access, such as NTP
See #7153Closes#7224
* syuu1228-offline_setup:
dist/common/scripts: skip internet access on offline installation
scylla_ntp_setup: use shutil.witch() to lookup command
- Add necessary changes to `scylla_util.py` in order to support Scylla Machine Image in GCE.
- Fixes and improvements for `curl` function.
Closes#7080
* bentsi-gce-image-support:
scylla_util.py: added GCE instance/image support
scylla_util.py: make max_retries as a curl function argument
scylla_util.py: adding timeout to curl function
scylla_util.py: styling fixes to curl function
scylla_util.py: change default value for headers argument in curl function
Let's build scylla-unified-package.tar.gz in build/<mode>/dist/tar for
symmetry. The old location is still kept for backward compatibility for
now. Also document the new official artifact location.
Message-Id: <20200917071131.126098-1-penberg@scylladb.com>
"
... and make sure nothing is left. Whith the help of fresh seastar
this can be done quickly.
Before doing this check -- unregister remaining verbs in repair and
storage_service and fix tests not to register verbs, because they
are all local.
tests: unit(dev), manual
"
* 'br-messaging-service-stop-all' of https://github.com/xemul/scylla:
messaging_service: Report still registered services as errors
repair: Move CHECKSUM_RANGE verb into repair/
repair: Toss messaging init/uninit calls
storage_service: Uninit RPC verbs
test: Do not init messaging verbs
On stop -- unregister the CLIENT_ID verb, which is registerd
in constructor, then check for any remaining ones.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The verb is sent by repair code, so it should be registered
in the same place, not in main. Also -- the verb should be
unregistered on stop.
The global messaging service instance is made similarly to the
row-level one, as there's no ready to use repair service.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
There goal is to make it possible to reg/unreg not only row-level
verbs. While at it -- equip the init call with sharded<database>&
argument, it will be needed by the next patch.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
This patch adds a "--list" option to test.py that shows a list of tests
instead of executing them. This is useful for people and scripts, which
need to discover the tests that will be run. For example, when Jenkins
needs to store failing tests, it can use "test.py --list" to figure out
what to archive.
Message-Id: <20200916135714.89350-1-penberg@scylladb.com>
And move-assign to _pending_ranges_interval_map[keyspace_name]
only when done.
This is more effient since there's no need to look up
_pending_ranges_interval_map[keyspace_name] for every insert to the
interval_map.
And it is exception safe in case we run out of memory mid-way.
Refs #7220
Test: unit(dev)
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20200916115059.788606-1-bhalevy@scylladb.com>
This patch fixes a bug noted in issue #7218 - where PutItem operations
sometimes lose part of the item's data - some attributes were lost,
and the name of other attributes replaced by empty strings. The problem
happened when the write-isolation policy was LWT and there was contention
of writes to the same partition (not necessarily the same item).
To use CAS (a.k.a. LWT), Alternator builds an alternator::rmw_operation
object with an apply() function which takes the old contents of the item
(if needed) and a timestamp, and builds a mutation that the CAS should
apply. In the case of the PutItem operation, we wrongly assumed that apply()
will be called only once - so as an optimization the strings saved in the
put_item_operation were moved into the returned mutation. But this
optimization is wrong - when there is contention, apply() may be called
again when the changed proposed by the previous one was not accepted by
the Paxos protocol.
The fix is to change the one place where put_item_operation *moved* strings
out of the saved operations into the mutations, to be a copy. But to prevent
this sort of bug from reoccuring in future code, this patch enlists the
compiler to help us verify that it can't happen: The apply() function is
marked "const" - it can use the information in the operation to build the
mutation, but it can never modify this information or move things out of it,
so it will be fine to call this function twice.
The single output field that apply() does write (_return_attributes) is
marked "mutable" to allow the const apply() to write to it anyway. Because
apply() might be called twice, it is important that if some apply()
implementation sometimes sets _return_attributes, then it must always
set it (even if to the default, empty, value) on every call to apply().
The const apply() means that the compiler verfies for us that I didn't
forget to fix additional wrong std::move()s. Additionally, a test I wrote
to easily reproduce issue #7218 (which I will submit as a dtest later)
passes after this fix.
Fixes#7218.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20200916064906.333420-1-nyh@scylladb.com>
Add a "Closes #$PR_NUM" annotation at the end of the commit
message to tell github to close the pull request, preventing
manual work and/or dangling pull requests.
Closes#7245
"
This series follows the suggestion from https://github.com/scylladb/scylla/pull/7203#issuecomment-689499773 discussion and deprecates a number of cluster features. The deprecation does not remove any features from the strings sent via gossip to other nodes, but it removes all checks for these features from code, assuming that the checks are always true. This assumption is quite safe for features introduced over 2 years ago, because the official upgrade path only allows upgrading from a previous official release, and these feature bits were introduced many release cycles ago.
All deprecated features were picked from a `git blame` output which indicated that they come from 2018:
```git
e46537b7d3 2016-05-31 11:44:17 +0200 RANGE_TOMBSTONES_FEATURE = "RANGE_TOMBSTONES";
85c092c56c 2016-07-11 10:59:40 +0100 LARGE_PARTITIONS_FEATURE = "LARGE_PARTITIONS";
02bc0d2ab3 2016-12-09 22:09:30 +0100 MATERIALIZED_VIEWS_FEATURE = "MATERIALIZED_VIEWS";
67ca6959bd 2017-01-30 19:50:13 +0000 COUNTERS_FEATURE = "COUNTERS";
815c91a1b8 2017-04-12 10:14:38 +0300 INDEXES_FEATURE = "INDEXES";
d2a2a6d471 2017-08-03 10:53:22 +0300 DIGEST_MULTIPARTITION_READ_FEATURE = "DIGEST_MULTIPARTITION_READ";
ecd2bf128b 2017-09-01 09:55:02 +0100 CORRECT_COUNTER_ORDER_FEATURE = "CORRECT_COUNTER_ORDER";
713d75fd51 2017-09-14 19:15:41 +0200 SCHEMA_TABLES_V3 = "SCHEMA_TABLES_V3";
2f513514cc 2017-11-29 11:57:09 +0000 CORRECT_NON_COMPOUND_RANGE_TOMBSTONES = "CORRECT_NON_COMPOUND_RANGE_TOMBSTONES";
0be3bd383b 2017-12-04 13:55:36 +0200 WRITE_FAILURE_REPLY_FEATURE = "WRITE_FAILURE_REPLY";
0bab3e59c2 2017-11-30 00:16:34 +0000 XXHASH_FEATURE = "XXHASH";
fbc97626c4 2018-01-14 21:28:58 -0500 ROLES_FEATURE = "ROLES";
802be72ca6 2018-03-18 06:25:52 +0100 LA_SSTABLE_FEATURE = "LA_SSTABLE_FORMAT";
71e22fe981 2018-05-25 10:37:54 +0800 STREAM_WITH_RPC_STREAM = "STREAM_WITH_RPC_STREAM";
```
Tests: unit(dev)
manual(verifying with cqlsh that the feature strings are indeed still set)
"
Closes#7234.
* psarna-clean_up_features:
gms: add comments for deprecated features
gms: remove unused feature bits
streaming: drop checks for RPC stream support
roles: drop checks for roles schema support
service: drop checks for xxhash support
service: drop checks for write failure reply support
sstables: drop checks for non-compound range tombstones support
service: drop checks for v3 schema support
repair: drop checks for large partitions support
service: drop checks for digest multipartition read support
sstables: drop checks for correct counter order support
cql3: drop checks for materialized views support
cql3: drop checks for counters support
cql3: drop checks for indexing support
We need to skip internet access on offline installation.
To do this we need following changes:
- prevent running yum/apt for each script
- set default "NO" for scripts it requires package installation
- set default "NO" for scripts it requires internet access, such as NTP
See #7153Fixes#7182
The command installed directory may different between distributions,
we can abstract the difference using shutil.witch().
Also the script become simpler than passing full path to os.path.exists().
Consider an unpaged query that consumes all of available memory, despite
fea5067dfa which limits them (perhaps the
user raised the limit, or this is a system query). Eventually we will see a
bad_alloc which will abort the query and destroy this reconcilable_result_builder.
During destruction, we first destroy _memory_accounter, and then _result.
Destroying _memory_accounter resumes some continuations which can then
allocate memory synchronously when increasing the task queue to accomodate
them. We will then crash. Had we not crashed, we would immediately afterwards
release _result, freeing all the memory that we would ever need.
Fix by making _result the last member, so it is freed first.
Fixes#7240.
Introduce new database config option `schema_registry_grace_period`
describing the amount of time in seconds after which unused schema
versions will be cleaned up from the schema registry cache.
Default value is 1 second, the same value as was hardcoded before.
Tests: unit(debug)
Refs: #7225
Signed-off-by: Pavel Solodovnikov <pa.solodovnikov@scylladb.com>
Message-Id: <20200915131957.446455-1-pa.solodovnikov@scylladb.com>
The set contains 3 small optimizations:
- avoid copying of partition key on lookup path
- reduce number of args carried around when creating a new entry
- save one partition key comparison on reader creation
Plus related satellite cleanups.
* https://github.com/xemul/scylla/tree/br-row-cache-less-copies:
row_cache: Revive do_find_or_create_entry concepts
populating reader: Do not copy decorated key too early
populating reader: Less allocator switching on population
populating reader: Fix indentation after previous patch
row_cache: Move missing entry creation into helper
test: Lookup an existing entry with its own helper
row_cache: Do not copy partition tombstone when creating cache entry
row_cache: Kill incomplete_tag
row_cache: Save one key compare on direct hit
"
This PR removes _pending_ranges and _pending_ranges_map in token_metadata.
This removal of makes copying of token_metadata faster and reduces the chance to cause reactor stall.
Refs: #7220
"
* asias-token_metadata_replication_config_less_maps:
token_metadata: Remove _pending_ranges
token_metadata: Get rid of unused _pending_ranges_map
from Asias.
This series follows "repair: Add progress metrics for node ops #6842"
and adds the metrics for the remaining node operations,
i.e., replace, decommission and removenode.
Fixes#1244, #6733
* asias-repair_progress_metrics_replace_decomm_removenode:
repair: Add progress metrics for removenode ops
repair: Add progress metrics for decommission ops
repair: Add progress metrics for replace ops
Change 94995acedb added yielding to abstract_replication_strategy::do_get_ranges.
And 07e253542d used get_ranges_in_thread in compaction_manager.
However, there is nothing to prevent token_metadata, and in particular its
`_sorted_tokens` from changing while iterating over them in do_get_ranges if the latter yields.
Therefore copy the the replication strategy `_token_metadata` in `get_ranges_in_thread(inet_address ep)`.
If the caller provides `token_metadata` to get_ranges_in_thread, then the caller
must make sure that we can safely yield while accessing token_metadata (like
in `do_rebuild_replace_with_repair`).
Fixes#7044
Test: unit(dev)
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20200915074555.431088-1-bhalevy@scylladb.com>
- Remove get_pending_ranges and introduce has_pending_ranges, since the
caller only needs to know if there is a pending range for the keyspace
and the node.
- Remove print_pending_ranges which is only used in logging. If we
really want to log the new pending token ranges, we can log when we
set the new pending token ranges.
This removal of _pending_ranges makes copying of token_metadata faster
and reduces the chance to cause reactor stall.
Refs: #7220
"
The range_tombstone_list provides an abstraction to work with
sorted list of range tombstones with methods to add/retrive
them. However, there's a tombstones() method that just returns
modifiable reference to the used collection (boost::intrusive_set)
which makes it hard to track the exact usage of it.
This set encapsulates the collaction of range tombstones inside
the mentioned ..._list class.
tests: unit(dev)
"
* 'br-range-tombstone-encapsulate-collection' of https://github.com/xemul/scylla:
range_tombstone_list: Do not expose internal collection
range_tombstone_list: Introduce and use pop-and-lock helper
range_tombstone_list: Introduce and use pop_as<>()
flat_mutation_reader: Use range_tombstone_list begin/end API
repair: Mark some partition_hasher methods noexcept
hashers: Mark hash updates noexcept
The histogram constructor has a `counts` parameter defaulted to
`defaultdict(int)`. Due to how default argument values work in
python -- the same value is passed to all invocations -- this results in
all histogram instances sharing the same underlying counts dict. Solve
it the way this is usually solved -- default the parameter to `None` and
when it is `None` create a new instance of `defaultdict(int)` local to
the histogram instance under construction.
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20200908142355.1263568-1-bdenes@scylladb.com>
Currently blobs are converted to python bytes objects and printed by
simply converting them to string. This results in hard to read blobs as
the bytes' __str__() attempts to interpret the data as a printable
string. This patch changes this to use bytes.hex() which prints blobs in
hex format. This is much more readable and it is also the format that
scylla uses when printing blobs.
Also the conversion to bytes is made more efficient by using gdb's
gdb.inferior.read_memory() function to read the data.
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20200911085439.1461882-1-bdenes@scylladb.com>
hugepages and libhugetlbfs-bin packages is only required for DPDK mode,
and unconditionally installation causes error on offline mode, so drop it.
Fixes#7182
data::cell targets 8KB as its maximum allocations size to avoid
pressuring the allocator. This 8KB target is used for internal storage
-- values small enough to be stored inside the cell itself -- as well
for external storage. Externally stored values use 8KB fragment sizes.
The problem is that only the size of data itself was considered when
making the allocations. For example when allocating the fragments
(chunks) for external storage, each fragment stored 8KB of data. But
fragments have overhead, they have next and back pointers. This resulted
in a 8KB + 2 * sizeof(void*) allocation. IMR uses the allocation
strategy mechanism, which works with aligned allocations. As the seastar
allocation only guarantees aligned allocations for power of two sizes,
it ends up allocating a 16KB slot. This results in the mutation fragment
using almost twice as much memory as would be required. This is a huge
waste.
This patch fixes the problem by considering the overhead of both
internal and external storage ensuring allocations are 8KB or less.
Fixes: #6043
Tests: unit(debug, dev, release)
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20200910171359.1438029-1-bdenes@scylladb.com>
Instead of using the default hasher, hasing specializations should
use the hasher type they were specialized for. It's not a correctness
issue now because the default hasher (xx_hasher) is compatible with
its predecessor (legacy_xx_hasher_without_null_digest), but it's better
to be future-proof and use the correct type in case we ever change the
default hasher in a backward-incompatible way.
Message-Id: <c84ce569d12d9b4f247fb2717efa10dc2dabd75b.1600074632.git.sarna@scylladb.com>
