because we build stripped package and non-stripped package in parallel
using ninja. there are chances that the non-stripped build job could
be adding build/node_exporter directory to the tarball while the job
building stripped package is using objcopy to extract the symbols from
the build/node_exporter/node_exporter executable. but objcopy creates
temporary files when processing the executables. and the temporary
files can be spotted by the non-stripped build job. there are two
consequences:
1. non-stripped build job includes the temporary files in its tarball,
even they are not supposed to be distributed
2. non-stripped build job fails to include the temporary file(s), as
they are removed after objcopy finishes its job. but the job did spot
them when preparing the tarball. so when the tarfile python module
tries to include the previous found temporary file(s), it throws.
neither of these consequences is expected. but fortunately, this only
happens when packaging the non-stripped package. when packaging the
stripped package, the build/node_exported directory is not in flux
anymore. as ninja ensures the dependencies between the jobs.
so, in this change, we do not add the whole directory when packaging
the non-stripped version. as all its ingredients have been added
separately as regular files. and when packaing the stripped version,
we still use the existing step, as we don't have to list all the
files created by strip.sh:
node_exporter{,.debug,.dynsyms,.funcsyms,.keep_symbols,.minidebug.xz}
we could do so in this script, but the repeatings is unnecessary and
error-prune. so, let's keep including the whole directory recursively,
so all the debug symbols are included.
Fixes https://github.com/scylladb/scylladb/issues/14079Closes#14081
* github.com:scylladb/scylladb:
create-relocatable-package.py: package build/node_export only for stripped version
create-relocatable-package.py: use positive condition when possible
this change is a follow-up of 4f5fcb02fd,
the goal is to avoid the programming oversights like
```c++
trace(trace_ptr, "foo {} with {} but {} is {}");
```
as `trace(const trace_state_ptr& p, const std::string& msg)` is
a better match than the templated one, i.e.,
`trace(const trace_state_ptr& p, fmt::format_string<T...> fmt, T&&...
args)`. so we cannot detect this with the compile-time format checking.
so let's just drop this overload, and update its callers to use
the other overload.
The change was suggested by Avi. the example also came from him.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#14188
sel is a local variable, and it is not shared with anybody else.
so make it a unique_ptr<> for better readability.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#14189
The estimated_partitions is estimated after the repair_meta is created.
Currently, the default estimated_partitions was used to create the
write which is not correct.
To fix, use the updated estimated_partitions.
Reported by Petr Gusev
Closes#14179
Return 400 Bad Request instead of 500 Internal Server Error
when user requests task or module which does not exist through
task manager and task manager test apis.
Closes#14166
* github.com:scylladb/scylladb:
test: add test checking response status when requested module does not exist
api: fix indentation
api: throw bad_param_exception when requested task/module does not exists
in this series, we use {fmt}'s compile-time formatting check, and avoid deep copy when creating sstring from std::string.
Closes#14169
* github.com:scylladb/scylladb:
tracing: use std::string instead of sstring for event_record::message
tracing: use compile-time formatting check
Compaction task test should only check the intended group of task.
Thus, the tasks are filtered in each test.
In order to be able to run the tests in parallel, checks for the tasks
of the same type are grouped together.
Fixes: #14131.
Closes#14161
* github.com:scylladb/scylladb:
test: put compaction task checks of the same type together
test: filter tasks of given compaction type
there are chances that a Boost::test test fails to generate a
valid XML file after the test finishes. and
xml.etree.ElementTree.parse() throws when parsing it.
see https://github.com/scylladb/scylla-pkg/issues/3196
before this change, the exception is not handled, and test.py
aborts in this case. this does not help and could be misleading.
after this change, the exception is handled and printed.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#14180
Containing two tables, describing all the possible operations seen in user, system and streaming semaphore diagnostics dumps.
Closes#14171
* github.com:scylladb/scylladb:
docs/dev/reader-concurrency-semaphore.md: add section about operations
docs/dev/reader-concurrency-semaphore.md: switch to # headers markings
reader_concurrency_semaphore: s/description/operation/ in diagnostics dumps
The helper if huge in the form of then-chain. Also it's generic enough not to limit itself in returning sstables' Data file names only.
refs: #14122 (detached from the one that needs more thinking about)
Closes#14174
* github.com:scylladb/scylladb:
table: Return shared sstable from get_sstables_by_partition_key()
table: Coroutinize get_sstables_by_partition_key()
* seastar afe39231...99d28ff0 (16):
> file/util: Include seastar.hh
> http/exception: Use http::reply explicitly
> http/client: Include lost condition-variable.hh
> util: file: drop unnecessary include of reactor.hh
> tests: perf: add a markdown printer
> http/client: Introduce unexpected_status_error for client requests
> sharded: avoid #include <seastar/core/reactor.hh> for run_in_background()
> code: Use std::is_invocable_r_v instead of InvokeReturns
> http/client: Add ability to change pool size on the fly
> http/client: Add getters for active/idle connections counts
> http/client: Count and limit the number of connections
> http/client: Add connection->client RAII backref
> build: use the user-specified compiler when building DPDK
> build: use proper toolchain based on specified compiler
> build: only pass CMAKE_C_COMPILER when building ingredients
> build: use specified compiler when building liburing
Two changes are folded into the commit:
1. missing seastar/core/coroutine.hh include in one .cc file that
got it indirectly included before seastar reactor.hh drop from
file.hh
2. http client now returns unexpected_status_error instead of
std::runtime_error, so s3 test is updated respectively
Closes#14168
There are some headers that include tracing/*.hh ones despite all they
need is forward-declared trace_state_ptr
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Closes#14155
There are some cases where we can deduce that the entry was committed,
but we were throwing `commit_status_unknown`. Handle one more such case.
The added comment explains it in detail.
Also add a FIXME for another case where we throw `commit_status_unknown`
but we could do better.
Fixes: #14029Fixes: #14072Closes#14167
* github.com:scylladb/scylladb:
raft: server: throw fewer `commit_status_unknown`s from `wait_for_entry`
raft: replication test: don't hang if `_seen` overshots `_apply_entries`
raft: replication test: print a warning when handling `commit_status_unknown`
In 297c75c6d8 I set the timeout to
5 minutes mainly due to debug mode which is often quite slow on Jenkins.
But 5 minutes is a bit of an overkill. It wouldn't be a problem but
there is a dtest that waits for a node to fail bootstrap; it's wasteful
for the test to sleep for an entire 5 minutes.
Set it to:
- 3 minutes in debug mode,
- 30 seconds in dev/release modes.
Ref: scylladb/scylla-dtest#3203Closes#14140
There are some cases where we can deduce that the entry was committed,
but we were throwing `commit_status_unknown`. Handle one more such case.
The added comment explains it in detail.
Also add a FIXME for another case where we throw `commit_status_unknown`
but we could do better.
Fixes: #14029
As in the previous commit, if a command gets doubly applied due to
`commit_status_unknown`, this will could lead to hard-to-debug failures;
one of them was the test hanging because we would never call
`_done.set_value()` in `state_machine::apply` due to `_seen`
overshooting `_apply_entries`.
Fix the problem and print a warning if we apply too many commands.
Fixes: #14072
`commit_status_unknown` may lead to double application and then a
hard-to-debug failure. But some tests actually rely on retrying it, so
print a warning and leave a FIXME for maybe a better future solution.
Ref: #14029
This commit adds a table (with 1 row) explaining Scylla-specific
materialized view options - which now consists just of
synchronous_updates.
Tested manually by running `make preview` from docs/ directory.
Closes#11150
The call is generic enough not to drop the sstable itself on return so
that callers can do whatever they need with it. The only today's caller
is API which will convert sstables to filenames on its own
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
when creating an event_record, the typical use case is to use a
string created using fmt::format(), which returns a std::string.
before this change, we always convert the std::string to a sstring,
and move this shinny new sstring into a new event_record. but
when creating sstring, we always performs a deep copy, which is not
necessary, as we own the std::string already.
so, in this change, instead of performing a deep copy, we just keep
the std::string and pass it all the way to where event_record is
created. please note, the std::string will be implicitly converted
to data_value, and it will be dropped on the floor after being
serialized in abstract_type::decompose(). so this deep copy is
inevitable.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
in this change we pass the fmt string using fmt::format_string<T...>
in order to {fmt}'s compile-time formatting. so we can identify
the bad format specifier or bad format placeholders at compile-time.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
In task manager and task manager test rest apis when a task or module
which does not exist is requested, we get Internal Server Error.
In such cases, wrap thrown exceptions in bad_param_exception
to respond with Bad Request code.
Modify test accordingly.
statement_restrictions: forbid IS NOT NULL on columns outside the primary key
IS NOT NULL is currently allowed only when creating materialized views.
It's used to convey that the view will not include any rows that would make the view's primary key columns NULL.
Generally materialized views allow to place restrictions on the primary key columns, but restrictions on the regular columns are forbidden. The exception was IS NOT NULL - it was allowed to write regular_col IS NOT NULL. The problem is that this restriction isn't respected, it's just silently ignored (see #10365).
Supporting IS NOT NULL on regular columns seems to be as hard as supporting any other restrictions on regular columns.
It would be a big effort, and there are some reasons why we don't support them.
For now let's forbid such restrictions, it's better to fail than be wrong silently.
Throwing a hard error would be a breaking change.
To avoid breaking existing code the reaction to an invalid IS NOT NULL restrictions is controlled by the `strict_is_not_null_in_views` flag.
This flag can have the following values:
* `true` - strict checking. Having an `IS NOT NULL` restriction on a column that doesn't belong to the view's primary key causes an error to be thrown.
* `warn` - allow invalid `IS NOT NULL` restrictions, but throw a warning. The invalid restrictions are silently ignored.
* `false` - allow invalid `IS NOT NULL` restricitons, without any warnings or errors. The invalid restrictions are silently ignored.
The default values for this flag are `warn` in `db::config` and `true` in scylla.yaml.
This way the existing clusters will have `warn` by default, so they'll get a warning if they try to create such an invalid view.
New clusters with fresh scylla.yaml will have the flag set to `true`, as scylla.yaml overwrites the default value in `db::config`.
New clusters will throw a hard error for invalid views, but in older existing clusters it will just be a warning.
This way we can maintain backwards compatibility, but still move forward by rejecting invalid queries on new clusters.
Fixes: #10365Closes#13013
* github.com:scylladb/scylladb:
boost/restriction_test: test the strict_is_not_null_in_views flag
docs/cql/mv: columns outside of view's primary key can't be restricted
cql-pytest: enable test_is_not_null_forbidden_in_filter
statement_restrictions: forbid IS NOT NULL on columns outside the primary key
schema_altering_statement: return warnings from prepare_schema_mutations()
db/config: add strict_is_not_null_in_views config option
statement_restrictions: add get_not_null_columns()
test: remove invalid IS NOT NULL restrictions from tests
This is a regression introduced in f26179cd27.
Fixes: #14136
* 'gleb/set_group0' of github.com:scylladb/scylla-dev:
test: restart first node to see if it can boot after restart
service: move setting of group0 point in storage_service earlier
In test_compaction_task.py tests concerning the same type of compaction
are squashed together so that they are run synchronously and there is
no data race when the tests are run in parallel.
Add unit tests for the strict_is_not_null_in_views flag.
This flag controls the behavior in case of an invalid
IS NOT NULL restrictions on a materialized view column.
Materialized views allow only restricting columns
that belong to the view's primary key, all other
restrictions should be rejected.
There was a bug where IS NOT NULL restrictions
weren't rejected, but simply ignored instead.
This flags controls what should happen when the user
runs a query with such an invalid IS NOT NULL restriction.
strict_is_not_null_in_views can have the following values:
* `true` - strict checking, invalid queries are rejected
* `warn` - the query is allowed, but a warning is printed
* `false` - the query is allowed, the invalid restrictions
are silently ignored.
The tests are based on the ones for strict_allow_filtering,
which reside in the lines preceding the newly added tests.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
We used to allow IS NOT NULL restrictions on columns
that were not part of the materialized view's primary key.
It runs out that such restrictions are silently ignored (see #10365),
so we no longer allow such restrictions.
Update the documentation to reflect that change.
Also there was a mistake in the documentation.
It said that restrictions are allowed on all columns
of the base table's primary key, but they are actually
allowed on all columns of the view table's primary key,
not the base tables.
This change also fixes that mistake.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
IS NOT NULL is now allowed only on the view's primary key columns,
so the xfail marker can be removed.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
IS NOT NULL is currently allowed only
when creating materialized views.
It's used to convey that the view will
not include any rows that would make the
view's primary key columns NULL.
Generally materialized views allow
to place restrictions on the primary key
columns, but restrictions on the regular
columns are forbidden. The exception was
IS NOT NULL - it was allowed to write
regular_col IS NOT NULL. The problem is
that this restriction isn't respected,
it's just silently ignored.
Supporting IS NOT NULL on regular columns
seems to be as hard as supporting
any other restrictions on regular columns.
It would be a big effort, and there are some
reasons why we don't support them.
For now let's forbid such restrictions,
it's better to fail than be wrong silently.
Throwing a hard error would be a breaking change.
To avoid breaking existing code the reaction to
invalid IS NOT NULL restrictions is controlled
by the `strict_is_not_null_in_views` flag.
The default values for this flag are `warn` in db::config
and `true` in scylla.yaml.
This way the existing clusters will have `warn` by default,
so they'll get a warning if they try to create such an
invalid view.
New clusters with fresh scylla.yaml will have the flag set
to `true`, as scylla.yaml overwrites the default value
in db::config.
New clusters will throw a hard error for invalid views,
but in older existing clusters it will just be a warning.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
Validation of a CREATE MATERIALIZED VIEW statement takes place inside
the prepare_schema_mutations() method.
I would like to generate warnings during this validation, but there's
currently no way to pass them.
Let's add one more return value - a vector of CQL warnings generated
during the execution of this statement.
A new alias is added to make it clear what the function is returning:
```c++
// A vector of CQL warnings generated during execution of a statement.
using cql_warnings_vec = std::vector<sstring>;
```
Later the warnings will be sent to the user by the function
schema_altering_statment::execute(), which is the only caller
of prepare_schema_mutations().
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
IS NOT NULL shouldn't be allowed on columns
which are outside of the materialized view's primary key.
It's currently allowed to create views with such restrictions,
but they're silently ignored, it's a bug.
In the following commits restricting regular columns
with IS NOT NULL will be forbidden.
This is a breaking change.
Some users might have existing code that creates
views with such restrictions, we don't want to break it.
To deal with this a new feature flag is introduced:
strict_is_not_null_in_views.
By default it's set to `warn`. If a user tries to create
a view with such invalid restrictions they will get a warning
saying that this is invalid, but the query will still go through,
it's just a warning.
The default value in scylla.yaml will be `true`. This way new clusters
will have strict enforcement enabled and they'll throw errors when the
user tries to create such an invalid view,
Old clusters without the flag present in scylla.yaml will
have the flag set to warn, so they won't break on an update.
There's also the option to set the flag to `false`. It's dangerous,
as it silences information about a bug, but someone might want it
to silence the warnings for a moment.
Signed-off-by: Jan Ciolek <jan.ciolek@scylladb.com>
we have a dedicated facility for loading sstables since
68dfcf5256, and column_family (i.e. table)
is not responsible for loading new sstables. so update the comment
to reflect this change.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#14154
In that level no io_priority_class-es exist. Instead, all the IO happens
in the context of current sched-group. File API no longer accepts prio
class argument (and makes io_intent arg mandatory to impls).
So the change consists of
- removing all usage of io_priority_class
- patching file_impl's inheritants to updated API
- priority manager goes away altogether
- IO bandwidth update is performed on respective sched group
- tune-up scylla-gdb.py io_queues command
The first change is huge and was made semi-autimatically by:
- grep io_priority_class | default_priority_class
- remove all calls, found methods' args and class' fields
Patching file_impl-s is smaller, but also mechanical:
- replace io_priority_class& argument with io_intent* one
- pass intent to lower file (if applicatble)
Dropping the priority manager is:
- git-rm .cc and .hh
- sed out all the #include-s
- fix configure.py and cmakefile
The scylla-gdb.py update is a bit hairry -- it needs to use task queues
list for IO classes names and shares, but to detect it should it checks
for the "commitlog" group is present.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Closes#13963
Problem can be reproduced easily:
1) wrote some sstables with smp 1
2) shut down scylla
3) moved sstables to upload
4) restarted scylla with smp 2
5) ran refresh (resharding happens, adds sstable to cleanup
set and never removes it)
6) cleanup (tries to cleanup resharded sstables which were
leaked in the cleanup set)
Bumps into assert "Assertion `!sst->is_shared()' failed", as
cleanup picks a shared sstable that was leaked and already
processed by resharding.
Fix is about not inserting shared sstables into cleanup set,
as shared sstables are restricted to resharding and cannot
be processed later by cleanup (nor it should because
resharding itself cleaned up its input files).
Dtest: https://github.com/scylladb/scylla-dtest/pull/3206Fixes#14001.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Closes#14147
group0 pointer in storage_service should be set when group0 starts.
After f26179cd27 we start group0 earlier,
so we need to move setting of the group0 pointer as well.
Due to a simple programming oversight, one of keyspace_metadata
constructors is using empty user_types_metadata instead of the
passed one. Fix that.
Fixes#14139Closes#14143
as an alternative to passing the link-args using the environmental variable,
we can also use build script to pass the "-C link-args=<FLAG>" to the compiler.
see https://doc.rust-lang.org/nightly/cargo/reference/build-scripts.html#cargorustc-link-argflag
to ensure that cargo is called again by ninja, after build.rs is
updated, build.rs is added as a dependency of {wasm} files along with
Cargo.lock.
this change is verified using following command
```
RUSTFLAGS='--print link-args' cargo build \
--target=wasm32-wasi \
--example=return_input \
--locked \
--manifest-path=Cargo.toml \
--target-dir=build/cmake/test/resource/wasm/rust
```
the output includes "-zstack-size=131072" in the argument passed to lld:
```
Compiling examples v0.0.0 (/home/kefu/dev/scylladb/test/resource/wasm/rust)
LC_ALL="C"
PATH="/usr/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/lib/rustlib/x86_64-unknown-linux-gnu/bin/self-contained:/home/kefu/.local/bin:/home/kefu/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin"
VSLANG="1033"
"lld"
"-flavor" "wasm" "--rsp-quoting=posix" "--export"
"_scylla_abi" "--export" "_scylla_free" "--export" "_scylla_malloc"
"--export" "return_input" "-z" "stack-size=1048576" "--stack-first"
"--allow-undefined" "--fatal-warnings" "--no-demangle"
...
"-L" "/usr/lib/rustlib/wasm32-wasi/lib"
"-L" "/usr/lib/rustlib/wasm32-wasi/lib/self-contained"
"-o"
"/home/kefu/dev/scylladb/build/cmake/test/resource/wasm/rust/wasm32-wasi/debug/examples/return_input-ef03083560989040.wasm"
"--gc-sections"
"--no-entry"
"-O0"
"-zstack-size=131072"
```
with this change, it'd be easier to build .wat files in CMake, so
we don't need to repeat the settings in both configure.py and
CMakeLists.txt
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#14123
