Mention executables (scylla, tools and tests) as well as how to build
individual object files and how to verify individual headers. Also
mention the not-at-all obvious trick of how to build tests with debug
symbols.
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20210416131950.175413-1-bdenes@scylladb.com>
By default, argparse will provide the value of the option as str.
Later, we compare it with int, which will be always False. Fix by
telling argparse to provide as int.
Message-Id: <20210415182149.686355-1-tgrabiec@scylladb.com>
Looks like in python 3, division automatically yields a double/float,
even if both operands are integers. This results in
get_base_class_offset() returning a double/float, which breaks pointer
arithmetics (which is what the returned value is used for), because now
instead of decrementing/incrementing the pointer, the pointer will be
converted to a double itself silently, then back to some corrupt pointer
value. One user visible effect is `intrusive_list` being broken, as it
uses the above method to calculate the member type pointer from the node
pointers.
Fix by coercing the returned value to int.
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20210415080034.167762-1-bdenes@scylladb.com>
The database.hh is the central recursive-headers knot -- it has ~50
includes. This patch leaves only 34 (it remains the champion though).
Similar thing for database.cc.
Both changes help the latter compile ~4% faster :)
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Message-Id: <20210414183107.30374-1-xemul@scylladb.com>
First commit:
In the first commit, add validation of `enable` and `postimage` CDC options. Both options are boolean options, but previously they were not validated, meaning you could issue a query:
```
CREATE TABLE ks.t(pk int, PRIMARY KEY(pk)) WITH cdc = {'enabled': 'dsfdsd'};
```
and it would be executed without any errors, silently interpreting `dsfdsd` as false.
The first commit narrows possible values of those boolean CDC options to `false`, `true`, `0`, `1`. After applying this change, issuing the query above would result in this error message:
```
ConfigurationException: Invalid value for CDC option "enabled": dsfdsd
```
I actually encountered this lacking validation myself, as I mistakenly issued a query:
```
CREATE TABLE ks.t(pk int, PRIMARY KEY(pk)) WITH cdc = {'enabled': true, 'preimage': true, 'postimage': 'full'};
```
incorrectly assigning `full` to `postimage`, instead of `preimage`. However, before this commit, this query ran correctly and it interpreted `full` as `false` and disabled postimages altogether.
Second commit:
The second commit improves the error message of invalid `ttl` CDC option:
Before:
```
CREATE TABLE ks.t(pk int, PRIMARY KEY(pk)) WITH cdc = {'enabled': true, 'ttl': 'invalid'};
ServerError: stoi
```
After:
```
CREATE TABLE ks.t(pk int, PRIMARY KEY(pk)) WITH cdc = {'enabled': true, 'ttl': 'kgjhfkjd'};
ConfigurationException: Invalid value for CDC option "ttl": kgjhfkjd
```
```
CREATE TABLE ks.t(pk int, PRIMARY KEY(pk)) WITH cdc = {'enabled': true, 'ttl': '75747885787487'};
ConfigurationException: Invalid CDC option: ttl too large
```
Closes#8486
* github.com:scylladb/scylla:
cdc: improve exception message of invalid "ttl"
cdc: add validation of "enable" and "postimage"
On Debian variants, sh -x ./install.sh will fail since our script in
written in bash, and /bin/sh in Debian variants is dash, not bash.
So detect non-bash shell and print error message, let users to run in
bash.
Fixes#8479Closes#8484
We start background reclaim after we bootstrap, so bootstrap doesn't
benefit from it, and sees long stalls.
Fix by moving background reclaim initialization early, before
storage_service::join_cluster().
(storage_service::join_cluster() is quite odd in that main waits
for it synchronously, compared to everything else which is just
a background service that is only initialized in main).
Fixes#8473.
Closes#8474
given that resharding is now a synchronous mandatory step, before
table is populated, snapshot() can now get rid of code which takes
into account whether or not a sstable is shared.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Reviewed-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20210414121549.85858-1-raphaelsc@scylladb.com>
* seastar d2dcda96bb...0b2c25d133 (4):
> reactor: reactor_backend_epoll: stop using signals for high resolution timers
> reactor: move task_quota_timer_thread_fn from reactor to reactor_backend_epoll
> Merge "Report maximum IO lenghts via file API" from Pavel E
> Merge "Improve efficiency of io-tester" from Pavel E
Improve the exception message of providing invalid "ttl" value to the
table.
Previously, if you executed a CREATE TABLE query with invalid "ttl"
value, you would get a non-descriptive error message:
CREATE TABLE ks.t(pk int, PRIMARY KEY(pk)) WITH cdc = {'enabled': true, 'ttl': 'invalid'};
ServerError: stoi
This commit adds more descriptive exception messages:
CREATE TABLE ks.t(pk int, PRIMARY KEY(pk)) WITH cdc = {'enabled': true, 'ttl': 'kgjhfkjd'};
ConfigurationException: Invalid value for CDC option "ttl": kgjhfkjd
CREATE TABLE ks.t(pk int, PRIMARY KEY(pk)) WITH cdc = {'enabled': true, 'ttl': '75747885787487'};
ConfigurationException: Invalid CDC option: ttl too large
Add validation of "enable" and "postimage" CDC options. Both options
are boolean options, but previously they were not validated, meaning
you could issue a query:
CREATE TABLE ks.t(pk int, PRIMARY KEY(pk)) WITH cdc = {'enabled': 'dsfdsd'};
and it would be executed without any errors, silently interpreting
"dsfdsd" as false.
This commit narrows possible values of those boolean CDC options to
false, true, 0, 1. After applying this change, issuing the query above
would result in this error message:
ConfigurationException: Invalid value for CDC option "enabled": dsfdsd
This patch fixes cql-pytest/run-cassandra to work on systems which
default to Java 11, including Fedora 33.
Recent versions of Cassandra can run on Java 11 fine, but requires a
bunch of weird JVM options to work around its JPMS (Java Platform Module
System) feature. Cassandra's start scripts require these options to
be listd in conf/jvm11-server.options, which is read by the startup
script cassandra.in.sh.
Because our "run-cassandra" builds its own "conf" directory, we need
to create a jvm11-server.options file in that directory. This is ugly,
but unfortunately necessary if cql-pytest/run-cassandra is to run with
on systems defaulting to Java 11.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20210406220039.195796-1-nyh@scylladb.com>
We currently only update the failure detector for a node when a higher
version of application state is received. Since gossip syn messages do
not contain application state, so this means we do not update the
failure detector upon receiving gossip syn messages, even if a message
from peer node is received which implies the peer node is alive.
This patch relaxes the failure detector update rule to update the
failure detector for the sender of gossip messages directly.
Refs #8296Closes#8476
When a particular partition exists in at least one sstable, the cache
expects any single-partition query to this partition to return a `partition_start`
fragment, even if the result is empty.
In `time_series_sstable_set::create_single_key_sstable_reader` it could
happen that all sstables containing data for the given query get
filtered out and only sstables without the relevant partition are left,
resulting in a reader which immediately returns end-of-stream (while it
should return a `partition_start` and if not in forwarding mode, a
`partition_end`). This commit fixes that.
We do it by extending the reader queue (used by the clustering reader
merger) with a `dummy_reader` which will be returned by the queue as
the very first reader. This reader only emits a `partition_start` and,
if not in forwarding mode, a `partition_end` fragment.
Fixes#8447.
Closes#8448
Support native building & unit testing in the Nix ecosystem under
nix-shell.
Actual dist packaging for Nixpkgs/NixOS is not there (yet?), because:
* Does not exactly seem like a huge priority.
* I don't even have a firm idea of how much work it would entail (it
certainly does not need the ld.so trickery, so there's that. But at
least some work would be needed, seeing how ScyllaDB needs to
integrate with its environment and NixOS is a little unorthodox).
Signed-off-by: Michael Livshin <michael.livshin@scylladb.com>
Message-Id: <20210413110508.5901-4-michael.livshin@scylladb.com>
"
These methods are generic ways to query a mutation source. At least they
used to be, but nowadays they are pretty specific to how tables are
queried -- they use a querier cache to lookup queriers from and save
them into. With the coming changes to how permits are obtained, they are
about to get even more specific to tables. Instead of forcing the
genericity and keep adding new parameters, this patchset bites the
bullet and moves them to table. `data_query()` is inlined into
`table::query()`, while `mutation_query()` is replaced with
`table::mutation_query()`.
The only other users besides table are tests and they are adjusted to
use similarly named local methods that just combine the right querier
with the right result builder. This combination is what the tests really
want to test, as this is also what is used by the table methods behind
the scenes.
Tests: unit(release, debug)
"
* 'mutation-query-move-query-methods-into-table/v1' of https://github.com/denesb/scylla:
mutation_query: remove now unused mutation_query()
test: mutation_query_test: use local mutation_query() implementation
database: mutation_query(): use table::mutation_query()
table: add mutation_query()
query: remove the now unused data_query()
test: mutation_query_test: use local data_query() implementation
table: query(): inline data_query() code into query()
table: make query() a coroutine
It is possible that a partition is in cache but is not present in sstables that are underneath.
In such case:
1. cache_flat_mutation_reader will fast forward underlying reader to that partition
2. The underlying reader will enter the state when it's empty and its is_end_of_stream() returns true
3. Previously cache_flat_mutation_reader::do_fill_buffer would try to fast forward such empty underlying reader
4. This PR fixes that
Test: unit(dev)
Fixes#8435Fixes#8411Closes#8437
* github.com:scylladb/scylla:
row_cache: remove redundant check in make_reader
cache_flat_mutation_reader: fix do_fill_buffer
read_context: add _partition_exists
read_context: remove skip_first_fragment arg from create_underlying
read_context: skip first fragment in ensure_underlying
This check is always true because a dummy entry is added at the end of
each cache entry. If that wasn't true, the check in else-if would be
an UB.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Make sure that when a partition does not exist in underlying,
do_fill_buffer does not try to fast forward withing this nonexistent
partition.
Test: unit(dev)
Fixes#8435Fixes#8411
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
This new state stores the information whether current partition
represented by _key is present in underlying.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
This was previously done in create_underlying but ensure_underlying is
a better place because we will add more related logic to this
consumption in the following patches.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
alternator/expressions.g had both AGPL and proprietary licensing. The
proprietary one is removed.
gms/inet_address_serializer.hh had only a proprietary license; it is
replaced by the AGPL.
Fixes#8465.
Closes#8466
This series introduces service level syntax borrowed from https://docs.scylladb.com/using-scylla/workload-prioritization/ , but without workload prioritization itself - just for the sake of using identical syntax to provide different parameters later. The new parameters may include:
* per-service-level timeouts
* oltp/olap declaration, which may change the way Scylla treats long requests - e.g. time them out (the oltp way) or keep them sustained with empty pages (the olap way)
Refs #7617Closes#7867
* github.com:scylladb/scylla:
transport: initialize query state with service level controller
main: add initializing service level data accessor
service: make enable_shared_from_this inheritance public
cql3: add SERVICE LEVEL syntax (without an underscore)
unit test: Add unit test for per user sla syntax
cql: Add support for service level cql queries
auth: Add service_level resource for supporting in authorization of cql service_level
cql: Support accessing service_level_controller from query state
instantiate and initialize the service_level_controller
qos: Add a standard implementation for service level data accessor
qos: add waiting for the updater future
service/qos: adding service level controller
service_levels: Add documentation for distributed tables
service/qos: adding service level table to the distributed keyspace
service/qos: add common definitions
auth: add support for role attributes
In order for the syntax to be more natural, it's now possible
to use SERVICE LEVEL instead of SERVICE_LEVEL in all appropriate
places. The old syntax is supported as well.
This commit adds the infrastructure needed to test per user sla,
more specificaly, a service level accessor that triggers the
update_service_levels_from_distributed_data function uppon any
change to the dystributed sla data.
A test was added that indirectly consumes this infrastructure by
changing the distributed service level data with cql queries.
Message-Id: <23b2211e409446c4f4e3e57b00f78d9ff75fc978.1609249294.git.sarna@scylladb.com>
This patch adds support for new service level cql queries.
The queries implemented are:
CREATE SERVICE_LEVEL [IF NOT EXISTS] <service_level_name>
ALTER SERVICE_LEVEL <service_level_name> WITH param = <something>
DROP SERVICE_LEVEL [IF EXISTS] <service_level_name>
ATTACH SERVICE_LEVEL <service_level_name> TO <role_name>
DETACH SERVICE_LEVEL FROM <role_name>
LIST SERVICE_LEVEL <service_level_name>
LIST ALL SERVICE_LEVELS
LIST ATTACHED SERVICE_LEVEL OF <role_name>
LIST ALL ATTACHED SERVICE_LEVELS
queries
In order to be able to manage service_level configuration one must be authorized
to do so, or to be a superuser. This commit adds the support for service_levels
resource. Since service_levels are relative, reconfiguring one service level is not locallized
only to that service level and will affect the QOS for all of the service levels,
so there is not much sense of granting permissions to manage individual service_levels.
This is why only root resource named service_levels that represents all service levels is used.
This commit also implements the unit test additions for the newly introduced resource.
Message-Id: <81ab16fa813b61be117155feea405da6266921e3.1609237687.git.sarna@scylladb.com>
In order to implement service level cql queries, the queries objects
needs access to the service_level_controller object when processing.
This patch adds this access by embedding it into the query state object.
In order to accomplish the above the query processor object needs an
access to service_level_controller in order to instantiate the query state.
Message-Id: <68f5a7796068a49d9cd004f1cbf34bdf93b418bc.1609234193.git.sarna@scylladb.com>
service_level_controller defines an interface for accessing the service
level distributed data, this patch implements a standard implementation
of the interface that delegates to the system distributed keyspace.
Message-Id: <25e68302f6f4d4fe5fcb66ea19159ad68506ba64.1609175314.git.sarna@scylladb.com>
The distributed data updated used to spawn a future without waiting
for it. It was quite safe, since the future had its own abort source,
but it's better to remember it and wait for it during stop() anyway.
In the general case roles might come with attributes attached to them
these attributes can originate in mechanisms such as LDAP where in
the undelying directory each entity can have a key:value data structure.
This patch add support for such attributes in the role manager interface,
it also implements the attribute support in the standard role
manager in the form of a table with an attribute map in the distributed system keyspace.
Message-Id: <f53c74a7ac315c4460ff370ea6dbb1597821edc2.1609158013.git.sarna@scylladb.com>
fixes AddressSanitizer: stack-buffer-underflow on address 0x7ffd9a375820 at pc 0x555ac9721b4e bp 0x7ffd9a374e70 sp 0x7ffd9a374620
Backend registry holds a unique pointer to the backend implementation
that must outlive the whole tracing lifetime until the shutdown call.
So it must be catched/moved before the program exits its scope by
passing out the lambda chain.
Regarding deletion of the default destructor: moving object requires
a move constructor (for do_with) that is not implicitly provided if
there is a user-defined object destructor defined even tho its impl
is default.
Signed-off-by: Ivan Prisyazhnyy <ivan@scylladb.com>
Closes#8461
This pull request adds a "ninja help" build target in hopes of making
the different build targets more discoverable to developers.
Closes#8454
* github.com:scylladb/scylla:
building.md: Document "ninja help" target
configure.py: "ninja help" target
building.md: Document "ninja <mode>-dist" target
configure.py: Add <mode>-dist target as alias for dist-<mode>
Stop use seastar::pipe and use seastar::queue directly to pass log
entries to apply_fiber. The pipe is a layer above queue anyway and it
adds functionality that we do not need (EOS) and hinds functionality that
we do (been able to abort()). This fixes a crash during abort where the
pipe was uses after been destroyed.
Message-Id: <YHLkPZ9+sdLhwcjZ@scylladb.com>
* seastar 1c1f610ceb...d2dcda96bb (3):
> closeable: add with_closeable and with_stoppable helpers
> circleci: relax concurrency of the build process
> logger: failed_to_log: print source location and format string
