Fixes#5902 by making the LIKE restriction keep a vector
of matchers and apply them all to the column value.
Tests: unit (dev)
* dekimir/multiple-likes:
cql3: Allow repeated LIKE on same column
cql3: Forbid calling LIKE::values()
cql3: Move LIKE::_last_pattern to matcher
In order to properly validate not only network topology strategy,
but also other strategies, the checks are moved straight to
validate_replication_factor().
Also, the test case is extended with a too long integer
and a check for SimpleStrategy replication factor.
Fixes#3801
Tests: unit(dev)
Message-Id: <e0c3c3c36c589e1d440c9708a6dce820c111b8da.1583483602.git.sarna@scylladb.com>
In order to prevent users from creating a network topology
strategy instance with invalid inputs, it's not enough to use
std::stol() on the input: a string "3abc" still returns the number '3',
but will later confuse cqlsh and other drivers, when they ask for
topology strategy details.
The error message is now more human readable, since for incorrect
numeric inputs it used to return a rather cryptic message:
ServerError: stol()
This commit fixes the issue and comes with a simple test.
Fixes#3801
Tests: unit(dev)
Message-Id: <7aaae83d003738f047d28727430ca0a5cec6b9c6.1583478000.git.sarna@scylladb.com>
If a batch update is performed with a sequence of changes with a single
timestamp, they will now show up in CDC with a single timeuuid in the
`time` column, distinguished by different `batch_seq_no` values.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Currently, writes to a static row in a base table are not reflected
at all in the corresponding cdc log. This patch causes such writes
to be properly logged.
Fixes: #5744
Tests: unit(dev)
* piodul/5744-handle-static-row-correctly-in-cdc:
cdc_test: add tests for handling static row
cdc: fix indentation in transformer::transform
cdc: handle static rows separately in transformer::transform
cdc: move process_cells higher (and fix captured variables)
cdc: reduce dependencies on captured variables in process_cells
cdc: fix preimage query for static rows
We use boost test logging primarily to generate nice XML xunit
files used in Jenkins. These XML files can be bloated
with messages from BOOST_TEST_MESSAGE(), hundreds of megabytes
of build archives, on every build.
Let's use seastar logger for test logging instead, reserving
the use of boost log facilities for boost test markup information.
It is possible to produce an empty mutation using CQL. For example, the
following query:
DELETE FROM ks.tbl WHERE pk = 0 AND ck < 1 AND ck > 2;
will attempt to delete from an empty range of rows. This is translated
to the following mutation:
{ks.tbl {key: pk{000400000000}, token:-3485513579396041028}
{mutation_partition:
static: cont=1 {row: },
clustered: {}}}
Such mutation does not contain any timestamp, therefore it is difficult
to determine what timestamp was used while making the query. This is
problematic for CDC, because an entry in CDC log should be written with
the same timestamp as a part of the mutation.
Because an empty mutation does not modify the table in any way, we can
safely skip logging such mutations in CDC and still preserve the
ability to reconstruct the current state of the base table from full
CDC log.
Tests: unit(dev)
Most test-methods log a message with their names upon entering them.
This helps in identifying the test-method a failure happened in in the
logs. Two methods were missing this log line, so add it.
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20200304155235.46170-1-bdenes@scylladb.com>
Merged patch series from Avi Kivity:
boost/multiprecision is a heavyweight library, pulling in 20,000 lines of code into
each header that depends on it. It is used by converting_mutation_partition_applier
and types.hh. While the former is easy to put out-of-line, the latter is not.
All we really need is to forward-declare boost::multiprecision::cpp_int, but that
is not easy - it is a template taking several parameters, among which are non-type
template parameters also defined in that header. So it's quite difficult to
disentangle, and fragile wrt boost changes.
This patchset introduces a wrapper type utils::multiprecision_int which _can_
be forward declared, and together with a few other small fixes, manages to
uninclude boost/multiprecision from most of the source files. The total reduction
in number of lines compiled over a full build is 324 * 23,227 or around 7.5
million.
Tests: unit (dev)
Ref #1https://github.com/avikivity/scylla uninclude-boost-multiprecision/v1
Avi Kivity (5):
converting_mutation_partition_applier: move to .cc file
utils: introduce multiprecision_int
tests: cdc_test: explicitly convert from cdc::operation to uint8_t
treewide: use utils::multiprecision_int for varint implementation
types: forward-declare multiprecision_int
configure.py | 2 +
concrete_types.hh | 2 +-
converting_mutation_partition_applier.hh | 163 ++-------------
types.hh | 12 +-
utils/big_decimal.hh | 3 +-
utils/multiprecision_int.hh | 256 +++++++++++++++++++++++
converting_mutation_partition_applier.cc | 188 +++++++++++++++++
cql3/functions/aggregate_fcts.cc | 10 +-
cql3/functions/castas_fcts.cc | 28 +--
cql3/type_json.cc | 2 +-
lua.cc | 38 ++--
mutation_partition_view.cc | 2 +
test/boost/cdc_test.cc | 6 +-
test/boost/cql_query_test.cc | 16 +-
test/boost/json_cql_query_test.cc | 12 +-
test/boost/types_test.cc | 58 ++---
test/boost/user_function_test.cc | 2 +-
test/lib/random_schema.cc | 14 +-
types.cc | 20 +-
utils/big_decimal.cc | 4 +-
utils/multiprecision_int.cc | 37 ++++
21 files changed, 627 insertions(+), 248 deletions(-)
create mode 100644 utils/multiprecision_int.hh
create mode 100644 converting_mutation_partition_applier.cc
create mode 100644 utils/multiprecision_int.cc
The goal is to forward-declare utils::multiprecision_int, something
beyond my capabilities for boost::multiprecision::cpp_int, to reduce
compile time bloat.
The patch is mostly search-and-replace, with a few casts added to
disambiguate conversions the compiler had trouble with.
After the varint data type starts using the new multiprecision_int type,
this code fails to compile. I expect that somehow the conversion from enum
class to cpp_int was allowed to succeed, and we ended up with a data_value
of type varint. The tests succeeded because the serialized representation
happened to be the same.
Previously we had stream_id_1 and stream_id_2 columns
of type long each. They were forming a partition key.
In a new format we want a single stream_id column that
forms a partition key. To be able to still store two
longs, the new column will have type blob and its value
will be concatenated bytes of two longs that
partition key is composed of.
We still want partition key to logically be two longs
because those two values will be used by a custom partitioner
later once we implement it.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
I found that a few variables in cql_test_env were wrapping sharded in
shared_ptr for no apparent reason. These patches convert them to plain
sharded<...>.
"
This set cleans sstable_writer_config and surrounding sstables
code from using global storage_ and feature_ service-s and database
by moving the configuration logic onto sstables_manager (that
was supposed to do it since eebc3701a5).
Most of the complexity is hidden around sstable_writer_config
creation, this set makes the sstables_manager create this object
with an explicit call. All the rest are consequences of this change.
Tests: unit(debug), manual start-stop
"
* 'br-clean-sstables-manager-2' of https://github.com/xemul/scylla:
sstables: Move get_highest_supported_format
sstables: Remove global get_config() helper
sstables: Use manager's config() in .new_sstable_component_file()
sstable_writer_config: Extend with more db::config stuff
sstables_manager: Don't use global helper to generate writer config
sstable_writer_config: Sanitize out some features fields initialization
sstable_writer_config: Factor out some field initialization
sstables: Generate writer config via manager only
sstables: Keep reference on manager
test: Re-use existing global sstables_manager
table: Pass sstable_writer_config into write_memtable_to_sstable
If the mutation contains separate logical changes (e.g. with different
timestamps and/or ttls), it will be split into multiple mutations, each
passed into transform.
Merged pull request https://github.com/scylladb/scylla/pull/5910 by
Calle Wilund:
Rename metadata and data columns according to new spec
Also use transformation methods for names in all code + tests
to make switching again easier
Break up data column tuple
Data column is now pure frozen original type.
If column is deleted (set to null), a metadata column cdc$deleted_ is set to true, to distinguish null column == not involved in row operation
For non-atomic collections, a cdc$deleted_elements_ column is added, and when removing elements from collection this is where they are shown.
For non-atomic assign, the "cdc$deleted_" is true, and is set to new value.
column_op removed.
According to "new" spec:
Data column is now pure frozen original type.
If column is deleted (set to null), a metadata column
cdc$deleted_<name> is set to true, to distinguish
null column == not involved in row operation
For non-atomic collections, a cdc$deleted_elements_<name>
column is added, and when removing elements from collection
this is where they are shown.
For non-atomic assign, the "cdc$deleted_<name>" is true,
and <name> is set to new value.
column_op removed.
Support LIKE operator condition on column expressions.
NOTE: following the existing code, the LIKE pattern value is converted
to raw bytes and passed straight as bytes_view to like_matcher
without type checking; it should be checked/sanitized by caller.
Refs: #5777
Branch URL: https://github.com/alecco/scylla/tree/as_like_condition_2
Tests: unit ({dev}), unit ({debug})
NOTE: fail for unrelated test test_null_value_tuple_floating_types_and_uuids
`table` is not registered with the database, and hence will not be
waited on during shutdown.
Stop it explicitly to prevent any asynchronous operation on it racing
with shutdown.
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20200302142845.569638-1-bdenes@scylladb.com>
When incoming mutation contains live row marker the `operation` is
described as "insert", not as an "update".
Also, I extended the test case "test_row_delete" with one insert,
which is expected to log different value of `operation` than update
or delete. Renamed the test case accordingly.
Test cases that relied on "update" being the same as "insert" are
updated accordingly (`test_pre_image_logging`, `test_cdc_across_shards`,
`test_add_columns`).
Fixes#5723
"
Timeouts defaulted to `db::no_timeout` are dangerous. They allow any
modifications to the code to drop timeouts and introduce a source of
unbounded request queue to the system.
This series removes the last such default timeouts from the code. No
problems were found, only test code had to be updated.
tests: unit(dev)
"
* 'no-default-timeouts/v1' of https://github.com/denesb/scylla:
database: database::query*(), database::apply*(): remove default timeouts
database: table::query(): remove default timeout
mutation_query: data_query(): remove default timeout
mutation_query: mutation_query(): remove default timeout
multishard_mutation_query: query_mutations_on_all_shards(): remove default timeout
reader_concurrency_semaphore: wait_admission(): remove default timeout
utils/logallog: run_when_memory_available(): remove default timeout
"
As part of avoiding static initialization order problems I want to
switch a few global sstring to constexpr std::string_view. The
advantage being that a constexpr variable doesn't need runtime
initialization and therefore cannot be part of a static initialization
order problem.
In order to do the conversion I needed to convert a few APIs to use
std::string_view instead of sstring and const sstring&.
These patches are the simple cases that are also an improvement in
their own right.
"
* 'espindola/string_view' of https://github.com/espindola/scylla: (22 commits)
test: Pass a string_view to create_table's callback
Pass string_view to the schema constructor
cql3: Pass string_view to the column_specification constructor
Pass string_view to keyspace_metadata::new_keyspace
Pass string_view to the keyspace_metadata constructor
utils: Use std::string as keys in nonstatic_class_registry
utils: Pass a string_view to class_registry::to_qualified_class_name
auth: Return a string_view from authorizer::qualified_java_name
auth: Return a string_view from authenticator::qualified_java_name
utils: Pass string_view to is_class_name_qualified
test: Pass a string_view to create_keyspace
Pass string_view to no_such_column_family's constructor
perf_simple_query: Pass a string_view to make_counter_schema
Pass string_view to the schema_builder constructor
types: Add more data_value constructors
transport: Pass a string_view to cql_server::connection::make_autheticate
transport: Pass a string_view to cql_server::response::write_string
cql3: Pass std::string_view to query_processor::compute_id
cql3: Remove unused variable
cql3: Pass a string_view to cf_statement::prepare_keyspace
...
If the reversing requires more memory than the limit, the read is
aborted. All users are updated to get a meaningful limit, from the
respective table object, with the exception of tests of course.
Currently reverse reads just pass a flag to
`flat_mutation_reader::consume()` to make the read happen in reverse.
This is deceptively simple and streamlined -- while in fact behind the
scenes a reversing reader is created to wrap the reader in question to
reverse partitions, one-by-one.
This patch makes this apparent by exposing the reversing reader via
`make_reversing_reader()`. This now makes how reversing works more
apparent. It also allows for more configuration to be passed to the
reversing reader (in the next patches).
This change is forward compatible, as in time we plan to add reversing
support to the sstable layer, in which case the reversing reader will
go.
