The previous patch changed the interface and callers, this one updates
the implementation to actually work with fragmented buffers. Most types
just use with_linearized() to linearize the fragmented input buffer for
parsing. This is fine, as most types have a fixed or bounded-size string
representation that is small.
Importantly, the input is not linearized for the 3 types which have
unbounded values: ascii, bytes and text. The tuple type can contain any
of these types itself, so it is also converted to avoid linearization.
Change input: str::string_view -> utils::chunked_string_view.
Change return value: bytes -> managed_bytes.
This patch only changes the interface, with some to_bytes() sprinkled in
the internals to deal with recursive calls.
Internals will be updated in the next patch, to keep the churn of
updating callers separate from the actually important changes.
std::string_view is not guaranteed to point to null-terminated string
literals, it may point to a substring of such a string or a string which
is not null-terminated.
std::strtoll() assumes a null terminated string and triggers heap buffer
overflow if this is not true.
Use std::from_chars() -- which doesn't assume or require null-terminated
strings -- to parse numbers from strings instead of std:strtoll().
While at it: fix a small mistake in error reporting. When reporting
failure to parse the number, include the original string in the error
report, instead of the (failed-to-parse) number.
Not a problem on current master, as all callers pass null-terminated
strings.
Add a is_arithmetic() method for types, which can be used to check if
this is a numeric type on which arithmentic operators will allowed -
for example in the following patch to support `SET x = x + 1`.
The arithmetic types are byte, short, int, long, varint, float, double
and decimal.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Good practice in general. Also prepares the ground for calling
serialize_for_cql() from serialize_for_cql_with_timestamps(). The latter
already switched to throwing_assert(), avoid regressing to a crash.
When reserialize_value() is called on a vector type (which happens only
when the vector's element type contains sets or maps), the old code
materialized all elements via split_fragmented() into a
std::vector<managed_bytes>, then iterated them calling
reserialize_value() on each — discarding the intermediate copy.
Use split_fragmented_view() to obtain zero-copy views of elements, and
pass those directly to reserialize_value(). This avoids one managed_bytes
allocation per element.
Additionally, wrap the call with with_simplified() so that when the
input is a single contiguous fragment (the common case), the compiler
receives a single_fragmented_view and can eliminate fragment-boundary
checks at compile time.
Also generalize build_value_fragmented() to accept any forward range of
FragmentedView elements (not just managed_bytes), and write directly
into the output buffer via with_linearized instead of going through an
intermediate read_simple_bytes copy. This benefits all callers including
evaluate_vector() on the INSERT path for vector<float, N>.
The with_simplified() dispatch instantiates reserialize_value with
single_fragmented_view, which in turn instantiates
partially_deserialize_listlike and partially_deserialize_map with that
type. Add explicit template instantiations in types/types.cc since those
function templates are defined there and only previously instantiated for
managed_bytes_view and fragmented_temporary_buffer::view.
Note: the reserialization path is only exercised for vectors whose
element type contains sets or maps (e.g. vector<frozen<map<int,int>>, N>).
The common vector<float, N> case never enters reserialize_value() because
bound_value_needs_to_be_reserialized() returns false at the call site.
However, the build_value_fragmented() improvement applies to all vector
INSERTs.
References: SCYLLADB-471
Fixes: SCYLLADB-1799
Closesscylladb/scylladb#28559
In commit 727f68e0f5 we added the ability to SELECT:
* Individual elements of a map: `SELECT map_col[key]`.
* Individual elements of a set: `SELECT set_col[key]` returns key if the key exists in the set, or null if it doesn't, allowing to check if the element exists in the set.
* Individual pieces of a UDT: `SELECT udt_col.field`.
But at the time, we didn't provide any way to retrieve the **meta-data** for this value, namely its timestamp and TTL. We did not support `SELECT TIMESTAMP(collection[key])`, or `SELECT TIMESTAMP(udt.field)`.
Users requested to support such SELECTs in the past (see issue #15427), and Cassandra 5.0 added support for this feature - for both maps and sets and udts - so we also need this feature for compatibility. This feature was also requested recently by vector-search developers, who wanted to read Alternator columns - stored as map elements, not individual columns - with their WRITETIME information.
The first four patches in this series adds the feature (in four smaller patches instead one big one), the fifth and sixth patches add tests (cqlpy and boost tests, respectively). The seventh patch adds documentation.
All the new tests pass on Cassandra 5, failed on Scylla before the present fix, and pass with it.
The fix was surprisingly difficult. Our existing implementation (from 727f68e0f5 building on earlier machinery) doesn't just "read" `map_col[key]` and allow us to return just its timestamp. Rather, the implementation reads the entire map, serializes it in some temporary format that does **not** include the timestamps and ttls, and then takes the subscript key, at which point we no longer have the timestamp or ttl of the element. So the fix had to cross all these layers of the implementation.
While adding support for UDT fields in a pre-existing grammar nonterminal "subscriptExpr", we unintentionally added support for UDT fields also in LWT expressions (which used this nonterminal). LWT missing support for UDT fields was a long-time known compatibility issue (#13624) so we unintentionally fixed it :-) Actually, to completely fix it we needed another small change in the expression implementation, so the eighth patch in this series does this.
Fixes#15427Fixes#13624Closesscylladb/scylladb#29134
* github.com:scylladb/scylladb:
cql3: support UDT fields in LWT expressions
cql3: document WRITETIME() and TTL() for elements of map, set or UDT
test/boost: test WRITETIME() and TTL() on map collection elements
test/cqlpy: test WRITETIME() and TTL() on element of map, set or UDT
cql3: prepare and evaluate WRITETIME/TTL on collection elements and UDT fields
cql3: parse per-element timestamps/TTLs in the selection layer
cql3: add extended wire format for per-element timestamps and TTLs
cql3: extend WRITETIME/TTL grammar to accept collection and UDT elements
Introduce the infrastructure needed to transport per-element timestamps
and TTL expiry times from replicas to coordinators, required for
WRITETIME(col[key]) / TTL(col[key]) and WRITETIME(col.field) /
TTL(col.field).
* Add a 'writetime_ttl_individual_element' cluster feature flag that
guards usage of the new wire format during rolling upgrades: the
extended format is only emitted and consumed when every node in the
cluster supports it.
* Implement serialize_for_cql_with_timestamps() (types/types.cc), a
variant of serialize_for_cql() that appends a per-element section to
the regular CQL bytes, listing each live element's serialized key,
timestamp, and expiry. The format is:
[uint32 cql_len][cql bytes]
[int32 entry_count]
[per entry: (int32 key_len)(key bytes)(int64 timestamp)(int64 expiry)]
expiry is -1 when the element has no TTL.
* Add partition_slice::option::send_collection_timestamps and modify
write_cell() (mutation_partition.cc) to use the new function
serialize_for_cql_with_timestamps() when this option is available.
This commit stands alone with no user-visible effect: nothing yet sets
the new partition-slice option. The next patch adds the selection-layer
code that sets the option and parses the extended response.
There was a redundant work in split_fragmented(): value_length_if_fixed() was
called inside the loop (N virtual calls), and no reserve() was done
on the output vector causing repeated reallocations.
This patch reserves the output vector to _dimension and
caches value_length_if_fixed() before the loop.
Additionally, split read_vector_element() into two specialized functions:
read_vector_element_fixed() and read_vector_element_variable(), and hoist
the branch on fixed_len outside the loop in split_fragmented() and
deserialize_loop(). This avoids a conditional branch per element in the
hot path.
Benchmark results (1024-dim float vector, release build, -O3 -flto):
10.34 us -> 7.45 us (1.39x, 28% faster)
One of the performance bottlenecks while deserializing vectors was
per-element virtual dispatch in deserialize(): each of the N elements
went through visit() which switches on ~28 type variants. For a
1024-dimension float vector, that's 1024 redundant type switches when
the element type is the same for all of them.
This patch introduces deserialize_vector_visitor that dispatches on the element
type once for the entire vector, then loops inside the resolved
handler. Simple numeric types (float, int, etc.) call
deserialize_value() directly with no virtual dispatch per element.
String types (ascii, utf8) get a dedicated handler that skips
make_empty() (sstring has no empty_t constructor). Complex types
(list, map, tuple, etc.) fall back to per-element dispatch.
Benchmark results (1024-dim float vector, release build, -O3 -flto):
15.73 us -> 11.70 us (1.34x, 26% faster)
Switch vector dimension handling to fixed-width `uint32_t` type,
update parsing/validation, and add boundary tests.
The dimension is parsed as `unsigned long` at first which is guaranteed
to be **at least** 32-bit long, which is safe to downcast to `uint32_t`.
Move `MAX_VECTOR_DIMENSION` from `cql3_type::raw_vector` to `cql3_type`
to ensure public visibility for checks outside the class.
Add tests to verify the type boundaries.
Fixes: https://scylladb.atlassian.net/browse/SCYLLADB-223
Signed-off-by: Yaniv Kaul <yaniv.kaul@scylladb.com>
Co-authored-by: Dawid Pawlik <dawid.pawlik@scylladb.com>
Closesscylladb/scylladb#28762
When describing a table, we need to do it carefully: if some
columns were dropped, we must specify that explicitly by
```
ALTER TABLE {table} DROP {column} USING TIMESTAMP ...
```
in the result of the DESCRIBE statement. Failing to do so
could lead to data resurrection.
However, if a table has been altered many, many times,
we might end up with a huge create statement. Constructing
it could, in turn, trigger an oversized allocation.
Some tests ran into that very problem in fact.
In this commit, we want to mitigate the problem: instead of
allocating a contiguous chunk of memory for the create
statement, we use `fragmented_ostringstream` and `managed_string`
to possibly keep data scattered in memory. It makes handling
`cql3::description` less convenient in the code, but since
the struct is pretty much immediately serialized after
creating it, it's a very good trade-off.
We provide a reproducer. It consistently passes with this commit,
while having about 50% chance of failure before it (based on my
own experiments). Playing with the parameters of the test
doesn't seem to improve that chance, so let's keep it as-is.
Fixesscylladb/scylladb#24018
The vector is a fixed-length array of non-null
specified type elements.
Implement serialization, deserialization, comparison,
JSON and Lua support, and other functionalities.
Co-authored-by: Dawid Pawlik <501149991dp@gmail.com>
The definition of the template is in a source translation unit, but there
are also uses outside the translation unit. Without lto/pgo it worked due
to the definition in the translation unit, but with lto/pgo we can presume
the definition was inlined, so callers outside the translation unit did not
have anything to link with.
Fix by explicitly instantiating the template function.
Closesscylladb/scylladb#22136
All CQL type implementations have a from_sstring(sstring_view) method.
The "sstring_view" type is just an historic alias for std::string_view,
so this patch switches to use the standard type as suggested in #4062,
and also renames these functions from_string_view() to emphesize they can
take any string view, and not necessarily a "sstring" as their old name
suggested.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
The later includes the former and in addition to `seastar::format()`,
`print.hh` also provides helpers like `seastar::fprint()` and
`seastar::print()`, which are deprecated and not used by scylladb.
Previously, we include `seastar/core/print.hh` for using
`seastar::format()`. and in seastar 5b04939e, we extracted
`seastar::format()` into `seastar/core/format.hh`. this allows us
to include a much smaller header.
In this change, we just include `seastar/core/format.hh` in place of
`seastar/core/print.hh`.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#21574
The interface is not used anywhere anymore, so we can
remove it safely. It has been replaced by custom
functions for each keyspace element and `cql3::description`.
We add a new parameter in functions used to generate instances
of `cql3::description` for types related to situations where we
might not need a create statement. An example of such a scenario
could be `DESCRIBE TYPES`.
We're removing `data_dictionary::keyspace_element`.
Before we can do that, we need to substitute the existing
methods used for describing keyspace elements with their
new versions returning `cql3::description`.
That's what happens in this commit.
The introduced function returns the actual name
of the type represented by `abstract_type`.
It circumvents name processing like wrapping a type
within `frozen<>` or using Cassandra's syntax.
We add the function to be able to describe UDFs
in the upcoming commits that require that their
arguments not be `frozen<>`.
We also test the implementation.
before this change, we rely on `using namespace seastar` to use
`seastar::format()` without qualifying the `format()` with its
namespace. this works fine until we changed the parameter type
of format string `seastar::format()` from `const char*` to
`fmt::format_string<...>`. this change practically invited
`seastar::format()` to the club of `std::format()` and `fmt::format()`,
where all members accept a templated parameter as its `fmt`
parameter. and `seastar::format()` is not the best candidate anymore.
despite that argument-dependent lookup (ADT for short) favors the
function which is in the same namespace as its parameter, but
`using namespace` makes `seastar::format()` more competitive,
so both `std::format()` and `seastar::format()` are considered
as the condidates.
that is what is happening scylladb in quite a few caller sites of
`format()`, hence ADT is not able to tell which function the winner
in the name lookup:
```
/__w/scylladb/scylladb/mutation/mutation_fragment_stream_validator.cc:265:12: error: call to 'format' is ambiguous
265 | return format("{} ({}.{} {})", _name_view, s.ks_name(), s.cf_name(), s.id());
| ^~~~~~
/usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/format:4290:5: note: candidate function [with _Args = <const std::basic_string_view<char> &, const seastar::basic_sstring<char, unsigned int, 15> &, const seastar::basic_sstring<char, unsigned int, 15> &, const utils::tagged_uuid<table_id_tag> &>]
4290 | format(format_string<_Args...> __fmt, _Args&&... __args)
| ^
/__w/scylladb/scylladb/seastar/include/seastar/core/print.hh:143:1: note: candidate function [with A = <const std::basic_string_view<char> &, const seastar::basic_sstring<char, unsigned int, 15> &, const seastar::basic_sstring<char, unsigned int, 15> &, const utils::tagged_uuid<table_id_tag> &>]
143 | format(fmt::format_string<A...> fmt, A&&... a) {
| ^
```
in this change, we
change all `format()` to either `fmt::format()` or `seastar::format()`
with following rules:
- if the caller expects an `sstring` or `std::string_view`, change to
`seastar::format()`
- if the caller expects an `std::string`, change to `fmt::format()`.
because, `sstring::operator std::basic_string` would incur a deep
copy.
we will need another change to enable scylladb to compile with the
latest seastar. namely, to pass the format string as a templated
parameter down to helper functions which format their parameters.
to miminize the scope of this change, let's include that change when
bumping up the seastar submodule. as that change will depend on
the seastar change.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
assert() is traditionally disabled in release builds, but not in
scylladb. This hasn't caused problems so far, but the latest abseil
release includes a commit [1] that causes a 1000 insn/op regression when
NDEBUG is not defined.
Clearly, we must move towards a build system where NDEBUG is defined in
release builds. But we can't just define it blindly without vetting
all the assert() calls, as some were written with the expectation that
they are enabled in release mode.
To solve the conundrum, change all assert() calls to a new SCYLLA_ASSERT()
macro in utils/assert.hh. This macro is always defined and is not conditional
on NDEBUG, so we can later (after vetting Seastar) enable NDEBUG in release
mode.
[1] 66ef711d68Closesscylladb/scylladb#20006
this change was created in the same spirit of ebff5f5d.
despite that we include Seastar as a submodule, Seastar is not a
part of scylla project. so we'd better include its headers using
brackets.
ebff5f5d addressed this cosmetic issue a while back. but probably
clangd's header-insertion helped some of contributor to insert
the missing headers with `"`. so this style of `include` returned
to the tree with these new changes.
unfortunately, clangd does not allow us to configure the style
of `include` at the time of writing.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#19406
since we've switched almost all callers of the operator<< to {fmt},
let's drop the unused operator<<:s.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
without `FMT_DEPRECATED_OSTREAM` macro, `UUID::to_sstring()` is
implemented using its `fmt::formatter`, which is not available
at the end of this header file where `UUID` is defined. at this moment,
we still use `FMT_DEPRECATED_OSTREAM` and {fmt} v9, so we can
still use `UUID::to_sstring()`, but in {fmt} v10, we cannot.
so, in this change, we change all callers of `UUID::to_sstring()`
to `fmt::to_string()`, so that we don't depend on
`FMT_DEPRECATED_OSTREAM` and {fmt} v9 anymore.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
C++20 introduced a new overload of std::ostringstream::str() that is selected when the mentioned member function is called on r-value.
The new overload returns a string, that is move-constructed from the underlying string instead of being copy-constructed.
This change applies std::move() on stringstream objects before calling str() member function to avoid copying of the underlying buffer.
It also removes a helper function `inet_addr_type_impl::to_sstring()` - it was used only in two places. It was replaced with `fmt::to_string()`.
Closesscylladb/scylladb#16991
* github.com:scylladb/scylladb:
use fmt::to_string() for seastar::net::inet_address
types/types.cc: move stringstream content instead of copying it
This change removes inet_addr_type_impl::to_sstring()
and replaces its usages with fmt::to_string().
The removed helper performed an uneeded copying via
std::ostringstream::str().
Signed-off-by: Patryk Wrobel <patryk.wrobel@scylladb.com>
C++20 introduced a new overload of std::ofstringstream::str()
that is selected when the mentioned member function is called
on r-value.
The new overload returns a string, that is move-constructed
from the underlying string instead of being copy-constructed.
This change applies std::move() on stringstream objects before
calling str() member function to avoid copying of the underlying
buffer.
Signed-off-by: Patryk Wrobel <patryk.wrobel@scylladb.com>
before this change, we rely on the default-generated fmt::formatter
created from operator<<, but fmt v10 dropped the default-generated
formatter.
in this change, we define a formatter for data_value, but its
its operator<<() is preserved as we are still using the generic
homebrew formatter for formatting std::vector, which in turn uses
operator<< of the element type.
Refs #13245
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#16767
Compilation fails with recent boost versions (>=1.79.0) due to an
ambiguity with the align_up function call. Fix that by adding type
inference to the function call.
Fixes#16746
Signed-off-by: Lakshmi Narayanan Sreethar <lakshmi.sreethar@scylladb.com>
Closesscylladb/scylladb#16747
before this change, we print
marshaling error: Value not compatible with type org.apache.cassandra.db.marshal.AsciiType: '...'
but the wording is not quite user friendly, it is a mapping of the
underlying implementation, user would have difficulty understanding
"marshaling" and/or "org.apache.cassandra.db.marshal.AsciiType"
when reading this error message.
so, in this change
1. change the error message to:
Invalid ASCII character in string literal: '...'
which should be more straightforward, and easier to digest.
2. update the test accordingly
please note, the quoted non-ASCII string is preserved instead of
being printed in hex, as otherwise user would not be able to map it
with his/her input.
Refs #14320
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#15678
