Good practice in general. Also prepares the ground for calling
serialize_for_cql() from serialize_for_cql_with_timestamps(). The latter
already switched to throwing_assert(), avoid regressing to a crash.
Use the collection_mutation_view directly.
Also use managed_bytes_view for collection keys to match
collection_mutation_view::iterator::value_type and to avoid
unnecessary key linearization.
Use the collection_mutation_view directly.
Requires changing key type from bytes_view to managed_bytes_view, to
match collection_mutation_view::iterator::value_type and more
importantly to avoid unnecessary key linearization.
Work with collection_mutation_view directly instead
Use collection_mutation_writer to generate the serialized output
directly, instead of accumulating cells in temporary storage.
Work with collection_mutation_view directly instead.
Use collection_mutation_writer to generate the serialized output
directly, instead of accumulating cells in temporary storage.
Change the collection overload of compaction_garbage_collector::collect()
to accept a collection_mutation (a serialized blob) instead of
collection_mutation_description (an owned, deserialized
representation).
Change collection_mutation_description::compact_and_expire() (the only
user) to build a serialized collection_mutation, with the new
collection_mutation_writer directly.
Allows dropping collection_mutation_description from the
compaction_garbage_collector interface, getting us one step closer to
removing it.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add size, tombstone getters and begin()/end() to iterate over the cells.
Allows inspecting the content of the collection in-place, without
deserializing into an intermediate representation
(collection_mutation_description[_view]). This will be used to gradually
replace all usage of collection_mutation_description[_view].
Deliberately avoding the use of collection_mutation_input_stream as that
one is also a target for elimination.
The names for the accessors are tomb() and size(), following existing
conventions in mutation/. Also rename is_empty() -> empty() to align
with this convention. There is a single caller to update only.
All accessors (new or pre-existing) are made to work with
default-constructed `collection_mutation` (i.e. one containing empty
buffer).
No users yet.
Create a serialized collection element-by-element, with size not known
up-front. Appended elements are written into the buffer directly.
Allows replacing the current practice of first creating a
vector of cells, then serializing at the end.
Currently the collection_mutation default constructor just
default-initializes the underlying managed_bytes, which is not a valid
collection mutation value. This is an accident waiting to happen,
because attempting to deserialize or interpret this value would result
in buffer overflow. Generate a valid collection value in the default
constructor. This is just 5 bytes, small enough to fit into
managed_byte's small buffer, so it even avoids allocation.
The abstract_type parameter was needed by the old IMR-based atomic-cell
serialization format. When we reverted to the simpler managed_bytes
representation, the parameter became a no-op but was never removed.
Drop it from all overloads of atomic_cell_view::from_bytes() and
atomic_cell_mutable_view::from_bytes() and update all call sites.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
In SCYLLADB-2058 we observed a timeout exception while querying the base
table after restarting nodes 2 and 3.
Unfortunately, logs don't give us much useful information about the
root cause.
This patch adds basic checks that nodes see each other after the restart
and that the cql connection sees restarted node.
It doesn't guarantee that the error won't occur again - in logs from
SCYLLADB-2058 we see that each node sees other via gossip after part of
the cluster is restarted.
In case the error will occur again, this commit also increases logging
level of `cql_server` and `storage_proxy`.
Refs SCYLLADB-2058
Closesscylladb/scylladb#29951
This patch series adds `audit_rules`, a new audit configuration option for fine-grained, role-aware audit filtering with per-rule sink routing. Rules can be configured in `scylla.yaml` or updated live through `system.config` without restarting the node. Each rule specifies target sinks (`table`, `syslog`), statement categories, qualified table name patterns, and role patterns. Table and role patterns use POSIX `fnmatch` with extended glob syntax. For table-scoped categories (`DML`, `DDL`, `QUERY`), a rule matches only when the category, role, and qualified table name all match. For table-independent categories (`AUTH`, `ADMIN`, `DCL`), the table filter is ignored. Empty category or role lists match nothing; an empty table list matches nothing only for table-scoped categories. The new rules are additive with the existing `audit_categories`, `audit_keyspaces`, and `audit_tables` settings: both mechanisms are evaluated for each audit event, and the final sink set is the union of all matches.
To avoid evaluating glob patterns on every audit event, audit rules use a preprocessed cache of known roles and tables. The cache is kept in sync through group0 role/table snapshots, role-change notifications, and schema migration notifications. For known entities, rule matching uses precomputed role/table rule sets; unknown entities fall back to direct rule evaluation. When `audit_rules` is empty, per-event rule matching returns immediately and does not evaluate glob patterns. Audit still keeps known role/table metadata in sync while audit is enabled, so rules can be enabled later through live configuration updates without restarting the node.
**Performance**
Measured with `perf-simple-query --smp 1 --duration 100` against a null syslog socket. Results show no regression when audit is disabled, and audit-rules performance has at most 1% more instructions than legacy config for equivalent workloads:
```
===============================================================================================================================================================================
Configuration | Binary | throughput (tps) | insns/op | cpu_cycles/op | alloc/op | logal/op | task/op
===============================================================================================================================================================================
audit=none [1] | baseline | 206922.4 | 36591.6 | 15348.3 | 58.1 | 0.0 | 14.1
audit=none [1] | this PR | 207856.4 (+0.5%) | 36544.9 (-0.1%) | 15274.0 (-0.5%) | 58.1 | 0.0 | 14.1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
audit=syslog keyspaces=ks [2] | baseline | 94871.8 | 54163.0 | 27172.4 | 72.0 | 0.0 | 24.0
audit=syslog keyspaces=ks [2] | this PR | 96138.4 (+1.3%) | 54072.3 (-0.2%) | 26699.3 (-1.7%) | 72.0 | 0.0 | 24.0
audit=syslog audit-rules=ks [3] | this PR | 95142.1 (+0.3%) | 54457.8 (+0.5%) | 26953.8 (-0.8%) | 72.0 | 0.0 | 24.0
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
audit=syslog keyspaces=ks-non-existent [4] | baseline | 213997.8 | 36735.6 | 14848.1 | 58.1 | 0.0 | 14.1
audit=syslog keyspaces=ks-non-existent [4] | this PR | 219297.2 (+2.5%) | 36667.3 (-0.2%) | 14500.1 (-2.3%) | 58.1 | 0.0 | 14.1
audit=syslog audit-rules=ks-non-existent [5] | this PR | 211038.7 (-1.4%) | 36999.7 (+0.7%) | 15048.6 (+1.4%) | 58.1 | 0.0 | 14.1
===============================================================================================================================================================================
[1] ./scylla perf-simple-query --smp 1 --duration 100 --audit "none"
[2] ./scylla perf-simple-query --smp 1 --duration 100 --audit "syslog" --audit-keyspaces "ks" --audit-categories "DCL,DDL,AUTH,DML,QUERY" --audit-unix-socket-path "/tmp/audit-null.sock"
[3] ./scylla perf-simple-query --smp 1 --duration 100 --audit "syslog" --audit-rules '[{"sinks":["syslog"],"categories":["DCL","DDL","AUTH","DML","QUERY"],"qualified_table_names":["ks.*"],"roles":["*"]}]' --audit-unix-socket-path "/tmp/audit-null.sock"
[4] ./scylla perf-simple-query --smp 1 --duration 100 --audit "syslog" --audit-keyspaces "ks-non-existent" --audit-categories "DCL,DDL,AUTH,DML,QUERY" --audit-unix-socket-path "/tmp/audit-null.sock"
[5] ./scylla perf-simple-query --smp 1 --duration 100 --audit "syslog" --audit-rules '[{"sinks":["syslog"],"categories":["DCL","DDL","AUTH","DML","QUERY"],"qualified_table_names":["ks-non-existent.*"],"roles":["*"]}]' --audit-unix-socket-path "/tmp/audit-null.sock"
audit-null.sock was created with `socat -u UNIX-RECV:/tmp/audit-null.sock,type=2 OPEN:/dev/null`
```
Fixes: SCYLLADB-1430
No backport: new feature
Closesscylladb/scylladb#29267
* github.com:scylladb/scylladb:
test: alternator: audit: rules filtering and batch bypass
test: perf: add --audit-rules option to perf-simple-query
docs: add audit rules section to the auditing guide
test: audit: cover role and schema cache notifications
test: audit: cover audit rules cluster behavior
audit: rebuild rule caches on group0 snapshot and role changes
audit: refresh rule caches on schema, role, and config changes
audit: route matching rules to configured sinks
test: cover preprocessed audit rule cache
audit: add preprocessed rule matching cache
audit: pass sink targets to storage helpers
test: audit: cover rule matching semantics
audit: add rule matching and sink helpers
test: audit: cover audit_rules configuration
config: add live audit_rules option
test: cover audit rule parsing and validation
audit: define audit_rule type with parsing and validation
The test was starting Scylla with --write-request-timeout-in-ms=500 on the
command line. This tight timeout also applied to paxos state table creation,
which goes through raft and can take longer than 500ms on slow platforms
(e.g. aarch64/dev). When the first batch of CAS requests triggered paxos
state table creation under error injection, the raft schema change could
still be in-flight when the second batch fired, causing spurious WriteTimeout
failures unrelated to the semaphore bug being tested.
Fix by changing the write timeout at runtime via the REST API: lower it to
500ms only for the error-injection CAS phase (after table creation is done),
then restore it to 10000ms before the second batch that must succeed.
Fixes https://scylladb.atlassian.net/browse/SCYLLADB-2104Closesscylladb/scylladb#29969
Changed seastar::http::experimental to seastar::http to reflect
graduation of the seastar http API.
Changed call to seastar::rename_file() (in sstables/storage.cc,
sstables/sstable_directory.cc, sstable/sstables.cc and
db/hints/internal/hint_storage.cc) to reflect new default parameter.
Updated scylla_gdb test helper get_task() to work with updated
accept loop in Seatar. This is just test code (attempts to find
a task to operate on), not used in real scylla-gdb.py work, but
nevertheless the adjustment keeps backward compatibility.
Fixes https://scylladb.atlassian.net/browse/SCYLLADB-1798
Fixes https://scylladb.atlassian.net/browse/SCYLLADB-2043
* seastar 485a62b2...510f3148 (43):
> reactor_backend: fix iocb double-free and shutdown hang during AIO teardown
> file: fix default DMA alignment
> http: add to_reply() to redirect_exception with extra-header support
> core: propagate syscall errors via `coroutine::exception`
> file: assert dma alignments are powers of two
> doc: Document undocumented io_tester features and fix output example
> backtrace: print the build_id along with the backtrace
> reactor: default to oneline backtraces
> Merge 'json: formatter: support types with user-defined conversion to sstring' from Benny Halevy
tests: json_formatter: test formatter::write with string types
json: formatter: support types with user-defined conversion to sstring
> httpd_test: fix build failure with Seastar_SSTRING=OFF
> net/tls: introduce ssl_call wrapper for SSL I/O
> build: disable unused command line argument error for C++ module
> coroutine/generator: fix setup of generator's waiting task
> tests/tls: set 1000-day validity for self-signed CA cert
> net: tls: openssl: disable certificate compression
> reactor: reduce steady_clock::now() calls per scheduling quantum
> fair_queue: remove notify_request_finished()
> loop: use small_vector for parallel_for_each_state incomplete futures
> dodge false sharing in spinlock
> Merge 'Handle nowait support for reads and writes independently' from Pavel Emelyanov
file: Change nowait_works mode detection
file: Introduce read-only nowait_mode
filesystem: Make nowait_works bit a enum class too
file: Make nowait_works bit a enum class
> Merge 'net/tls: improve OpenSSL error queue hygiene' from Gellért Peresztegi-Nagy
net/tls: assert clean error queue before SSL operations
net/tls: clear error queue after successful SSL operations
net/tls: clear error queue after successful SSL_CTX_new
net/tls: drain error queue on unexpected error codes
net/tls: use make_openssl_error for BIO creation failure
> vla.hh: add missing includes
> Merge 'smp: make smp::count non-static' from Avi Kivity
smp: convert all smp::count usages to instance-aware alternatives
smp: add per-instance shard_count and this_smp() infrastructure
disk_params: document pre-init smp::count access with explicit 0
reactor_backend: document pre-init smp::count access with explicit 0
tests: alien_test: pass shard count to alien thread explicitly
> build: fix cmake missing ninja on Ubuntu 26.04
> rpc: Fix uint64 wraparound of expired timeout in send_entry()
> Merge 'Generalize some RPC tests' from Pavel Emelyanov
tests: Generalize async connection-based scheduling RPC tests
tests: Generalize sync connection-based scheduling RPC tests
tests: Remove redundant variadic/nonvariadic RPC tuple tests
tests: Generalize max timeout RPC tests
> net: tls: openssl: Share BIO ptrs across shards
> http: fix compilation on clang 22 with c++26
> build: openssl tools needed for test cert generation
> reactor: support rename2
> future: fix forwarding of reference types
> Merge 'Zero-copy http chunked data sink' from Pavel Emelyanov
http: Make chunked data sink zero-copy
tests/prometheus_http: Rewrite on top of http::client
tests/httpd: Rewrite content_length_limit on top of http::client
> tests: Replace ad-hoc http_consumer with production HTTP parser
> Merge 'co_return to accept same expressions and types as return' from Alexey Bashtanov
tests/unit/{coroutines,futures}: strict types on co_return and set_value
api: introduce version 10:
core/{coroutine,future}: make `co_return` more strict with types
core/{coroutine,future}: preparations to fix `co_return` type semantics
> Merge 'Perftune.py: add special handling for mlx5 rss queues number calculation' from Vladislav Zolotarov
perftune.py: NetPerfTuner: enhance RSS (a.k.a. "Rx") queues accounting for mlx5 devices
perftune.py: update docstring of NetPerfTuner.__get_rps_cpus() method
perftune.py: add a method that parses and models the output of the 'ethtool -l' command for a given interface
> httpd: rewrite do_accepts/do_accept_one as coroutines
> file: add mmap support to file
> http: Move client code out of experimental namespace
> file: add hugetlbfs support to file system detection
> tests: Replace test_source_impl with util::as_input_stream
> tests: Replace buf_source_impl with util::as_input_stream
> Merge 'rpc_tester: expose throuput for rpc tester' from Marcin Szopa
rpc_tester: remove unused payload size variable from job_rpc_streaming class
rpc_tester: add start time tracking for throughput calculation, print throughput and msg/s for job_rpc
rpc_tester: refactor result emission to use dedicated functions for messages and throughput
> iostream: cast first argument of `std::min` to `size_t`
Closesscylladb/scylladb#29952
Make pytest logging config robust when the ini is missing and prevents
crashing Pytest on the configuration stage in case the wrong tests path
is provided.
Fixes: SCYLLADB-1998
Closesscylladb/scylladb#29941
When reserialize_value() is called on a vector type (which happens only
when the vector's element type contains sets or maps), the old code
materialized all elements via split_fragmented() into a
std::vector<managed_bytes>, then iterated them calling
reserialize_value() on each — discarding the intermediate copy.
Use split_fragmented_view() to obtain zero-copy views of elements, and
pass those directly to reserialize_value(). This avoids one managed_bytes
allocation per element.
Additionally, wrap the call with with_simplified() so that when the
input is a single contiguous fragment (the common case), the compiler
receives a single_fragmented_view and can eliminate fragment-boundary
checks at compile time.
Also generalize build_value_fragmented() to accept any forward range of
FragmentedView elements (not just managed_bytes), and write directly
into the output buffer via with_linearized instead of going through an
intermediate read_simple_bytes copy. This benefits all callers including
evaluate_vector() on the INSERT path for vector<float, N>.
The with_simplified() dispatch instantiates reserialize_value with
single_fragmented_view, which in turn instantiates
partially_deserialize_listlike and partially_deserialize_map with that
type. Add explicit template instantiations in types/types.cc since those
function templates are defined there and only previously instantiated for
managed_bytes_view and fragmented_temporary_buffer::view.
Note: the reserialization path is only exercised for vectors whose
element type contains sets or maps (e.g. vector<frozen<map<int,int>>, N>).
The common vector<float, N> case never enters reserialize_value() because
bound_value_needs_to_be_reserialized() returns false at the call site.
However, the build_value_fragmented() improvement applies to all vector
INSERTs.
References: SCYLLADB-471
Fixes: SCYLLADB-1799
Closesscylladb/scylladb#28559
The audit_rules path was not covered at all by alternator
tests. Add focused coverage that single-table operations
respect audit_rules qualified_table_names filtering, and
that cross-table batches bypass the table filter because
the audit path receives an empty keyspace for multi-table
batch operations.
Refs SCYLLADB-1430
Operators need a reference for the new rule schema, its
relationship to audit_categories/audit_tables/
audit_keyspaces, and the live-update path so they can
adopt the feature without reading the source.
Refs SCYLLADB-1430
Verify on a multi-node cluster that role creation/alter/
drop and table/materialized-view create/drop trigger
updates to the preprocessed audit-rules cache on every
node, and that a matching DML on the newly created table
is audited via the cache.
Refs SCYLLADB-1430
Cluster-level tests should validate rule matching, live
updates, sink routing, role filtering, and error handling
without rerunning the broader audit suite.
Add audit_rules to LIVE_AUDIT_KEYS so the test framework
tracks it as a live-updatable config key. Test that rules
with empty categories or roles match nothing, that DML
rules coexist with legacy audit config, AUTH rules fire
on login events, CQL and REST API update paths reject
invalid JSON, per-rule sink routing works for table and
syslog, role-based filtering works across sessions, and
sink mismatch produces a warning in server logs.
Refs SCYLLADB-1430
Nodes can join or reload snapshots after roles and tables
already exist, so the cache cannot rely only on
incremental notifications.
Bulk-load all known roles and tables into the rule cache
on Raft state reload and snapshot transfer. Detect
incremental role creates and drops in reload_modules() by
comparing the loaded roles against the auth cache, and
forward the changes to every shard.
Each shard rebuilds the fnmatch cache locally from its own
rules to avoid cross-shard races when rules are updated
concurrently with entity sync.
Refs SCYLLADB-1430
Schema, role, and config changes must refresh the
preprocessed rule cache, otherwise the fast path serves
stale matches after reconfiguration or metadata changes.
Register a migration listener for table/view create/drop.
Observe audit_rules config changes through a serialized
action so concurrent rebuilds collapse. Add hooks for role
create/drop and a set_known_entities() bulk-load method.
Implement real cleanup in shutdown() (previously a no-op)
and roll back cleanly on start failure.
Refs SCYLLADB-1430
Rule-based routing must coexist with legacy
category/keyspace/table filtering so operators who have
not opted into rules keep their existing behavior.
Merge rule-matched sinks into the event's sink set
alongside legacy matches. Add a username parameter to
should_log_login/sinks_for_login so rules can match the
authenticated role. Use a conservative over-approximation
for the fast will-log check since the role is not yet
known at that call site. Log an error at startup when
rules reference sinks not enabled globally. Log a warning
when rules are configured but audit is disabled.
Refs SCYLLADB-1430
The rule cache is the fast path for matching, so its hit,
fallback, refresh, and category-bypass behavior needs
focused unit coverage.
Test transparent hash consistency, cached and uncached
lookup paths, incremental entity add/remove, rule
refresh, and empty-rules short circuit.
Refs SCYLLADB-1430
Running fnmatch on every audit event would hurt hot-path
latency.
Precompute per-role and per-table bitsets and intersect
them at query time. Rebuild from snapshots with a
generation counter to avoid partial state after yielding.
Unknown roles/tables fall back to linear fnmatch until
metadata notifications populate the cache.
Refs SCYLLADB-1430
Per-rule routing needs each audit event to carry its
target sinks so storage helpers can self-filter without
duplicating writes.
Replace should_log() with sinks_for() returning an
audit_sink_set and add sinks_for_login() for the login
path. Move the early-return filtering check from the
static inspect() caller into audit::log() so it uses the
new sinks_for() directly. Pass the sink set to
storage_helper::write() so each helper only fires when its
sink is included. Rename parse_audit_modes to
parse_audit_sinks.
Refs SCYLLADB-1430
Rule matching is reused by both the preprocessed cache and
the fallback path -- unit-test it separately so coupling
failures do not mask matching bugs.
Cover category bitmask, glob patterns for tables and
roles, AUTH/ADMIN/DCL table bypass, empty-keyspace batch
bypass, and sink bitmask conversion.
Refs SCYLLADB-1430
Rule matching must be shared between the preprocessed cache
and the fallback path to avoid divergent semantics.
Introduce audit_sink enum and audit_sink_set bitmask for
routing. Match categories via bitmask, tables and roles via
fnmatch with extended globs. AUTH/ADMIN/DCL bypass table
matching. Empty category or role lists match nothing. Empty
keyspace (e.g. cross-table batches) bypasses table matching
for table-scoped categories. Convert validated sink names
to an audit_sink_set bitmask for routing.
Refs SCYLLADB-1430
Audit rules enter through three paths (YAML, CQL, CLI),
each with its own parsing and tracking -- cover all entry
points before routing can depend on them.
Test loading from YAML, live update via CQL and server
API, CLI parsing, invalid value rejection at each path,
and observer notification on live update.
Refs SCYLLADB-1430
Operators need to configure audit rules through YAML, CQL,
and CLI with live-update support so routing can be
reconfigured without restart.
Add audit_rules as a LiveUpdate config option with YAML
decoding, JSON parsing for CQL updates, CLI --audit-rules
flag, and a custom serializer that avoids double-quoting
the JSON array.
Refs SCYLLADB-1430
Parsing and validation are the first consumer-visible
surface of audit rules -- cover them before building
higher layers.
Test JSON parsing (valid, malformed, missing fields),
rule validation (unknown sinks, invalid categories),
and JSON round-trip serialization.
Refs SCYLLADB-1430
Audit rules provide more granular control over which
statements are audited, filtering by tables, roles, and
categories. Typos in sink or category names should be
caught at parse time rather than silently disabling rules
at runtime.
Define the audit_rule struct with JSON parsing, validation
of sink and category names, serialization, and fmt support.
Move statement_category, category_set, and
category_to_string out of audit.hh/audit.cc so the rule
type is self-contained.
Refs SCYLLADB-1430
Add a per-scheduling-group latency histogram on the transport level that measures the full CQL request lifetime: from fetching the request buffer until the response is written to the socket.
Today latencies are accounted only on the storage proxy level, leaving the time spent in the transport layer (response queue wait + actual I/O) unaccounted. Having both transport and storage proxy latencies allows operators to tell where latency accumulates.
The metric is exposed as scylla_transport_cql_request_latency_histogram with the scheduling_group_name label, following the cql_ prefix convention of all other per-SG transport metrics.
Fixes: SCYLLADB-1691
New feature, no backport.
Closesscylladb/scylladb#29878
* github.com:scylladb/scylladb:
test/cluster: add test for per-service-level transport request latency histogram
transport: add per-service-level transport request latency histogram
This PR adds the schema-level validation required for `CREATE INDEX` and `DROP INDEX` on fulltext indexes, mirroring what vector indexes already enforce.
Fulltext indexes are viewless custom indexes (no backing materialized view) that rely on CDC for change tracking. The validation ensures these prerequisites are met at index creation time and cannot be violated afterwards via `ALTER TABLE`.
**Tablet storage**: Fulltext indexes require the keyspace to use tablet storage. Creation is rejected otherwise.
**CDC requirements**: Fulltext indexes need a CDC log with a minimum TTL of 24 hours and either `delta = 'full'` or `postimage = true`. The PR enforces this in three places:
- `CREATE INDEX` rejects creation when existing CDC options don't meet the requirements.
- auto-enables CDC for tables with a fulltext index (same as vector indexes) and validates CDC options on schema updates.
- `ALTER TABLE` blocks disabling CDC while a fulltext index exists.
**Viewless index generalization**: The `vector_index`-specific checks in `create_index_statement` (rejecting `WITH` view properties, name-based duplicate detection for issue #26672) are replaced with a generic `is_viewless_custom_class()` helper that queries the index factory. This automatically covers both vector and fulltext indexes without duplicating logic.
**DROP INDEX** reuses the existing path with no changes needed - the standard drop logic works for viewless indexes as-is.
Added tests covering all validation paths above. All existing tests are updated to require the `skip_without_tablets` fixture.
Fixes: SCYLLADB-1516
Closesscylladb/scylladb#29739
* github.com:scylladb/scylladb:
external_index: fix require CDC options for disabled CDC
test/cqlpy: add duplicate and view tests for fulltext index
cql3: generalize viewless index handling in CREATE INDEX statement
test/cqlpy: add CDC validation tests for fulltext index
fulltext_index: enforce CDC requirements for fulltext indexes
test/cqlpy: add tablet requirement test for fulltext index
fulltext_index: require tablet storage for fulltext indexes
index: introduce `external_index` base class for VS/FTS indexes
Between stopping a server and excluding it, wait for other nodes to see
the server as down, otherwise exclude may see the server as alive and
fail.
Fixes SCYLLADB-2110
Closesscylladb/scylladb#29966
Verify that the new scylla_transport_cql_request_latency_histogram metric
correctly records transport-level request latencies per service level.
Uses error injection to pause a request mid-flight and verifies that the
histogram is not updated while the request is paused (since the response
has not been written yet), and is updated after the request completes.
Co-authored-by: Marcin Maliszkiewicz <marcinmal@scylladb.com>
Add a per-scheduling-group latency histogram that tracks the full
transport-level CQL request lifetime: from fetching the request buffer
until the response is written to the socket.
Today latencies are accounted only on the storage proxy level, which
leaves the time spent in the transport layer unaccounted. The time
spent by a response waiting to be sent out can be significant. Having
both the transport and the storage proxy latencies allows operators to
tell where latency is accumulated.
The histogram uses utils::time_estimated_histogram (range 0.5ms to 33s)
and is exposed as scylla_transport_cql_request_latency_histogram with
the scheduling_group_name label, following the cql_ prefix convention
used by all other per-scheduling-group transport metrics.
The start time is captured at the beginning of process_request().
The latency is recorded after the response is successfully written to
the socket, ensuring the measurement covers processing time, response
queue wait time, and actual I/O time.
Co-authored-by: Marcin Maliszkiewicz <marcinmal@scylladb.com>
