Commit Graph

39003 Commits

Author SHA1 Message Date
Pavel Emelyanov
becd960ae8 view_update_generator: Add logging to do_abort()
Just tell the logs that the guy is aborting
refs: #10941

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-09-21 13:34:21 +03:00
Pavel Emelyanov
967ebacaa4 view_update_generator: Move abort kicking to do_abort()
When v.u.g. stops is first aborts the generation background fiber by
requesting abort on the internal abort source and signalling the fiber
in case it's waiting. Right now v.u.g.::stop() is defer-scheduled last
in main(), so this move doesn't change much -- when stop_signal fires,
it will kick the v.u.g.::do_abort() just a bit earlier, there's nothing
that would happen after it before real ::stop() is called that depends
on it.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-09-21 13:32:45 +03:00
Pavel Emelyanov
e34220ebb7 view_update_generator: Add early abort subscription
Subscribe v.u.g. to the main's stop_signal. For now a no-op callback.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
2023-09-21 13:32:45 +03:00
Botond Dénes
7f03ef07c8 Merge 'build: use default value of --with-* option s' from Kefu Chai
in this series, we use the default values of options specifying the paths to tools for better readability. and also to ease the migration to CMake

Refs #15379

Closes scylladb/scylladb#15500

* github.com:scylladb/scylladb:
  build: do not check for args.ragel_exec
  build: set default value of --with-antlr3 option
2023-09-21 10:51:08 +03:00
Pavel Emelyanov
e6fe18ca55 s3: Handle piece flushing exception
When a piece is uploaded it's first flushed, then upload-copy is issued.
Both happen in the background and if piece flush calls resolves with
exception the exception remains unhandled. That's OK, since upload
finalization code checks that some pieces didn't complete (for whatever
reason) and fails the whole uploading, however, the ignored exception is
reported in logs. Not nice.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>

Closes scylladb/scylladb#15491
2023-09-21 10:39:04 +03:00
Botond Dénes
ac8005a102 Merge 'build: extract code fragments into functions' from Kefu Chai
more structured this way. this also allows us to quickly identify the part which should/can be reused when migrating to CMake based building system.

Refs #15379

Closes scylladb/scylladb#15501

* github.com:scylladb/scylladb:
  build: extract check_for_lz4() out
  build: extract check_for_boost() out
  build: extract check_for_minimal_compiler_version() out
  build: extract write_build_file() out
2023-09-21 09:36:14 +03:00
Botond Dénes
f6575344df Merge 'Collect dangling object-store sstables' from Pavel Emelyanov
Sstables in transitional states are marked with the respective 'status' in the registry. Currently there are two of such -- 'creating' and 'removing'. And the 'sealed' status for sstables in use.

On boot the distributed loader tries to garbage collect the dangling sstables. For filesystem storage it's done with the help of temorary sstables' dirs and pending deletion logs. For s3-backed sstables, the garbage collection means fetching all non-sealed entries and removing the corresponding objects from the storage.

Test included (last patch)

fixes #13024

Closes scylladb/scylladb#15318

* github.com:scylladb/scylladb:
  test: Extend object_store test to validate GC works
  sstable_directory: Garbage collect S3 sstables on reboot
  sstable_directory: Pass storage to garbage_collect()
  sstable_directory: Create storage instance too
2023-09-21 09:15:00 +03:00
Benny Halevy
e8f720315d gossiper: run: hold background_gate when sending gossip in background
So it would be waited on in shutdown().

Although gossiper::run holds the `_callback_running` semaphore
which is acquired in `do_stop_gossiping`, the gossip messages
it initiates in the background are never waited on.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

Closes scylladb/scylladb#15493
2023-09-21 08:54:35 +03:00
Kefu Chai
fe4caeb77f utils/s3/client: do not allocate rapidxml::xml_document on stack
as the size of `rapidxml::xml_document` size quite large, let's
allocate it on the heap. otherwise GCC 13.2.1 warns us like:
```
utils/s3/client.cc: In function ‘seastar::sstring s3::parse_multipart_copy_upload_etag(seastar::sstring&)’:
utils/s3/client.cc:455:9: warning: stack usage is 66208 bytes [-Wstack-usage=]
  455 | sstring parse_multipart_copy_upload_etag(sstring& body) {
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#15472
2023-09-21 08:51:08 +03:00
Kefu Chai
8802364b5b build: extract check_for_lz4() out
more structured this way. this also allows us to quickly identify
the part which should/can be reused when migrating to CMake based
building system.

Refs #15379

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-09-21 11:11:00 +08:00
Kefu Chai
cb02a56421 build: extract check_for_boost() out
more structured this way. this also allows us to quickly identify
the part which should/can be reused when migrating to CMake based
building system.

Refs #15379

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-09-21 11:05:31 +08:00
Kefu Chai
9996503f56 build: extract check_for_minimal_compiler_version() out
more structured this way. this also allows us to quickly identify
the part which should/can be reused when migrating to CMake based
building system.

Refs #15379
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-09-21 11:05:31 +08:00
Kefu Chai
7236b81efc build: extract write_build_file() out
more structured this way. also, this will allow us to switch over
to the CMake building system.

Refs #15379
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-09-21 10:33:05 +08:00
Kefu Chai
f3d6e91287 build: do not check for args.ragel_exec
args.ragel_exec defaults to "ragel" already, so unless user specifies
an empty ragel using `--with-ragel=""`, we won't have an
`args.ragel_exec` which evaluates to `False`, so drop this check.

Refs #15379

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-09-21 10:28:57 +08:00
Kefu Chai
4632609a1c build: set default value of --with-antlr3 option
so we don't need to check if this option is specified.
this option will also be used even after switching to CMake.

Refs #15379
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2023-09-21 10:28:57 +08:00
Botond Dénes
a56a4b6226 Merge 'compaction_backlog_tracker: do not allow moving registered trackers' from Benny Halevy
Currently, the moved-object's manager pointer is moved into the
constructed object, but without fixing the registration to
point to the moved-to object, causing #15248.

Although we could properly move the registration from
the moved-from object to the moved-to one, it is simpler
to just disallow moving a registered tracker, since it's
not needed anywhere. This way we just don't need to mess
with the trackers' registration.

The move-assignment operator has a similar problem,
therefore it is deleted in this series, and the function is
renamed to `transfer_backlog` that just doesn't deal with the
moved-from registration.  This is safe since it's only used internally
by the compaction manager.

Fixes #15248

Closes scylladb/scylladb#15445

* github.com:scylladb/scylladb:
  compaction_state: store backlog_track in std::optional
  compaction_backlog_tracker: do not allow moving registered trackers
2023-09-20 16:41:10 +03:00
Kefu Chai
6fc171b9cf main: use fallback parameter when converting a YAML node
as yaml-cpp returns an invalid node when the node to be indexed
does not exist all. but it allows us to provide a fallback value
which is returned when the node is not valid. so, let's just use
this helper for accessing a node which does not necessarily exist.

simpler this way

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#15488
2023-09-20 16:01:45 +03:00
Nadav Har'El
f239849563 Merge 'doc: add a note that counters do not support TTL' from Anna Stuchlik
This PR adds the information that counters do not support data expiration with TTL, plus the link to the TTL page.

Fixes https://github.com/scylladb/scylladb/issues/15479

Closes scylladb/scylladb#15489

* github.com:scylladb/scylladb:
  doc: improve TTL limitation info on Counters page
  doc: add a note that counters do not support TTL
2023-09-20 15:49:44 +03:00
Anna Stuchlik
5073609366 doc: improve TTL limitation info on Counters page
This commit improves the information about
counters not supporting TTL on the Counters
page.
2023-09-20 14:38:35 +02:00
Anna Stuchlik
715b1a80c7 doc: add a note that counters do not support TTL
This commit adds the information that counters
do not support data expiration wtih TTL, plus
the link to the TTL page.

Fixes https://github.com/scylladb/scylladb/issues/15479
2023-09-20 13:28:33 +02:00
Benny Halevy
72a5ac9ce7 gossiper: get_or_create_endpoint_state: create empty endpoint_state
Currently, the endpoint address is set as the new
endpoint_state RPC_ADDRESS.  This is wrong since
it should be assigned with the `broadcast_rpc_address`
rather than `broadcast_address`.
This was introduced in b82c77ed9c

Instead just create an empty endpoint_state.
The RPC_ADDRESS (as well as HOST_ID) application states
are set later.

Fixes scylladb/scylladb#15458

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

Closes scylladb/scylladb#15475
2023-09-20 13:20:44 +02:00
Avi Kivity
47a1dc8d01 Update seastar submodule
* seastar 576ee47d...bab1625c (13):
  > build: s/{dpdk_libs}/${dpdk_libs}/
  > build: build with dpdk v23.07
  > scripts: Fix escaping of regexes in addr2line
  > linux-aio: print more specific error when setup_aio fails
  > linux-aio: correct the error message raised when io_setup() fails
  > build: reenable -Warray-bound compiling option
  > build: error out if find_program() fails
  > build: enable systemtap only if it is available
  > build: check if libucontext is necessary for using ucontext functions
  > smp: reference correct variable when fetch_or()
  > build: use target_compile_definitions() for adding -D...
  > http/client: pass tls_options to tls::connect()
  > Merge 'build, process: avoid using stdout or stderr as C++ identifiers' from Kefu Chai

Frozen toolchain regenerated for new Seastar depdendencies.

configure.py adjusted for new Seastar arch names.

Closes scylladb/scylladb#15476
2023-09-20 10:43:40 +02:00
Tomasz Grabiec
3d4398d1b2 Merge 'Don't calculate hashes for schema versions in Raft mode' from Kamil Braun
When performing a schema change through group 0, extend the schema mutations with a version that's persisted and then used by the nodes in the cluster in place of the old schema digest, which becomes horribly slow as we perform more and more schema changes (#7620).

If the change is a table create or alter, also extend the mutations with a version for this table to be used for `schema::version()`s instead of having each node calculate a hash which is susceptible to bugs (#13957).

When performing a schema change in Raft RECOVERY mode we also extend schema mutations which forces nodes to revert to the old way of calculating schema versions when necessary.

We can only introduce these extensions if all of the cluster understands them, so protect this code by a new cluster/schema feature, `GROUP0_SCHEMA_VERSIONING`.

Fixes: #7620
Fixes: #13957

Closes scylladb/scylladb#15331

* github.com:scylladb/scylladb:
  test: add test for group 0 schema versioning
  test/pylib: log_browsing: fix type hint
  feature_service: enable `GROUP0_SCHEMA_VERSIONING` in Raft mode
  schema_tables: don't delete `version` cell from `scylla_tables` mutations from group 0
  migration_manager: add `committed_by_group0` flag to `system.scylla_tables` mutations
  schema_tables: use schema version from group 0 if present
  migration_manager: store `group0_schema_version` in `scylla_local` during schema changes
  migration_manager: migration_request handler: assume `canonical_mutation` support
  system_keyspace: make `get/set_scylla_local_param` public
  feature_service: add `GROUP0_SCHEMA_VERSIONING` feature
  schema_tables: refactor `scylla_tables(schema_features)`
  migration_manager: add `std::move` to avoid a copy
  schema_tables: remove default value for `reload` in `merge_schema`
  schema_tables: pass `reload` flag when calling `merge_schema` cross-shard
  system_keyspace: fix outdated comment
2023-09-20 10:43:40 +02:00
Botond Dénes
45dfce6632 Merge 'compaction: change behaviour of compaction task executors' from Aleksandra Martyniuk
Compaction tasks executors serve two different purposes - as compaction
manager related entity they execute compaction operation and as task
manager related entity they track compaction status.

When one role depends on the other, as it currently is for
compaction_task_impl::done() and compaction_task_executor::compaction_done(),
requirements of both roles need to be satisfied at the same time in each
corner case. Such complexity leads to bugs.

To prevent it, compaction_task_impl::done() of executors no longer depends
on compaction_task_executor::compaction_done().

Fixes: #14912.

Closes scylladb/scylladb#15140

* github.com:scylladb/scylladb:
  compaction: warn about compaction_done()
  compaction: do not run stopped compaction
  compaction: modify lowest compaction tasks' run method
  compaction: pass do_throw_if_stopping to compaction_task_executor
2023-09-19 15:15:14 +03:00
Botond Dénes
844a0e426f Merge 'Mark counters with skip when empty' from Amnon Heiman
This series mark multiple high cardinality counters with skip_when_empty flag.
After this patch the following counters will not be reported if they were never used:
```
scylla_transport_cql_errors_total
scylla_storage_proxy_coordinator_reads_local_node
scylla_storage_proxy_coordinator_completed_reads_local_node
scylla_transport_cql_errors_total
```
Also marked, the CAS related CQL operations.
Fixes #12751

Closes scylladb/scylladb#13558

* github.com:scylladb/scylladb:
  service/storage_proxy.cc: mark counters with skip_when_empty
  cql3/query_processor.cc: mark cas related metrics with skip_when_empty
  transport/server.cc: mark metric counter with skip_when_empty
2023-09-19 15:02:39 +03:00
Benny Halevy
7ca91d719c compaction_state: store backlog_track in std::optional
So that replacing it will destroy the previous tracker
and unregister it before assigning the new one and
then registering it.

This is safer than assiging it in place.

With that, the move assignment operator is not longer
used and can be deleted.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-09-19 13:59:54 +03:00
Benny Halevy
4ad4b632b8 compaction_backlog_tracker: do not allow moving registered trackers
Currently, the moved-object's manager pointer is moved into the
constructed object, but without fixing the registration to
point to the moved-to object, causing #15248.

Although we could properly move the registration from
the moved-from object to the moved-to one, it is simpler
to just disallow moving a registered tracker, since it's
not needed anywhere. This way we just don't need to mess
with the trackers' registration.

With that in mind, when move-assigning a compaction_backlog_tracker
the existing tracker can remain registered.

Fixes scylladb/scylladb#15248

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2023-09-19 13:24:36 +03:00
Benny Halevy
e784930dd7 storage_service: fix comment about when group0 is set
Since 8598cebb11
it is set earlier, before join_cluster.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-ID: <20230919063951.1424924-1-bhalevy@scylladb.com>
2023-09-19 13:20:58 +03:00
Kefu Chai
ba002de263 build: enable more warnings
these options for disabling warnings are not necessary anymore, for
one of the following reasons:

* the code which caused the warning were either fixed or removed
* the toolchain were updated, so the false alarms do not exist
  with the latest frozen toolchain.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#15450
2023-09-19 13:02:34 +03:00
Kefu Chai
484d02da14 cql3: expr: do not use multi-line comment
do not use muti-line comment. this silences the warning from GCC:
```
In file included from ./cql3/prepare_context.hh:19,
                 from ./cql3/statements/raw/parsed_statement.hh:14,
                 from build/debug/gen/cql3/CqlParser.hpp:62,
                 from build/debug/gen/cql3/CqlParser.cpp:44:
./cql3/expr/expression.hh:490:1: error: multi-line comment [-Werror=comment]
  490 | /// Custom formatter for an expression.  Supports multiple modes:\
      | ^
```

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#15471
2023-09-19 12:00:09 +03:00
Kefu Chai
4b53a70d76 build: cmake: add tests target
this target mirrors the target named `{mode}e-test` in the
`build.ninja` build script created by `configure.py`.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#15448
2023-09-19 11:20:02 +03:00
Kefu Chai
da7de887d6 build: cmake: bump the minimum required CMake version
because we should have a frozeon toolchain built with fedora38, and f38
provides cmake v3.27.4, we can assume the availability of cmake v3.27.4
when building scylla with the toolchain.

in this change, the minimum required CMake version is changed to
3.27.

this also allows us to simplify the implementation of
`add_whole_archive()`, and remove the buggy branch for supporting
CMake < 3.24, as we should have used `${name}` in place of `auth` there.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#15446
2023-09-19 10:57:57 +03:00
Botond Dénes
111cdce2e1 Merge 'db/hints: Modularize manager.hh' from Dawid Mędrek
This PR modularizes `manager.{hh, cc}` by dividing the files into separate smaller units. The changes improve overall readability of code and help reason about it. Each file has a specific purpose now.

This is the first step in refactoring the Hinted Handoff module.

Refs scylladb/scylla#15358

Closes scylladb/scylladb#15378

* github.com:scylladb/scylladb:
  db/hints: Remove unused aliases from manager.hh
  db/hints: Rename end_point_hints_manager
  db/hints: Rename sender to hint_sender
  db/hints: Move the rebalancing logic to hint_storage
  db/hints: Move the implementation of sender
  db/hints: Move the declaration of sender to hint_sender.hh
  db/hints: Move sender::replay_allowed() to the source file
  db/hints: Put end_point_hints_manager in internal namespace
  db/hints: Move the implementation of end_point_hints_manager
  db/hints: Move the declaration of end_point_hints_manager
  db/hints: Move definitions of functions using shard hint manager
  db/hints: Introduce hint_storage.hh
  db/hints: Extract the logger from manager.cc
  db/hints: Extract common types from manager.hh
2023-09-19 10:56:16 +03:00
Michael Huang
62a8a31be7 cdc: use chunked_vector for topology_description entries
Lists can grow very big. Let's use a chunked vector to prevent large contiguous
allocations.
Fixes: #15302.

Closes scylladb/scylladb#15428
2023-09-18 23:17:01 +03:00
Avi Kivity
ab6988c52f Merge "auth: do not grant permissions to creator without actually creating" from Wojciech Mitros
Currently, when creating the table, permissions may be mistakenly
granted to the user even if the table is already existing. This
can happen in two cases:

The query has a IF NOT EXISTS clause - as a result no exception
is thrown after encountering the existing table, and the permission
granting is not prevented.
The query is handled by a non-zero shard - as a result we accept
the query with a bounce_to_shard result_message, again without
preventing the granting of permissions.
These two cases are now avoided by checking the result_message
generated when handling the query - now we only grant permissions
when the query resulted in a schema_change message.

Additionally, a test is added that reproduces both of the mentioned
cases.

CVE-2023-33972

Fixes #15467.

* 'no-grant-on-no-create' of github.com:scylladb/scylladb-ghsa-ww5v-p45p-3vhq:
  auth: do not grant permissions to creator without actually creating
  transport: add is_schema_change() method to result_message
2023-09-18 21:47:28 +03:00
Avi Kivity
16a72a81fc Merge 'build: cmake: add "dist-server-debuginfo" target' from Kefu Chai
this target mirrors the "dist-server-debuginfo-{mode}" target in the `build.ninja` created by `configure.py`.

Closes scylladb/scylladb#15441

* github.com:scylladb/scylladb:
  build: cmake: add "dist-server-debuginfo" target
  build: cmake: remove debian dep from relocatable pkg
2023-09-18 20:54:21 +03:00
Avi Kivity
146e49d0dd Merge 'Rewrap keyspace population loop' from Pavel Emelyanov
Populating of non-system keyspaces is now done by listing datadirs and assuming that each subdir found is a keyspace. For S3-backed keyspaces this is also true, but it's a bug (#13020). The loop needs to walk the list of known keyspaces instead, and try to find the keyspace storage later, based on the storage option.

Closes scylladb/scylladb#15436

* github.com:scylladb/scylladb:
  distributed_loader: Indentation fix after previous patch
  distributed_loader: Generalize datadir parallelizm loop
  distributed_loader: Provide keyspace ref to populate_keyspace
  distributed_loader: Walk list of keyspaces instead of directories
2023-09-18 20:51:01 +03:00
Kefu Chai
cf5400bc75 cql.g: always initialize returned values
always initialize returned values. the branches which
return these unitiailized returned values handles the
unmatched cases, so this change should not have any
impact on the behavior.

ANTLR3's C++ code generator does not assign any value
to the value, if it runs into failure or encounter
exceptions. for instance, following rule assigns the
value of `isStatic` to `isStaticColumn` only if
nothing goes wrong.

```
cfisStatic returns [bool isStaticColumn]
    @init{
        bool isStatic = false;
    }
    : (K_STATIC { isStatic=true; })?
    {
        $isStaticColumn = isStatic;
    }
    ;
```

as shown in the generated C++ code:
```c++
                switch (alt118)
                {
            	case 1:
            	    // build/debug/gen/cql3/Cql.g:989:8: K_STATIC
            	    {
            	         this->matchToken(K_STATIC, &FOLLOW_K_STATIC_in_cfisStatic5870);
            	        if  (this->hasException())
            	        {
            	            goto rulecfisStaticEx;
            	        }
            	        if (this->hasFailed())
            	        {
            	            return isStaticColumn;
            	        }

            	        if ( this->get_backtracking()==0 )
            	        {
            	             isStaticColumn=isStatic;

            	        }

            	    }
            	    break;

                }
```

when `this->hasException()` or `this->hasFailed()`,
`isStaticColumn` is returned right away without being
initialized, because we don't assign any initial value
to it, neither do we customize the exception handling
for this rule.

and, the parser bails out when its smells something bad
after it tries to match the specified rule. also, the
parser is a stateful tokenizer, its failure state is
carried by the parser itself. also, the matchToken()
*could* fail when trying to find the matched token,
this is the runtime behavior of parser, that's why the
compiler cannot be certain that the error path won't
be taken.

anyway, let's always initialize the values without
default constructor. the return values whose type
is of scoped enum are initialized with zero
initialization, because their types don't provide an
"invalid" value.

this change should silence warnings like:

```
clang++ -MD -MT build/debug/gen/cql3/CqlParser.o -MF build/debug/gen/cql3/CqlParser.o.d -I/home/kefu/dev/scylladb/seastar/include -I/home/kefu/dev/scylladb/build/debug/seastar/gen/include -U_FORTIFY_SOURCE -DSEASTAR_SSTRING -Werror=unused-result -fstack-clash-protection -fsanitize=address -fsanitize=undefined -fno-sanitize=vptr -DSEASTAR_API_LEVEL=7 -DSEASTAR_BUILD_SHARED_LIBS -DSEASTAR_SCHEDULING_GROUPS_COUNT=16 -DSEASTAR_DEBUG -DSEASTAR_DEFAULT_ALLOCATOR -DSEASTAR_SHUFFLE_TASK_QUEUE -DSEASTAR_DEBUG_SHARED_PTR -DSEASTAR_LOGGER_TYPE_STDOUT -DSEASTAR_TYPE_ERASE_MORE -DBOOST_NO_CXX98_FUNCTION_BASE -DFMT_SHARED -I/usr/include/p11-kit-1   -ffile-prefix-map=/home/kefu/dev/scylladb=. -march=westmere -DDEBUG -DSANITIZE -DDEBUG_LSA_SANITIZER -DSCYLLA_ENABLE_ERROR_INJECTION -Og -DSCYLLA_BUILD_MODE=debug -g -gz -iquote. -iquote build/debug/gen --std=gnu++20  -ffile-prefix-map=/home/kefu/dev/scylladb=. -march=westmere  -DBOOST_TEST_DYN_LINK   -DNOMINMAX -DNOMINMAX -fvisibility=hidden  -Wall -Werror -Wextra -Wno-deprecated-copy -Wno-mismatched-tags -Wno-missing-field-initializers -Wno-c++11-narrowing -Wno-ignored-qualifiers -Wno-overloaded-virtual -Wno-unsupported-friend -Wno-unused-parameter -Wno-implicit-int-float-conversion -Wno-error=deprecated-declarations -DXXH_PRIVATE_API -DSEASTAR_TESTING_MAIN -DFMT_DEPRECATED_OSTREAM -Wno-parentheses-equality -O1 -fno-sanitize-address-use-after-scope -c -o build/debug/gen/cql3/CqlParser.o build/debug/gen/cql3/CqlParser.cpp
build/debug/gen/cql3/CqlParser.cpp:26645:28: error: variable 'perm' is uninitialized when used here [-Werror,-Wuninitialized]
                    return perm;
                           ^~~~
build/debug/gen/cql3/CqlParser.cpp:26616:5: note: variable 'perm' is declared here
    auth::permission perm;
    ^
build/debug/gen/cql3/CqlParser.cpp:52577:28: error: variable 'op' is uninitialized when used here [-Werror,-Wuninitialized]
                    return op;
                           ^~
build/debug/gen/cql3/CqlParser.cpp:52518:5: note: variable 'op' is declared here
    oper_t op;
    ^
```

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#15451
2023-09-18 16:45:50 +03:00
Kefu Chai
ece45c9f70 build: cmake: use find_program(.. REQUIRED) when appropriate
instead of checking the availability of a required program, let's
use the `REQUIRED` argument introduced by CMake 3.18, simpler this
way.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#15447
2023-09-18 16:35:46 +03:00
Kefu Chai
9de00c1c5a build: cmake: add node_ops
node_ops source files was extracted into /node_ops directory in
d0d0ad7aa4, so let's update the building system accordingly.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#15442
2023-09-18 16:27:02 +03:00
Kefu Chai
4d285590f0 utils/config_file: document config_file::value_status
add doxygen style comment to document `value_status` members.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#15277
2023-09-18 16:20:06 +03:00
Benny Halevy
8a56050507 main: handle abort_requested_exception on startup
Handle abort_requested_exception exactly like
sleep_aborted, as an expected error when startup
is aborted mid-way.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

Closes scylladb/scylladb#15443
2023-09-18 15:05:52 +03:00
Botond Dénes
f7557a4891 Merge 'updating presto integration page documentation' from Guy Shtub
null

Closes scylladb/scylladb#15342

* github.com:scylladb/scylladb:
  Update integration-presto.rst
  Update integration-presto.rst
  Update docs/using-scylla/integrations/integration-presto.rst
  updating presto integration page
2023-09-18 14:41:16 +03:00
Botond Dénes
edb50c27ec Merge 'Use sstable_state in sstables populator' from Pavel Emelyanov
Some time ago populating of tables from sstables was reworked to use sstable states instead of full paths (#12707). Since then few places in the populator was left that still operate on the state-based subdirectory name. This PR collects most of those dangling ends

refs: #13020

Closes scylladb/scylladb#15421

* github.com:scylladb/scylladb:
  distributed_loader: Print sstable state explicitly
  distributed_loader: Move check for the missing dir upper
  distributed_loader: Use state as _sstable_directories key
2023-09-18 14:38:49 +03:00
Kefu Chai
054beb6377 tests: tablets: do not compare signed integer with unsigned integer
when compiling the tests with -Wsign-compare, the compiler complains like:
```
/home/kefu/.local/bin/clang++ -DBOOST_ALL_DYN_LINK -DBOOST_NO_CXX98_FUNCTION_BASE -DDEBUG -DDEBUG_LSA_SANITIZER -DFMT_DEPRECATED_OSTREAM -DFMT_SHARED -DSANITIZE -DSCYLLA_BUILD_MODE=debug -DSCYLLA_ENABLE_ERROR_INJECTION -DSEASTAR_API_LEVEL=7 -DSEASTAR_BROKEN_SOURCE_LOCATION -DSEASTAR_DEBUG -DSEASTAR_DEBUG_SHARED_PTR -DSEASTAR_DEFAULT_ALLOCATOR -DSEASTAR_LOGGER_TYPE_STDOUT -DSEASTAR_SCHEDULING_GROUPS_COUNT=16 -DSEASTAR_SHUFFLE_TASK_QUEUE -DSEASTAR_TESTING_MAIN -DSEASTAR_TYPE_ERASE_MORE -DXXH_PRIVATE_API -I/home/kefu/dev/scylladb -I/home/kefu/dev/scylladb/build/cmake/gen -I/home/kefu/dev/scylladb/seastar/include -I/home/kefu/dev/scylladb/build/cmake/seastar/gen/include -isystem /home/kefu/dev/scylladb/build/cmake/rust -Wall -Werror -Wextra -Wno-error=deprecated-declarations -Wimplicit-fallthrough -Wno-c++11-narrowing -Wno-mismatched-tags -Wno-overloaded-virtual -Wno-unsupported-friend -Wno-unused-parameter -Wno-missing-field-initializers -Wno-deprecated-copy -Wno-ignored-qualifiers -march=westmere  -Og -g -gz -std=gnu++20 -fvisibility=hidden -U_FORTIFY_SOURCE -DSEASTAR_SSTRING -Wno-error=unused-result "-Wno-error=#warnings" -fstack-clash-protection -fsanitize=address -fsanitize=undefined -fno-sanitize=vptr -MD -MT test/boost/CMakeFiles/tablets_test.dir/tablets_test.cc.o -MF test/boost/CMakeFiles/tablets_test.dir/tablets_test.cc.o.d -o test/boost/CMakeFiles/tablets_test.dir/tablets_test.cc.o -c /home/kefu/dev/scylladb/test/boost/tablets_test.cc
/home/kefu/dev/scylladb/test/boost/tablets_test.cc:1335:53: error: comparison of integers of different signs: 'int' and 'size_t' (aka 'unsigned long') [-Werror,-Wsign-compare]
            for (int log2_tablets = 0; log2_tablets < tablet_count_bits; ++log2_tablets) {
                                       ~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~
```

in this case, it should be safe to use an signed int as the loop
variable to be compared with `tablet_count_bits`, but let's just
appease the compiler so we can enable the warning option project-wide
to prevent any potential issues caused by signed-unsigned comparision.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#15449
2023-09-18 13:17:16 +02:00
Kamil Braun
bc6f7d1b20 Merge 'raft topology: add garbage collection for internal CDC generations table' from Patryk Jędrzejczak
We add garbage collection for the `CDC_GENERATIONS_V3` table to prevent
it from endlessly growing. This mechanism is especially needed because
we send the entire contents of `CDC_GENERATIONS_V3` as a part of the
group 0 snapshot.

The solution is to keep a clean-up candidate, which is one of the
already published CDC generations. The CDC generation publisher
introduced in #15281 continually uses this candidate to remove all
generations with timestamps not exceeding the candidate's and sets a new
candidate when needed.

We also add `test_cdc_generation_clearing.py` that verifies this new
mechanism.

Fixes #15323

Closes scylladb/scylladb#15413

* github.com:scylladb/scylladb:
  test: add test_cdc_generation_clearing
  raft topology: remove obsolete CDC generations
  raft topology: set CDC generation clean-up candidate
  topology_coordinator: refactor publish_oldest_cdc_generation
  system_keyspace: introduce decode_cdc_generation_id
  system_keyspace: add cleanup_candidate to CDC_GENERATIONS_V3
2023-09-18 11:30:10 +02:00
Pavel Emelyanov
30959fc9b1 lsa, test: Extend memory footprint test with per-type total sizes
When memory footprint test is over it prints total size taken by row
cache, memtable and sstables as well as individual objects' sizes. It's
also nice to know the details on the row-cache's individual objects.
This patch extends the printing with total size of allocated object
types according to migrator_fn types.

Sample output:

    mutation footprint:
     - in cache:     11040928
     - in memtable:  9142424
     - in sstable:
       mc:   2160000
       md:   2160000
       me:   2160000
     - frozen:       540
     - canonical:    827
     - query result: 342

     sizeof(cache_entry) = 64
     sizeof(memtable_entry) = 64
     sizeof(bptree::node) = 288
     sizeof(bptree::data) = 72
     -- sizeof(decorated_key) = 32
     -- sizeof(mutation_partition) = 96
     -- -- sizeof(_static_row) = 8
     -- -- sizeof(_rows) = 24
     -- -- sizeof(_row_tombstones) = 40

     sizeof(rows_entry) = 144
     sizeof(evictable) = 24
     sizeof(deletable_row) = 72
     sizeof(row) = 16
     radix_tree::inner_node::node_sizes =  48 80 144 272 528 1040
     radix_tree::leaf_node::node_sizes =  120 216 416 816 3104
     sizeof(atomic_cell_or_collection) = 16
     btree::linear_node_size(1) = 24
     btree::inner_node_size = 216
     btree::leaf_node_size = 120
    LSA stats:
      N18compact_radix_tree4treeI13cell_and_hashjE9leaf_nodeE: 360
      N5bplus4dataIl15intrusive_arrayI11cache_entryEN3dht25raw_token_less_comparatorELm16ELNS_10key_searchE0ELNS_10with_debugE0EEE: 5040
      N5bplus4nodeIl15intrusive_arrayI11cache_entryEN3dht25raw_token_less_comparatorELm16ELNS_10key_searchE0ELNS_10with_debugE0EEE: 19296
      17partition_version: 952416
      N11intrusive_b4nodeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EEE: 317472
      10rows_entry: 1429056
      12blob_storage: 254

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>

Closes scylladb/scylladb#15434
2023-09-18 11:23:18 +02:00
Guy Shtub
5d833b2ee7 Update integration-presto.rst 2023-09-18 11:29:38 +03:00
Botond Dénes
bb7121a1fb Merge 'tools/scylla-nodetools: do not create unowned bpo::value ' from Kefu Chai
in other words, do not create bpo::value unless transfer it to an
option_description.

`boost::program_options::value()` create a new typed_value<T> object,
without holding it with a shared_ptr. boost::program_options expects
developer to construct a `bpo::option_description` right away from it.
and `boost::program_options::option_description` takes the ownership
of the `type_value<T>*` raw pointer, and manages its life cycle with
a shared_ptr. but before passing it to a `bpo::option_description`,
the pointer created by `boost::program_options::value()` is a still
a raw pointer.

before this change, we initialize `operations_with_func` as global
variables using `boost::program_options::value()`. but unfortunately,
we don't always initialize a `bpo::option_description` from it --
we only do this on demand when the corresponding subcommand is
called.

so, if the corresponding subcommand is not called, the created
`typed_value<T>` objects are leaked. hence LeakSanitizer warns us.

after this change, we create the option map as a static
local variable in a function so it is created on demand as well.
as an alternative, we could initialize the options map as local
variable where it used. but to be more consistent with how
`global_option` is specified. and to colocate them in a single
place, let's keep the existing code layout.

this change is quite similar to 374bed8c3d

Fixes https://github.com/scylladb/scylladb/issues/15429

Closes scylladb/scylladb#15430

* github.com:scylladb/scylladb:
  tools/scylla-nodetools: reindent
  tools/scylla-nodetools: do not create unowned bpo::value
2023-09-18 11:09:46 +03:00
Kefu Chai
a51b14d4c4 sstables/metadata_collector: drop unused functions
column_stats::update_local_deletion_time() is not used anywhere,
what is being used is
`column_stats::update_local_deletion_time_and_tombstone_histogram(time_point)`.
while `update_local_deletion_time_and_tombstone_histogram(int32_t)`
is only used internally by a single caller.

neither is `column_stats::update(const deletion_time&)` used.

so let's drop them. and merge
`update_local_deletion_time_and_tombstone_histogram(int32_t)`
into its caller.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes scylladb/scylladb#15189
2023-09-18 10:18:56 +03:00