When starting `Build` job we have a situation when `x86` and `arm`
starting in different dates causing the all process to fail
As suggested by @avikivity , adding a date-stamp parameter and will pass
it through downstream jobs to get one release for each job
Ref: scylladb/scylla-pkg#3008
Closes#11234
We would like to define more distinct types
that are currently defined as aliases to utils::UUID to identify
resources in the system, like table id and schema
version id.
As with counter_id, the motivation is to restrict
the usage of the distinct types so they can be used
(assigned, compared, etc.) only with objects of
the same type. Using with a generic UUID will
then require explicit conversion, that we want to
expose.
This series starts with cleaning up the idl header definition
by adding support for `import` and `include` statements in the idl-compiler.
These allow the idl header to become self-sufficient
and then remove manually-added includes from source files.
The latter usually need only the top level idl header
and it, in turn, should include other headers if it depends on them.
Then, a UUID_class template was defined as a shared boiler plate
for the various uuid-class. First, we convert counter_id to use it,
rather than mimicking utils::UUID on its own.
On top of utils::UUID_class<T>, we define table_id,
table_schema_version, and query_id.
Following up on this series, we should define more commonly used
types like: host_id, streaming_plan_id, paxos_ballot_id.
Fixes#11207Closes#11220
* github.com:scylladb/scylladb:
query-request, everywhere: define and use query_id as a strong type
schema, everywhere: define and use table_schema_version as a strong type
schema, everywhere: define and use table_id as a strong type
schema: include schema_fwd.hh in schema.hh
system_keyspace: get_truncation_record: delete unused lambda capture
utils: uuid: define appending_hash<utils::tagged_uuid<Tag>>
utils: tagged_uuid: rename to_uuid() to uuid()
counters: counter_id: use base class create_random_id
counters: base counter_id on utils::tagged_uuid
utils: tagged_uuid: mark functions noexcept
utils: tagged_uuid: bool: reuse uuid::bool operator
raft: migrate tagged_id definition to utils::tagged_uuid
utils: uuid: mark functions noexcept
counters: counter_id delete requirement for triviality
utils: bit_cast: require TriviallyCopyable To
repair: delete unused include of utils/bit_cast.hh
bit_cast: use std::bit_cast
idl: make idl headers self-sufficient
db: hints: sync_point: do not include idl definition file
db/per_partition_rate_limit: tidy up headers self-sufficiency
idl-compiler: include serialization impl and visitors in generated dist.impl.hh files
idl-compiler: add include statements
idl_test: add a struct depending on UUID
Define table_schema_version as a distinct tagged_uuid class,
So it can be differentiated from other uuid-class types,
in particular table_id.
Added reversed(table_schema_version) for convenience
and uniformity since the same logic is currently open coded
in several places.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Define table_id as a distinct utils::tagged_uuid modeled after raft
tagged_id, so it can be differentiated from other uuid-class types,
in particular from table_schema_version.
Fixes#11207
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than defining generate_random,
and use respectively in unit tests.
(It was inherited from raft::internal::tagged_id.)
This allows us to shorten counter_id's definition
to just using utils::tagged_uuid<struct counter_id_tag>.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Use the common base class for uuid-based types.
tagged_uuid::to_uuid defined here for backward
compatibility, but it will be renamed in the next patch
to uuid().
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
So it can be used for other types in the system outside
of raft, like counter_id, table_id, table_schema_version,
and more.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
This stemmed from utils/bit_cast overly strict requirement.
Now that it was relaxed, these is no need for this static assert
as counter_id is trivially copyable, and that is checked
by bit_cast {read,write}_unaligned
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Now that scylla requries c++20 there's no
need to define our own implementation in utils/bit_cast.hh
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Add include statements to satisfy dependencies.
Delete, now unneeded, include directives from the upper level
source files.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
idl definition files are not intended for direct
inclusion in .cc files.
Data types it represents are supposed to be defined
in regular C++ header, so define them in db/hints/scyn_point.hh
and include it rather then idl/hinted_handoff.idl.hh.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
They are generally required by the serialization implementation.
This will simplify using them without having to hand pick
what header to include in the .cc file that includes them.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
For generating #include directives in the generated files,
so we don't have to hand-craft include the dependencies
in the right order.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
This series is aimed at fixing #11132.
To get there, the series untangles the functions that currently depend on the the cross-shard coordination in table::snapshot,
namely database::truncate and consequently database::drop_column_family.
database::get_table_on_all_shards is added here as a helper to get a foreign shared ptr of the the table shard from all shards,
and it is later used by multiple functions to truncate and then take a snapshot of the sharded table.
database::truncate_table_on_all_shards is defined to orchestrate the truncate process end-to-end, flushing or clearing all table shards before taking a snapshot if needed, using the newly defined table::snapshot_on_all_shards, and by that leaving only the discard_sstables job to the per-shard database::truncate function.
The latter, snapshot_on_all_shards, orchestrates the snapshot process on all shards - getting rid of the per-shard table::snapshot function (after refactoring take_snapshot and finalize_snapshot out of it), and the associated dreaded data structures: snapshot_manager and pending_snapshots.
Fixes#11132.
Closes#11133
* github.com:scylladb/scylladb:
table: reindent write_schema_as_cql
table: coroutinize write_schema_as_cql
table: seal_snapshot: maybe_yield when iterating over the table names
table: reindent seal_snapshot
table: coroutinize seal_snapshot
table: delete unused snapshot_manager and pending_snapshots
table: delete unused snapshot function
table: snapshot_on_all_shards: orchestrate snapshot process
table: snapshot: move pending_snapshots.erase from seal_snapshot
table: finalize_snapshot: take the file sets as a param
table: make seal_snapshot a static member
table: finalize_snapshot: reindent
table: refactor finalize_snapshot out of snapshot
table: snapshot: keep per-shard file sets in snapshot_manager
table: take_snapshot: return foreign unique ptr
table: take_snapshot: maybe yield in per-sstable loop
table: take_snapshot: simplify tables construction code
table: take_snapshot: reindent
table: take_snapshot: simplify error handling
table: refactor take_snapshot out of snapshot
utils: get rid of joinpoint
database: get rid of timestamp_func
database: truncate: snapshot table in all-shards layer
database: truncate: flush table and views in all-shards layer
database: truncate: stop and disable compaction in all-shards layer
database: truncate: move call to set_low_replay_position_mark to all-shards layer
database: truncate: enter per-shard table async_gate in all-shards layer
database: truncate: move check for schema_tables keyspace to all-shards layer.
database: snapshot_table_on_all_shards: reindent
table: add snapshot_on_all_shards
database: add snapshot_table_on_all_shards
database: rename {flush,snapshot}_on_all and make static
database: drop_table_on_all_shards: truncate and stop table in upper layer
database: drop_table_on_all_shards: get all table shards before drop_column_family on each
database: drop_column_family: define table& cf
database: drop_column_family: reuse uuid for evict_all_for_table
database: drop_column_family: move log message up a layer
database: truncate: get rid of the unused ks param
database: add truncate_table_on_all_shards
database: drop_table_on_all_shards: do not accept a truncated_at timestamp_func
database: truncate: get optional snapshot_name from caller
database: truncate: fix assert about replay_position low_mark
database_test: apply_mutation on the correct db shard
Add maybe_yield calls in tight loop, potentially
over thousands of sstable names to prevent reactor stalls.
Although the per-sstable cost is very small, we've experienced
stalls realted to printing in O(#sstables) in compaction.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Handle exceptions, making sure the output
stream is properly closed in all cases,
and an intermediate error, if any, is returned as the
final future.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Now that snapshot orchestration in snapshot_on_all_shards
doesn't use snapshot_manager, get rid of the data structure.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Now that snapshot orchestration is done solely
in snapshot_on_all_shards, the per-shard
snapshot function can be deleted.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Call take_snapshot on each shard and collect the
returns snapshot_file_set.
When all are done, move the vector<snapshot_file_set>
to finalize_snapshot.
All that without resorting to using the snapshot_manager
nor calling table::snapshot.
Both will deleted in the following patches.
Fixes#11132
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Now that seal_snapshot doesn't need to lookup
the snapshot_manager in pending_snapshots to
get to the file_sets, erasing the snapshot_manager
object can be done in table::snapshot which
also inserted it there.
This will make it easier to get rid of it in a later patch.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
and pass it to seal_snapshot, so that the latter won't
need to lookup and access the snapshot_manager object.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Write schema.cql and the files manifest in finalize_snapshot.
Currently call it from table::snapshot, but it will
be called in a later patch by snapshot_on_all_shards.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
To simplify processing of the per-shard file names
for generating the manifest.
We only need to print them to the manifest at the
end of the process, so there's no point in copying
them around in the process, just move the
foreign unique unordered_set.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Currently copying the sstable file names are created
and destroyed on each shard and are copied by the
"coordinator" shards using submit_to, while the
coroutine holds the source on its stack frame.
To prepare for the next patches that refactor this
code so that the coordinator shard will submit_to
each shard to perform `take_snapshot` and return
the set of sstrings in the future result, we need
to wrap the result in a foreign_ptr so it gets
freed on the shard that created it.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
There could be thousands of sstables so we better
cosider yielding in the tight loop that copies
the sstable names into the unordered_set we return.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Don't catch exception but rather just return
them in the return future, as the exception
is handled by the caller.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Do the actual snapshot-taking code in a per-shard
take_snapshot function, to be called from
snapshot_on_all_shards in a following patch.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
