In its operations the fs storage carefully generates full filename from
all sstable parameters -- version, format, generation, keyspace and
table names and component type or name. However, in all of the cases
format, version and keyspace:table names are inherited from the sstable
being operated on. This calls for a filename generation helper that
wraps most of the arguments thus making the lines shorter.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Closesscylladb/scylladb#23384
Fixes#23225Fixes#23185
Moved wrapping component files/sinks to storage provider. Also ensures
to wrap data_sinks as well as actual files. This ensures that we actually
write encryption if active.
Similarly to toc_, index_ and data filenames, make the generic component
name getter return back not string, but a wrapper object. Most of
callers are log messages and exception generations. Other than that
there are tests, filesystem storage driver and few more places in
generic code who "know" that they work with real files, so make them use
explicit fmt::to_string().
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The class in question is a wrapper around output_stream that writes,
flushes and closes the stream in async context. For logging it also
keeps the component filename on board, and now it's good time to patch
it and keep the component_filename instead.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Most of the callers use the returned value as log message parameter,
some construct malformed_sstable_exception that was prepared by previous
patch.
The remaining callers explicitly use fmt::to_string(), these are
- pending deletion log creation
- filesystem storage code
- tests
- stream-blob code that re-loads sstable
All but the last one are OK to use string toc name, the last one is not
very correct in its usage of toc_filename string, but it needs more care
to be fixed properly.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Sometimes a component object name looks like
s3://bucket/prefix/component. For that the path formatting code formats
bucket name with the result of sstable->filename() invocation. This
patch changes it to format bucket name, prefix itself and
sstable->component_filename().
The change is idempotent, as sstable::filename() just concatenates prefix
with sstable::component_filename(). This change will help to remove the
former method from sstable soon.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
When filesystem storage throws an exception about failure to create
components hardlinks, it includes three paths into it -- source file
name, destination file name and the directory name. The directory name
is excessive, source file name already has it. Also, this change will
make it possible to remove one of malformed_sstable_exception
constructors soon.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
... to where it belongs -- to the filesystem storage driver itself.
Continuation of the previous patch.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The method in question walks the list of sstables and accumulates
sstables' prefixes into a set on pending_delete_result object. The set
in question is not used at all in this method and is in fact alien to it
-- the p.d._result object is used by the filesystem storage driver as
atomic deletion prepare/commit transparent context.
Said that, move the whole pending_delete_result to where it belongs and
relax the create_pending_deletion_log() to only return the log
directory path string.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
these unused includes were identifier by clang-include-cleaner. after
auditing these source files, all of the reports have been confirmed.
please note, because quite a few source files relied on
`utils/to_string.hh` to pull in the specialization of
`fmt::formatter<std::optional<T>>`, after removing
`#include <fmt/std.h>` from `utils/to_string.hh`, we have to
include `fmt/std.h` directly.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Fixes#20717
Enables abortable interface and propagates abort_source to all s3 objects used for reading the restore data.
Note: because restore is done on each shard, we have to maintain a per-shard abort source proxy for each, and do a background per-shard abort on abort call. This is synced at the end of "run()".
Abort source is added as an optional parameter to s3 storage and the s3 path in distributed loader.
There is no attempt to "clean up" an aborted restore. As we read on a mutation level from remote sstables, we should not cause incomplete sstables as such, even though we might end up of course with partial data restored.
Closesscylladb/scylladb#21567
* github.com:scylladb/scylladb:
test_backup: Add restore abort test case
sstables_loader: Make restore task abortable
distributed_loader: Add optional abort_source to get_sstables_from_object_store
s3_storage: Add optional abort_source to params/object
s3::client: Make "readable_file" abortable
This helper is small enough and it's easier to understand how table
directory name is formatted without it.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Now this helper is not needed in replica code, as all manipulations of
tables' sstables now sit in the sstables/storage.cc.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
As mentioned in the previous patch, there are several places that need
to scan all datafile directories for a given table. This list is
currently stored on table.config.all_datadirs, this patch stops using
one and instead generates it from db::config::data_file_directories and
table's storage options.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The s3_storage backend driver has a method that generates object path
within the bucket. Depending on options alternative it picks one of two
formats:
- for string prefix, it uses it implicitly via sstable::filename() call
that calls storage->prefix() which, in turn, returns prefix value
- for registry-backed sstables, the /bucket/generation/component path is
generated
This patch bruses this place up. Similarly to previous patch, this
change also makes the selection based on the location alternative, not
on the sstable state. As well it's idempotent change, as S3 sstables
with 'upload' state only appear when restoring from object store, and in
this case the string location is in use.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Today, the system.sstables schema uses string as partition key. Callers,
in turn, use table's datadir value to reference entries in it. That's
wrong, S3-backed sstables don't have any local paths to work with. The
table's ID is better in this role.
This patch only changes the field type to be table_id and fixes the
callers to provide one. In particular, see init_table_storage() change
-- instead of generating a datadir string, it sets table.id() as the
options' location. Other fixed places are tests. Internally, this id
value is propagated via s3_storage::owner() method, that's fixed as
well.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Previous patch put variant<string, table_id> as location of S3 options.
This patch makes the S3 sstables backend driver keep variant as sstable
location. As with the previous patch, driver only keeps variant, but
continues using its string alternative internally. This will be changed
later on.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Describing S3 storage for an sstables nowadays has two options -- via
sstables registry entry and by using the direct prefix string. The
former is used when putting a keyspace on S3. In this case each sstable
has the corresponding entry in the system.sstables table. The latter is
used by "restore from object storage" code. In that case, sstables don't
have entries in the registry, but are accessed by a specific S3 object
path.
This patch reflects this difference by making s3_options::location be
variant of string prefix and table_id owner. The owner needs more
explanation, here it is.
Today, the system.sstables schema defines partition key to be "string
location" and clustering key to be "UUID generation". The partition key
is table's datadir string, but it's wrong to use it this way. Next
patches will change the partition key to be table's ID (there's table_id
type for it), and before doing it storage options must be prepared to
carry it onboard. This patch does it, but the table_id alternative of
the location is still unused, the rest of the code keeps using the
string location to reference a row in the registry table. Next patches
will eventually make use of the table_id value.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
This driver uses sstring _location as part of the lookup key in the
sstables registry. Next patches will need to change that and put more
checks on the registry access, so introduce a helper method beforehand.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The datadir keeps path to directory where local sstables can be. The very same information is now kept in table's storage options (#20542). This set fixes the remaining places that still use table::config::datadir and table::dir() and removes the datadir field.
Closesscylladb/scylladb#20675
* github.com:scylladb/scylladb:
treewide: Remove table::config::datadir
distributed_loader: Print storage options, not datadir
data_dictionary: Add formatter for storage_options
test: Construct table_for_tests with table storage options
test: Generalize pair of make_table_for_tests helpers
tests: Add helper to get snapshot directory from storage options
table: snapshot_exists: Get directory from storage options
table: snapshot_on_all_shards: Get directory from storage options
This reverts commit 44bd183187 and
moves the base directory back on filesystem_storage. The mentioned
commit says
> so we can use the base (table) directory for
> e.g. pending_delete logs, in the next patch.
but "next patch" doesn't use it outside of the filesystem-storage
anyway.
This field doesn't make sense for S3 backend. Its "location" is not
location, but a key in the system.sstables, which should rather be
schema ID, not /var/lib/.../keyspace/table-uuid string.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Closesscylladb/scylladb#20642
There are several things that are changed here
- The target directory for snapshot is evaluated using table directory
taken from its storage options, not from config
- If the storage options are not "local", the snapshot_on_all_shards is
failed early, it's impossible to snapshot sstables anyway
- If the storage is not configured for the obtained local options,
snapshotting is skilled, because it's a virtual table that's probably
not supposed to have snapshots
- The late failure to snapshot non-local sstables is converted into
internal error, as this functionality cannot be executed as per
previous change
- The target path is created using fs::path operator/ overload, not by
concatenating strings (it's minor change)
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
New sstables for a table are created by the table::make_sstable() method. The method then calls sstables_manager::make_sstable() and passes there a path to component files which, in turn, sits on table::config. Since some time ago having an on-disk path for an sstable had become optional, as sstables could be put on S3 storage without local paths involved. In that case the aforementioned "path" is ~~ab~~used as a key in the system.sstables registry, that references a record with information used to retrieve URLs of sstables' objects.
This PR removes the "path" argument from sstables_manager::make_sstable() and its sstable_sdirectory peer. The details of sstables' location are moved onto storage_options and depend on storage type. For now in both storage types this location is still the good-old $datadir/$keyspace/$table-$uuid string. S3 storage needs to be patched more to use more elegant "location" value.
Eventually the `table::config::{datadir|all_datadirs}` will be removed, this PR is the step towards it.
closes: #12707Closesscylladb/scylladb#20542
* github.com:scylladb/scylladb:
table: Use storage options to clean the storage
sstables/storage: Re-use ocally generated vector of paths
sstables/storage: Visit options once to initialize storage
sstables_manager: Return table storage options when initalizing storage
sstables/storage: Fix indentation after previous patch
table: Move datadirs initialization parallelism to storage level
sstables/storage: Split the visitor's overloaded functor
restore: Don't use table_dir to construct sstable_directory
sstable_directory: Remove table_dir field
sstable_directory: Use options details in lister
sstables_manager: Remove table_dir from make_sstable()
sstables: Remove table_dir from sstable constructor
sstables/storage: Remove sstring dir from make_storage()
sstables/storage: Use options to construct
tests: Properly initialize storage options with "dir"
distributed_loader: Create S3 options with prefix for restore
storage_options: Add special-purpose local options maker
storage_options: Keep local path / s3 prefix onboard
table: Get another options when initializing storage
Like it was done for table::init_storage(), patch the
table::destroy_storage() not to mess with datadir path and rely on
storage options only.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
A cleanup after prefious patch -- in order to create storage options for
table the local initialization code can re-use the vector of paths that
it hag generated in the same call to create table directory layout.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The init_table_storage() method now does it twice -- one time to
initialize the storage, another one to create new options for table.
Both can be merged, thus making table storage options initialization
better encapsulated for local/s3 cases.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Now the table::init_storage() calls sstables manager two times -- first,
to get storage options, second, to initialize the storage with obtained
options. Merge two calls into one.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The table::init_table_storage() calls sstables_manager's storage
initialization for each of the datadirs found on config. That's not
great, it's sstables manager (and its storage) that know if table needs
to mess with datadirs or not. This patch moves the loop to storage.cc.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The main goal is to have init_table_storage() overload for local options
as standalone function. This makes next patching simpler.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
All callers of make_sstable are now patched to provide correct storage
options with path/prefix set. The make_storage() helper can switch to
using it. Respectively, it's good to make sure that the storage is
created with table options that have path/prefix.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Now when tables keep their own copy of storage options, it's possible
for each table to add table-specific information on it. Namely -- path
for local storage and prefix for S3 one (in fact, it's not a "prefix",
but a key in sstables registry, but fixing it is beyond the scope of
this set).
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Right now the table's storage_options life starts in cql, and shortly
after the lw-shared-pointer to options is put on keyspace metadata.
Later, when the table is created the pointer from keyspace is copied on
the table via its contructor.
Next patches will extend the options pointed to by a table, and the
extension is going to be different for different tables. For that, each
table needs to have its private options and this patch prepares for
that.
For now table directly calls sstables/storage code to get the options
from, but it's temporary, soon the options will be created via sstables
manager together with initialising the storage itself.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
To be able to atomically delete sstables both in
base table directory and in its sub-directories,
like `staging/`, use a shared pending_delete_dir
under under the base directory.
Note that this requires loading and processing
the base directory first.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
in 372a4d1b79, we introduced a change
which was for debugging the logging message. but the logging message
intended for printing the temp_dir not prints an `optional<int>`. this
is both confusing, and more importantly, it hurts the debuggability.
in this change, the related change is reverted.
Fixesscylladb/scylladb#20408
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#20409
Current S3 storage driver keeps sstables in bucket in a form of
/bucket/generation/component-name
To get sstables that are backed up on S3 this format doesn't apply,
because components are uploaded with their names unmodified. This patch
makes S3 storage driver account for that and not re-format component
paths for upload sstable state.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Commit ad0e6b79 (replica: Remove all_datadir from keyspace config) removed all_datadirs from keyspace config, now it's datadir turn. After this change keyspace no longer references any on-disk directories, only the sstables's storage driver attached to keyspace's tables does.
refs #12707Closesscylladb/scylladb#19866
* github.com:scylladb/scylladb:
replica: Remove keyspace::config::datadir
sstables/storage: Evaluate path for keyspace directory in storage
sstables/storage: Add sstables_manager arg to init_keyspace_storage()
assert() is traditionally disabled in release builds, but not in
scylladb. This hasn't caused problems so far, but the latest abseil
release includes a commit [1] that causes a 1000 insn/op regression when
NDEBUG is not defined.
Clearly, we must move towards a build system where NDEBUG is defined in
release builds. But we can't just define it blindly without vetting
all the assert() calls, as some were written with the expectation that
they are enabled in release mode.
To solve the conundrum, change all assert() calls to a new SCYLLA_ASSERT()
macro in utils/assert.hh. This macro is always defined and is not conditional
on NDEBUG, so we can later (after vetting Seastar) enable NDEBUG in release
mode.
[1] 66ef711d68Closesscylladb/scylladb#20006
Currently, delete_atomically can be called with
a list of sstables from mixed prefixes in two cases:
1. truncate: where we delete all the sstables in the table directory
2. tablet cleanup: similar to truncate but restricted to sstables in a
single tablet replica
In both cases, it is possible that sstables in staging (or quarantine)
are mixed with sstables in the base directory.
Until a more comprehensive fix is in place,
(see https://github.com/scylladb/scylladb/pull/19555)
this change just lifts the ban on atomic deletion
of sstables from different prefixes, and acknowledging
that the implementation is not atomic across
prefixes. This is better than crashing for now,
and can be backported more easily to branches
that support tablets so tablet migration can
be done safely in the presence of repair of
tables with views.
Refs scylladb/scylladb#18862
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Closesscylladb/scylladb#19816
Currently the init_keyspace_storage() expects that the caller would
tell it where the ks directory is, but it's not nice as keyspace may
not necessarity keep its sstables in any directory.
This patch moves the directory path evaluation into storage code,
specifically to the lambda that is called for on-disk sstables. The
way directory is evaluated mirrors the one from make_keyspace_config()
that will be removed by next patch.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
filesystem_storage::create_links_common() runs on directories that
generally differ from "_dir", thus, we can't replace its sync_directory()
calls with _dir.sync(). We can still use a common (temporary)
"opened_directory" object for synching "dst_dir" three times, saving two
open and two close operations.
This patch is best viewed with "git show -W".
Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
In "filesystem_storage", change_dir_for_test() and move() replace "_dir"
with "opened_directory(new_dir)" using the move assignment operator.
Consequently, the file descriptor underlying "_dir" is closed
synchronously as a part of object destruction.
Expose the async file::close() function through "opened_directory".
Introduce filesystem_storage::change_dir() as a common async workhorse for
both change_dir_for_test() and move(). In change_dir(), close the old
directory asynchronously.
Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
Currently change_dir_for_test() is synchronous. Make it return a future,
so that we can use async operations in change_dir_for_test() overrides.
Signed-off-by: Laszlo Ersek <laszlo.ersek@scylladb.com>
