Describing S3 storage for an sstables nowadays has two options -- via
sstables registry entry and by using the direct prefix string. The
former is used when putting a keyspace on S3. In this case each sstable
has the corresponding entry in the system.sstables table. The latter is
used by "restore from object storage" code. In that case, sstables don't
have entries in the registry, but are accessed by a specific S3 object
path.
This patch reflects this difference by making s3_options::location be
variant of string prefix and table_id owner. The owner needs more
explanation, here it is.
Today, the system.sstables schema defines partition key to be "string
location" and clustering key to be "UUID generation". The partition key
is table's datadir string, but it's wrong to use it this way. Next
patches will change the partition key to be table's ID (there's table_id
type for it), and before doing it storage options must be prepared to
carry it onboard. This patch does it, but the table_id alternative of
the location is still unused, the rest of the code keeps using the
string location to reference a row in the registry table. Next patches
will eventually make use of the table_id value.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
before this change, we enumerate the sstables tracked by the
system.sstables table, and restore them when serving
requests to "storage_service/restore" API. this works fine with
"storage_service/backup" API. but this "restore" API cannot be
used as a drop-in replacement of the rclone based API currently
used by scylla-manager.
in order to fill the gap, in this change:
* add the "prefix" parameter for specifying the shared prefix of
sstables
* add the "sstables" parameter for specifying the list of TOC
components of sstables
* remove the "snapshot" parameter, as we don't encode the prefix
on scylla's end anymore.
* make the "table" parameter mandatory.
Fixesscylladb/scylladb#20461
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Lost of code (in tools and tests) explicitly deal with local sstables
and need to create options for it. Currently default-constructing
options generates local ones, but without the directory path. Add a
helper that creates local options with path and patch callers.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Now when tables keep their own copy of storage options, it's possible
for each table to add table-specific information on it. Namely -- path
for local storage and prefix for S3 one (in fact, it's not a "prefix",
but a key in sstables registry, but fixing it is beyond the scope of
this set).
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
otherwise when compiling with the new seastar, which removed
`#include <variant>` from `std-compat.hh`, the {mode}-headers
target would fail to build, like:
```
./data_dictionary/storage_options.hh:34:29: error: no template named 'variant' in namespace 'std'
10:45:15 using value_type = std::variant<local, s3>;
10:45:15 ~~~~~^
10:45:15 ./data_dictionary/storage_options.hh:35:5: error: unknown type name 'value_type'; did you mean 'std::_Bit_const_iterator::value_type'?
10:45:15 value_type value = local{};
10:45:15 ^~~~~~~~~~
10:45:15 std::_Bit_const_iterator::value_type
```
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closesscylladb/scylladb#18921
instead of dispatching and implementing the per-option handling
right in `storage_option`, define these helpers in the dedicated
option themselves, so `storage_option` is only responsible for
dispatching.
much cleaner this way. this change also makes it easier to add yet
another storage backend.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
as the only user of these comparison operators is
`storage_options::can_update_to()`, which just check if the given
`storage_options` is equal to the stored one. so no need to define
the <=> operator.
also, no need to add the `friend` specifier, as the options are plain
struct, all the member variables are public.
make the comparison operator a member function instead of a free
function, as in C++20 comparision operators are symmetric.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Description of storage options is important for S3, as one
needs to know if underlying storage is either local or
remote, and if the latter, details about it.
This relies on server-side desc statement.
$ ./bin/cqlsh.py -e "describe keyspace1;"
CREATE KEYSPACE keyspace1 WITH replication = { ... } AND
storage = {'type': 'S3', 'bucket': 'sstables',
'endpoint': '127.0.0.1:9000'} AND
durable_writes = true;
Fixes#13507.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Closes#13510
The STORAGE option is designed to hold a map of options
used for customizing storage for given keyspace.
The option is kept in a system_schema.scylla_keyspaces table.
The option is only available if the whole cluster is aware
of it - guarded by a cluster feature.
Example of the table contents:
```
cassandra@cqlsh> select * from system_schema.scylla_keyspaces;
keyspace_name | storage_options | storage_type
---------------+------------------------------------------------+--------------
ksx | {'bucket': '/tmp/xx', 'endpoint': 'localhost'} | S3
```
