9bd9ee9f36
this commit moves the object storage configuration guide from the developer documentation to the user-facing admin documentation. the change reflects the increasing importance of object storage integration in user-facing features. in this change: - move relevant content from `docs/dev/object_storage.md` to `docs/operating-scylla/admin.rst` - reformat the content from Markdown to reStructuredText (RST) - reword and restructure the content to be more user-friendly - add explanations and context suitable for a broader audience this change makes the object storage configuration information more accessible to Scylla administrators and end-users, supporting the adoption of new features built on top of object storage integration. Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
2.5 KiB
Keeping sstables on S3
On of the ways to use object storage is to keep sstables directly on it as objects.
Enabling the feature
Currently the object-storage backend works if keyspace-storage-options is listed
in experimental_features in scylla.yaml. like:
experimental_features:
- keyspace-storage-options
It can also be enabled with --experimental-features=keyspace-storage-options
command line option when launchgin scylla.
Creating keyspace
Sstables location is keyspace-scoped. In order to create a keyspace with S3
storage use CREATE KEYSPACE with STORAGE = { 'type': 'S3', 'endpoint': '$endpoint_name', 'bucket': '$bucket' }
parameters, where $endpoint_name should match with the corresponding name
of the configured endpoint in the YAML file above.
In the following example, an endpoint named "s3.us-east-2.amazonaws.com" is
defined in object_storage.yaml, and this endpoint is used when creating the
keyspace "ks".
in object_storage.yaml:
endpoints:
- name: s3.us-east-2.amazonaws.com
port: 443
https: true
aws_region: us-east-2
aws_access_key_id: EXAMPLE_ACCESS_KEY_ID
aws_secret_access_key: EXAMPLE_SECRET_ACCESS_KEY
and when creating the keyspace:
CREATE KEYSPACE ks
WITH REPLICATION = {
'class' : 'NetworkTopologyStrategy',
'replication_factor' : 1
}
AND STORAGE = {
'type' : 'S3',
'endpoint' : 's3.us-east-2.amazonaws.com',
'bucket' : 'bucket-for-testing'
};
Copying sstables on S3 (backup)
It's possible to upload sstables from data/ directory on S3 via API. This is good to do because in that case all the resources that are needed for that operation (like disk IO bandwidth and IOPS, CPU time, networking bandwidth) will be under Seastar's control and regular Scylla workload will not be randomly affected.
The API endpoint name is /storage_service/backup and its Swagger description can be
found here. Accepted parameters are
- keyspace: the keyspace to copy sstables from
- table: the table to copy sstables from
- snapshot: the snapshot name to copy sstables from
- endpoint: the key in the object storage configuration file
- bucket: bucket name to put sstables' files in
- prefix: prefix to put sstables' files under
Currently only snapshot backup is possible, so first one needs to take snapshot
All tables in a keyspace are uploaded, the destination object names will look like
s3://bucket/some/prefix/to/store/data/.../sstable