On boot it's very useful to know which storage a table comes from, so
add the respective info to existing log messages.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
When sstables are stored on object storage, they are "registered" in the
system.sstables_registry ownership table. The sstable_directory is
supposed to list sstables from this table, so here's the respective
components lister.
The lister is created by sstables_manager, by the time it's requested
from the the system keyspace is already plugged. The lister only handles
"sealed" sstables. Dangling ones are still ignored, this is to be fixed
later.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Now the lister is filesystem-specific. There will soon come another one
for S3, so the sstable_directory should be prepared for that by making
the lister an abstract class.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The directory's lister is storage-specific and should be created
differently for different storage options.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The driver puts all componenets into
s3://bucket/uuid/component_name
objects where 'bucket' is the keyspace options configuration parameter,
and the 'uuid' is the value obtained from the ownership table.
E.g.
s3://test_bucket/d0a743b0-ad38-11ed-85b5-39b6b0998182/Data.db
The life-time is straightforward. Until sealed, the sstable has
'creating' status in the table, then it's updated to be 'sealed'. Prior
to removing the objects the status is set to 'deleting' thus allowing
the distributed loader to pick up the dangling objects un re-load (not
yet implemented). Finally, the entry is deleted from the table.
It needs the PR #12648 not to generate empty ks/cf directories on the
local filesystem.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The schema is
CREATE TABLE system.sstables (
location text,
generation bigint,
format text,
status text,
uuid uuid,
version text,
PRIMARY KEY (location, generation)
)
A sample entry looks like:
location | generation | format | status | uuid | version
---------------------------------------------------------------------+------------+--------+--------+--------------------------------------+---------
/data/object_storage_ks/test_table-d096a1e0ad3811ed85b539b6b0998182 | 2 | big | sealed | d0a743b0-ad38-11ed-85b5-39b6b0998182 | me
The uuid field points to the "folder" on the storage where the sstable
components are. Like this:
s3
`- test_bucket
`- f7548f00-a64d-11ed-865a-0c1fbc116bb3
`- Data.db
- Index.db
- Filter.db
- ...
It's not very nice that the whole /var/lib/... path is in fact used as
location, it needs the PR #12707 to fix this place.
Also, the "status" part is not yet fully functional, it only supports
three options:
- creating -- the same as TemporaryTOC file exists on disk
- sealed -- default state
- deleting -- the analogy for the deletion log on disk
The latter needs support from the distributed_loader, which's not yet
there. In fact, distributes_loader also needs to be patched to actualy
select entries from this table on load. Also it needs the mentioned
PR #12707 to support staging and quarantine sstables.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The sharded<sys_ks> instances are plugged to large data handler and
compaction manager to maintain the circular dependency between these
components via the interposing database instance. Do the same for user
sstables manager, because S3 driver will need to update the local
ownership table.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
This patch adds storage options lw-ptr to sstables_manager::make_sstable
and makes the storage instance creation depend on the options. For local
it just creates the filesystem storage instance, for S3 -- throws, but
next patch will fix that.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The class in question will need to know the table's storage it will need
to list sstables from. For that -- construct it with the storage options
taken from table.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
When opening an existing (or just sealed) sstable its components are
stat()-ed to get the on-disk sizes and a bit more. Stat-ing a file by
name on S3 is not (yet) implemented and doing it file-by-file can be
quite terrible. So add a method to return sstable stats in a
storage-specific manner. For S3 this can be implemented by getting the
info from the ownership table (in the future).
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
This time sstable needs to create a data sink for a component without
having the file at hand. That's pretty much the same as in previous
patch, but the mathod declaration differs slightly.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The method needs to create two data sinks -- for Data and for Index
files -- and then wrap it with more stuff (compression, checksums,
streams, etc.). With S3 backend using file-output-stream won't work,
becase S3 storage cannot provide writable file API (it has data_sink
instead).
This patch extracts file_data_sink creation so that it could be
virtualized with storage API later.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Currently sstable carries a filesystem_storage instance on board. Next
patches will make it possible to use some other storage with different
data accessing methods. This patch makes sstable carry abstract storage
interface and make the existing filesystem_storage implement it.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Sometimes an sstable is used for random read, sometimes -- for streamed
read using the input stream. For both cases the file API can be
provided, because S3 API allows random reads of arbitrary lengths.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Putting a large object into S3 using plain PUT is bad choice -- one need
to collect the whole object in memory, then send it as a content-length
request with plain body. Less memory stress is by using multipart
upload, but multipart upload has its limitation -- each part should be
at least 5Mb in size. For that reason using file API doesn't work --
file IO API operates with external memory buffers and the file impl
would only have raw pointers to it. In order to collect 5Mb of chunk in
RAM the impl would have to copy the memory which is not good. Unlike the
file API data_sink API is more flexible, as it has temporary buffers at
hand and can cache them in zero-copy manner.
Having sad that, the S3 data_sink implementation is like this:
* put(buffer):
move the buffer into local cache, once the local cache grows above 5Mb
send out the part
* flush:
send out whatever is in cache, then send upload completion request
* close:
check that the upload finihsed (in flush), abort the upload otherwise
User of the API may (actually should) wrap the sink with output_stream
and use it as any other output_stream.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Those include -- HEAD to get size, PUT to upload object in one go, GET
to read the object as contigious buffer and DELETE to drop one.
The client uses http client from seastar and just implements the S3
protocol using it.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
The facilities in run.py script allow launching scylla over temporary
directory, waiting for it to come alive, killing, etc. The limitation of
those is that the work-dir create for scylla is tighly coupled with its
pid. The object-storage test in next patches will need to check that the
sstables are preserved on scylla restart and this hard binding of
workdir to pid won't work.
This patch generalizes the scylla run/abort helpers to accept an
external directory to work on and adds a call to restart scylla process
over existing directory.
And one small related change here -- log file is opened in O_APPEND mode
so that restarted scylla process continues writing into the old file.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
When test.py starts it activates a minio server inside test-dir and
configures an anonymous bucket for test cases to run on
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
When sstable is opened it generates a certain content into TOC file. In
filesystem storage this first gets into TemporaryTOC one. Future S3
driver will need the same to put into TOC object. Not to produce
duplicate code detach the content generation into a helper. Next patches
will make use of it.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
in c642ca9e73, a reference to the
a parameter `config` passed to the `thrift_server` 's constructor is
passed down to `create_handler_factory()`, which keeps it so it can
create connection handler on demand. but unfortunately,
- the `config` parameter is a temporary variable
- the `config` parameter is moved away in the constructor after
`create_handler_factory()` is called
hence we have a dangling reference when the factory created by
`create_handler_factory()` tries to deference the reference when
handling a new incoming connection.
in this change,
- the definitions of `_config` and `_handler_factory` member
variables are transposed, so that the former is initialized
first.
- `_handler_factory` now keeps a reference to `_config`'s member
variable, so that the weak reference it holds is always valid.
Fixes#13455
Branches: none
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#13456
The system_keyspace.hh now includes raft stuff, topology changes stuff, task_manager stuff, etc. It's going to include tablets.hh (but maybe not). Anything that deals with system keyspace, and includes system_keyspace.hh, would transitively pull these too. This header is becoming a central hub for all the features.
This PR removes all the headers from system_keyspace.hh that correspond to other "subsystems" keeping only generic mutations/querying and seastar ones.
Closes#13450
* github.com:scylladb/scylladb:
system_keyspace.hh: Remove unneeded headers
system_keyspace: Move topology_mutation_builder to storage_service
system_keyspace: Move group0_upgrade_state conversions to group0 code
After a failed topology operation, like bootstrap / decommission /
removenode, the cluster might contain a garbage entry in either token
ring or group 0. This entry can be cleaned-up by executing removenode on
any other node, pointing to the node that failed to bootstrap or leave
the cluster.
Document this procedure, including a method of finding the host ID of a
garbage entry.
Add references in other documents.
Fixes: #13122Closes#13186
As a first step towards using host_id to identify nodes instead of ip addresses
this series introduces a node abstraction, kept in topology,
indexed by both host_id and endpoint.
The revised interface also allows callers to handle cases where nodes
are not found in the topology more gracefully by introducing `find_node()` functions
that look up nodes by host_id or inet_address and also get a `must_exist` parameter
that, if false (the default parameter value) would return nullptr if the node is not found.
If true, `find_node` throws an internal error, since this indicates a violation of an internal
assumption that the node must exist in the topology.
Callers that may handle missing nodes, should use the more permissive flavor
and handle the !find_node() case gracefully.
Closes#11987
* github.com:scylladb/scylladb:
topology: add node state
topology: remove dead code
locator: add class node
topology: rename update_endpoint to add_or_update_endpoint
topology: define get_{rack,datacenter} inline
shared_token_metadata: mutate_token_metadata: replicate to all shards
locator: endpoint_dc_rack: refactor default_location
locator: endpoint_dc_rack: define default operator==
test: storage_proxy_test: provide valid endpoint_dc_rack
The latter is the only user of the class. This keeps system keyspace
code free from unrelated logic and from raft::server_id type.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
In order to keep system keyspace free from group0 logic and from the
service::group0_upgrade_state type
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
before this change, we suse `git submodule summary ${submodule}`
for collecting the titles of commits in between current HEAD and
origin/master. normally, this works just fine. but it fails to
collect all commits if the origin/master happens to reference
a merge commit. for instance, if we have following history like:
1. merge foo
2. bar
3. foo
4. baz <--- submodule is pointing here.
`git submodule summary` would just print out the titles of commits
of 1 and 3.
so, in this change, instead of relying on `git submodule summary`,
we just collect the commits using `git log`. but we preserve the
output format used by `git submodule summary` to be consistent with
the previous commits bumping up the submodules. please note, in
this change instead of matching the output of `git submodule summary`,
we use `git merge-base --is-ancestor HEAD origin/master` to check
if we are going to create a fastforward change, this is less fragile.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Closes#13366
A problem in compaction reevaluation can cause the SSTable set to be left uncompacted for indefinite amount of time, potentially causing space and read amplification to be suboptimal.
Two revaluation problems are being fixed, one after off-strategy compaction ended, and another in compaction manager which intends to periodically reevaluate a need for compaction.
Fixes https://github.com/scylladb/scylladb/issues/13429.
Fixes https://github.com/scylladb/scylladb/issues/13430.
Closes#13431
* github.com:scylladb/scylladb:
compaction: Make compaction reevaluation actually periodic
replica: Reevaluate regular compaction on off-strategy completion
Writing into sstable component output stream should be done with care. In particular -- flushing can happen only once right before closing the stream. Flushing the stream in between several writes is not going to work, because file stream would step on unaligned IO and S3 upload stream would send completion message to the server and would lose any subsequent write.
Most of the file_writer users already obey that and flush the writer once right before closing it. The do_write_simple() is extra careful about exceptions handling, but it's an overkill (see first patch).
It's better to make file_writer API explicitly lack the ability to flush itself by flushing the stream when closing the writer.
Closes#13338
* github.com:scylladb/scylladb:
sstables: Move writer flush into close (and remove it)
sstables: Relax exception handling in do_write_simple
This test currently uses `test/lib/test_table.hh` to generate data for its test cases. This data generation facility is used by no other tests. Worse, it is redundant as we already have a random data generator with fixed schema, in `test/lib/mutation_source_test.hh`. So in this series, we migrate the test cases in said test file to random schema and its random data generation facilities. These are used by several other test cases and using random schema allows us to cover a wider (quasi-infinite) number of possibilities.
After migrating all tests away from it, `test/lib/test_table.hh` is removed.
This series also reduces the runtime of `fuzzy_test` drastically. It should now run in a few minutes or even in seconds (depending on the machine).
Fixes: #12944Closes#12574
* github.com:scylladb/scylladb:
test/lib: rm test_table.hh
test/boos/multishard_mutation_query_test: migrate other tests to random schema
test/boost/multishard_mutation_query_test: use ks keyspace
test/boost/multishard_mutation_query_test: improve test pager
test/boost/multishard_mutation_query_test: refactor fuzzy_test
test/boost: add multishard_mutation_query_test more memory
types/user: add get_name() accessor
test/lib/random_schema: add create_with_cql()
test/lib/random_schema: fix udt handling
test/lib/random_schema: type_generator(): also generate frozen types
test/lib/random_schema: type_generator(): make static column generation conditional
test/lib/random_schema: type_generator(): don't generate duration_type for keys
test/lib/random_schema: generate_random_mutations(): add overload with seed
test/lib/random_schema: generate_random_mutations(): respect range tombstone count param
test/lib/random_schema: generate_random_mutations(): add yields
test/lib/random_schema: generate_random_mutations(): fix indentation
test/lib/random_schema: generate_random_mutations(): coroutinize method
test/lib/random_schema: generate_random_mutations(): expand comment
Every tracker insertion has to have a corresponding removal or eviction,
(otherwise the number of rows in the tracker will be misaccounted).
If we add the row to the tracker before adding it to the tree,
and the tree insertion fails (with bad_alloc), this contract will be violated.
Fix that.
Note: the problem is currently irrelevant because an exception during
sentinel insertion will abort the program anyway.
Closes#13336
Compaction group is responsible for deleting SSTables of "in-strategy"
compactions, i.e. regular, major, cleanup, etc.
Both in-strategy and off-strategy compaction have their completion
handled using the same compaction group interface, which is
compaction_group::table_state::on_compaction_completion(...,
sstables::offstrategy offstrategy)
So it's important to bring symmetry there, by moving the responsibility
of deleting off-strategy input, from manager to group.
Another important advantage is that off-strategy deletion is now throttled
and gated, allowing for better control, e.g. table waiting for deletion
on shutdown.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Closes#13432
this is the 15th changeset of a series which tries to give an overhaul to the CMake building system. this series has two goals:
- to enable developer to use CMake for building scylla. so they can use tools (CLion for instance) with CMake integration for better developer experience
- to enable us to tweak the dependencies in a simpler way. a well-defined cross module / subsystem dependency is a prerequisite for building this project with the C++20 modules.
also, i just found that the scylla executable built with cmake building system segfault in master HEAD. like
```
AddressSanitizer:DEADLYSIGNAL
=================================================================
==3974496==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000000000 bp 0x7ffd48549f70 sp 0x7ffd48549728 T0)
==3974496==Hint: pc points to the zero page.
==3974496==The signal is caused by a READ memory access.
==3974496==Hint: address points to the zero page.
#0 0x0 (<unknown module>)
#1 0x14e785a5 in wasmtime_runtime::traphandlers::unix::trap_handler::h1f510afc2968497f /home/kefu/.cargo/registry/src/mirrors.sjtug.sjtu.edu.cn-7a04d2510079875b/wasmtime-runtime-5.0.1/src/traphandlers/unix.rs:159:9
#2 0x7f3462e5eb9f (/lib64/libc.so.6+0x3db9f) (BuildId: 6107835fa7d4725691b2b7f6aaee7abe09f493b2)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (<unknown module>)
==3974496==ABORTING
Aborting on shard 0.
Backtrace:
0xd16c38a
0x13c5aab0
0x13b9821e
0x13c2fdc7
/lib64/libc.so.6+0x3db9f
/lib64/libc.so.6+0x8eb93
/lib64/libc.so.6+0x3daed
/lib64/libc.so.6+0x2687e
0xd1e5f8a
0xd1e3d34
0xd1ca059
0xd1c5e29
0xd1c5605
0x14e785a5
/lib64/libc.so.6+0x3db9f
```
decoded:
```
__interceptor_backtrace at ??:?
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /home/kefu/dev/scylladb/seastar/include/seastar/util/backtrace.hh:60
seastar::backtrace_buffer::append_backtrace() at /home/kefu/dev/scylladb/seastar/src/core/reactor.cc:778
(inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /home/kefu/dev/scylladb/seastar/src/core/reactor.cc:808
seastar::print_with_backtrace(char const*, bool) at /home/kefu/dev/scylladb/seastar/src/core/reactor.cc:820
(inlined by) seastar::sigabrt_action() at /home/kefu/dev/scylladb/seastar/src/core/reactor.cc:3882
(inlined by) operator() at /home/kefu/dev/scylladb/seastar/src/core/reactor.cc:3858
(inlined by) __invoke at /home/kefu/dev/scylladb/seastar/src/core/reactor.cc:3854
/lib64/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6107835fa7d4725691b2b7f6aaee7abe09f493b2, for GNU/Linux 3.2.0, not stripped
__GI___sigaction at :?
__pthread_kill_implementation at ??:?
__GI_raise at :?
__GI_abort at :?
__sanitizer::Abort() at ??:?
__sanitizer::Die() at ??:?
__asan::ScopedInErrorReport::~ScopedInErrorReport() at ??:?
__asan::ReportDeadlySignal(__sanitizer::SignalContext const&) at ??:?
__asan::AsanOnDeadlySignal(int, void*, void*) at ??:?
wasmtime_runtime::traphandlers::unix::trap_handler at /home/kefu/.cargo/registry/src/mirrors.sjtug.sjtu.edu.cn-7a04d2510079875b/wasmtime-runtime-5.0.1/src/traphandlers/unix.rs:159
__GI___sigaction at :?
```
this led me to this change. but unfortunately, this changeset does not address the segfault. will continue the investigation in my free cycles.
Closes#13434
* github.com:scylladb/scylladb:
build: cmake: include cxx.h with relative path
build: cmake: set stack frame limits
build: cmake: pass -fvisibility=hidden to compiler
build: cmake: use -O0 on aarch64, otherwise -Og
S3 client cannot perform anonymous multipart uploads into any real S3
buckets regardless of their configuration. Since multipart upload is
essential part of the sstables backend, we need to implement the
authorisation support for the client early.
(side note): with minio anonymous multipart upload works, with aws s3
anonymous PUT and DELETE can be configured, it's exactly the combination
of aws + multipart upload that does need authorization.
Fortunately, the signature generation and signature checking code is
symmetrical and we have the checking option already in alternator :) So
what this patch does is just moves the alternator::get_signature()
helper into utils/. A sad side effect of that is all tests now need to
link with gnutls :( that is used to compute the hash value itself.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Closes#13428
This PR reverts the scylla sstable schema loading improvements as they fail in CI every other run. I am already working on fixes for these but I am not sure I understand all the failures so it is best to revert and re-post the series later.
Fixes: #13404Fixes: #13410Closes#13419
* github.com:scylladb/scylladb:
Revert "Merge 'tool/scylla-sstable: more flexibility in obtaining the schema' from Botond Dénes"
Revert "tools/schema_loader: don't require results from optional schema tables"
The manager intended to periodically reevaluate compaction need for
each registered table. But it's not working as intended.
The reevaluation is one-off.
This means that compaction was not kicking in later for a table, with
low to none write activity, that had expired data 1 hour from now.
Also make sure that reevaluation happens within the compaction
scheduling group.
Fixes#13430.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
When off-strategy compaction completes, regular compaction is not triggered.
If off-strategy output causes the table's SSTable set to not conform the strategy
goal, it means that read and space amplification will be suboptimal until the next
compaction kicks in, which can take undefinite amount of time (e.g. when active
memtable is flushed).
Let's reevaluate compaction on main SSTable set when off-strategy ends.
Fixes#13429.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
before this change, the wasm binding source files includes the
cxxbridge header file of `cxx.h` with its full path.
to better mirror the behavior of configure.py, let's just
include this header file with relative path.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
* transpose include(mode.common) and include (mode.${build_mode}),
so the former can reference the value defined by the latter.
* set stack_usage_threshold for supported build modes.
please note, this compiler option (-Wstack-usage=<bytes>) is only
supported by GCC so far.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
this addresses an oversight in b234c839e4,
which is supposed to mirror the behavior of `configure.py`.
Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
Related: https://github.com/scylladb/scylla-enterprise/issues/2770
This commit adds the upgrade guide from ScyllaDB Open Source 5.2
to ScyllaDB Enterprise 2023.1.
This commit does not cover metric updates (the metrics file has no
content, which needs to be added in another PR).
As this is an upgrade guide, this commit must be merged to master and
backported to branch-5.2 and branch-2023.1 in scylla-enterprise.git.
Closes#13294
Task manager compaction tasks that cover compaction group
compaction need access to compaction_manager::tasks.
To avoid circular dependency and be able to rely on forward
declaration, task needs to be moved out of compaction manager.
To avoid naming confusion compaction_manager::task is renamed.
Closes#13226
* github.com:scylladb/scylladb:
compaction: use compaction namespace in compaction_manager.cc
compaction: rename compaction::task
compaction: move compaction_manager::task out of compaction manager
compaction: move sstable_task definition to source file
This reverts commit 32fff17e19, reversing
changes made to 164afe14ad.
This series proved to be problematic, the new test introduced by it
failing quite often. Revert it until the problems are tracked down and
fixed.
There are two occasions in scylla_cluster
where we read the node logs, and in both of
them we read the entire file in memory.
This is not efficient and may cause an OOM.
In the first case we need the last line of the
log file, so we seek at the end and move backwards
looking for a new line symbol.
In the second case we look through the
log file to find the expected_error.
The readlines() method returns a Python
list object, which means it reads the entire
file in memory. It's sufficient to just remove
it since iterating over the file instance
already yields lines lazily one by one.
This is a follow-up for #13134.
Closes#13399
