Fixes#6202
Distributed loader sstable opening is gated through the
database::sstable_load_concurrency_sem() semaphore
(at a concurrency of 3).
This is (according to creation comment) to reduce memory footprint
during bootstrap, by partially serializing the actual opening of
existing sstables.
However, in certain versions of the product, there exist circular
dependencies between data in some sstables and the ability to actually
read others. Thus when gated as above, we can end up with the
dependents acquiring the semaphore fully, and once stuck waiting for
population of their dependency effectively blocking this from ever
happening.
Since we probably do not want to remove the concurrency control,
and increasing it would only push the problem further away,
we solve the issue by adding the ability to mark certain keyspaces
as "prioritized" (pre-bootstrap), and allow them to populate outside
the normal concurrency control semaphore. Concurrency increase is
however limited to one extra sstable per shard and prio keyspace.
Message-Id: <20200415102431.20816-1-calle@scylladb.com>
This is the last place where database code needs the migration_manager
instance to be alive, so now the mutual dependency between these two
is gone, only the migration_manager needs the database, but not the
vice-versa.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Files and directories must be owned by the process uid.
Files must have read access and directories must have
read, write, and execute access.
Refs #3117
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Scan the table's pending_delete sub-directory if it exists.
Remove any temporary pending_delete log files to roll back the respective
delete_atomically operation.
Replay completed pending_delete log files to roll forward the respective
delete_atomically operation, and finally delete the log files.
Cleanup of temporary sstable directories and pending_delete
sstables are done in a preliminary scan phase when populating the column family
so that we won't attempt to load the to-be-deleted sstables.
Fixes#4082
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
In preparation for replaying pending_delete log files,
we would like to first remove any temporary sst dirs
and later handle pending_delete log files, and only
then populate the column family.
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
SSTables loaded to the system via /upload dir may sometimes be needed
to generate view updates from them (if their table has accompanying
views).
Fixes#4047
distributed_loader is a sizeable fraction of database.cc, so moving it
out reduces compile time and improves readability.
Message-Id: <20181230200926.15074-1-avi@scylladb.com>
