Currently, we control incremental backups behavior from the storage service.
This creates some very concrete problems, since the storage service is not
always available and initialized.
The solution is to move it to the column family (and to the keyspace so we can
properly propagate the conf file value). When we change this from the api, we will
have to iterate over all of them, changing the value accordingly.
Signed-off-by: Glauber Costa <glommer@scylladb.com>
Align with rest of file (for better or worse). This allows calls from
entity without query_processor handy (i.e. storage_proxy).
Added "minimal" setup method for the "global" state, to facilitate
tests. Doing a full setup either in cql_test_env or after it is created
breaks badly. (Not sure why). So quick workaround.
Updated the current two users (batchlog_manager and commitlog_replayer)
callsites to conform.
Since row_cache::populate() uses allocating_section now, the trick
with populating under relcaim lock no longer works, resulting in
assertion failure inside allocating_section:
row_cache_alloc_stress: utils/logalloc.hh:289: auto logalloc::allocating_section::operator()(logalloc::region&, Func&&) [with Func = row_cache::populate(const mutation&)::<lambda()>::<lambda()>]: Assertion `r.reclaiming_enabled()' failed.
Use the trick with populating until eviction is detected by comapring
region occupancy.
Refs #356
Pre-allocates N segments from timer task. N is "adaptive" in that it is
increased (to a max) every time segement acquisition is forced to allocate
a new instead of picking from pre-alloc (reserve) list. The idea is that it is
easier to adapt how many segments we consume per timer quanta than the timer
quanta itself.
Also does disk pressure check and flush from timer task now. Note that the
check is still only done max once every new segment.
Some logging cleanup/betterment also to make behaviour easier to trace.
Reserve segments start out at zero length, and are still deleted when finished.
This is because otherwise we'd still have to clear the file to be able to
properly parse it later (given that is can be a "half" file due to power fail
etc). This might need revisiting as well.
With this patch, there should be no case (except flush starvation) where
"add_mutation" actually waits for a (potentially) blocking op (disk).
Note that since the amount of reserve is increased as needed, there will
be occasional cases where a new segment is created in the alloc path
until the system finds equilebrium. But this should only be during a breif
warmup.
v2: Fixed timestamp not being reset on reserve acquire
Refs #356
Pre-allocates N segments from timer task. N is "adaptive" in that it is
increased (to a max) every time segement acquisition is forced to allocate
a new instead of picking from pre-alloc (reserve) list. The idea is that it is
easier to adapt how many segments we consume per timer quanta than the timer
quanta itself.
Also does disk pressure check and flush from timer task now. Note that the
check is still only done max once every new segment.
Some logging cleanup/betterment also to make behaviour easier to trace.
Reserve segments start out at zero length, and are still deleted when finished.
This is because otherwise we'd still have to clear the file to be able to
properly parse it later (given that is can be a "half" file due to power fail
etc). This might need revisiting as well.
With this patch, there should be no case (except flush starvation) where
"add_mutation" actually waits for a (potentially) blocking op (disk).
Note that since the amount of reserve is increased as needed, there will
be occasional cases where a new segment is created in the alloc path
until the system finds equilebrium. But this should only be during a breif
warmup.
Patch "Fix some timing/latency issues with sync" changed new_segment to
_not_ wait for flush to finish. This means that checking actual files on
disk in the test case might race.
Lucklily, we can more or less just check the segment list instead
(added recently-ish)
Currently, we create a temporary TOC file after we are done writing
all the other components. However, we want to create a temporary
TOC before starting to write any other component.
So if there is a missing TOC, there is likely to be a corruption,
so we should refuse to boot and provide the sysadmin with a
detailed message. If there is a temporary TOC, it means that there
was a sudden shutdown while the sstable was being written.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
"The motivation is to exercise more code during tests, and possibly also avoid
some special casing just for tests in the future. Sstables will be persisted
in a unique temporary directory which is auto-removed when environment is
torn down."
It's easier to set members directly rather than pass them to the
constructor of in_memory_cql_env. Plus, stop() now matches start() and
not an external function.
Refs #293
Even more horrible that the shutdown patch. Tests using cql_test_env
are dependant on init.cc functions, but then scylla stopped being shut down
properly, those tests did to -> assert in sharded.hh
Yet another temp patch, simply duplicating the init.cc code for clq_test_env
to ensure we get what we think.
From Pawel:
This series makes compaction remove items that are no longer items:
- expired cells are changed into tombstones
- items covered by higher level tombstones are removed
- expired tombstones are removed if possible
Fixes#70.
Fixes#71.
Also at seastar-dev: calle/commitlog_flush_v3
(And, yes, this time I _did_ update the remote!)
Refs #262
Commit of original series was done on stale version (v2) due to authors
inability to multitask and update git repos.
v3:
* Removed future<> return value from callbacks. I.e. flush callback is now
only fully syncronous over actual call
"Fixes #262
Handles CL disk size exceeding configured max size by calling flush handlers
for each dirty CF id / high replay_position mark. (Instead of uncontrolled
delete as previously).
* Increased default max disk size to 8GB. Same as Origin/scylla.yaml (so no
real change, but synced).
* Divide the max disk size by cpus (so sum of all shards == max)
* Abstract flush callbacks in CL
* Handler in DB that initiates memtable->sstable writes when called.
Note that the flush request is done "syncronously" in new_segment() (i.e.
when getting a new segment and crossing threshold). This is however more or
less congruent with Origin, which will do a request-sync in the corresponding
case.
Actual dealing with the request should at least in production code however be
done async, and in DB it is, i.e. we initiate sstable writes. Hopefully
they finish soon, and CL segments will be released (before next segment is
allocated).
If the flush request does _not_ eventually result in any CF:s becoming
clean and segments released we could potentially be issuing flushes
repeatedly, but never more often than on every new segment."
* Do not throw away commitlog segments on disk size overflow.
Issue a flush request (i.e. calculate RP we want to free unto,
and for all dirty CF:s, do a request).
"Abstracted" as registerable callback. I.e. DB:s responsibility
to actually do something with it.
Right now, gossip returns hard coded cluster and partitioner name.
sstring get_cluster_name() {
// FIXME: DatabaseDescriptor.getClusterName()
return "my_cluster_name";
}
sstring get_partitioner_name() {
// FIXME: DatabaseDescriptor.getPartitionerName()
return "my_partitioner_name";
}
Fix it by setting the correct name from configure option.
With this
cqlsh 127.0.0.$i -e "SELECT * from system.local;
returns correct cluster_name.
Fixes#291
In some cases region may be in a state where it is not empty and
nothing could be evicted from it. For example when creating the first
entry, reclaimer may get invoked during creation before it gets
linked. We therefore can't rely on emptiness as a stop condition for
reclamation, the evction function shall signal us if it made forward
progress.
It's a singleton, so we can't attempt to init it more than once.
Fixes cql_query_test failure:
/home/tgrabiec/src/urchin2/seastar/core/future.hh:315: void future_state<>::set(): Assertion `_u.st == state::future' failed.
unknown location(0): fatal error in "test_create_table_statement": signal: SIGABRT (application abort requested)
seastar/tests/test-utils.cc(31): last checkpoint
This patch tests that collection within a mutation behave properly.
That is what lead to #188.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Disabling compaction of a region is currently done in order to keep
the references valid. But disabling only compaction is not enough, we
also need to disable eviction, as it also invalidates
references. Rather than introducing another type of lock, compaction
and eviction are controlled together, generalized as "reclaiming"
(hence the reclaim_lock).
It relies on the fact that the process has a fixed amount of memory
assigned and std::bad_alloc is thrown in a timely manner when it fills
up, which is the case for seastar's allocator, but not with the
default allocator. With the latter the OOM killer kills the process.
