Refs #356
Pre-allocates N segments from timer task. N is "adaptive" in that it is
increased (to a max) every time segement acquisition is forced to allocate
a new instead of picking from pre-alloc (reserve) list. The idea is that it is
easier to adapt how many segments we consume per timer quanta than the timer
quanta itself.
Also does disk pressure check and flush from timer task now. Note that the
check is still only done max once every new segment.
Some logging cleanup/betterment also to make behaviour easier to trace.
Reserve segments start out at zero length, and are still deleted when finished.
This is because otherwise we'd still have to clear the file to be able to
properly parse it later (given that is can be a "half" file due to power fail
etc). This might need revisiting as well.
With this patch, there should be no case (except flush starvation) where
"add_mutation" actually waits for a (potentially) blocking op (disk).
Note that since the amount of reserve is increased as needed, there will
be occasional cases where a new segment is created in the alloc path
until the system finds equilebrium. But this should only be during a breif
warmup.
Refs #356
* Move sync time setting to sync initiate to help prevent double syncs
* Change add_mutation to only do explicit sync with wait if time elapsed
since last is 2x sync window
* Do not wait for sync when moving to new segment in alloc path
* Initiate _sync_time properly.
* Add some tracing log messages to help debug
Also at seastar-dev: calle/commitlog_flush_v3
(And, yes, this time I _did_ update the remote!)
Refs #262
Commit of original series was done on stale version (v2) due to authors
inability to multitask and update git repos.
v3:
* Removed future<> return value from callbacks. I.e. flush callback is now
only fully syncronous over actual call
* Do not throw away commitlog segments on disk size overflow.
Issue a flush request (i.e. calculate RP we want to free unto,
and for all dirty CF:s, do a request).
"Abstracted" as registerable callback. I.e. DB:s responsibility
to actually do something with it.
Fixes#99
Adding missing commitlog metrics to the rest API.
v2: Mis-send (clumsy fingers)
v3: Use map_reduce0 + subroutine for nicer code
v4: rebased on current master
v5: rebased yet again.
Since the _second_ file in this previous patch set was commited, and is
dependent on this very change below to even compile, some expediency might be
warranted.
* Fixes#247
* Re-introduce test_allocation_failure, but allow for the "failure" to not
happen. I.e. if run with low memory settings, the test will check that
allocation failure is graceful. With lots of memory it will check partial
write.
* Make it more like origin, i.e. based on wall clock time of app start
* Encode shard ID in the, RP segement ID, to ensure RP:s and segement names
are unique per shard
Origin
* Note: removed commitlog_test:test_allocation_failure because with
segments limited to 4GB -> mutation limited to 2GB, actually forcing
a fail is not guaranteed or even likely.
* Make it more like origin, i.e. based on wall clock time of app start
* Encode shard ID in the, RP segement ID, to ensure RP:s and segement names
are unique per shard
recently, "file" started to use a shared_ptr internally, and is already
copy-able and reference counted, and there is no reason to use
lw_shared_ptr<file>. This patch cleans up a few remaining places where
lw_shared_ptr<file> was used.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
This adds a method to return a vector with full-path to the active
segment names. It will be used by the API.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Should fix use-after-free when a frozen_mutation is applied to the
local shard.
Includes two adjustments to urchin collectd usage from Calle:
- Updated thrift collectd registration to use proper move semantics
- Commitlog: Fix collectd registration to use move semantics + test
Signed-off-by: Calle Wilund <calle@cloudius-systems.com>
- # segments
- # allocting segments
- # unused segments
- # allocations
- # cycles (disk writes)
- # flush
- # total bytes allocated
- # total bytes disk slack (due to dma blocks)
Counters are per-commitlog (shard). Can be extended to be per-segment also,
but would be transient and probably not much more useful.
Signed-off-by: Calle Wilund <calle@cloudius-systems.com>
