Add a Dockerfile for building a ScyllaDB Docker image. The image is
based on Fedora 22 and ScyllaDB is installed from our RPM repository.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Since row_cache::populate() uses allocating_section now, the trick
with populating under relcaim lock no longer works, resulting in
assertion failure inside allocating_section:
row_cache_alloc_stress: utils/logalloc.hh:289: auto logalloc::allocating_section::operator()(logalloc::region&, Func&&) [with Func = row_cache::populate(const mutation&)::<lambda()>::<lambda()>]: Assertion `r.reclaiming_enabled()' failed.
Use the trick with populating until eviction is detected by comapring
region occupancy.
During cql connection removal we wait for all outstanding sends to
complete by waiting for _ready_to_respond future to resolve, but if
at this point connection is in _pending_responders then poller my call
do_flush() and try to reuse same _ready_to_respond future that already has
a continuation attached to it. The fix is to remove connection from
the poller before waiting for _ready_to_respond. The special measures
should be taken to prevent the connection from been added to the poller
again, so we set _flush_requested to avoid exactly that.
Cache has a tendency to eat up all available memory. It is evicted
on-demand, but this happens at certain points in time (during large
allocation requests). Small allocations which are served from small
object pools won't usually trigger this. Large allocations happen for
example when LSA region needs a new segment, eg. when row cache is
populated. If large allocations happen for certain period only inside
row_cache::update(), then eviction will not be able to make forward
progress because cache's LSA region is locked inside
row_cache::update(). While it's locked, data can't be evicted from
it.
The solution is to use allocating_section.
Fixes#376.
map_reduce() can run the reducer out-of-order which breaks the MD5 hash.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Fixes#357. [tgrabiec]
scanning_reader has a bug in its range support when it iterates over a
memtable which is still open, and thus might still be modified between
calls to the read function.
This caused, among other things, issue #368 - where repair was reading
a memtable which was still open and being written to (by a stream from a
a remote node).
The problem is that scanning_reader has an optimization so it can avoid
comparing the current partition with the range's end on every iteration:
It finds, once, a pointer to the element past the end of the range (the
so-called "upper bound"), and saves this pointer in _end. Then at every
iteration, we can just compare pointers.
But If partitions are added to the memtable, the _end we saved is no longer
relevant: It still points to a valid partition, but this partition which
was once the first partition *after* the range, may now be precedeed by
many new partitions, which may be now returned despite being after the
range's end.
The fix is to re-calculate "_end" if partitions were added to the memtable.
Moreover, we also need to re-calculate "_i" in this case - the current code
calculates in one iteration a pointer, _i, to the element to be returned in
the *next* iteration. If additional partitions were added in the meantime,
we may need to return them.
Because it's impossible to delete partitions from a memtable (just to
add new ones or modify existing ones), we can trivially figure out if
new partitions were added, using _memtable->partition_count(). Because
boost::intrusive::set defaults to constant_time_size(true), using this
count is efficient.
Fixes#368.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
The debug build uncovered this typo. It was setting a class member with
itself (with an undefined value) instead from the parameter, which I was
surprised the compiler didn't catch at compile time.
Discovered in issue #368.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
For unknown reasons, I saw gossip syn message got rpc timeout erros when
the cluster is under heavy cassandra-strss stress.
Using a standalone tcp connection seems to fix the issue.
Restrict the impact of flushing a memtable to row_cache to 20% of the
cpu. This is accomplished by converting the code to a thread (with
bad indentation to improve patch readability) and using a thread
scheduling group.
Using the base fedora22 image there are many updates - for an unknown
reason after doing all the rpm installs we are getting
amazon-ebs:
amazon-ebs: Complete!
amazon-ebs: Failed to execute operation: Access denied
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: No AMIs to cleanup
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' errored: Script exited with non-zero exit status: 1
The workaround is to create fedora22 image that already pulled the
updates
Signed-off-by: Shlomi Livne <shlomi@cloudius-systems.com>
- no need to create the binary rpm twice - we are using the mock version
- this is causing issues on jenkins as we build rpms on it only via mock
Signed-off-by: Shlomi Livne <shlomi@cloudius-systems.com>
We remove the poller too early; after _ready_to_respond becomes ready,
it is likely to have been inserted again.
Fix by moving it after _ready_to_respond.
Refs #356
Pre-allocates N segments from timer task. N is "adaptive" in that it is
increased (to a max) every time segement acquisition is forced to allocate
a new instead of picking from pre-alloc (reserve) list. The idea is that it is
easier to adapt how many segments we consume per timer quanta than the timer
quanta itself.
Also does disk pressure check and flush from timer task now. Note that the
check is still only done max once every new segment.
Some logging cleanup/betterment also to make behaviour easier to trace.
Reserve segments start out at zero length, and are still deleted when finished.
This is because otherwise we'd still have to clear the file to be able to
properly parse it later (given that is can be a "half" file due to power fail
etc). This might need revisiting as well.
With this patch, there should be no case (except flush starvation) where
"add_mutation" actually waits for a (potentially) blocking op (disk).
Note that since the amount of reserve is increased as needed, there will
be occasional cases where a new segment is created in the alloc path
until the system finds equilebrium. But this should only be during a breif
warmup.
This address issue #154
Flush command should wait for the commmand completion before returning.
This change replaces the for loop with a parallel_for_each, it will now
wait for all the flushes to complete before returning.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
the tar file prefix needs to be only the version without the release
without this bug I get
.
.
.
Finish: build setup for scylla-server-0.8-20150917.2d99476.fc21.src.rpm
Start: rpmbuild scylla-server-0.8-20150917.2d99476.fc21.src.rpm
Building target platforms: x86_64
Building for target x86_64
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.fA7nBm
+ umask 022
+ cd /builddir/build/BUILD
+ cd /builddir/build/BUILD
+ rm -rf scylla-server-0.8
+ /usr/bin/tar -xf
/builddir/build/SOURCES/scylla-server-0.8-20150917.2d99476.tar
+ cd scylla-server-0.8
/var/tmp/rpm-tmp.fA7nBm: line 33: cd: scylla-server-0.8: No such file or
directory
RPM build errors:
error: Bad exit status from /var/tmp/rpm-tmp.fA7nBm (%prep)
Bad exit status from /var/tmp/rpm-tmp.fA7nBm (%prep)
ERROR:
Exception(build/rpmbuild/SRPMS/scylla-server-0.8-20150917.2d99476.fc21.src.rpm)
Config(fedora-21-x86_64) 4 minutes 17 seconds
Signed-off-by: Shlomi Livne <shlomi@cloudius-systems.com>
When the cluster is under heavy load, the time to exchange a gossip
message might take longer than 1s. Let's make the timeout longer for now
before we can solve the large delay of gossip message issue.
Patch "Fix some timing/latency issues with sync" changed new_segment to
_not_ wait for flush to finish. This means that checking actual files on
disk in the test case might race.
Lucklily, we can more or less just check the segment list instead
(added recently-ish)
Refs #356
* Move sync time setting to sync initiate to help prevent double syncs
* Change add_mutation to only do explicit sync with wait if time elapsed
since last is 2x sync window
* Do not wait for sync when moving to new segment in alloc path
* Initiate _sync_time properly.
* Add some tracing log messages to help debug
Race condition happens when two or more shards will try to delete
the same partial sstable. So the problem doesn't affect scylla
when it boots with a single shard.
To fix this problem, shard 0 will be made the responsible for
deleting a partial sstable.
fixes#359.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
From Avi:
We currently send out each cql transport response in its own packet, which
is very inefficient.
Use a poller to schedule responses to be flushed out, which allows multiple
responses to be sent out in one packet, reducing tcp stack overhead.
I see ~50% improvement with this on my desktop (single core).
