Leveled compaction strategy is doing a lot of work whenever it's asked to get
a list of sstables to be compacted. It's checking if a sstable overlaps with
another sstable in the same level twice. First, when adding a sstable to a
list with sstables at the same level. Second, after adding all sstables to
their respective lists.
It's enough to check that a sstable creates an overlap in its level only once.
So I am changing the code to unconditionally insert a sstable to its respective
list, and after that, it will call repair_overlapping_sstables() that will send
any sstable that creates an overlap in its level to L0 list.
By the way, the optimization isn't in the compaction itself, instead in the
strategy code that gets a set of sstables to be compacted.
Reviewed-by: Nadav Har'El <nyh@scylladb.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <8c8526737277cb47987a3a5dbd5ff3bb81a6d038.1461965074.git.raphaelsc@scylladb.com>
(cherry picked from commit ae95ce1bd7)
Refresh will rewrite statistics of any migrated sstable with level
> 0. However, this operation is currently not working because O_EXCL
flag is used, meaning that create will fail.
It turns out that we don't actually need to change on-disk level of
a sstable by overwriting statistics file.
We can only set in-memory level of a sstable to 0. If Scylla reboots
before all migrated sstables are compacted, leveled strategy is smart
enough to detect sstables that overlap, and set their in-memory level
to 0.
Fixes#1124.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
(cherry picked from commit ee0f66eef6)
Avi says:
"During shutdown, we prevent new compactions, but perhaps too late.
Memtables are flushed and these can trigger compaction."
To solve that, let's stop compaction manager at a very early step
of shutdown. We will still try to stop compaction manager in
database::stop() because user may ask for a shutdown before scylla
was fully started. It's fine to stop compaction manager twice.
Only the first call will actually stop the manager.
Fixes#1238.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <c64ab11f3c91129c424259d317e48abc5bde6ff3.1462496694.git.raphaelsc@scylladb.com>
(cherry picked from commit bf18025937)
In perf-flame, I saw in
service::storage_proxy::create_write_response_handler (2.66% cpu)
gossiper::is_alive takes 0.72% cpu
locator::token_metadata::pending_endpoints_for takes 1.2% cpu
After this patch:
service::storage_proxy::create_write_response_handler (2.17% cpu)
gossiper::is_alive does not show up at all
locator::token_metadata::pending_endpoints_for takes 1.3% cpu
There is no need to copy the endpoint_state from the endpoint_state_map
to check if a node is alive. Optimize it since gossiper::is_alive is
called in the fast path.
Message-Id: <2144310aef8d170cab34a2c96cb67cabca761ca8.1463540290.git.asias@scylladb.com>
(cherry picked from commit eb9ac9ab91)
pending_endpoints_for is called frequently by
storage_proxy::create_write_response_handler when doing cql query.
Before this patch, each call to pending_endpoints_for involves
converting a multimap (std::unordered_multimap<range<token>,
inet_address>>) to map (std::unordered_map<range<token>,
std::unordered_set<inet_address>>).
To speed up the token to pending endpoint mapping search, a interval map
is introduced. It is faster than searching the map linearly and can
avoid caching the token/pending endpoint mapping.
With this patch, the operations per second drop during adding node
period gets much better.
Before:
45K to 10K
After:
45k to 38K
(The number is measured with the streaming code skipping to send data to
rule out the streaming factor.)
Refs: #1223
(cherry picked from commit 089734474b)
Spotted during code review.
If it doesn't defer, we may execute then_wrapped() body before we
change the state. Fix by moving then_wrapped() body after state changes.
(cherry picked from commit 443e5aef5a)
The problem was that "s" would not be marked as synced-with if it came from
shard != 0.
As a result, mutation using that schema would fail to apply with an exception:
"attempted to mutate using not synced schema of ..."
The problem could surface when altering schema without changing
columns and restarting one of the nodes so that it forgets past
versions.
Fixes#1258.
Will be covered by dtest:
SchemaManagementTest.test_prepared_statements_work_after_node_restart_after_altering_schema_without_changing_columns
(cherry picked from commit 8703136a4f)
Currently we only do that when column set changes. When prepared
statements are executed, paramaters like read repair chance are read
from schema version stored in the statement. Not invalidating prepared
statements on changes of such parameters will appear as if alter took
no effect.
Fixes#1255.
Message-Id: <1462985495-9767-1-git-send-email-tgrabiec@scylladb.com>
(cherry picked from commit 13d8cd0ae9)
1.) It most likely is not, i.e. either tcp or more likely, ssl
negotiation failure. In any case, we can still try next
connection.
2.) Not retrying will cause us to "leak" the accept, and then hang
on shutdown.
Also, promote logging message on accept exception to "warn", since
dtest(s?) depend on seeing log output.
Message-Id: <1462283265-27051-4-git-send-email-calle@scylladb.com>
(cherry picked from commit 917bf850fa)
To simplify init of msg service, use credendials_builder
to encapsulate tls options so actual credentials can be
more easily created in each shard.
Message-Id: <1462283265-27051-2-git-send-email-calle@scylladb.com>
(cherry picked from commit 58f7edb04f)
* seastar 73d5583...85bdfb7 (4):
> tests/mkcert.gmk: Fix makefile bug in snakeoil cert generator
> tls_test: Add case to do a little checking of credentials_builder
> tls: Add credentials_builder - copyable credentials "factory"
> tls_test: Add test for large-ish buffer send/recieve
* seastar dab58e4...73d5583 (2):
> iotune: don't coredump when directory fails to be created
> iotune: improve recommendation in case we timeout
Fixes#1243.
The patch calculates row count during result building and while merging.
If one of results that are being merged does not have row count the
merged result will not have one either.
(cherry picked from commit db322d8f74)
In initial implementation I figured this was not required, but
we get issues communicating across nodes if system tables
don't have the same UUID, since creation is forcefully local, yet
shared.
Just do a manual re-create of the scema with a name UUID, and
use migration manager directly.
Message-Id: <1462194588-11964-1-git-send-email-calle@scylladb.com>
(cherry picked from commit 6d2caedafd)
Fixes: #1220
While the server_credentials object is technically immutable
(esp with last change in seastar), the ::shared_ptr holding them
is not safe to share across shards.
Pre-create cpu x credentials and then move-hand them out in service
start-up instead.
Fixes assertion error in debug builds. And just maybe real memory
corruption in release.
Requires seastar tls change:
"Change server_credentials to copy dh_params input"
Message-Id: <1462187704-2056-1-git-send-email-calle@scylladb.com>
(cherry picked from commit 751ba2f0bf)
On Ubuntu 15.04 and newer, official g++ package is >= g++-4.9.
So we don't need to use development repository, just use official package.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
To handle scylla startup correctly on systemd versions of Ubuntu, scylla requires to build with libsystemd-dev.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Since 16.04LTS does not support this argument anymore, drop it on recent version of Ubuntu which does not uses Upstart.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Use distribution's thrift if version > 14.04LTS.
14.04LTS doesn't have thrift-compiler-0.9.1, use our version.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Amnon reports that current code fails to compile on gcc 4.9:
distcc[9700] ERROR: compile /home/amnon/.ccache/tmp/query.tmp.localhost.localdomain.9673.ii on localhost failed
In file included from query.cc:30:0:
query-result-reader.hh: In instantiation of ‘void query::result_view::consume(const query::partition_slice&, ResultVisitor&&) [with ResultVisitor = query::result::calculate_row_count(const query::partition_slice&)::<anonymous struct>&]’:
query.cc:196:32: required from here
query-result-reader.hh:184:21: error: cannot pass objects of non-trivially-copyable type ‘class clustering_key_prefix’ through ‘...’
visitor.accept_new_row(*row.key(), static_row, view);
^
query-result-reader.hh:184:21: error: cannot pass objects of non-trivially-copyable type ‘class query::result_row_view’ through ‘...’
query-result-reader.hh:184:21: error: cannot pass objects of non-trivially-copyable type ‘class query::result_row_view’ through ‘...’
query-result-reader.hh:186:21: error: cannot pass objects of non-trivially-copyable type ‘class query::result_row_view’ through ‘...’
visitor.accept_new_row(static_row, view);
^
query-result-reader.hh:186:21: error: cannot pass objects of non-trivially-copyable type ‘class query::result_row_view’ through ‘...’
Work around the problem by not using '...'.
Message-Id: <1460964042-2867-1-git-send-email-tgrabiec@scylladb.com>
Drop dependency to libthrift0 on installation time, link libthrift statically.
With this fix, we don't need to distribute libthrift0 deb package anymore to install scylla-server binary package.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1461594460-2403-2-git-send-email-syuu@scylladb.com>
* seastar 2b3c363...15a92cf (2):
> smp: allow more than 128 in-flight operations on core-to-core queue
> future: balance constructors and destructors in future_state<>
Fixes#1205.
