Commit Graph

9287 Commits

Author SHA1 Message Date
Tomasz Grabiec
dce549f44f tests: Add unit tests for schema_registry
(cherry picked from commit 90c31701e3)
scylla-1.1
2016-05-16 10:52:02 +03:00
Tomasz Grabiec
26a3302957 schema_registry: Fix possible hang in maybe_sync() if syncer doesn't defer
Spotted during code review.

If it doesn't defer, we may execute then_wrapped() body before we
change the state. Fix by moving then_wrapped() body after state changes.

(cherry picked from commit 443e5aef5a)
2016-05-16 10:51:54 +03:00
Tomasz Grabiec
f796d8081b migration_manager: Fix schema syncing with older version
The problem was that "s" would not be marked as synced-with if it came from
shard != 0.

As a result, mutation using that schema would fail to apply with an exception:

  "attempted to mutate using not synced schema of ..."

The problem could surface when altering schema without changing
columns and restarting one of the nodes so that it forgets past
versions.

Fixes #1258.

Will be covered by dtest:

  SchemaManagementTest.test_prepared_statements_work_after_node_restart_after_altering_schema_without_changing_columns

(cherry picked from commit 8703136a4f)
2016-05-16 10:51:48 +03:00
Pekka Enberg
b850cb991c release: prepare for 1.1.0 2016-05-16 09:33:26 +03:00
Tomasz Grabiec
734cfa949a migration_manager: Invalidate prepared statements on every schema change
Currently we only do that when column set changes. When prepared
statements are executed, paramaters like read repair chance are read
from schema version stored in the statement. Not invalidating prepared
statements on changes of such parameters will appear as if alter took
no effect.

Fixes #1255.
Message-Id: <1462985495-9767-1-git-send-email-tgrabiec@scylladb.com>

(cherry picked from commit 13d8cd0ae9)
2016-05-12 09:18:00 +03:00
Calle Wilund
3606e3ab29 transport::server: Do not treat accept exception as fatal
1.) It most likely is not, i.e. either tcp or more likely, ssl
    negotiation failure. In any case, we can still try next
    connection.
2.) Not retrying will cause us to "leak" the accept, and then hang
    on shutdown.

Also, promote logging message on accept exception to "warn", since
dtest(s?) depend on seeing log output.

Message-Id: <1462283265-27051-4-git-send-email-calle@scylladb.com>
(cherry picked from commit 917bf850fa)
2016-05-10 19:26:29 +03:00
Calle Wilund
014284de00 cql_server: Use credentials_builder to init tls
Slightly cleaner, and shard-safe tls init.

Message-Id: <1462283265-27051-3-git-send-email-calle@scylladb.com>
(cherry picked from commit 437ebe7128)
2016-05-10 19:26:23 +03:00
Calle Wilund
f17764e74a messaging_service: Change tls init to use credentials_builder
To simplify init of msg service, use credendials_builder
to encapsulate tls options so actual credentials can be
more easily created in each shard.

Message-Id: <1462283265-27051-2-git-send-email-calle@scylladb.com>
(cherry picked from commit 58f7edb04f)
2016-05-10 19:26:18 +03:00
Avi Kivity
8643028d0c Update seastar submodule
* seastar 73d5583...85bdfb7 (4):
  > tests/mkcert.gmk: Fix makefile bug in snakeoil cert generator
  > tls_test: Add case to do a little checking of credentials_builder
  > tls: Add credentials_builder - copyable credentials "factory"
  > tls_test: Add test for large-ish buffer send/recieve
2016-05-10 19:24:49 +03:00
Avi Kivity
a35f1d765a Backport seastar iotune fixes
* seastar dab58e4...73d5583 (2):
  > iotune: don't coredump when directory fails to be created
  > iotune: improve recommendation in case we timeout

Fixes #1243.
2016-05-09 10:49:15 +03:00
Avi Kivity
3116a92b0e Point seastar submodule at scylla-seastar repository
Allows us to backport seastar fixes to branch-1.1.
2016-05-08 14:49:02 +03:00
Gleb Natapov
dad312ce0a tests: test for result row counting
Message-Id: <1462377579-2419-2-git-send-email-gleb@scylladb.com>
(cherry picked from commit f1cd52ff3f)
2016-05-06 13:32:36 +03:00
Gleb Natapov
3cae56f3e3 query: fix result row counting for results with multiple partitions
Message-Id: <1462377579-2419-1-git-send-email-gleb@scylladb.com>
(cherry picked from commit b75475de80)
2016-05-06 13:32:29 +03:00
Calle Wilund
656a10c4b8 storage_service: Add logging to match origin
Pointing out if CQL server is listing in SSL mode.
Message-Id: <1462368016-32394-2-git-send-email-calle@scylladb.com>

(cherry picked from commit 709dd82d59)
2016-05-06 13:30:29 +03:00
Calle Wilund
c04b3de564 messaging_service: Add logging to match origin
To announce rpc port + ssl if on.

Message-Id: <1462368016-32394-1-git-send-email-calle@scylladb.com>
(cherry picked from commit d8ea85cd90)
2016-05-06 13:26:01 +03:00
Gleb Natapov
4964fe4cf0 storage_proxy: stop range query with limit after the limit is reached
(cherry picked from commit 3039e4c7de)
2016-05-06 12:50:51 +03:00
Gleb Natapov
e3ad3cf7d9 query: put live row count into query::result
The patch calculates row count during result building and while merging.
If one of results that are being merged does not have row count the
merged result will not have one either.

(cherry picked from commit db322d8f74)
2016-05-06 12:50:47 +03:00
Gleb Natapov
cef40627a7 storage_proxy: fix calculation of concurrency queried ranges
(cherry picked from commit 41c586313a)
2016-05-06 12:50:37 +03:00
Gleb Natapov
995820c08a storage_proxy: add logging for range query row count estimation
(cherry picked from commit c364ab9121)
2016-05-06 12:50:32 +03:00
Calle Wilund
b78abd7649 auth: Make auth.* schemas use deterministic UUIDs
In initial implementation I figured this was not required, but
we get issues communicating across nodes if system tables
don't have the same UUID, since creation is forcefully local, yet
shared.

Just do a manual re-create of the scema with a name UUID, and
use migration manager directly.
Message-Id: <1462194588-11964-1-git-send-email-calle@scylladb.com>

(cherry picked from commit 6d2caedafd)
2016-05-03 10:49:16 +03:00
Calle Wilund
d47c62b51c messaging_service: Change init to use per-shard tls credentials
Fixes: #1220

While the server_credentials object is technically immutable
(esp with last change in seastar), the ::shared_ptr holding them
is not safe to share across shards.

Pre-create cpu x credentials and then move-hand them out in service
start-up instead.

Fixes assertion error in debug builds. And just maybe real memory
corruption in release.

Requires seastar tls change:
"Change server_credentials to copy dh_params input"

Message-Id: <1462187704-2056-1-git-send-email-calle@scylladb.com>
(cherry picked from commit 751ba2f0bf)
2016-05-02 15:37:43 +03:00
Pekka Enberg
bef19e7f9e release: prepare for 1.1.rc1 2016-04-29 08:49:10 +03:00
Takuya ASADA
3ec47fbcf0 dist/ubuntu: unofficial support Debian 8.4
Unofficial support for Debian 8.4.
Now we supported both ubuntu and debian, but keep directory name as 'dist/ubuntu' for now.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1461006868-28273-1-git-send-email-syuu@scylladb.com>
2016-04-27 15:39:20 +03:00
Pekka Enberg
31090f3116 Merge "Fix for systemd support on Ubuntu, add Ubuntu 16.04 support" from Takuya
"This is bug fix for systemd support on Ubuntu, and add Ubuntu 16.04 support."
2016-04-27 15:37:25 +03:00
Takuya ASADA
1cfde50102 dist/ubuntu: support 16.04
Drop 'unsupported release' message on 16.04.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-04-27 18:06:59 +09:00
Takuya ASADA
988b7bcd3d dist/ubuntu: don't use ubuntu-toolchain-r/test ppa repo on recent versions of Ubuntu, since it has newer g++
On Ubuntu 15.04 and newer, official g++ package is >= g++-4.9.
So we don't need to use development repository, just use official package.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-04-27 18:06:59 +09:00
Takuya ASADA
fa0b90b727 dist/ubuntu: add dependency for libsystemd-dev to handle startup correctly on recent versions of Ubuntu
To handle scylla startup correctly on systemd versions of Ubuntu, scylla requires to build with libsystemd-dev.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-04-27 18:06:59 +09:00
Takuya ASADA
eae881ff70 dist/ubuntu: skip dh_installinit --upstart-only on recent versions of Ubuntu
Since 16.04LTS does not support this argument anymore, drop it on recent version of Ubuntu which does not uses Upstart.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-04-27 18:06:59 +09:00
Takuya ASADA
d5efa02eab dist/ubuntu/dep: Drop python-support on Ubuntu 16.04
Ubuntu 16.04 seems dropped python-support, so remove it from thrift package.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-04-27 18:06:59 +09:00
Takuya ASADA
e733c2aae8 dist/ubuntu/dep: use distribution's thrift-compiler-0.9.1 on newer versions of Ubuntu
Use distribution's thrift if version > 14.04LTS.
14.04LTS doesn't have thrift-compiler-0.9.1, use our version.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-04-27 18:06:59 +09:00
Avi Kivity
ad9e75a3fa Merge seastar upstream
* seastar 15a92cf...dab58e4 (6):
  > tls: Fix tls sink::put so it deals with larger packets
  > tls: Change server_credentials to copy dh_params input
  > seastar thread: allow the thread_scheduling_group's usage fraction to change
  > seastar::async allow passing an attribute
  > thread: document undocumented classes
  > fair_queue: fix inconsistency during renormalization
2016-04-27 10:39:09 +03:00
Avi Kivity
454512a272 dist/redhat: package scylla_kernel_check
Can't build rpm without this.
Message-Id: <1461683947-30356-1-git-send-email-avi@scylladb.com>
2016-04-27 08:38:48 +03:00
Tomasz Grabiec
61435108a5 query: Do not take arguments via ... in the visitor
Amnon reports that current code fails to compile on gcc 4.9:

distcc[9700] ERROR: compile /home/amnon/.ccache/tmp/query.tmp.localhost.localdomain.9673.ii on localhost failed
In file included from query.cc:30:0:
query-result-reader.hh: In instantiation of ‘void query::result_view::consume(const query::partition_slice&, ResultVisitor&&) [with ResultVisitor = query::result::calculate_row_count(const query::partition_slice&)::<anonymous struct>&]’:
query.cc:196:32:   required from here
query-result-reader.hh:184:21: error: cannot pass objects of non-trivially-copyable type ‘class clustering_key_prefix’ through ‘...’
                     visitor.accept_new_row(*row.key(), static_row, view);
                     ^
query-result-reader.hh:184:21: error: cannot pass objects of non-trivially-copyable type ‘class query::result_row_view’ through ‘...’
query-result-reader.hh:184:21: error: cannot pass objects of non-trivially-copyable type ‘class query::result_row_view’ through ‘...’
query-result-reader.hh:186:21: error: cannot pass objects of non-trivially-copyable type ‘class query::result_row_view’ through ‘...’
                     visitor.accept_new_row(static_row, view);
                     ^
query-result-reader.hh:186:21: error: cannot pass objects of non-trivially-copyable type ‘class query::result_row_view’ through ‘...’

Work around the problem by not using '...'.
Message-Id: <1460964042-2867-1-git-send-email-tgrabiec@scylladb.com>
2016-04-26 14:50:35 +03:00
Takuya ASADA
eb9bd3ee21 dist/common/scripts: show knowledge base URL when kernel is too old
To explain why this kernel is not supported, we need to show kb URL here.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1461644708-32078-1-git-send-email-syuu@scylladb.com>
2016-04-26 14:43:10 +03:00
Takuya ASADA
05ac4bb99d dist/common/scripts: notice restart required after changing bootparameters
Fixes #1115

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1459330851-32470-1-git-send-email-syuu@scylladb.com>
2016-04-26 14:41:49 +03:00
Tomasz Grabiec
88bb5fcb53 api: Fix error message
Keyspace and table names are separated by a single colon.
Message-Id: <1461600269-4070-1-git-send-email-tgrabiec@scylladb.com>
2016-04-26 08:40:28 +03:00
Takuya ASADA
e7f438eeae dist/ubuntu: Drop dependency to libthrift0, link it statically
Drop dependency to libthrift0 on installation time, link libthrift statically.
With this fix, we don't need to distribute libthrift0 deb package anymore to install scylla-server binary package.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1461594460-2403-2-git-send-email-syuu@scylladb.com>
2016-04-25 17:44:46 +03:00
Takuya ASADA
ec2ef467c8 configure.py: configure.py: add --static-thrift option to link libthrift statically
This is needed for Ubuntu packaging, to drop dependency to libthrift0 on installation time.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1461594460-2403-1-git-send-email-syuu@scylladb.com>
2016-04-25 17:44:44 +03:00
Avi Kivity
af803a9149 Merge seastar upstream
* seastar 2b3c363...15a92cf (2):
  > smp: allow more than 128 in-flight operations on core-to-core queue
  > future: balance constructors and destructors in future_state<>

Fixes #1205.
2016-04-25 13:34:27 +03:00
Calle Wilund
cdd0f00de5 client_state: Remove unwarranted keyspace check
"has_keyspace_access" is not supposed to (according to origin)
verify that a keyspace exists. Remove.
It (and all others) are however supposed to check "ks" (name)
not empty. Add this.
Message-Id: <1461578072-24113-1-git-send-email-calle@scylladb.com>
2016-04-25 13:16:36 +03:00
Calle Wilund
49d3d79dfe sstables: Fix compilation error on boost 1.55
Message-Id: <1461067254-526-2-git-send-email-calle@scylladb.com>
2016-04-25 12:54:44 +03:00
Calle Wilund
9130b0de16 database.cc: Fix compilation error with boost 1.55
Message-Id: <1461067254-526-1-git-send-email-calle@scylladb.com>
2016-04-25 12:54:43 +03:00
Takuya ASADA
c657a431dc dist/common/scripts: Fix incorrect order to run scylla_sysconfig_setup on scylla_setup
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1461174517-12441-2-git-send-email-syuu@scylladb.com>
2016-04-25 11:09:49 +03:00
Takuya ASADA
9a99231f6b dist/common/scripts: On scylla_setup, skip showing 'lo' interface on sysconfig prompt
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1461174517-12441-1-git-send-email-syuu@scylladb.com>
2016-04-25 11:09:48 +03:00
Takuya ASADA
611b0a3400 dist/common/scripts: Add kernel version check
Check kernel version at beginning of scylla_setup, show error when kernel is too old.
Use iotune --fs-check to check kernel.

Fixes #1116

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1459886738-10882-1-git-send-email-syuu@scylladb.com>
2016-04-24 17:47:13 +03:00
Vlad Zolotarov
813ad4024f query_processor: account unprepared statements executions
Add the statistics counter for a number of unprepared statements
executions and expose it with collectd.

Since in our implementation a number of unprepared statements executions
equals to a number of executions of prepare() function we may simply
increment the new statistics counter every time query_processor::get_statement()
is called.

Fixes #1068

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Message-Id: <1461503492-32228-1-git-send-email-vladz@cloudius-systems.com>
2016-04-24 16:55:15 +03:00
Avi Kivity
c6b5890eb2 Merge 2016-04-24 16:17:00 +03:00
Pekka Enberg
f6da9bc92b Merge "Additional mutations/queries related collectd metrics" from Vlad
"This series introduces some additional metrics (mostly) in a storage_proxy and
a database level that are meant to create a better picture of how data flows
in the cluster.

First of all where possible counters of each category (e.g. total writes in the storage
proxy level) are split into the following categories:
   - operations performed on a local Node
   - operations performed on remote Nodes aggregated per DC

In a storage_proxy level there are the following metrics that have this "split"
nature (all on a sending side):
   - total writes (attempts/errors)
   - writes performed as a result of a Read Repair logic
   - total data reads (attempts/completed/errors)
   - total digest reads (attempts/completed/errors)
   - total mutations data reads (attempts/completed/errors)

In a batchlog_manager:
   - writes performed as a result of a batchlog replay logic

Thereby if for instance somebody wants to get an idea of how many writes
the current Node performs due to user requested mutations only he/she has
to take a counter of total writes and subtract the writes resulted by Read
Repairs and batchlog replays.

On a receiving side of a storage_proxy we add the two following counters:
   - total number of received mutations
   - total number of forwarded mutations (attempts/errors)

In order to get a better picture of what is going on on a local Node
we are adding two counters on a database level:
   - total number of writes
   - total number of reads

Comparing these to total writes/reads in a storage_proxy may give a good
idea if there is an excessive access to a local DB for example."
2016-04-21 15:58:45 +03:00
Takuya ASADA
2bfc8e8c12 main: add tcp_syncookies sanity check
Check net.ipv4.tcp_syncookies, show error message when it set to 0.
Fixes #1118

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1460738415-3798-1-git-send-email-syuu@scylladb.com>
2016-04-21 14:55:26 +03:00
Pekka Enberg
3f1fcca3bc cql3: Fix DROP KEYSPACE error message when keyspace does not exist
Commit d3fe0c5 ("Refactor db/keyspace/column_family toplogy") changed
database::find_keyspace() to throw a std::nested_exception so the catch
block in migration_manager::announce_keyspace_drop() no longer catches
the exception. Fix the issue by explicitly checking if the keyspace
exists and throwing the correct exception type if it doesn't.

Fixes TestCQL.keyspace_test.
Message-Id: <1461218910-26691-1-git-send-email-penberg@scylladb.com>
2016-04-21 12:42:45 +02:00