Commit Graph

9276 Commits

Author SHA1 Message Date
Gleb Natapov
dad312ce0a tests: test for result row counting
Message-Id: <1462377579-2419-2-git-send-email-gleb@scylladb.com>
(cherry picked from commit f1cd52ff3f)
2016-05-06 13:32:36 +03:00
Gleb Natapov
3cae56f3e3 query: fix result row counting for results with multiple partitions
Message-Id: <1462377579-2419-1-git-send-email-gleb@scylladb.com>
(cherry picked from commit b75475de80)
2016-05-06 13:32:29 +03:00
Calle Wilund
656a10c4b8 storage_service: Add logging to match origin
Pointing out if CQL server is listing in SSL mode.
Message-Id: <1462368016-32394-2-git-send-email-calle@scylladb.com>

(cherry picked from commit 709dd82d59)
2016-05-06 13:30:29 +03:00
Calle Wilund
c04b3de564 messaging_service: Add logging to match origin
To announce rpc port + ssl if on.

Message-Id: <1462368016-32394-1-git-send-email-calle@scylladb.com>
(cherry picked from commit d8ea85cd90)
2016-05-06 13:26:01 +03:00
Gleb Natapov
4964fe4cf0 storage_proxy: stop range query with limit after the limit is reached
(cherry picked from commit 3039e4c7de)
2016-05-06 12:50:51 +03:00
Gleb Natapov
e3ad3cf7d9 query: put live row count into query::result
The patch calculates row count during result building and while merging.
If one of results that are being merged does not have row count the
merged result will not have one either.

(cherry picked from commit db322d8f74)
2016-05-06 12:50:47 +03:00
Gleb Natapov
cef40627a7 storage_proxy: fix calculation of concurrency queried ranges
(cherry picked from commit 41c586313a)
2016-05-06 12:50:37 +03:00
Gleb Natapov
995820c08a storage_proxy: add logging for range query row count estimation
(cherry picked from commit c364ab9121)
2016-05-06 12:50:32 +03:00
Calle Wilund
b78abd7649 auth: Make auth.* schemas use deterministic UUIDs
In initial implementation I figured this was not required, but
we get issues communicating across nodes if system tables
don't have the same UUID, since creation is forcefully local, yet
shared.

Just do a manual re-create of the scema with a name UUID, and
use migration manager directly.
Message-Id: <1462194588-11964-1-git-send-email-calle@scylladb.com>

(cherry picked from commit 6d2caedafd)
2016-05-03 10:49:16 +03:00
Calle Wilund
d47c62b51c messaging_service: Change init to use per-shard tls credentials
Fixes: #1220

While the server_credentials object is technically immutable
(esp with last change in seastar), the ::shared_ptr holding them
is not safe to share across shards.

Pre-create cpu x credentials and then move-hand them out in service
start-up instead.

Fixes assertion error in debug builds. And just maybe real memory
corruption in release.

Requires seastar tls change:
"Change server_credentials to copy dh_params input"

Message-Id: <1462187704-2056-1-git-send-email-calle@scylladb.com>
(cherry picked from commit 751ba2f0bf)
2016-05-02 15:37:43 +03:00
Pekka Enberg
bef19e7f9e release: prepare for 1.1.rc1 2016-04-29 08:49:10 +03:00
Takuya ASADA
3ec47fbcf0 dist/ubuntu: unofficial support Debian 8.4
Unofficial support for Debian 8.4.
Now we supported both ubuntu and debian, but keep directory name as 'dist/ubuntu' for now.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1461006868-28273-1-git-send-email-syuu@scylladb.com>
2016-04-27 15:39:20 +03:00
Pekka Enberg
31090f3116 Merge "Fix for systemd support on Ubuntu, add Ubuntu 16.04 support" from Takuya
"This is bug fix for systemd support on Ubuntu, and add Ubuntu 16.04 support."
2016-04-27 15:37:25 +03:00
Takuya ASADA
1cfde50102 dist/ubuntu: support 16.04
Drop 'unsupported release' message on 16.04.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-04-27 18:06:59 +09:00
Takuya ASADA
988b7bcd3d dist/ubuntu: don't use ubuntu-toolchain-r/test ppa repo on recent versions of Ubuntu, since it has newer g++
On Ubuntu 15.04 and newer, official g++ package is >= g++-4.9.
So we don't need to use development repository, just use official package.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-04-27 18:06:59 +09:00
Takuya ASADA
fa0b90b727 dist/ubuntu: add dependency for libsystemd-dev to handle startup correctly on recent versions of Ubuntu
To handle scylla startup correctly on systemd versions of Ubuntu, scylla requires to build with libsystemd-dev.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-04-27 18:06:59 +09:00
Takuya ASADA
eae881ff70 dist/ubuntu: skip dh_installinit --upstart-only on recent versions of Ubuntu
Since 16.04LTS does not support this argument anymore, drop it on recent version of Ubuntu which does not uses Upstart.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-04-27 18:06:59 +09:00
Takuya ASADA
d5efa02eab dist/ubuntu/dep: Drop python-support on Ubuntu 16.04
Ubuntu 16.04 seems dropped python-support, so remove it from thrift package.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-04-27 18:06:59 +09:00
Takuya ASADA
e733c2aae8 dist/ubuntu/dep: use distribution's thrift-compiler-0.9.1 on newer versions of Ubuntu
Use distribution's thrift if version > 14.04LTS.
14.04LTS doesn't have thrift-compiler-0.9.1, use our version.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-04-27 18:06:59 +09:00
Avi Kivity
ad9e75a3fa Merge seastar upstream
* seastar 15a92cf...dab58e4 (6):
  > tls: Fix tls sink::put so it deals with larger packets
  > tls: Change server_credentials to copy dh_params input
  > seastar thread: allow the thread_scheduling_group's usage fraction to change
  > seastar::async allow passing an attribute
  > thread: document undocumented classes
  > fair_queue: fix inconsistency during renormalization
2016-04-27 10:39:09 +03:00
Avi Kivity
454512a272 dist/redhat: package scylla_kernel_check
Can't build rpm without this.
Message-Id: <1461683947-30356-1-git-send-email-avi@scylladb.com>
2016-04-27 08:38:48 +03:00
Tomasz Grabiec
61435108a5 query: Do not take arguments via ... in the visitor
Amnon reports that current code fails to compile on gcc 4.9:

distcc[9700] ERROR: compile /home/amnon/.ccache/tmp/query.tmp.localhost.localdomain.9673.ii on localhost failed
In file included from query.cc:30:0:
query-result-reader.hh: In instantiation of ‘void query::result_view::consume(const query::partition_slice&, ResultVisitor&&) [with ResultVisitor = query::result::calculate_row_count(const query::partition_slice&)::<anonymous struct>&]’:
query.cc:196:32:   required from here
query-result-reader.hh:184:21: error: cannot pass objects of non-trivially-copyable type ‘class clustering_key_prefix’ through ‘...’
                     visitor.accept_new_row(*row.key(), static_row, view);
                     ^
query-result-reader.hh:184:21: error: cannot pass objects of non-trivially-copyable type ‘class query::result_row_view’ through ‘...’
query-result-reader.hh:184:21: error: cannot pass objects of non-trivially-copyable type ‘class query::result_row_view’ through ‘...’
query-result-reader.hh:186:21: error: cannot pass objects of non-trivially-copyable type ‘class query::result_row_view’ through ‘...’
                     visitor.accept_new_row(static_row, view);
                     ^
query-result-reader.hh:186:21: error: cannot pass objects of non-trivially-copyable type ‘class query::result_row_view’ through ‘...’

Work around the problem by not using '...'.
Message-Id: <1460964042-2867-1-git-send-email-tgrabiec@scylladb.com>
2016-04-26 14:50:35 +03:00
Takuya ASADA
eb9bd3ee21 dist/common/scripts: show knowledge base URL when kernel is too old
To explain why this kernel is not supported, we need to show kb URL here.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1461644708-32078-1-git-send-email-syuu@scylladb.com>
2016-04-26 14:43:10 +03:00
Takuya ASADA
05ac4bb99d dist/common/scripts: notice restart required after changing bootparameters
Fixes #1115

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1459330851-32470-1-git-send-email-syuu@scylladb.com>
2016-04-26 14:41:49 +03:00
Tomasz Grabiec
88bb5fcb53 api: Fix error message
Keyspace and table names are separated by a single colon.
Message-Id: <1461600269-4070-1-git-send-email-tgrabiec@scylladb.com>
2016-04-26 08:40:28 +03:00
Takuya ASADA
e7f438eeae dist/ubuntu: Drop dependency to libthrift0, link it statically
Drop dependency to libthrift0 on installation time, link libthrift statically.
With this fix, we don't need to distribute libthrift0 deb package anymore to install scylla-server binary package.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1461594460-2403-2-git-send-email-syuu@scylladb.com>
2016-04-25 17:44:46 +03:00
Takuya ASADA
ec2ef467c8 configure.py: configure.py: add --static-thrift option to link libthrift statically
This is needed for Ubuntu packaging, to drop dependency to libthrift0 on installation time.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1461594460-2403-1-git-send-email-syuu@scylladb.com>
2016-04-25 17:44:44 +03:00
Avi Kivity
af803a9149 Merge seastar upstream
* seastar 2b3c363...15a92cf (2):
  > smp: allow more than 128 in-flight operations on core-to-core queue
  > future: balance constructors and destructors in future_state<>

Fixes #1205.
2016-04-25 13:34:27 +03:00
Calle Wilund
cdd0f00de5 client_state: Remove unwarranted keyspace check
"has_keyspace_access" is not supposed to (according to origin)
verify that a keyspace exists. Remove.
It (and all others) are however supposed to check "ks" (name)
not empty. Add this.
Message-Id: <1461578072-24113-1-git-send-email-calle@scylladb.com>
2016-04-25 13:16:36 +03:00
Calle Wilund
49d3d79dfe sstables: Fix compilation error on boost 1.55
Message-Id: <1461067254-526-2-git-send-email-calle@scylladb.com>
2016-04-25 12:54:44 +03:00
Calle Wilund
9130b0de16 database.cc: Fix compilation error with boost 1.55
Message-Id: <1461067254-526-1-git-send-email-calle@scylladb.com>
2016-04-25 12:54:43 +03:00
Takuya ASADA
c657a431dc dist/common/scripts: Fix incorrect order to run scylla_sysconfig_setup on scylla_setup
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1461174517-12441-2-git-send-email-syuu@scylladb.com>
2016-04-25 11:09:49 +03:00
Takuya ASADA
9a99231f6b dist/common/scripts: On scylla_setup, skip showing 'lo' interface on sysconfig prompt
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1461174517-12441-1-git-send-email-syuu@scylladb.com>
2016-04-25 11:09:48 +03:00
Takuya ASADA
611b0a3400 dist/common/scripts: Add kernel version check
Check kernel version at beginning of scylla_setup, show error when kernel is too old.
Use iotune --fs-check to check kernel.

Fixes #1116

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1459886738-10882-1-git-send-email-syuu@scylladb.com>
2016-04-24 17:47:13 +03:00
Vlad Zolotarov
813ad4024f query_processor: account unprepared statements executions
Add the statistics counter for a number of unprepared statements
executions and expose it with collectd.

Since in our implementation a number of unprepared statements executions
equals to a number of executions of prepare() function we may simply
increment the new statistics counter every time query_processor::get_statement()
is called.

Fixes #1068

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Message-Id: <1461503492-32228-1-git-send-email-vladz@cloudius-systems.com>
2016-04-24 16:55:15 +03:00
Avi Kivity
c6b5890eb2 Merge 2016-04-24 16:17:00 +03:00
Pekka Enberg
f6da9bc92b Merge "Additional mutations/queries related collectd metrics" from Vlad
"This series introduces some additional metrics (mostly) in a storage_proxy and
a database level that are meant to create a better picture of how data flows
in the cluster.

First of all where possible counters of each category (e.g. total writes in the storage
proxy level) are split into the following categories:
   - operations performed on a local Node
   - operations performed on remote Nodes aggregated per DC

In a storage_proxy level there are the following metrics that have this "split"
nature (all on a sending side):
   - total writes (attempts/errors)
   - writes performed as a result of a Read Repair logic
   - total data reads (attempts/completed/errors)
   - total digest reads (attempts/completed/errors)
   - total mutations data reads (attempts/completed/errors)

In a batchlog_manager:
   - writes performed as a result of a batchlog replay logic

Thereby if for instance somebody wants to get an idea of how many writes
the current Node performs due to user requested mutations only he/she has
to take a counter of total writes and subtract the writes resulted by Read
Repairs and batchlog replays.

On a receiving side of a storage_proxy we add the two following counters:
   - total number of received mutations
   - total number of forwarded mutations (attempts/errors)

In order to get a better picture of what is going on on a local Node
we are adding two counters on a database level:
   - total number of writes
   - total number of reads

Comparing these to total writes/reads in a storage_proxy may give a good
idea if there is an excessive access to a local DB for example."
2016-04-21 15:58:45 +03:00
Takuya ASADA
2bfc8e8c12 main: add tcp_syncookies sanity check
Check net.ipv4.tcp_syncookies, show error message when it set to 0.
Fixes #1118

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1460738415-3798-1-git-send-email-syuu@scylladb.com>
2016-04-21 14:55:26 +03:00
Pekka Enberg
3f1fcca3bc cql3: Fix DROP KEYSPACE error message when keyspace does not exist
Commit d3fe0c5 ("Refactor db/keyspace/column_family toplogy") changed
database::find_keyspace() to throw a std::nested_exception so the catch
block in migration_manager::announce_keyspace_drop() no longer catches
the exception. Fix the issue by explicitly checking if the keyspace
exists and throwing the correct exception type if it doesn't.

Fixes TestCQL.keyspace_test.
Message-Id: <1461218910-26691-1-git-send-email-penberg@scylladb.com>
2016-04-21 12:42:45 +02:00
Vlad Zolotarov
4ef5b11e9b batchlog_manager: add a counter for a total number of write attempts
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2016-04-21 11:29:21 +03:00
Vlad Zolotarov
97e5bfa815 database: add metrics for total writes and reads
This patch adds a counter of total writes and reads
for each shard.

It seems that nothing ensures that all database queries are
ready before database object is destroyed.
Make _stats lw_shared_ptr in order to ensure that the object is
alive when lambda gets to incrementing it.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2016-04-21 11:28:53 +03:00
Vlad Zolotarov
9bf8253412 storage_proxy: add read requests split counters
Add split (local Nodes, external Nodes aggregated per Nodes' DCs) counters
for the following read categories:
   - data reads
   - digest reads
   - mutation data reads

Each category is added attempts, completions and errors metrics.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2016-04-21 11:28:19 +03:00
Vlad Zolotarov
cbcbdc3b4a storage_proxy: add split counters for writes
Added split metrics for operations on a local Node and on external
Nodes aggregated per Nodes' DCs.

Added separate split counters for:
    - total writes attempts/errors
    - read repair write attempts (there is no easy way to separate errors
      at the moment)

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2016-04-21 11:28:15 +03:00
Vlad Zolotarov
c92654b281 storage_proxy: add counters for received and forwarded mutations
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2016-04-21 11:27:29 +03:00
Piotr Jastrzebski
8231385e0c sstables: Remove unused code from mp_row_consumer
_mutation_to_subscription is not used anywhere so
it should probably be removed.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <90ef62daee0c183b29dcb86d08843145d657ea38.1461179970.git.piotr@scylladb.com>
2016-04-20 23:10:43 +03:00
Raphael S. Carvalho
eb51c93a5a tests: fix use-after-free in sstable test
After commit a843aea547, a gate was introduced to make sure that
an asynchronous operation is finished before column family is
destroyed. A sstable testcase was not stopping column family,
instead it just removed column family from compaction manager.
That could cause an user-after-free if column family is destroyed
while the asynchronous operation is running. Let's fix it by
stopping column family in the test.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <ed910ec459c1752148099e6dc503e7f3adee54da.1461177411.git.raphaelsc@scylladb.com>
2016-04-20 22:08:08 +03:00
Pekka Enberg
7af9ac2880 Merge "Add support for User Defined Types" from Duarte
"This patchset enables support for user defined types,
 completing the functionality that was already in place.

 Fixes #426"
2016-04-20 21:26:03 +03:00
Yoav Kleinberger
1543253bfd scyllatop: differentiate metrics coming from different hosts
Fix issue #1173.
Previously scyllatop aggregated metrics coming from a cluster with many
hosts so that individual contributions could not be recognized. This is
now changed so that aggregation is also by hostname.

Signed-off-by: Yoav Kleinberger <yoav@scylladb.com>
Message-Id: <8a4d8b82216d8c1aa855026ff31bcfd8bfac7e47.1461150261.git.yoav@scylladb.com>
2016-04-20 20:20:09 +02:00
Duarte Nunes
c04f8c239e udt: Enable user type query test case
This patch enables the test case for user defined types in
cql_query_test.

Fixes #426

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-04-20 18:07:07 +02:00
Duarte Nunes
bc90d6a730 udt: type_parser handles user defined types
This patch ensures type_parser can handle user defined types. It also
prefixes user_type_impl::make_name() with
org.apache.cassandra.db.marshal.UserType.

Fixes #631

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-04-20 18:07:07 +02:00