Commit Graph

9342 Commits

Author SHA1 Message Date
Takuya ASADA
49cdd0b786 dist: move '--cpuset' and '--smp' configuration to scylla_cpuset_setup / cpuset.conf
These parameters are only required for AMI, not for non-AMI environment which want to enable SET_NIC, so split them to indivisual script / conf file, call it from AMI install script.
2016-05-19 06:25:28 +09:00
Takuya ASADA
46fa80a5a6 dist/common/scripts: replace IFNAME variable when --nic specified to scylla_sysconfig_setup
scylla_sysconfig_setup has bug that it not replaces IFNAME variable, so fixed.
2016-05-19 06:25:15 +09:00
Takuya ASADA
4972a72380 dist: drop 'sudo -E' and SETENV for security reason, source envfile from scripts
As Nadav pointed out, SETENV and sudo -E might be causes security hole:
https://github.com/scylladb/scylla/issues/1028#issuecomment-196202171
So drop them now, sourcing envfiles from scylla_prepare / scylla_stop scripts
instead.

Also on "[PATCH] ubuntu: Fix the init script variable sourcing" thread
we have problem to passing variables from envfiles to scylla_prepare /
scylla_stop on Ubuntu, it seems better to sourcing from these scripts.

Additionally, this fixes #1249

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1462989906-30062-1-git-send-email-syuu@scylladb.com>
2016-05-17 10:31:03 +03:00
Pekka Enberg
9c450f673c cql3: Clean up prepared_metadata class
Return vectors by const reference in prepared_metadata class and add a
FIXME to result_message class.

Message-Id: <1463425756-20225-1-git-send-email-penberg@scylladb.com>
2016-05-17 10:02:14 +03:00
Pekka Enberg
217c1ffa95 cql3: Specify result set flag ABI explicitly
As Avi points out, the flag values are an ABI. So specify them explicitly.

Message-Id: <1463413379-8355-1-git-send-email-penberg@scylladb.com>
2016-05-16 19:00:52 +03:00
Avi Kivity
a3b23d75b9 Merge "Fix Prepared message metadata serialization"
"The Prepared message has a metadata section that's similar to result set
metadata but not exactly the same. Fix serialization by introducing a
separate prepared_metadata class like Origin has and implement
serialization as per the CQL protocol specification. This fixes one CQL
binary protocol version 4 issue that we currently have.

The changes have been verified by running the gocql integration tests
using v4. Please note that this series does *not* enable v4 for clients
because Cassandra 2.1.x series only supports CQL binary protocol v3."
2016-05-16 18:59:54 +03:00
Pekka Enberg
868ff5107c cql3: Introduce prepared_metadata class
Introduce a new prepared_metadata class that holds prepared statement
metadata and implement CQL binary protocol serialization that works for
all versions.
2016-05-16 18:06:01 +03:00
Tomasz Grabiec
272e89846d Merge branch 'cache' from git@github.com:haaawk/scylla.git
From Piotr:

Fixes #656.

It makes it possible to slice using clustering ranges in mutation
readers.  We don't have row index yet so the slicing is just ignoring
data which is out of range.
2016-05-16 14:44:33 +02:00
Piotr Jastrzebski
dcba6f5c45 Pass clustering_row_ranges to mutation readers.
This will allow readers to reduce the amount of data read.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2016-05-16 14:36:57 +02:00
Pekka Enberg
a68671e247 cql3: Add column_specification::all_in_same_table() helper
We need it the prepared_metadata class that we're about to introduce.
2016-05-16 14:13:31 +03:00
Takuya ASADA
80037aa95b dist/common/scripts: don't proceed to run scylla_raid_setup when disks not selected, on interactive RAID setup
When disks not selected, run disk select prompt again.
Fixes #1260

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1463388933-3640-1-git-send-email-syuu@scylladb.com>
2016-05-16 13:45:17 +03:00
Pekka Enberg
adfb4d7bbd cql3: Move result_set class implementation to source file 2016-05-16 13:20:45 +03:00
Pekka Enberg
8552f222f5 cql3: Clean up result_set class
Kill some left-over ifdef'd code from the result_set class.

Message-Id: <1463392997-22921-1-git-send-email-penberg@scylladb.com>
2016-05-16 13:09:37 +03:00
Piotr Jastrzebski
23c23abe53 Make memtable mutation_reader slice using clustering ranges.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2016-05-16 11:46:41 +02:00
Piotr Jastrzebski
484d2ecd0a Slice data with clustering key range in sstable reader
Add additional parameters to mp_row_consumer to be able to fetch
only cells for given clustering key ranges

This will be used in row_cache when it will work on clustering key
level instead of partition key level.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2016-05-16 11:46:30 +02:00
Piotr Jastrzebski
8307681975 Introduce clustering_ranges type.
It will be used to slice data returned by mutation_readers.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2016-05-16 11:46:09 +02:00
Pekka Enberg
17765b6c06 Merge seastar upstream
* seastar 3dec26f...6a849ac (4):
  > seastar::socket: Be resilient against ENOTCONN
  > Merge " improve performance and predictability of syscall thread communications" from Glauber
  > rpc_test: Shutdown properly
  > [PATCH} future: better detect get_future() on already used promise
2016-05-16 08:04:47 +03:00
Yoav Kleinberger
de7952a8db tools/scyllatop: log input from collectd for easier debugging
When running with DEBUG verbosity, scyllatop will now log every single
value it receives from collectd. When you suspect that scyllatop is
somehow distorting values, this is a good way to check it.

Signed-off-by: Yoav Kleinberger <yoav@scylladb.com>
Message-Id: <1463320730-6631-1-git-send-email-yoav@scylladb.com>
2016-05-15 19:17:10 +03:00
Tomasz Grabiec
1eabe9b840 storage_proxy: Add trace-level logging for mutating
Message-Id: <1462978554-31217-1-git-send-email-tgrabiec@scylladb.com>
2016-05-12 13:52:56 +03:00
Tomasz Grabiec
7207cc8b1a storage_proxy: Improve error reporting
Knowing the source node can help in debugging the issue.
Message-Id: <1462978535-31164-1-git-send-email-tgrabiec@scylladb.com>
2016-05-12 13:52:39 +03:00
Pekka Enberg
b5d9aa866d Merge "Fixes for schema synchronization" from Tomek
"Writes may start to be rejected by replicas after issuing alter table
 which doesn't affect columns. This affects all versions with alter table
 support.

 Fixes #1258"
2016-05-12 09:43:25 +03:00
Duarte Nunes
7dbeef3c39 storage_service: Fix ignored future in on_alive
This patch ensures the future created by invoke_on_all is not ignored
by waiting on it, which is safe to do since we are within a
seastar::async context.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1462989837-7326-1-git-send-email-duarte@scylladb.com>
2016-05-12 09:03:46 +03:00
Tomasz Grabiec
13d8cd0ae9 migration_manager: Invalidate prepared statements on every schema change
Currently we only do that when column set changes. When prepared
statements are executed, paramaters like read repair chance are read
from schema version stored in the statement. Not invalidating prepared
statements on changes of such parameters will appear as if alter took
no effect.

Fixes #1255.
Message-Id: <1462985495-9767-1-git-send-email-tgrabiec@scylladb.com>
2016-05-12 08:58:40 +03:00
Tomasz Grabiec
90c31701e3 tests: Add unit tests for schema_registry 2016-05-11 17:31:22 +02:00
Tomasz Grabiec
443e5aef5a schema_registry: Fix possible hang in maybe_sync() if syncer doesn't defer
Spotted during code review.

If it doesn't defer, we may execute then_wrapped() body before we
change the state. Fix by moving then_wrapped() body after state changes.
2016-05-11 17:31:22 +02:00
Tomasz Grabiec
8703136a4f migration_manager: Fix schema syncing with older version
The problem was that "s" would not be marked as synced-with if it came from
shard != 0.

As a result, mutation using that schema would fail to apply with an exception:

  "attempted to mutate using not synced schema of ..."

The problem could surface when altering schema without changing
columns and restarting one of the nodes so that it forgets past
versions.

Fixes #1258.

Will be covered by dtest:

  SchemaManagementTest.test_prepared_statements_work_after_node_restart_after_altering_schema_without_changing_columns
2016-05-11 17:29:14 +02:00
Takuya ASADA
8503600e30 dist/common/systemd: drop hardcoded path
Stop using /var/lib/scylla, use $SCYLLA_HOME instead.
systemd seems does not extract variables on Environment="HOME=$SCYLLA_HOME", but both CentOS/Ubuntu able to run scylla-server without $HOME, so dropped it.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1462977871-26632-1-git-send-email-syuu@scylladb.com>
2016-05-11 17:53:53 +03:00
Calle Wilund
152bd82a05 alter_keyspace_statement: Handle missing replication strategy
ALTER KEYSPACE should allow no replication strategy to be set,
in which case old strategy should be kept.
Initial translation from origin missed this.

Fixes #1256

Message-Id: <1462967584-2875-2-git-send-email-calle@scylladb.com>
2016-05-11 16:02:22 +03:00
Calle Wilund
5604fb8aa3 cql3::statements::cf_prop_defs: Fix compation min/max not handled
Property parsing code was looking at wrong property level
for initial guard statement.

Fixes #1257

Message-Id: <1462967584-2875-1-git-send-email-calle@scylladb.com>
2016-05-11 16:02:16 +03:00
Takuya ASADA
c38b5fbb3d dist/common/scripts: On scylla_io_setup, run iotune on correct data directory which specified on scylla.yaml
Currently scylla_io_setup hardcoded to run iotune on /var/lib/scylla, but user may change data directory by modifying scylla.yaml, and it may on different block device.
So use scylla_config_get.py to get configuration from scylla.yaml, passes it to iotune.

Fixes #1167

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1462955824-21983-2-git-send-email-syuu@scylladb.com>
2016-05-11 13:02:25 +03:00
Takuya ASADA
53820393da dist/common/scripts: add scylla.yaml parser for scripts
To parse scylla.yaml, scylla_config_get.py is added.
It can be use like 'scylla_config_get.py [key name]' from shell script, or command line.
This is needed for scylla_io_setup, to get 'data_file_directories' from shellscript.
Currently it does not supported to specify key name of nested data structure, but enough for scyll_io_setup.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1462955824-21983-1-git-send-email-syuu@scylladb.com>
2016-05-11 13:02:23 +03:00
Pekka Enberg
d93d46e721 Merge "ALTER KEYSPACE" from Calle
"Implementation of ALTER KEYSPACE.
Fixes #429"
2016-05-10 22:07:06 +03:00
Takuya ASADA
a73924b4e0 dist/ubuntu/dep: introduce scylla-gdb-7.11 for Ubuntu 14.04LTS
Introduce scylla-gdb-7.11 for Ubuntu 14.04LTS, to get better support of recent version of g++ on gdb.

Fixes #969

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1462825880-20866-3-git-send-email-syuu@scylladb.com>
2016-05-10 17:53:32 +03:00
Takuya ASADA
9ff2efb28b dist/common/dep: add Ubuntu support for scylla-env
Since Ubuntu 14.04LTS needs scylla-gdb package which install to /opt/scylladb, we need to port scylla-env package to Ubuntu as well.
This change introduces scylla-env package to Ubuntu 14.04LTS.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1462825880-20866-2-git-send-email-syuu@scylladb.com>
2016-05-10 17:53:32 +03:00
Takuya ASADA
43cc77d1b8 dist/redhat/centos_dep: move scylla-env to dist/common to share with Ubuntu
Since Ubuntu 14.04LTS needs scylla-gdb package which install to /opt/scylladb, we need to port scylla-env package to Ubuntu as well.
To do it, share the package directory on dist/common/dep at first.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1462825880-20866-1-git-send-email-syuu@scylladb.com>
2016-05-10 17:53:31 +03:00
Calle Wilund
147aa81177 Cql.g: Handle ALTER KEYSPACE 2016-05-10 14:36:46 +00:00
Calle Wilund
5c36d2e09e alter_keyspace_statement: Implement
Note: Like create keyspace, we don't properly validate 
replication strategy yet.
2016-05-10 14:36:17 +00:00
Piotr Jastrzebski
240a185727 Stop scanning keyspace data directory when populating.
Iterate over column families and check/create directories for them
instead of scanning keyspace data directory and filtering directories
against column families that exist in system tables for this keyspace.

Fixes #1008

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <26da66eec67a1ab1318917a66161915cdef924ab.1462890592.git.piotr@scylladb.com>
2016-05-10 17:35:55 +03:00
Calle Wilund
63b6c6bb5a migration_manager: Implement announce_keyspace_update
More or less the same as create keyspace...
2016-05-10 14:34:51 +00:00
Calle Wilund
8cdf4e37fb schema_tables: Fix merge_keyspaces to handle alter keyspace
Must keep "altered" alive into the call chain.
2016-05-10 14:32:51 +00:00
Calle Wilund
6ef7885ae3 database: Implement update_keyspace
Reloads keyspace metadata and replaces in existing keyspace. 
Note: since keyspace metadata, and consequently, replication 
strategy now becomes volatile, keyspace::metadata now returns
shared pointer by value (i.e. keep-alive). 
Replication strategy should receive the same treatment, but
since it is extensively used, but never kept across a 
continuation, I've just added a comment for now.
2016-05-10 14:31:30 +00:00
Raphael S. Carvalho
d80d194873 compaction_manager: stop compaction tasks in parallel
Purpose is to speed up shutdown.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <a8db3492f1ceeea2a886d3920e5effa841ea155f.1462838670.git.raphaelsc@scylladb.com>
2016-05-10 10:03:35 +03:00
Avi Kivity
28cc6f97af Merge 2016-05-09 14:25:25 +03:00
Calle Wilund
917bf850fa transport::server: Do not treat accept exception as fatal
1.) It most likely is not, i.e. either tcp or more likely, ssl
    negotiation failure. In any case, we can still try next
    connection.
2.) Not retrying will cause us to "leak" the accept, and then hang
    on shutdown.

Also, promote logging message on accept exception to "warn", since
dtest(s?) depend on seeing log output.

Message-Id: <1462283265-27051-4-git-send-email-calle@scylladb.com>
2016-05-09 14:13:07 +03:00
Calle Wilund
437ebe7128 cql_server: Use credentials_builder to init tls
Slightly cleaner, and shard-safe tls init.

Message-Id: <1462283265-27051-3-git-send-email-calle@scylladb.com>
2016-05-09 14:12:59 +03:00
Calle Wilund
58f7edb04f messaging_service: Change tls init to use credentials_builder
To simplify init of msg service, use credendials_builder
to encapsulate tls options so actual credentials can be
more easily created in each shard.

Message-Id: <1462283265-27051-2-git-send-email-calle@scylladb.com>
2016-05-09 14:12:53 +03:00
Avi Kivity
29e103a2ae Merge seastar upstream
* seastar 7782ad4...3dec26f (3):
  > tests/mkcert.gmk: Fix makefile bug in snakeoil cert generator
  > tls_test: Add case to do a little checking of credentials_builder
  > tls: Add credentials_builder - copyable credentials "factory"
2016-05-09 14:12:29 +03:00
Tomasz Grabiec
1ca5ceadff Merge tag '1235-v2' from https://github.com/avikivity/scylla
From Avi:

When we shut down, we may have to give up on some pending atomic
sstable deletions, because not all shards may have agreed to delete
all members of the set.

This is expected, so silence these frightening error messages.

Fixes #1235.
2016-05-09 12:22:41 +02:00
Duarte Nunes
dada385826 rpc: Secure connection attempts can be cancelled
This patch adds support for secure connection attempts to be
cancellable.

Fixes #862

Includes seastar upstream merge:

* seastar f1a3520...7782ad4 (1):
  > Merge "rpc: Allow client connections to be cancelled" from Duarte

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1462783335-10731-1-git-send-email-duarte@scylladb.com>
2016-05-09 11:44:53 +03:00
Takuya ASADA
f7d41ba07a dist: Extract scylla.yaml and create metapackage
This patch create a scylla-conf package containing
scylla.yaml and a scylla package acting as a metapackage.

Fixes #421

Signed-off-by: Benoît Canet <benoit@scylladb.com>
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1462280987-26909-1-git-send-email-syuu@scylladb.com>
2016-05-09 11:23:28 +03:00