Commit Graph

7834 Commits

Author SHA1 Message Date
Takuya ASADA
6e8dd00535 dist: apply limits settings correctly on Ubuntu
Fixes #738

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-01-02 12:20:18 +02:00
Benoît Canet
de508ba6cc README: Add missing build dependencies
It's just a waste of time to find them manually
when compiling ScyllaDB on a fresh install: add them.

Also fix the ninja-build name.

Signed-off-by: Benoît Canet <benoit@scylladb.com>
2015-12-31 13:34:48 +02:00
Shlomi Livne
6316fedc0a Make sure the directory we are writting coredumps to exists
After upgrading an AMI and trying to stop and start a machine the
/var/lib/scylla/coredump is not created. Create the directory if it does
not exist prior to generating core

Signed-off-by: Shlomi Livne <shlomi@scylladb.com>
2015-12-31 13:21:39 +02:00
Pekka Enberg
32d22a1544 cql3: Fix relation grammar rule
There's two untranslated grammar subrules in the 'relation' rule. We
have all the necessary AST classes translated, so translate the
remaining subrules.

Refs #534.

Signed-off-by: Pekka Enberg <penberg@scylladb.com>
2015-12-31 11:44:15 +01:00
Tomasz Grabiec
e23adb71cf Merge tag 'asias/streaming/cleanup_verb/v2' from seastar-dev.git
messaging_service cleanups and improvements from Asias
2015-12-31 11:25:09 +01:00
Asias He
1b3d2dee8f streaming: Drop src_cpu_id parameter
Now that we can get the src_cpu_id from rpc::client_info.
No need to pass it as verb parameter.
2015-12-31 11:25:09 +01:00
Asias He
3ae21e06b5 messaging_service: Add src_cpu_id to CLIENT_ID verb
It is useful to figure out which shard to send messages back to the
sender.
2015-12-31 11:25:09 +01:00
Asias He
22d0525bc0 streaming: Get rid of the _from_ parameter
Get this from cinfo.retrieve_auxiliary inside the rpc handler.
2015-12-31 11:25:08 +01:00
Asias He
89b79d44de streaming: Get rid of the _connecting_ parameter
messaging_service will use private ip address automatically to connect a
peer node if possible. There is no need for the upper level like
streaming to worry about it. Drop it simplifies things a bit.
2015-12-31 11:25:08 +01:00
Amnon Heiman
8d31c27f7b api: stream_state should include the stream_info
The stream info was creted but was left out of the stream state. This
patch adds the created stream_info to the stream state vector.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-12-31 12:01:03 +02:00
Gleb Natapov
2bcfe02ee6 messaging: remove unused verbs 2015-12-30 15:06:35 +01:00
Gleb Natapov
f0e8b8805c messaging: constify some handlers 2015-12-30 15:06:35 +01:00
Avi Kivity
a318b02335 Merge seastar upstream
* seastar 8b2171e...de112e2 (4):
  > build: disable -fsanitize=vptr harder
  > Merge "Make RPC more robust against protocol changes" from Gleb
  > new when_all implementation
  > apps: memcached: Don't fiddle with the expiration time value in a usable range
2015-12-30 14:56:09 +02:00
Pekka Enberg
500f5b3f27 dist: Increase NOFILE rlimit to 200k
Commit 2ba4910 ("main: verify that the NOFILE rlimit is sufficient")
added a recommendation to set NOFILE rlimit to 200k. Update our release
binaries to do the same.
2015-12-30 11:18:23 +01:00
Avi Kivity
2ba4910385 main: verify that the NOFILE rlimit is sufficient
Require 10k files, recommend 200k.

Allow bypassing via --developer-mode.

Fixes #692.
2015-12-30 11:02:08 +02:00
Avi Kivity
c26689f325 init: bail out if running not on an XFS filesystem
Allow an override via '--developer-mode true', and use it in
the docker setup, since that cannot be expected to use XFS.

Fixes #658.
2015-12-30 10:56:21 +02:00
Pekka Enberg
0aa105c9cf Merge "load report a negative value" from Amnon
"This series solve an issue with the load broadcaster that reports negative
 values due to an integer wrap around.  While fixing this issue an additional
 change was made so that the load_map would return doubles and not formatted
 string.  This is a better API, safer and better documented."
2015-12-30 10:21:55 +02:00
Nadav Har'El
f0b27671a2 murmur3 partitioner: remove outdated comment, and code
Since commit 16596385ee, long_token() is already checking
t.is_minimum(), so the comment which explains why it does not (for
performance) is no longer relevant. And we no longer need to check
t._kind before calling long_token (the check we do here is the same
as is_minimum).

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
2015-12-30 10:01:29 +02:00
Nadav Har'El
de5a3e5c5a repair: check columnFamilies list
Check the list of column families passed as an option to repair, to
provide the user with a more meaningful exception when a non-existant
column family is passed.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
2015-12-30 09:59:54 +02:00
Nadav Har'El
3ae29216c8 repair: add missing ampersand
This was a plain bug - ranges_opt is supposed to parse the option into
the vector "var", but took the vector by value.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
2015-12-30 09:46:13 +02:00
Nadav Har'El
a0a649c1be repair: support "columnFamilies" parameter
Support the "columnFamilies" parameter of repair, allowing to repair
only some of the column families of a keyspace, instead of all of them.
For example, using a command like "nodetool repaire keyspace cf1 cf2".

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
2015-12-30 09:45:28 +02:00
Lucas Meneghel Rodrigues
43d39d8b03 scylla_coredump_setup: Don't call yum on scylla server spec file
The script scylla_coredump_setup was introduced in
9b4d0592, and added to the scylla rpm spec file, as a
post script. However, calling yum when there's one
yum instance installng scylla server will cause a deadlock,
since yum waits for the yum lock to be released, and the
original yum process waits for the script to end.

So let's remove this from the script. Debian shouldn't be
affected, since it was never added to the debian build
rules (to the best of my knowlege, after analyzing 9b4d0592),
hence I did not remove it. It should cause the same problem
with apt-get in case it was used.

CC: Takuya ASADA <syuu@scylladb.com>
[ penberg: Rebase and drop statement about 'abrt' package not in Fedora. ]
Signed-off-by: Lucas Meneghel Rodrigues <lmr@scylladb.com>
2015-12-30 09:38:36 +02:00
Nadav Har'El
ebebaa525d repair: fix missing default values
A default value was not set for the "incremental" and "parallelism"
repair parameters, so Scylla can wrongly decide that they have an
unsupported value.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
2015-12-29 15:39:47 +02:00
Amnon Heiman
ec379649ea API: repair to use documented params
The repair API use to have an undocumented parameter list similiar to
origin.

This patch changes the way repair is getting its parameters.
Instead of a one undocumented string it now lists all the different
optional parameters in the swagger file and accept them explicitely.

Reviewed-by: Nadav Har'El <nyh@scylladb.com>
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-12-29 15:38:44 +02:00
Amnon Heiman
f0d68e4161 main: start the http server in the first step
This change set the http server to start as the first step in the boot
order.

It is helpfull if some other step takes a long time or stuck.

Fixes #725

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-12-29 14:20:57 +02:00
Avi Kivity
c8b09a69a9 lsa: disable constant_time_size in binomial_heap implementation
Corrupts heap on boost < 1.60, and not needed.

Fixes #698.
2015-12-29 12:59:00 +01:00
Vlad Zolotarov
756de38a9d database: actually check that a snapshot directory exists
Actually check that a snapshot directory with a given tag
exists instead of just checking that a 'snapshot' directory
exists.

Fixes issue #689

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2015-12-29 12:59:00 +01:00
Amnon Heiman
71905081b1 API: report the load map as an unformatted double
In origin the storage_serivce report the load map as a formatted string.
As an API a better option is to report the load map as double and let
the JMX proxy do the formatting.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-12-29 11:55:34 +02:00
Amnon Heiman
06e1facc34 load_broadcaster report negative size
The map_reduce0 convert the result value to the init value type. In
load_bradcaster 0 is of type int.
This result with an int wrap around and negative results.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-12-29 11:55:34 +02:00
Avi Kivity
41bd266ddd db: provide more information on "Unrecognized error" while loading sstables
This information can be used to understand the root cause of the failure.

Refs #692.
2015-12-29 10:23:32 +02:00
Nadav Har'El
7247f055df repair: partial support for some options
Add partial support for the "incremental" option (only support the
"false" setting, i.e., not incremental repair) and the "parallelism"
option (the choice of sequential or parallel repair is ignored - we
always use our own technique).

This is needed because scylla-jmx passes these options by default
(e.g., "incremental=false" is passed to say this is *not* incremental
repair, and we just need to allow this and ignore it).

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
2015-12-29 09:38:09 +02:00
Nadav Har'El
3cfa39e1f0 repair: log repair options
When throwing an "unsupported repair options" exception to the caller
(such as "nodetool repair"), also list which options were not recognized.
Additionally, list the options when logging the repair operation.

This patch includes an operator<< implementation for pretty-printing an
std::unordered_map. We may want to move it later to a more central
location - even Seastar (like we have a pretty-printer for std::vector
in core/sstring.hh).

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
2015-12-29 09:37:30 +02:00
Raphael S. Carvalho
b7d36af26f compaction: fix max_purgeable calculation
max_purgeable was being incorrectly calculated because the code
that creates vector of uncompacted sstables was wrong.
This value is used to determine whether or not a tombstone can
be purged.
Operand < is supposed to be used instead in the callback passed
as third parameter to boost::set_difference.
This fix is a step towards closing the issue #676.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2015-12-29 09:30:08 +02:00
Takuya ASADA
46767fcacf dist: fix .rpm build error (File not found: scylla_extlinux_setup)
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-12-29 09:26:58 +02:00
Pekka Enberg
ca1f9f1c9a main: Fix implicitly disabled client encryption options
The start_native_transport() function in storage_service expects the
'enabled' option to be defined. If the option is not defined, it means
that encryption is implicitly disabled.

Fixes #718.
2015-12-28 16:24:49 +02:00
Pekka Enberg
a76b3a009b Merge "use steady_clock where monotonic clock is required" from Vlad
"The first patch in this series fixes the issue #638 in scylla.
 The second one fixes the tests to use the appropriate clock."
2015-12-28 13:35:50 +02:00
Avi Kivity
561bb79d22 Merge "CQL server SSL" from Calle
"* Update scylla.conf section
 * Add SSL capability to cql server
 * Use conf and initiate optional SSL cql server in
   main/storage_service"
2015-12-28 12:55:25 +02:00
Avi Kivity
72cb8d4461 Merge "Messaging service TLS" from Calle
"Adds support for TLS/SSL encrypted (and cert verified)
connections for message service

* Modify config option to match "native" style cerificate management
* Add SSL options to messaging service and generate SSL server/client
  endpoints when required
* Add config option handling to init/main"
2015-12-28 12:54:28 +02:00
Calle Wilund
fae3bb7a24 storage_service: Set up CQL server as SSL if specified
* Massage user options in main
* Use them in storage_service, and if needed, load certificates etc
  and pass to transport/cql server.

Conflicts:
	service/storage_service.cc
2015-12-28 10:13:48 +00:00
Calle Wilund
51d3990261 cql_server: Allow using SSL socket
Optional credentials argument determine if SSL or normal
server socket is created.

Note: This does not follow the pattern of "socket as argument", simply
because this is a distributed object, so only trivial or immutable
objects should be passed to it.
2015-12-28 10:13:48 +00:00
Calle Wilund
d8b2581a07 scylla.conf: Update client_encryption_options with scylla syntax
Using certificate+key directly
2015-12-28 10:13:48 +00:00
Calle Wilund
5f003f9284 scylla.conf: Modify server_encryption_options section
Describe scylla version of option.

Note, for test usage, the below should be workable:

server_encryption_options:
    internode_encryption: all
    certificate: seastar/tests/test.crt
    truststore: seastar/tests/catest.pem
    keyfile: seastar/tests/test.key

Since the seastar test suite contains a snakeoil cert + trust
combo
2015-12-28 10:10:35 +00:00
Calle Wilund
70f293d82e main/init: Use server_encryption_options
* Reads server_encryption_options
* Interpret the above, and load and initialize credentials
  and use with messaging service init if required
2015-12-28 10:10:35 +00:00
Calle Wilund
d1badfa108 messaging_service: Optionally create SSL endpoints
* Accept port + credentials + option for what to encrypt
* If set, enable a SSL listener at ssl_port
* Check outgoing connections by IP to determine if
  they should go to SSL/normal endpoint

Requires seastar RPC patch

Note: currently, the connections created by messaging service
does _not_ do certificate name verification. While DNS lookup
is probably not that expensive here, I am not 100% sure it is
the desired behaviour.
Normal trust is however verified.
2015-12-28 10:10:35 +00:00
Calle Wilund
1a9fb4ed7f config: Modify/use server_encryption_options
* Mark option used
* Make sub-options adapted to seastar-tls useable values (i.e. x509)

Syntax is now:

server_encryption_options:
	internode_encryption: <none, all, dc, rack>
	certificate: <path-to-PEM-x509-cert> (default conf/scylla.crt)
	keyfile: <path-to-PEM-x509-key> (default conf/scylla.key)
	truststore: <path-to-PEM-trust-store-file> (default empty,
                                                    use system trust)
2015-12-28 10:10:35 +00:00
Calle Wilund
b7baa4d1f5 config: clean up some style + move method to cc file 2015-12-28 10:10:35 +00:00
Takuya ASADA
fc29a341d2 dist: show usage and scylla-server status when login to AMI instance
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-12-28 11:40:34 +02:00
Avi Kivity
827a4d0010 Merge "streaming: Invalidate cache upon receiving of stream" from Asias
"When a node gain or regain responsibility for certain token ranges, streaming
will be performed, upon receiving of the stream data, the row cache
is invalidated for that range.

Refs #484."
2015-12-28 10:24:46 +02:00
Amnon Heiman
2c79fe1488 storage_service: describe_ring return full data
The describe_ring method in storage_service did not report the start and
end tokens.

Also for rpc addresses that are not the local address, it returned the
value representation (including the version) and not just the adress.

Fixes #695

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-12-28 09:56:12 +02:00
Takuya ASADA
0abcf5b3f3 dist: use readable time format on coredump file, instead of unix time
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-12-28 09:55:05 +02:00