Commit Graph

53948 Commits

Author SHA1 Message Date
Takuya ASADA
d2dd6b90a9 dist: support ./SCYLLA-VERSION-GEN on ubuntu package
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-04 11:50:38 +02:00
Avi Kivity
a41ffb3f4b Merge "support initial_token config option" from Asias
"This is needed by some tests to specify specific tokens."
2015-11-04 11:42:39 +02:00
Paweł Dziepak
a553c9236e transport: do not send response with unknown protocol version
All responses sent from the server have protocol version set to
connection::_version which is set to the version used by the client
in its first message. However, if the protocol version used by the
client is unuspported or invalide the server should use the latest
version it recognizes.

This solves problem with version negotiation with Java driver. The
driver first sends a request in the latest version it recognizes, if
that fails it retries with the version that server has used in the error
message. If that fails as well it gives up. However, since Scylla always
responds with the same version that the client has used the negotiation
always fails if the client supports more protocol version than the
server.

Refs #317.

Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-11-04 08:59:23 +02:00
Asias He
ed313160c2 storage_service: Add initial_token config option support 2015-11-04 10:42:17 +08:00
Asias He
20ecb0bede database: Introduce get_initial_tokens
Get initial tokens specified by the initial_token in scylla.conf.

E.g.,

   --initial-token "-1112521204969569328,1117992399013959838"
   --initial-token "1117992399013959838"

It can be multiple tokens split by comma.
2015-11-04 10:40:12 +08:00
Asias He
16596385ee token: Handle minimum token correctly in long_token
Fixes:

Exiting on unhandled exception of type 'runtime_exception': runtime
error: Invalid token. Should have size 8, has size 0
2015-11-04 09:01:06 +08:00
Avi Kivity
a060610347 Merge seastar upstream
* seastar 258daf9...05b41b8 (5):
  > resource: fix system memory reservation
  > Add .gitattributes to improve 'git diff' output
  > README: remove irrelevant OSv instructions
  > Add invocation of ninja in Ubuntu section of README
  > collectd: remove incorrect license
2015-11-04 00:18:06 +02:00
Avi Kivity
012f13eb53 Merge "Adding describering API" from Amnon
"This series adds the missing functionality that the nodetool describering would work.

It import the missing functionality from origin.
After this patch the API:
GET /storage_service/describe_ring/{keyspace}

will be available"
2015-11-03 16:54:13 +02:00
Amnon Heiman
ef3c6b2647 API: Add describe ring API implementation
This patch chanages the API to support describe ring instead of describe
ring jmx that will be implemented in the jmx server.

The API will return a list of objects instead of string.

An additional api was added as the equivelent to the jmx call with an
empty param.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-03 16:33:39 +02:00
Amnon Heiman
3809565dbd storage_service: Add method implementation from origin
This patch adds the following methods implementation:
getRpcaddress
getRangeToAddressMap
getRangeToAddressMapInLocalDC
describeRing
getAllRanges

Those methods are used as part of the describe_ring method
implementation.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-03 16:33:20 +02:00
Gleb Natapov
28bb6a3efe messaging_service: fix hanging reference access
Do not pass reference to an on-stack objects to a function that uses
its parameters asynchronously.
2015-11-03 12:10:38 +01:00
Tomasz Grabiec
7a2ac628a0 Fix compilation of types_test 2015-11-03 11:02:35 +01:00
Avi Kivity
74faaa4698 types: implement date_type::from_string()
Luckily all the hard work was already done for timestamp_type.

Fixes #522.
2015-11-03 10:33:20 +01:00
Amnon Heiman
b77ec2bd6a Importing token_range and endpoint_details from origin
The storage server uses the token_range in origin to return inforamtion
about the ring.

This import the structures. The functionality in origin is redundant in
this case and was not imported.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-03 10:17:05 +02:00
Asias He
dafdac6731 ami: Improve scylla raid setup
Use all the disks except the one for rootfs for RAID0 which stores
scylla data. If only one disk is available warn the user since currently
our AMI's rootfs is not XFS.

[fedora@ip-172-31-39-189 ~]$ cat WARN.TXT
WARN: Scylla is not using XFS to store data. Performance will suffer.

Tested on AWS with 1 disk, 2 disks, 7 disk case.

(cherry picked from commit 49d6cba471)
2015-11-03 08:16:09 +08:00
Avi Kivity
389e035d19 Update scylla-ami submodule
* dist/ami/files/scylla-ami c6ddbea...3f37184 (1):
  > Update reflector URL

(cherry picked from commit 440b403089)
2015-11-03 08:12:51 +08:00
Takuya ASADA
a0feaf801c dist: add scylla.repo to fetch scylla rpms on ami
Mistakenly didn't included on yum repository for AMI patchset, but it's needed

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
(cherry picked from commit 8587c4d6b3)
2015-11-03 08:01:09 +08:00
Takuya ASADA
a5a69279ef dist: update packages on ec2 instance first bootup
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
(cherry picked from commit aaeccdee60)
2015-11-03 07:50:43 +08:00
Takuya ASADA
db634dfe01 dist: use scylla repo on ami, instead of locally built rpms
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
(cherry picked from commit 8b98fe5a1c)
2015-11-03 07:50:33 +08:00
Takuya ASADA
63d16f5816 dist: move inline script to setup-ami.sh
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
(cherry picked from commit 2863b8098f)
2015-11-03 07:50:12 +08:00
Amnon Heiman
e562736e82 API: A workaround for nodetool cleanup
The nodetool cleanup command is used in many of the tests, because the
API call is not implemented it causes the tests to fail.

This is a workaround until the cleanup will be implemented, the method
return successfuly.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-02 17:38:48 +02:00
Amnon Heiman
80e5289359 Adding API as an unimplemented cause
Normally an API call that is not implemented should fail, there are
cases that as a workaround an API call is stub, in those cases a warning
is added to indicate that the API is not implemented.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-02 17:38:48 +02:00
Avi Kivity
c9b8c368bd Merge "Adding missing statistics to enable nodetool netstats" from Amnon
"The series adds read repair and storage_service server statistics."
2015-11-02 17:10:43 +02:00
Amnon Heiman
fc7ea6207c API: messaging service, Add getter for the pending and completed server
This patch adds getter for the messeging service server pending and sent
messages.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-02 16:17:47 +02:00
Amnon Heiman
55d7ddafe5 API: Add repond complete to the messaging service swagger
This patch do the following:
It adds a getter for the completed respond messages (i.e. the total
messages that were sent by the server)

It replaces the return mapping for the statistics to use the key, value
notation that is used in the jmx side.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-02 16:17:36 +02:00
Amnon Heiman
afb921a643 API: add read repair stats to the storage_proxy
This adds the API to the storage_proxy read repair stats that replaces
the stub implementation.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-02 16:17:22 +02:00
Amnon Heiman
b6034572dc storage_proxy: Add read repair statistics
This adds the read repair statistics to he storage_proxy stats and adds
to its implementation incrementing the counters value.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-02 16:16:40 +02:00
Amnon Heiman
d5d0653210 messaging_service: Add a function that goes over all the server stats
The API needs to get the stats from the rpc server, that is hidden from the
messaging service API.

This patch adds a foreach function that goes over all the server stats
without exposing the server implementation.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-02 16:15:52 +02:00
Avi Kivity
9c09f43311 Merge "add [background] read/write statistics" from Gleb
"The main objective of the series is to introduce statistics about ongoing
read/writes and especially those that are done in the background (acknowledged,
but uncompleted), but it contains some cleanups as well."
2015-11-02 15:46:07 +02:00
Avi Kivity
e2b661bb78 sstables: fix compile error 2015-11-02 15:32:44 +02:00
Gleb Natapov
9af55b3a7b storage_proxy: add ongoing read statistics
Add statistics for ongoing reads and ongoing background reads. Read is
a background one if it was acknowledged, but there still work to do to
complete it.
2015-11-02 15:02:13 +02:00
Gleb Natapov
463d330a24 storage_proxy: do not capture 'this' needlessly. 2015-11-02 15:01:16 +02:00
Gleb Natapov
5067d027ba storage_proxy: export statistics via collectd 2015-11-02 15:01:16 +02:00
Gleb Natapov
287cf894a0 storage_proxy: count background mutation writes
Count how many writes are running in the background. Write is a
background write if it was acknowledged, but there still work to do to
complete it.
2015-11-02 15:01:12 +02:00
Gleb Natapov
afa61cb87c storage_proxy: separate methods from members with access specifier 2015-11-02 15:00:47 +02:00
Gleb Natapov
62e5ad27d8 storage_proxy: devirtualize total_block_for()
total_block_for() is defined only in base class, so make it non virtual.
2015-11-02 15:00:47 +02:00
Gleb Natapov
59d8f9f392 storage_proxy: move proxy pointer to write response handler
We will need it later there.
2015-11-02 15:00:47 +02:00
Gleb Natapov
ca0177f5cc storage_proxy: simplify timeout handling
Use _cl_achieved instead of calculating if there is enough acks.
2015-11-02 15:00:47 +02:00
Gleb Natapov
4838de008f storage_proxy: get rid of semaphore in mutation write response handler
Acks are counted in _cl_acks, so there is no need to count them once
more in semaphore, just make promise available when there is enough of
them.
2015-11-02 15:00:47 +02:00
Gleb Natapov
9381ad0741 storage_proxy: initialize _stats 2015-11-02 15:00:47 +02:00
Avi Kivity
b1d13636bb Merge "locator: use the correct snitch instance in gossiping_property_file_snitch::reload_configuration()" from Vlad
"Commit 4cd9c4c0c5441cf55e280c6f2f2e5529426b9c98 introduced a minor
issue: a wrong snitch instance may be used when updating a Gossiper state
(if I/O CPU is different from CPU0).

In order to fix this issue a local snitch instance on CPU0 should be used,
just like a Gossiper local instance.

We have to move some interfaces to i_endpoint_snitch
from being private in a gossiping_property_file_snitch in order to be
able to access it using snitch_ptr handle."
2015-11-02 14:09:27 +02:00
Vlad Zolotarov
b654d942b4 locator::gossiping_property_file_snitch: don't ignore a returned future
Don't ignore yet another returned future in reload_configuration().

Since commit 5e8037b50a
storage_service::gossip_snitch_info() returns a future.

This patch takes this into an account.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2015-11-02 13:44:53 +02:00
Vlad Zolotarov
689d5fb000 locator::gossiping_property_file_snitch: fix in reload_configuration()
When we access a gossiper instance we use a _gossip_started
state of a snitch, which is set in a gossiper_starting() method.

gossiper_starting() method however is invoked by a gossiper on CPU0
only therefore the _gossip_started snitch state will be set for an
instance on CPU0 only.

Therefore instead of synchronizing the _gossip_started state between
all shards we just have to make sure we check it on the right CPU,
which is CPU0.

This patch fixes this issue.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2015-11-02 13:44:53 +02:00
Vlad Zolotarov
5da4e62a59 locator::i_endpoint_snitch: align the _prefer_local parameter with _my_dc and _my_rack
Adjust the interface and distribution of prefer_local parameter read
from a snitch property file with the rest of similar parameters (e.g. dc and rack):
they are read and their values are distributed (copied) across all shards'
instances.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2015-11-02 13:44:53 +02:00
Vlad Zolotarov
5042f3c952 locator::i_endpoint_snitch_base: make reload_gossiper_state() a virtual function
Make reload_gossiper_state() be a virtual method
of a base class in order to allow calling it using a snitch_ptr
handle.

A base class already has a ton of virtual methods so no harm is
done performance-wise. Using virtual methods instead of doing
dynamic_cast results in a much cleaner code however.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2015-11-02 13:44:53 +02:00
Vlad Zolotarov
926ce145db locator::i_endpoint_snitch_base: move _gossip_started to the base class
Move the member and add an access method.
This is needed in order to be able to access this state using
snitch_ptr handle.

This also allows to get rid of ec2_multi_region_snitch::_helper_added
member since it duplicates _gossip_started semantics.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2015-11-02 13:44:31 +02:00
Takuya ASADA
919d94a52b dist: update Ubuntu's g++ to 5.x
Now ubuntu-toolchain-r repo has g++-5 package for 14.04, so switch to it

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-02 13:31:19 +02:00
Avi Kivity
4e3d281402 Merge 2015-11-02 13:31:02 +02:00
Avi Kivity
d436617adc Merge seastar upstream
* seastar 9ae6407...258daf9 (6):
  > rpc server: Add pending and sent messages to server
  > scripts: posix_net_conf.sh: Use a generic logic for RPS configuring
  > scripts: posix_net_conf.sh: allow passing a NIC name as a parameter
  > doc: link to the tutorial
  > tutorial: begin documenting the network API
  > slab: remove bogus uintptr_t definition
2015-11-02 13:30:26 +02:00
Avi Kivity
c6968b2940 Make abstract_type::serialize(data_value) a member of data_value
Reduces duplication.  Fixes #517.
2015-11-02 12:25:35 +01:00