Commit Graph

11716 Commits

Author SHA1 Message Date
Tomasz Grabiec
43a69e64f6 serializer: Add serializer<std::chrono::duration<T, Ratio>> 2016-07-25 17:20:08 +02:00
Tomasz Grabiec
445f763fa3 serializer: Add serializer<std::vector<T>> 2016-07-25 17:20:08 +02:00
Tomasz Grabiec
953fce3f7b serializer: Define serializer<> specializations for integral types 2016-07-25 17:19:50 +02:00
Avi Kivity
c908e630f0 Merge seastar upstream
* seastar 2425aab...64ae228 (1):
  > Merge "Adding prometheus API support" from Amnon
2016-07-25 15:32:40 +03:00
Avi Kivity
45e1274064 Merge "Add GDB commands for working with thread lists" from Tomasz 2016-07-25 15:32:02 +03:00
Tomasz Grabiec
0b8aafd72c scylla-gdb.py: Introduce "scylla thread apply all ..."
Similar to gdb's "thread apply all". Executes given command in the
context of each seastar thread.

For example to print backtrace of all threads:

  scylla thread apply-all bt
2016-07-25 12:40:47 +02:00
Tomasz Grabiec
2d6341d4cb scylla-gdb.py: Introduce "scylla threads" command
Lists all seastar threads.

Example:

(gdb) scylla threads
[shard  1] (seastar::thread_context*) 0x602008c9aa00
[shard  1] (seastar::thread_context*) 0x602008c9ca00
[shard  1] (seastar::thread_context*) 0x602008cf5800
[shard  1] (seastar::thread_context*) 0x602008cbe000
[shard  1] (seastar::thread_context*) 0x602008c4bc00
[shard  2] (seastar::thread_context*) 0x601008d6b800
[shard  2] (seastar::thread_context*) 0x601008d89400
[shard  2] (seastar::thread_context*) 0x601008c95a00
[shard  0] (seastar::thread_context*) 0x600008d84400
[shard  0] (seastar::thread_context*) 0x6000000a1600
[shard  0] (seastar::thread_context*) 0x600008ce9a00
[shard  0] (seastar::thread_context*) 0x600008df9a00
[shard  0] (seastar::thread_context*) 0x600008dfee00
[shard  0] (seastar::thread_context*) 0x600008d85800
[shard  0] (seastar::thread_context*) 0x600008df9000
[shard  0] (seastar::thread_context*) 0x600008d82c00
[shard  0] (seastar::thread_context*) 0x600008cece00
2016-07-25 12:32:48 +02:00
Tomasz Grabiec
0e1b21eab6 scylla-gdb.py: Make intrusive_list() work with newer version of boost 2016-07-25 12:32:48 +02:00
Duarte Nunes
d8a4bd6b1a sstables: Remove duplication in extract_clustering_key
This patch removes some duplicated code in extract_clustering_key(),
which is already handled in composite_view.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1469397806-8067-1-git-send-email-duarte@scylladb.com>
2016-07-25 10:07:25 +02:00
Duarte Nunes
b2278697ce sstables: Remove superfluous call to check_static()
When building a column we're calling check_static() two times;
refector things a bit so that this doesn't happen and we reuse the
previous calculation.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1469397748-7987-1-git-send-email-duarte@scylladb.com>
2016-07-25 10:06:56 +02:00
Duarte Nunes
7a81553d17 compound_compat: Only compound values can be static
If a composite is not a compound, then it doesn't carry a length
prefix where static information is encoded. In its absence, a
non-compound composite can never be static.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1469397561-7748-1-git-send-email-duarte@scylladb.com>
2016-07-25 10:05:19 +02:00
Avi Kivity
f2939b125e dist: reduce kernel's inclincation to OOM-kill Scylla
Scylla is likely the most important process on the machine, make the kernel
less inclined to kill it, on systemd-enabled hosts.

See #1393.
Message-Id: <1468847986-9429-1-git-send-email-avi@scylladb.com>
2016-07-25 10:00:19 +02:00
Duarte Nunes
5c4a2044d5 thrift: Fail when creating mixed CF
This patch ensures we fail when creating a mixed column family, either
when adding columns to a dynamic CF through updated_column_family() or
when adding a dynamic column upon insertion.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1469378658-19853-1-git-send-email-duarte@scylladb.com>
2016-07-25 10:35:37 +03:00
Duarte Nunes
560cc12fd7 thrift: Correctly translate no_such_column_family
The no_such_column_family exception is translated to
InvalidRequestException instead of to NotFoundException.

8991d35231 exposed this problem.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1469376674-14603-1-git-send-email-duarte@scylladb.com>
2016-07-25 10:35:07 +03:00
Avi Kivity
df346d8b49 Merge "thrift: Implements describe_splits verbs" from Duarte
"This patchset implements the describe_splits and describe_splits_ex verbs
in Thrift. It also contains a couple of related fixes."
2016-07-25 10:08:00 +03:00
Raphael S. Carvalho
56ef3d4fde sstables: get rid of magic number
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <c0b3acf58aa6f7632bf15a2dd0dcac4d1b45d444.1469389290.git.raphaelsc@scylladb.com>
2016-07-25 10:06:51 +03:00
Takuya ASADA
b5bb702b35 dist: add dependency for lspci
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1469429320-13645-1-git-send-email-syuu@scylladb.com>
2016-07-25 10:02:30 +03:00
Duarte Nunes
ab08561b89 thrift: Implement describe_splits verb
This patch implements the describe_splits verb on top of
describe_splits_ex.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-24 22:44:07 +00:00
Duarte Nunes
472c23d7d2 thrift: Implement describe_splits_ex verb
This patch implements the describe_splits_ex verbs by querying the
size_estimates system table for all the estimates in the specified
token range.

If the keys_per_split argument is bigger then the
estimated partitions count, then we merge ranges until keys_per_split
is met. Note that the tokens can't be split any further,
keys_per_split might be less than the reported number of keys in one
or more ranges.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-24 22:44:01 +00:00
Duarte Nunes
ecfa04da77 system_keyspace: Add query_size_estimates() function
The query_size_estimates() function queries the size_estimates system
table for a given keyspace and table, filtering out the token ranges
according to the specified tokens.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-24 22:43:58 +00:00
Duarte Nunes
d984cc30bf size_estimates_recorder: Fix stop()
This patch fixes stop() by checking if the current CPU instead of
whether the service is active (which it won't be at the time stop() is
called).

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-24 22:43:58 +00:00
Duarte Nunes
e16f3f2969 system_keyspace: Avoid pointers in range_estimates
This patch makes range_estimates a proper struct, where tokens are
represented as dht::tokens rather than dht::ring_position*.

We also pass other arguments to update_ and clear_size_estimates by
copy, since one will already be required.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-24 22:43:35 +00:00
Asias He
2f4cd86809 random_partitioner: Implement random_partitioner
Cassandra 1.x clusters often use RandomPartitioner. Supporting
RandomPartitioner will allow easier migration to Scylla

Tests are added to make sure scylla generates the same token as
Cassandra does for the same partition key.

Fixes #1438

Message-Id: <3bc8b7f06fad16d59aaaa96e2827198ce74214c6.1469166766.git.asias@scylladb.com>
2016-07-24 16:25:25 +03:00
Avi Kivity
a0369a6d5b Merge seastar upstream
* seastar 9d1db3f...2425aab (5):
  > Merge "Track all threads globally" from Tomasz
  > net: provide remote address on native_server_socket_impl<Protocol>::accept()
  > Introduce install-dependencies.sh
  > reactor: make sure a poll cycle always happens when later is called
  > Merge "rpc: various fixes and cleanups" from Gleb
2016-07-24 16:22:05 +03:00
Avi Kivity
b339337287 Merge "Add more tools to the GDB script" from Tomasz 2016-07-24 16:21:31 +03:00
Duarte Nunes
2be45c4806 thrift: Handle and convert invalid_request_exception
This patch converts an exceptions::invalid_request_exception
into a Thrift InvalidRequestException instead of into a generic one.

This makes TitanDB work correctly, which expects an
InvalidRequestException when setting a non-existent keyspace.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1469362086-1013-1-git-send-email-duarte@scylladb.com>
2016-07-24 14:08:58 +02:00
Duarte Nunes
b5968ac244 sstables: Fix format string
This patch fixes a format string by using {} instead of %s.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1469352081-2167-1-git-send-email-duarte@scylladb.com>
2016-07-24 14:01:04 +02:00
Avi Kivity
de62285591 bloom_filter: use correct types
'long' is not a defined size.  It happens to match Java's long on Linux x86_64,
but may not on other platforms (e.g. Windows x64).
Message-Id: <1469352705-1079-1-git-send-email-avi@scylladb.com>
2016-07-24 14:00:37 +02:00
Avi Kivity
900639915d bloom_filter: fix overflow for large filters
We use ::abs(), which has an int parameter, on long arguments, resulting
in incorrect results.

Switch to std::abs() instead, which has the correct overloads.

Fixes #1494.

Message-Id: <1469347802-28933-1-git-send-email-avi@scylladb.com>
2016-07-24 11:31:26 +03:00
Vlad Zolotarov
9423c13419 cql_server::connection::process_prepare(): don't std::move() a shared_ptr captured by reference in value_of() lambda
A seastar::value_of() lambda used in a trace point was doing the unthinkable:
it called std::move() on a value captured by reference. Not only it compiled(!!!)
but it also actually std::move()ed the shared_ptr before it was used in a make_result()
which naturally caused a SIGSEG crash.

Fixes #1491

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Message-Id: <1469193763-27631-1-git-send-email-vladz@cloudius-systems.com>
2016-07-22 16:32:21 +03:00
Tomasz Grabiec
0db79c7153 scylla-gdb.py: Introduce "scylla mem-range" command
Prints memory range belonging to current shard.
2016-07-22 15:19:48 +02:00
Tomasz Grabiec
26e66bf49e scylla-gdb.py: Introduce "scylla thread" command
Switches into a seastar thread. "scylla unthread" switches back.

Example:

```
  (gdb) scylla thread 0x6000000e1800
  Switched to thread 1, (seastar::thread_context*) 0x6000000e1800

  (gdb) bt
  #0  seastar::thread_context::switch_out (this=0x6000000e1800) at core/thread.cc:104
  #1  0x00000000004cfb07 in future<>::wait() (this=0x600008ca2c70) at core/future.hh:817
  #2  0x0000000000f7752c in future<>::get() (this=0x600008ca2c70) at /home/tgrabiec/src/scylla2/seastar/core/future.hh:787
  ...
  #16 seastar::thread_context::main (this=0x6000000e1800) at core/thread.cc:166
  #17 0x000000000051a702 in seastar::thread_context::s_main (lo=<optimized out>, hi=<optimized out>) at core/thread.cc:157
  #18 0x00007f2c34861f20 in ?? () from /lib64/libc.so.6
  #19 0x0000000000000000 in ?? ()

  (gdp) scylla unthread
  Switched to thread 1
```
2016-07-22 15:19:48 +02:00
Tomasz Grabiec
d6fc9ad48e scylla-gdb.py: Break down code into finer abstractions 2016-07-22 15:19:48 +02:00
Raphael S. Carvalho
c4f34f5038 compaction: do not convert timestamp resolution to uppercase
C* only allows timestamp resolution in uppercase, so we shouldn't
be forgiving about it, otherwise migration to C* will not work.
Timestamp resolution is stored in compaction strategy options of
schema BTW.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <d64878fc9bbcf40fd8de3d0f08cce9f6c2fde717.1469133851.git.raphaelsc@scylladb.com>
2016-07-22 07:03:27 +01:00
Avi Kivity
106e3703d9 sstables: stop using unaligned_cast
unaligned_cast violates strict aliasing, and causes code misgeneration on
gcc 6.  Replace it with read_be/write_be, which are nicer anyway.
Message-Id: <1469122850-7511-1-git-send-email-avi@scylladb.com>
2016-07-22 07:03:08 +01:00
Raphael S. Carvalho
56a50784f8 compaction_manager: make registration of sstables and weight exception safe
Compacting sstables and weight could be left unregistered in event of an
exception. Let's make it safe by using a RAII approach.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <f2cf9d0c12f22046293bd2185ef14ede3f4d63d4.1469114161.git.raphaelsc@scylladb.com>
2016-07-22 07:02:48 +01:00
Vlad Zolotarov
4647ad9d8a tracing: set a default TTL for system_traces tables when they are created
Fixes #1482

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Message-Id: <1469104164-4452-1-git-send-email-vladz@cloudius-systems.com>
2016-07-22 07:02:29 +01:00
Vlad Zolotarov
57b58cad8e SELECT tracing instrumentation: improve inter-nodes communication stages messages
Add/fix "sending to"/"received from" messages.

With this patch the single key select trace with a data on an external node
looks as follows:

Tracing session: 65dbfcc0-4f51-11e6-8dd2-000000000001

 activity                                                                                                                        | timestamp                  | source    | source_elapsed
---------------------------------------------------------------------------------------------------------------------------------+----------------------------+-----------+----------------
                                                                                                              Execute CQL3 query | 2016-07-21 17:42:50.124000 | 127.0.0.2 |              0
                                                                                                   Parsing a statement [shard 1] | 2016-07-21 17:42:50.124127 | 127.0.0.2 |             --
                                                                                                Processing a statement [shard 1] | 2016-07-21 17:42:50.124190 | 127.0.0.2 |             64
 Creating read executor for token 2309717968349690594 with all: {127.0.0.1} targets: {127.0.0.1} repair decision: NONE [shard 1] | 2016-07-21 17:42:50.124229 | 127.0.0.2 |            103
                                                                            read_data: sending a message to /127.0.0.1 [shard 1] | 2016-07-21 17:42:50.124234 | 127.0.0.2 |            108
                                                                           read_data: message received from /127.0.0.2 [shard 1] | 2016-07-21 17:42:50.124358 | 127.0.0.1 |             14
                                                          read_data handling is done, sending a response to /127.0.0.2 [shard 1] | 2016-07-21 17:42:50.124434 | 127.0.0.1 |             89
                                                                               read_data: got response from /127.0.0.1 [shard 1] | 2016-07-21 17:42:50.124662 | 127.0.0.2 |            536
                                                                                  Done processing - preparing a result [shard 1] | 2016-07-21 17:42:50.124695 | 127.0.0.2 |            569
                                                                                                                Request complete | 2016-07-21 17:42:50.124580 | 127.0.0.2 |            580

Fixes #1481

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Message-Id: <1469112271-22818-1-git-send-email-vladz@cloudius-systems.com>
2016-07-21 19:46:43 +03:00
Tomasz Grabiec
5e8f0efc85 schema_tables: Fix hang during keyspace drop
Fixes #1484.

We drop tables as part of keyspace drop. Table drop starts with
creating a snapshot on all shards. All shards must use the same
snapshot timestamp which, among other things, is part of the snapshot
name. The timestamp is generated using supplied timestamp generating
function (joinpoint object). The joinpoint object will wait for all
shards to arrive and then generate and return the timestamp.

However, we drop tables in parallel, using the same joinpoint
instance. So joinpoint may be contacted by snapshotting shards of
tables A and B concurrently, generating timestamp t1 for some shards
of table A and some shards of table B. Later the remaining shards of
table A will get a different timestamp. As a result, different shards
may use different snapshot names for the same table. The snapshot
creation will never complete because the sealing fiber waits for all
shards to signal it, on the same name.

The fix is to give each table a separate joinpoint instance.

Message-Id: <1469117228-17879-1-git-send-email-tgrabiec@scylladb.com>
2016-07-21 19:14:57 +03:00
Vlad Zolotarov
e1480cd00d tracing: add a word "shard" to a "thread" value
Add a word "shard" to a "thread" column value.
From now its format is "shard X", where X is a shard index.

Fixes #1480

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Message-Id: <1469099732-23334-1-git-send-email-vladz@cloudius-systems.com>
2016-07-21 15:08:09 +03:00
Paweł Dziepak
1c7c944eee Merge "Thrift: small cleanups" from Duarte
"This patchset adds several small thrift related cleanups."
2016-07-21 11:56:51 +01:00
Duarte Nunes
8991d35231 thrift: Use database::find_schema directly
This patch changes lookup_schema() so it directly calls
database::find_schema() instead of going through
database::find_column_family(). It also drops conversion of the
no_such_column_family exeption, as that is already handled at a higher
layer.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-21 10:37:38 +00:00
Duarte Nunes
038d42c589 thrift: Remove hardcoded version constant
...and use the one in thrift_server.hh instead.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-21 10:37:33 +00:00
Duarte Nunes
8bb43d09b1 thrift: Remove unused with_cob_dereference function
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-21 10:13:55 +00:00
Paweł Dziepak
8a386a51bd Merge "Don't cache wide partitions" from Piotr
"When reading a partition try to read it all
but once more bytes are read than a given limit
we decide that partition is wide and we don't cache it.
Instead we retry the read with clustering key filtering
applied."
2016-07-21 10:24:25 +01:00
Benoît Canet
4ce7bced27 docker: Add documentation page for Docker Hub
Signed-of-by: Benoît Canet <benoit@scylladb.com>
Message-Id: <1466438296-5593-1-git-send-email-benoit@scylladb.com>
2016-07-21 12:22:57 +03:00
Yoav Kleinberger
d1d1be4c1a docker: bring docker image closer to a more 'standard' scylla installation
Previously, the Docker image could only be run interactively, which is
not conducive for running clusters. This patch makes the docker image
run in the background (using systemd). This makes the docker workflow
similar to working with virtual machines, i.e. the user launches a
container, and once it is running they can connect to it with

       docker exec -it <container_name> bash

and immediately use `cqlsh` to control it.

In addition, the configuration of scylla is done using established
scripts, such as `scylla_dev_mode_setup`, `scylla_cpuset_setup` and
`scylla_io_setup`, whereas previously code from these scripts was
duplicated into the docker startup file.

To specify seeds for making a cluster, use the --seeds command line
argument, e.g.

    docker run -d --privileged scylladb/scylla
    docker run -d --privileged scylladb/scylla --seeds 172.17.0.2

other options include --developer-mode, --cpuset, --broadcast-address

The --developer-mode option mode is on by default - so that we don't fail users
who just want to play with this.

The Dockerfile entrypoint script was rewritten as a few Python modules.
The move to Python is meritted because:

    * Using `sed` to manipulate YAML is fragile
    * Lack of proper command line parsing resulted in introducing ad-hoc environment variables
    * Shell scripts don't throw exceptions, and it's easy to forget to check exit codes for every single command

I've made an effort to make the entrypoint `go' script very simple and readable.
The goary details are hidden inside the other python modules.

Signed-off-by: Yoav Kleinberger <yoav@scylladb.com>
Message-Id: <1468938693-32168-1-git-send-email-yoav@scylladb.com>
2016-07-21 12:20:39 +03:00
Duarte Nunes
a436cf945c thrift: Omit regular columns for dynamic CFs
This patch skips adding the auto-generated regular column when
describing a dynamic Column family for the describe_keyspace(s) verbs.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1469091720-10113-1-git-send-email-duarte@scylladb.com>
2016-07-21 12:06:10 +03:00
Pekka Enberg
d90f8dc2d9 Merge "tracing: cleanup" from Vlad 2016-07-21 11:57:54 +03:00
Avi Kivity
a9e07b292b Merge seastar upstream
* seastar 103543a...9d1db3f (8):
  > reactor: limit task backlog
  > iotune: Fix SIGFPE with some executions
  > Merge "Preparation for protobuf" from Amnon
  > byteorder: add missing cpu_to_be(), be_to_cpu() functions
  > rpc: fix gcc-7 compilation error
  > reactor: Register the smp metrics disabled
  > scollectd: Allow creating metric that is disabled
  > Merge "Propagate timeout to a server" from Gleb
2016-07-21 10:54:48 +03:00