Commit Graph

10041 Commits

Author SHA1 Message Date
Avi Kivity
b339337287 Merge "Add more tools to the GDB script" from Tomasz 2016-07-24 16:21:31 +03:00
Duarte Nunes
2be45c4806 thrift: Handle and convert invalid_request_exception
This patch converts an exceptions::invalid_request_exception
into a Thrift InvalidRequestException instead of into a generic one.

This makes TitanDB work correctly, which expects an
InvalidRequestException when setting a non-existent keyspace.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1469362086-1013-1-git-send-email-duarte@scylladb.com>
2016-07-24 14:08:58 +02:00
Duarte Nunes
b5968ac244 sstables: Fix format string
This patch fixes a format string by using {} instead of %s.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1469352081-2167-1-git-send-email-duarte@scylladb.com>
2016-07-24 14:01:04 +02:00
Avi Kivity
de62285591 bloom_filter: use correct types
'long' is not a defined size.  It happens to match Java's long on Linux x86_64,
but may not on other platforms (e.g. Windows x64).
Message-Id: <1469352705-1079-1-git-send-email-avi@scylladb.com>
2016-07-24 14:00:37 +02:00
Avi Kivity
900639915d bloom_filter: fix overflow for large filters
We use ::abs(), which has an int parameter, on long arguments, resulting
in incorrect results.

Switch to std::abs() instead, which has the correct overloads.

Fixes #1494.

Message-Id: <1469347802-28933-1-git-send-email-avi@scylladb.com>
2016-07-24 11:31:26 +03:00
Vlad Zolotarov
9423c13419 cql_server::connection::process_prepare(): don't std::move() a shared_ptr captured by reference in value_of() lambda
A seastar::value_of() lambda used in a trace point was doing the unthinkable:
it called std::move() on a value captured by reference. Not only it compiled(!!!)
but it also actually std::move()ed the shared_ptr before it was used in a make_result()
which naturally caused a SIGSEG crash.

Fixes #1491

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Message-Id: <1469193763-27631-1-git-send-email-vladz@cloudius-systems.com>
2016-07-22 16:32:21 +03:00
Tomasz Grabiec
0db79c7153 scylla-gdb.py: Introduce "scylla mem-range" command
Prints memory range belonging to current shard.
2016-07-22 15:19:48 +02:00
Tomasz Grabiec
26e66bf49e scylla-gdb.py: Introduce "scylla thread" command
Switches into a seastar thread. "scylla unthread" switches back.

Example:

```
  (gdb) scylla thread 0x6000000e1800
  Switched to thread 1, (seastar::thread_context*) 0x6000000e1800

  (gdb) bt
  #0  seastar::thread_context::switch_out (this=0x6000000e1800) at core/thread.cc:104
  #1  0x00000000004cfb07 in future<>::wait() (this=0x600008ca2c70) at core/future.hh:817
  #2  0x0000000000f7752c in future<>::get() (this=0x600008ca2c70) at /home/tgrabiec/src/scylla2/seastar/core/future.hh:787
  ...
  #16 seastar::thread_context::main (this=0x6000000e1800) at core/thread.cc:166
  #17 0x000000000051a702 in seastar::thread_context::s_main (lo=<optimized out>, hi=<optimized out>) at core/thread.cc:157
  #18 0x00007f2c34861f20 in ?? () from /lib64/libc.so.6
  #19 0x0000000000000000 in ?? ()

  (gdp) scylla unthread
  Switched to thread 1
```
2016-07-22 15:19:48 +02:00
Tomasz Grabiec
d6fc9ad48e scylla-gdb.py: Break down code into finer abstractions 2016-07-22 15:19:48 +02:00
Raphael S. Carvalho
c4f34f5038 compaction: do not convert timestamp resolution to uppercase
C* only allows timestamp resolution in uppercase, so we shouldn't
be forgiving about it, otherwise migration to C* will not work.
Timestamp resolution is stored in compaction strategy options of
schema BTW.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <d64878fc9bbcf40fd8de3d0f08cce9f6c2fde717.1469133851.git.raphaelsc@scylladb.com>
2016-07-22 07:03:27 +01:00
Avi Kivity
106e3703d9 sstables: stop using unaligned_cast
unaligned_cast violates strict aliasing, and causes code misgeneration on
gcc 6.  Replace it with read_be/write_be, which are nicer anyway.
Message-Id: <1469122850-7511-1-git-send-email-avi@scylladb.com>
2016-07-22 07:03:08 +01:00
Raphael S. Carvalho
56a50784f8 compaction_manager: make registration of sstables and weight exception safe
Compacting sstables and weight could be left unregistered in event of an
exception. Let's make it safe by using a RAII approach.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <f2cf9d0c12f22046293bd2185ef14ede3f4d63d4.1469114161.git.raphaelsc@scylladb.com>
2016-07-22 07:02:48 +01:00
Vlad Zolotarov
4647ad9d8a tracing: set a default TTL for system_traces tables when they are created
Fixes #1482

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Message-Id: <1469104164-4452-1-git-send-email-vladz@cloudius-systems.com>
2016-07-22 07:02:29 +01:00
Vlad Zolotarov
57b58cad8e SELECT tracing instrumentation: improve inter-nodes communication stages messages
Add/fix "sending to"/"received from" messages.

With this patch the single key select trace with a data on an external node
looks as follows:

Tracing session: 65dbfcc0-4f51-11e6-8dd2-000000000001

 activity                                                                                                                        | timestamp                  | source    | source_elapsed
---------------------------------------------------------------------------------------------------------------------------------+----------------------------+-----------+----------------
                                                                                                              Execute CQL3 query | 2016-07-21 17:42:50.124000 | 127.0.0.2 |              0
                                                                                                   Parsing a statement [shard 1] | 2016-07-21 17:42:50.124127 | 127.0.0.2 |             --
                                                                                                Processing a statement [shard 1] | 2016-07-21 17:42:50.124190 | 127.0.0.2 |             64
 Creating read executor for token 2309717968349690594 with all: {127.0.0.1} targets: {127.0.0.1} repair decision: NONE [shard 1] | 2016-07-21 17:42:50.124229 | 127.0.0.2 |            103
                                                                            read_data: sending a message to /127.0.0.1 [shard 1] | 2016-07-21 17:42:50.124234 | 127.0.0.2 |            108
                                                                           read_data: message received from /127.0.0.2 [shard 1] | 2016-07-21 17:42:50.124358 | 127.0.0.1 |             14
                                                          read_data handling is done, sending a response to /127.0.0.2 [shard 1] | 2016-07-21 17:42:50.124434 | 127.0.0.1 |             89
                                                                               read_data: got response from /127.0.0.1 [shard 1] | 2016-07-21 17:42:50.124662 | 127.0.0.2 |            536
                                                                                  Done processing - preparing a result [shard 1] | 2016-07-21 17:42:50.124695 | 127.0.0.2 |            569
                                                                                                                Request complete | 2016-07-21 17:42:50.124580 | 127.0.0.2 |            580

Fixes #1481

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Message-Id: <1469112271-22818-1-git-send-email-vladz@cloudius-systems.com>
2016-07-21 19:46:43 +03:00
Tomasz Grabiec
5e8f0efc85 schema_tables: Fix hang during keyspace drop
Fixes #1484.

We drop tables as part of keyspace drop. Table drop starts with
creating a snapshot on all shards. All shards must use the same
snapshot timestamp which, among other things, is part of the snapshot
name. The timestamp is generated using supplied timestamp generating
function (joinpoint object). The joinpoint object will wait for all
shards to arrive and then generate and return the timestamp.

However, we drop tables in parallel, using the same joinpoint
instance. So joinpoint may be contacted by snapshotting shards of
tables A and B concurrently, generating timestamp t1 for some shards
of table A and some shards of table B. Later the remaining shards of
table A will get a different timestamp. As a result, different shards
may use different snapshot names for the same table. The snapshot
creation will never complete because the sealing fiber waits for all
shards to signal it, on the same name.

The fix is to give each table a separate joinpoint instance.

Message-Id: <1469117228-17879-1-git-send-email-tgrabiec@scylladb.com>
2016-07-21 19:14:57 +03:00
Vlad Zolotarov
e1480cd00d tracing: add a word "shard" to a "thread" value
Add a word "shard" to a "thread" column value.
From now its format is "shard X", where X is a shard index.

Fixes #1480

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Message-Id: <1469099732-23334-1-git-send-email-vladz@cloudius-systems.com>
2016-07-21 15:08:09 +03:00
Paweł Dziepak
1c7c944eee Merge "Thrift: small cleanups" from Duarte
"This patchset adds several small thrift related cleanups."
2016-07-21 11:56:51 +01:00
Duarte Nunes
8991d35231 thrift: Use database::find_schema directly
This patch changes lookup_schema() so it directly calls
database::find_schema() instead of going through
database::find_column_family(). It also drops conversion of the
no_such_column_family exeption, as that is already handled at a higher
layer.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-21 10:37:38 +00:00
Duarte Nunes
038d42c589 thrift: Remove hardcoded version constant
...and use the one in thrift_server.hh instead.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-21 10:37:33 +00:00
Duarte Nunes
8bb43d09b1 thrift: Remove unused with_cob_dereference function
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-21 10:13:55 +00:00
Paweł Dziepak
8a386a51bd Merge "Don't cache wide partitions" from Piotr
"When reading a partition try to read it all
but once more bytes are read than a given limit
we decide that partition is wide and we don't cache it.
Instead we retry the read with clustering key filtering
applied."
2016-07-21 10:24:25 +01:00
Benoît Canet
4ce7bced27 docker: Add documentation page for Docker Hub
Signed-of-by: Benoît Canet <benoit@scylladb.com>
Message-Id: <1466438296-5593-1-git-send-email-benoit@scylladb.com>
2016-07-21 12:22:57 +03:00
Yoav Kleinberger
d1d1be4c1a docker: bring docker image closer to a more 'standard' scylla installation
Previously, the Docker image could only be run interactively, which is
not conducive for running clusters. This patch makes the docker image
run in the background (using systemd). This makes the docker workflow
similar to working with virtual machines, i.e. the user launches a
container, and once it is running they can connect to it with

       docker exec -it <container_name> bash

and immediately use `cqlsh` to control it.

In addition, the configuration of scylla is done using established
scripts, such as `scylla_dev_mode_setup`, `scylla_cpuset_setup` and
`scylla_io_setup`, whereas previously code from these scripts was
duplicated into the docker startup file.

To specify seeds for making a cluster, use the --seeds command line
argument, e.g.

    docker run -d --privileged scylladb/scylla
    docker run -d --privileged scylladb/scylla --seeds 172.17.0.2

other options include --developer-mode, --cpuset, --broadcast-address

The --developer-mode option mode is on by default - so that we don't fail users
who just want to play with this.

The Dockerfile entrypoint script was rewritten as a few Python modules.
The move to Python is meritted because:

    * Using `sed` to manipulate YAML is fragile
    * Lack of proper command line parsing resulted in introducing ad-hoc environment variables
    * Shell scripts don't throw exceptions, and it's easy to forget to check exit codes for every single command

I've made an effort to make the entrypoint `go' script very simple and readable.
The goary details are hidden inside the other python modules.

Signed-off-by: Yoav Kleinberger <yoav@scylladb.com>
Message-Id: <1468938693-32168-1-git-send-email-yoav@scylladb.com>
2016-07-21 12:20:39 +03:00
Duarte Nunes
a436cf945c thrift: Omit regular columns for dynamic CFs
This patch skips adding the auto-generated regular column when
describing a dynamic Column family for the describe_keyspace(s) verbs.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1469091720-10113-1-git-send-email-duarte@scylladb.com>
2016-07-21 12:06:10 +03:00
Pekka Enberg
d90f8dc2d9 Merge "tracing: cleanup" from Vlad 2016-07-21 11:57:54 +03:00
Avi Kivity
a9e07b292b Merge seastar upstream
* seastar 103543a...9d1db3f (8):
  > reactor: limit task backlog
  > iotune: Fix SIGFPE with some executions
  > Merge "Preparation for protobuf" from Amnon
  > byteorder: add missing cpu_to_be(), be_to_cpu() functions
  > rpc: fix gcc-7 compilation error
  > reactor: Register the smp metrics disabled
  > scollectd: Allow creating metric that is disabled
  > Merge "Propagate timeout to a server" from Gleb
2016-07-21 10:54:48 +03:00
Piotr Jastrzebski
7d29cdf81f Add tests for wide partiton handling in cache.
They shouldn't be cached.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2016-07-21 09:48:09 +02:00
Piotr Jastrzebski
37a7d49676 Add collectd counter for uncached wide partitions.
Keep track of every read of wide partition that's
not cached.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2016-07-21 09:47:49 +02:00
Piotr Jastrzebski
636a4acfd0 Add flag to configure
max size of a cached partition.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2016-07-21 09:47:20 +02:00
Piotr Jastrzebski
98c12dc2e2 Try to read whole streamed_mutation up to limit
If limit is exceeded then return the streamed_mutation
and don't cache it.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2016-07-21 09:35:35 +02:00
Piotr Jastrzebski
0d39bb1ad0 Implement mutation_from_streamed_mutation_with_limit
If mutation is bigger than this limit
it won't be read and mutation_from_streamed_mutation
will return empty optional.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2016-07-21 09:35:23 +02:00
Avi Kivity
5e3b019447 Merge "Fix sstable reader for duplicated range tombstones" from Paweł
"This series fixes sstable reader so that it can handle duplicated range
tombstones which may appear if promoted index is used."
2016-07-21 10:13:29 +03:00
Vlad Zolotarov
276ef041d2 tracing: use "trace" log level instead of "debug".
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2016-07-20 19:50:40 +03:00
Vlad Zolotarov
d7d72c4cd4 tracing: "inline" cleanup
- Don't use inline for templates.
   - Put "inline" qualifier for out-of-class defined methods
     where they are defined and not where they are declared.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2016-07-20 19:49:30 +03:00
Vlad Zolotarov
5376b053f9 tracing: use seastar::format() for formatted trace()
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2016-07-20 19:49:30 +03:00
Duarte Nunes
8d87976fa1 tracing: Downgrade debug level log messages
Two periodic debug log level messages triggered by
tracing::write_timer_callback() quickly fill the logs, so this patch
downgrades them to trace level.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Reviewed-by: Vlad Zolotarov <vladz@scylladb.com>
Message-Id: <1469003034-1147-1-git-send-email-duarte@scylladb.com>
2016-07-20 11:34:02 +03:00
Pekka Enberg
aff8cf319d db/config: Start Thrift server by default
We have Thrift support now so start the server by default.
Message-Id: <1469002000-26767-1-git-send-email-penberg@scylladb.com>
2016-07-20 09:25:44 +01:00
Avi Kivity
5ceb55827d Merge "Add more tools to the gdb script" from Tomasz 2016-07-20 11:21:58 +03:00
Tomasz Grabiec
d2f0711608 scylla-gdb: Fix bounds checking in scylla ptr command
Message-Id: <1468951987-10184-1-git-send-email-tgrabiec@scylladb.com>
2016-07-20 11:20:45 +03:00
Duarte Nunes
64dff69077 thrift: Actually concatenate strings
This patch fixes concatenating a char[] with an int by using sprint
instead of just increasing the pointer.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1468971542-9600-1-git-send-email-duarte@scylladb.com>
2016-07-20 11:08:44 +03:00
Tomasz Grabiec
0d26294fac database: Add table name to log message about sealing
Message-Id: <1468917744-2539-1-git-send-email-tgrabiec@scylladb.com>
2016-07-20 10:12:31 +03:00
Tomasz Grabiec
a0832f08d2 schema_tables: Add more logging
Message-Id: <1468917771-2592-1-git-send-email-tgrabiec@scylladb.com>
2016-07-20 10:12:00 +03:00
Pekka Enberg
7b5c2266b6 Merge "date tiered compaction strategy options" from Raphael
"After this patchset, user can now tune date tiered compaction strategy by
playing with its parameters."
2016-07-20 09:13:08 +03:00
Raphael S. Carvalho
cf54af9e58 tests: add new test for date tiered strategy
This test set the time window to 1 hour and checks that the strategy
works accordingly.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2016-07-19 19:20:40 -03:00
Raphael S. Carvalho
eaa6e281a2 compaction: implement date tiered compaction strategy options
Now date tiered compaction strategy will take into account the
strategy options which are defined in the schema.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2016-07-19 19:20:35 -03:00
Tomasz Grabiec
93a942dc53 scylla-gdb.py: Add "scylla mem-ranges" command
Prints memory ranges corresponding to seastar heap.

The ouput can be easily turned into arguemnts to "find".
2016-07-19 20:23:40 +02:00
Tomasz Grabiec
724f99641f scylla-gdb.py: Add "scylla shard" command
Beams you up to the thread corresponding to given seastar shard.

Example:

 (gdb) scylla shard 0
 Switched to thread 49
2016-07-19 20:23:40 +02:00
Tomasz Grabiec
d5fb452e94 scylla-gdb.py: Add "scylla apply" command
Executes given command on all shards.
2016-07-19 20:23:40 +02:00
Tomasz Grabiec
720d8149ba scylla-gdb.py: Add "scylla timers" command
Lists all active timers on current shard.
2016-07-19 20:23:40 +02:00
Tomasz Grabiec
6e4506a1f2 scylla-gdb.py: Add std::array wrapper which allows iteration 2016-07-19 20:23:40 +02:00