Commit Graph

9949 Commits

Author SHA1 Message Date
Tomasz Grabiec
93a942dc53 scylla-gdb.py: Add "scylla mem-ranges" command
Prints memory ranges corresponding to seastar heap.

The ouput can be easily turned into arguemnts to "find".
2016-07-19 20:23:40 +02:00
Tomasz Grabiec
724f99641f scylla-gdb.py: Add "scylla shard" command
Beams you up to the thread corresponding to given seastar shard.

Example:

 (gdb) scylla shard 0
 Switched to thread 49
2016-07-19 20:23:40 +02:00
Tomasz Grabiec
d5fb452e94 scylla-gdb.py: Add "scylla apply" command
Executes given command on all shards.
2016-07-19 20:23:40 +02:00
Tomasz Grabiec
720d8149ba scylla-gdb.py: Add "scylla timers" command
Lists all active timers on current shard.
2016-07-19 20:23:40 +02:00
Tomasz Grabiec
6e4506a1f2 scylla-gdb.py: Add std::array wrapper which allows iteration 2016-07-19 20:23:40 +02:00
Tomasz Grabiec
29af5b22da scylla-gdb.py: Import intrusive_list() function
Imported from:

  https://github.com/cloudius-systems/osv/blob/master/scripts/loader.py
2016-07-19 20:23:40 +02:00
Raphael S. Carvalho
7b9cf528ad tests: fix occassional failure in date tiered test
That was a bug in the test itself. It could happen that a sstable would
incorrectly belong to the next time window if the current minute is
approaching its end. Fix is about having all sstables that we want in
the same time window with the same min/max timestamp.

Fixes #1448.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <ee25d49e7ed12b4cf7d018a08163404c3d122e56.1468782787.git.raphaelsc@scylladb.com>
2016-07-18 15:18:29 +02:00
Paweł Dziepak
4497204b7d streamed_mutation: do not leave mutation in an invalid state
This patch avoids moving entries from range tombstones and clustering
rows sets in streamed_mutation_from_mutation(). Such action breaks these
sets as the entries will be left in some unknown state.

Instead, the sets are being broken in a supported and predictable way
using unlink_leftmost_without_rebalance().

Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
Message-Id: <1468843205-18852-1-git-send-email-pdziepak@scylladb.com>
2016-07-18 15:14:21 +02:00
Avi Kivity
9d1b813f45 Merge seastar upstream
* seastar d699205...a45823a (5):
  > rpc: do not call shutdown function on already closed fd
  > log: Do not crash if logger is invoked from non-reactor thread
  > rpc: remove unaligned_cast and reinterpret_cast uses
  > unaligned: note unaligned_cast<> is deprecated
  > byteorder: add unaligned read/write helpers

Fixes #1463.
2016-07-18 15:24:43 +03:00
Avi Kivity
60491476e3 Merge "thrift: Add authentication and authorization" from Duarte
"This patchset implements the login verb to enable authentication
in the thrift API, and it adds access control to the already
implemented verbs."
2016-07-18 11:32:32 +03:00
Duarte Nunes
b6663f050d thrift: Add authorization for DML verbs
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-17 17:38:23 +00:00
Duarte Nunes
63354320b8 thrift: Add authorization to thrift DDL verbs
This patch adds authorization to the DDL thrift verbs. Since checking
for authorization is asynchronous, we now need to copy the verb
arguments so they can be accessed from the continuations.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-17 17:38:23 +00:00
Duarte Nunes
3c389ba871 client_state: Add has_schema_access function
This function is similar to has_column_family_access, but skips
validating if the specified keyspace and column family names map to a
valid schema, as it already takes one as an argument.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-17 17:38:23 +00:00
Duarte Nunes
dbbf4b3cc2 thrift: Group mutation map by column family
This patch transforms the mutation map, a map of keys to a map of columns
families to mutations, into a map of column families to a map of keys
to mutations. This makes is a more natural organization, as things
like checking access permissions are done by column family.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-17 17:38:23 +00:00
Duarte Nunes
f14628dc49 thrift: Introduce with_schema function
This is a wrapper around with_cob, which fetches a schema and forwards
it to a supplied function.

The patch also removes superfluous return instructions.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-17 17:38:23 +00:00
Duarte Nunes
09a5560b1b thrift: Validate login
This patch validates that a user is correctly logged in (if
authentication is required) for the required verbs.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-17 17:38:23 +00:00
Duarte Nunes
a3e507eb1c thrift: Implement login verb
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-17 17:38:23 +00:00
Amnon Heiman
d096a6762a scylla_setup: Ask if to start scylla-housekeeping
The scylla-server.service will try to start the scylla-housekeeping.

This patch adds a question to the scylla_setup if to enable the version
check, if the answer is no, the scylla-housekeeping will be masked.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1468741129-1977-1-git-send-email-amnon@scylladb.com>
2016-07-17 17:57:12 +03:00
Nadav Har'El
c647d917e0 sstables: move to_bytes_view to header file
Move the to_bytes_view(temporary_buffer<char>) function from source file
to header file where is can be used in more places.

This saves one use of reinterpret_cast (which we are no re-evaluating),
and moreover, we want to use this function also in the promoted index
code (to return a bytes_view from the promoted index which was saved as a
temporary_buffer).

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <1468761437-27046-1-git-send-email-nyh@scylladb.com>
2016-07-17 16:29:26 +03:00
Avi Kivity
b6b35a986a Merge seastar upstream
* seastar 5e97d5f...d699205 (3):
  > rpc: fix race between send loop and expiration timer
  > rpc: fix cancellable type move operations
  > reactor: create new files with a more reasonable default mode
2016-07-17 13:27:23 +03:00
Paweł Dziepak
81e4952c78 row_cache: fix marking last entry as continuous
Range queries need to take special care when transitioning between
ranges that are read from sstables and ranges that are already in the
cache.

Original code in such case just started a secondary reader and told it
to unconditionally mark the last entry as continuous (primary reader has
already returned an element tha immediately follows the range that is
going to be read form sstables).

However, that information may get stale. For instance, by the time
secondary reader finish reading its range the element immediately
following it may get evicted from the cache thus causing continuity flag
to be incorrectly set.

The solution is to ensure that the element immediately after the range
read from sstables is still in the cache.

Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
Message-Id: <1468586893-15266-1-git-send-email-pdziepak@scylladb.com>
2016-07-15 15:15:02 +02:00
Tomasz Grabiec
7328a8eff8 cql: modification_statement: Avoid copying keyspace and table names
Message-Id: <1468574135-4701-1-git-send-email-tgrabiec@scylladb.com>
2016-07-15 10:36:53 +01:00
Duarte Nunes
aaa76d58ba query: Move to_partition_range to dht namespace
This patch moves to_partition_range, from the query namespace
to the dht namespace, where it is a more natural fit.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <1468498060-19251-1-git-send-email-duarte@scylladb.com>
2016-07-15 10:41:52 +02:00
Tomasz Grabiec
32937f354e Merge branch 'duarten/thrift/dml/v9' from git@github.com:duarten/scylla.git
From Duarte:

This patchset adds support for the data manipulation verbs. It defers support
for super columns and mixed CFs (a static CF treated as dynamic) to later
patchsets.

Everything is done on top of storage_proxy; it was only necessary to modify the
layers below to add support for different kinds of limits: per partition row
limit, which corresponds to limiting the number of columns returned when
querying a dynamic CF, and limit on the number of partitions returned, so that
we can emulate the one thrift row per key model when querying dynamic CFs.

Ref #399
2016-07-14 18:26:07 +02:00
Duarte Nunes
df1234d86a thrift: Mark static CFs as non-compound
By default, the schema is marked as compound regardless of the
comparator. Since a composite comparator for static CFs is currently
unsupported (otherwise thrift column families would be
indistinguishable from CQL ones), just mark them as non-compound.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 18:20:55 +02:00
Duarte Nunes
901d4d1628 thrift: Skip CQL3 column families
This patch prevents CQL3 column families from being returned to
clients or subject to updates from thrift.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 18:20:54 +02:00
Duarte Nunes
92adbaab0a thrift: Warn about unimplemented features
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 18:20:54 +02:00
Duarte Nunes
a924f14441 thrift: Validate thrift Columns
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 18:20:54 +02:00
Duarte Nunes
7c1bf41b0d thrift: Implement truncate verb
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 18:20:54 +02:00
Duarte Nunes
4f440217e5 thrift: Implement remove verb
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 18:20:54 +02:00
Duarte Nunes
237e3b28d6 thrift: Implement insert verb
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 18:20:54 +02:00
Duarte Nunes
5c5056e4f9 thrift: Implement atomic_batch_mutate verb
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 18:20:54 +02:00
Duarte Nunes
f237b5ff19 thrift: Implement batch_mutate on top of storage_proxy
So that the specified consistency level can be respected.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 18:20:54 +02:00
Duarte Nunes
12dca9fdc9 thrift: Convert thrift Mutation to internal one
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 18:20:54 +02:00
Duarte Nunes
822a315dfa thrift: Implement get_multi_slice verb
The get_multi_slice verb is used to perform multiple slices on a
single row key in one operation. It takes a set of column_slices,
which we normalize to not contain any overlapping ranges.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 18:20:54 +02:00
Duarte Nunes
9792a77266 range: Add deoverlap function
This patch adds the deoverlap function to range.hh, which takes in a
vector of possibly overlapping ranges and returns a vector of
non-overlapping ranges covering the same values.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 18:20:41 +02:00
Duarte Nunes
c910a4639c thrift: Implement get_paged_slice verb
The get_paged_slice verb is similar to the get_range_slices verb,
except that it doesn't take a SlicePredicate. Instead, it takes a
column from which to start the query.

For dynamic CFs, we use the partition_slice::specific_ranges to single
out the first partition, and query starting from the start_column row.
For static CFs, we issue an initial query to fetch the remainder of
columns from the first partition, and at least one more query to fetch
the subsequent columns until the limit is reached. This implies a
performance penalty for static CFs.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 15:36:23 +02:00
Duarte Nunes
370572884c thrift: Implement get_range_slices verb
The get_range_slices verb is similar to the multiget_slice verb,
except that it operates on a range of partition keys (or tokens).

In origin, empty partitions are returned as part of the KeySlice, for
which the key will be filled in but the columns vector will be empty.
Since in our case we don't return empty partitions, we don't know which
partition keys in the specified range we should return back to the client.
So for now, our behavior differs from Origin.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 15:36:23 +02:00
Duarte Nunes
b872db55bd thrift: Implement get_count verb on top of multiget_count
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 15:36:23 +02:00
Duarte Nunes
a42b7ba3f7 thrift: Implement multiget_count verb
This patch implements the multiget_count verb in a similar fashion as
multiget_slice, but using an accumulator that counts the returned
columns instead of create thrift ColumnOrSuperColumn objects.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 15:36:23 +02:00
Duarte Nunes
a44561870a thrift: Implement get verb in terms of get_slice
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 15:36:23 +02:00
Duarte Nunes
db4c26d5b8 thrift: Implement get_slice in terms of multiget_slice
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 15:36:23 +02:00
Duarte Nunes
cd3a12535e thrift: Implement multiget_slice verb
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 15:36:23 +02:00
Duarte Nunes
acd39d871f thrift: Validate column names
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 15:36:23 +02:00
Duarte Nunes
4e9af0dc8e thrift: Make read_command from SlicePredicate
This patch build a query::read_command from a SlicePredicate,
for both dynamic and static column families.

For dynamic CFs, restrictions on the clustering columns are added, and
for static CFs, limits and ordering is defined inline by selecting the
correct regular columns.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 15:36:23 +02:00
Duarte Nunes
21d0a2c764 query: Optionally send cell ttl
This patch adds support to send a cell's ttl as part of a query's
result. This is needed for thrift support.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 15:36:23 +02:00
Duarte Nunes
eb8f5fafb2 thrift: Add partition key validation
This patch validates whether the specified partition key is not empty
and under the size limit.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 15:36:23 +02:00
Duarte Nunes
f57136f2f3 thrift: Make key_from_thrift take schema ref
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 15:36:23 +02:00
Duarte Nunes
e2b4cc4849 types: Add to_bytes_view function
This patch adds a function that converts a reference to an std::string
to a bytes_view.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 15:36:23 +02:00
Duarte Nunes
a647fea30b schema: Add is_dynamic to thrift_schema
This patch adds the is_dynamic() function to thrift_schema, which
tells whether the underlying column family is dynamic or not,
according to thrift rules.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
2016-07-14 15:36:23 +02:00