All cql_statement::execute_internal() overrides now either throw or
call execute(). Since we shouldn't be calling the throwing overrides
internally, we can safely call execute() instead. This allows us to
get rid of execute_internal().
execute_internal(), for some code paths, differs from execute by the
following:
1. it uses CL_ONE unconditionally
2. it has no query timeout
3. it doesn't use execution stages
for other code paths, it just calls execute.
As preparation for getting rid of execute_internal(), unify the two
code paths.
Commit 4859b759b9 caused the consistency level and timeouts
to be provided by the caller, so using the caller provided parameters
instead of overriding them does not change behavior.
"
This patchset makes all users of query_processor specify their timeouts
explicitly, in preparation for the removal of
cql_statement::execute_internal() (whose main function was to override
timeouts).
"
* tag 'cql-explicit-timeouts/v1' of https://github.com/avikivity/scylla:
query_processor: require clients to specify timeout configuration
query_processor: un-default consistency level in make_internal_options
"
Firstly, this patchset removes the is_fixed_length() function of
abstract_type in favour of value_length_if_fixed().
Secondly, it fixed the byte_type to be compatible with Cassandra which
erroneously treats it as a variable-length data type.
Lastly, it adds a unit test covering all non-composite CQL data types
for writing.
Tests: unit {release}
"
* 'projects/sstables-30/different-data-types/v1' of https://github.com/argenet/scylla:
tests: Add a unit test for writing different data types to SSTables 3.x format.
types: Treat byte_type as a variable-length type for compatibility reasons.
types: Remove is_value_fixed() and use value_length_if_fixed() instead.
Although values of the byte_type that corresponds to CQL TINYINT type
always occupy only a single byte, Cassandra treats this it as a
variable-length type for SSTables 3.0 reading and writing.
While it is clearly a mistake at Cassandra side, we have to stay
compatible.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
"
This patch series fixes#3405: secondary-index search only provided
correct results in certain cases, where entire partitions or contiguous
partition slices matched the query. When this was not the case, and
individual clustering rows match or do not match the query, the wrong
results were returned.
To fix this bug, we need to fix the two stages of secondary-index search:
1. In the first stage, we read from the index MV a list of row keys
(i.e., primary keys) matching the query. We can no longer remember
just the partition keys, and need to keep the list of full primary keys.
2. In the second stage, we have a list of rows (not partitions) and need
to read their selected contents to return to the user. Since CQL queries
do not have a syntax to select an arbitrary list of rows, we have to
add new code to do such a selection.
Because we provide an ad-hoc, inefficient, implementation for the row
selection described in stage 2, these patches leave two paths in the code:
The old path, efficiently selecting entire partitions, and the new path,
selecting individual rows. The old path is still used when it is applicable,
which is when a partition key column or the first clustering key column
is searched.
"
* 'si-fix-v4' of http://github.com/nyh/scylla:
secondary index: test multiple clustering column
secondary index: fix wrong results returned in certain cases
secondary index: method for fetching list of rows from base table
secondary index: method for fetching list of rows from index
select_statement.cc: refactor find_index_partition_ranges()
select_statement.cc: fix variable lifetime errors
This patch adds a test for secondary indexes on a table which has many
columns - two partition key column, two clustering key columns, and two
regular columns. We add a bunch of data in various rows and partitions,
index all columns and search on this data and verify the results.
This test exposed various bugs in secondary index search, including
issue #3405. After we fixed those bugs, the test now passes.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
The current secondary-index search code, in
indexed_table_select_statement::do_execute(), begins by fetching a list
of partitions, and then the content of these partitions from the base
table. However, in some cases, when the table has clustering columns and
not searching on the first one of them, doing this work in partition
granularity is wrong, and yields wrong results as demonstrated in
issue #3405.
So in this patch, we recognize the cases where we need to work in
clustering row granularity, and in those cases use the new functions
introduced in the previous patches - find_index_clustering_rows() and
the execute() variant taking a list of primary-keys of rows.
Fixes#3405.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
We add a new variant of select_statement::execute() which allows selecting
an arbitrary list of clustering rows. The existing execute() variant can't
do that - it can only take a list of *partitions*, and read the same
clustering rows from all of them.
The new select variant is not needed for regular CQL queries (which do
not have a syntax allowing reading a list of rows with arbitrary primary
keys), but we will need it for secondary index search, for solving
issue #3405.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
We already have a method find_index_partition_ranges(), to fetch a list
of partition keys from the secondary index. However, as we shall see in
the following patches (and see also issue #3405), getting a list of entire
partitions is not always enough - the secondary index actually holds a list
of primary keys, which includes clustering keys, and in some queries we
can't just ignore them.
So this patch provides a new method find_index_clustering_rows(), to
query the secondary index and get a list of matching clustering keys.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
The function find_index_partition_ranges() is used in secondary index
searches for fetching a list of matching partition. In a following patch,
we want to add a similar function for getting a list of *rows*. To avoid
duplicate code, in this patch we split parts of find_index_partition_ranges()
into two new functions:
1. get_index_schema() returns a pointer to the index view's schema.
2. read_posting_list() reads from this view the posting list (i.e., list
of keys) for the current searched value.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
do_with() provides code a *reference* to an object which will be kept
alive. It is a mistake to make a copy of this object or of parts of it,
because then the lifetime of this copy will have to be maintained as well.
In particular, it is a mistake to do do_with(..., [] (auto x) { ... }) -
note how "auto x" appears instead of the correct "auto& x". This causes
the object to be copied, and its lifetime not maintained.
This patch fixes several cases where this rule was broken in
select_statement.cc. I could not reproduce actual crashes caused by
these mistakes, but in theory they could have happened.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
"
This patchset implements reading row columns from SSTable 3 format data file.
Tests: units (release)
"
* 'haaawk/sstables3/read-columns-v4' of ssh://github.com/scylladb/seastar-dev: (21 commits)
Add test for reading column values of different types.
Support all fixed size column types from SSTable 3.x
Add abstract_type::value_length_if_fixed
Add test for simple table with value
flat_reader_assertions: Add produces_row taking column values
Implement reading rows and columns in data_consume_rows_context_m
Introduce column_flags_m
Add column_translation to data_consume_rows_context_m
Pass schema to data_consume_context
Add column_translation.hh
consumer_m: Add consume methods for consuming rows and columns
Extract make_atomic_cell from mp_row_consumer_k_l
Rename NON_STATIC_ROW_* states to ROW_BODY_*
Add liveness_info and use it in reading sstables
Add helper methods for parsing simple types.
Add unfiltered_flags_m::has_all_columns
data_consume_context: use make_unique instead of new
Pass serialization_header to data_consume_rows_context*
Use disk_string_vint_size for bytes_array_vint_size
Introduce disk_string_vint_size type
...
We need to specify --configfile on pdebuild too, otherwise we will
always fail to build .deb on newly created build environment.
Only reason why we still able to build .deb is we already copied
.pbuilderrc to home directory on existing build environment.
Fixes#3456
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20180523204112.24669-1-syuu@scylladb.com>
It will be needed to obtain column_translation that will
be added to data_consume_context in the next patch.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
New name describes the states in a better way as those states
will be used both for static and non-static rows.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
"
This series introduces a cache of already authenticated prepared statements which
is meant to optimize the prepared statement lookup when authentication is enabled.
This cache allows to perform a single cache lookup per EXECUTE operation as opposed
to at least 2 lookups: one in the prepared statements cache and one in the authentication
cache.
Tests:
- cql_query_test {debug, release}.
- cassandra-stress with authentication enabled and with short eviction timeout.
- Manual (with printouts) checks:
- Tested the eviction due to eviction in the prepared_statements_cache:
- Artificially decreased the prepared_statements_cache size and ran c-s with different keyspaces.
- Verified that the corresponding authorized_prepared_statements_cache entry is evicted and re-populated.
- Tested the BATCH of prepared statements (with dtest infrastructure):
- Verified that for each prepared statement authorized_prepared_statements_cache is updated only once:
- The batch contained a few entries of the same prepared statement.
"
* 'authorized_prepared_statements_cache-v3' of https://github.com/vladzcloudius/scylla:
cql3: use authorized_prepared_statements_cache in the BATCH processing
cql3::statements::batch_statement: introduce a single_statement class
cql3: introduce the authorized_prepared_statements_cache class
loading_shared_values: introduce the templated find() overload
tests: loading_cache_test: add a tests for a loading_cache::remove(key)/remove(iterator)
utils::loading_cache: add remove(key)/remove(iterator) methods
cql3::query_processor: properly stop() prepared_statements_cache object
Since Seastar no longer (1f005fb434) requires libunwind, we can
drop it from our dependency list. This helps the power build, for
which no libunwind is available.
Fixes#3453.
Message-Id: <20180523114750.10753-1-avi@scylladb.com>
"
This series implements the backlog tracker for TWCS, allowing it to
be controlled. The backlog for a TWCS colum family is just the sum of
the SizeTiered backlogs for all the windows that we know about.
A possible optimization for this is to stop tracking windows after
they become old enough and revert to zero backlog. I reverted that
last minute, though, since this will probably cause the backlog to
completely misrepresent reality if we import SSTables into old buckets
with things like repairs or nodetool refresh.
"
* 'twcs-backlog-v4.1' of github.com:glommer/scylla:
backlog: implement backlog tracker for the TWCS
STCS_backlog: allow users to query for the total bytes managed
backlog: keep track of maximum timestamp in write monitor
memtable: also keep track of max timestamp
The TWCS backlog is relatively simple: we just need to keep track of
which SSTable belong to which time window (and actually as usual,
just their sizes). That is an easy thing to do since we can statically
calculate the time bound from the timestamp.
Once we do that we can just sum the backlogs for each individual window.
Time windows that are well enough into the past can be at some point
discarded when their backlogs become zero.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
The exploded_clustering_prefix type has a convenient is_empty() method
and an even more convenient "operator bool" shortcut. Unfortunately,
the other clustering prefix types (clustering_key_prefix,
clustering_key_prefix_view) have, for historic reasons, an is_empty
method which takes a schema parameter. That also means they can't
have an "operator bool" shortcut.
But checking if a prefix doesn't really need the schema - all we need to
check is whether the byte representation is empty. The result is simpler
and more efficient code, and easier to use. It is also more consistent -
all clustering-key-related types will have an "operator bool" instead of
just some of them.
To avoid massive code changes, we leave a is_empty(schema) variant, which
simply calls is_empty(). There's already precedent for that - various
methods which have a variant taking schema (and ignoring it) and one
taking nothing.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20180521174220.13262-1-nyh@scylladb.com>
Like with the EXECUTE command avoid authorizing the same prepared
statement twice - this time in the context of processing the BATCH
command.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
This is a helper class needed to control the handling process of a single
statement in the current batch. In particular it has the boolean defining
if the authorization is needed for this statement.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Add a cache that would store the checked weak pointer to already authorized prepared statements
and which key is a tuple of an authenticated_user and key of the prepared_statements_cache.
The entries will be held as long as the corresponding prepared statement is valid (cached)
and will be discarded with the period equal to the refresh period of the permissions cache.
Entries are also going to be discarded after 60 minutes if not used.
The purpose of this new cache is to save the lookup in the permissions cache for already authenticated
resource (whatever is needed to be authenticated for the particular prepared statement).
This is meant to improve the cache coherency as well (since we are going to look in a single cache
instead of two).
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
