"
This patchset makes all users of query_processor specify their timeouts
explicitly, in preparation for the removal of
cql_statement::execute_internal() (whose main function was to override
timeouts).
"
* tag 'cql-explicit-timeouts/v1' of https://github.com/avikivity/scylla:
query_processor: require clients to specify timeout configuration
query_processor: un-default consistency level in make_internal_options
Like with the EXECUTE command avoid authorizing the same prepared
statement twice - this time in the context of processing the BATCH
command.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Add a cache that would store the checked weak pointer to already authorized prepared statements
and which key is a tuple of an authenticated_user and key of the prepared_statements_cache.
The entries will be held as long as the corresponding prepared statement is valid (cached)
and will be discarded with the period equal to the refresh period of the permissions cache.
Entries are also going to be discarded after 60 minutes if not used.
The purpose of this new cache is to save the lookup in the permissions cache for already authenticated
resource (whatever is needed to be authenticated for the particular prepared statement).
This is meant to improve the cache coherency as well (since we are going to look in a single cache
instead of two).
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Remove implicit timeouts and replace with caller-specified timeouts.
This allows removing the ambiguity about what timeout a statement is
executed with, and allows removing cql_statement::execute_internal(),
which mostly overrode timeouts and consistency levels.
Timeout selection is now as follows:
query_processor::*_internal: infinite timeout, CL=ONE
query_processor::process(), execute(): user-specified consisistency level and timeout
All callers were adjusted to specify an infinite timeout. This can be
further adjusted later to use the "other" timeout for DCL and the
read or write timeout (as needed) for authentication in the normal
query path.
Note that infinite timeouts don't mean that the query will hang; as
soon as the failure detector decides that the node is down, RPC
responses will termiante with a failure and the query will fail.
Make the consistency level explicit in the caller in order to clarify
what is going on.
An "internal" query used to mean that it was accessing local tables,
so infinite timeouts and a consistency level of ONE were indicated,
but authentication accesses non-local tables so explicit consistency
level and timeouts are needed.
Since storage_proxy provides access to the entire cluster, a local shard
reference is sufficient. Adjust query_processor to store a reference to
just the local shard, rather than a seastar::sharded<storage_proxy> and
adjust callers.
This simplifies the code a little.
Message-Id: <20180415142656.25370-3-avi@scylladb.com>
"
The original motivation for the "utils: introduce a loading_shared_values" series was a hinted handoff work where
I needed an on-demand asynchronously loading key-value container (a replica address to a commitlog instance map).
It turned out that we already have the classes that do almost what I needed:
- utils::loading_cache
- sstables::shared_index_lists
Therefore it made sense to find a common ground, unify this functionality and reuse the code both in the classes above and in the
new hinted handoff code.
This series introduces the utils::loading_shared_values that generalizes the sstables::shared_index_lists
API on top of bi::unordered_set with the rehashing logic from the utils::loading_cache triggered by an addition
of an entry to the set (PATCH1).
Then it reworks the sstables::shared_index_lists and utils::loading_cache on top of the new class (PATCH2 and PATCH3).
PATCH4 optimizes the loading_cache for the long timer period use case.
But then we have discovered that we have another "customer" for the loading_cache. Apparently our prepared statements cache
had a birth flaw - it was unlimited in size - unless the corresponding keyspace and/or table are modified/dropped the entries
are never evicted. We clearly need to limit its size and it would also make sense to evict the cache entries that haven't been
used long enough.
This seems like a perfect match for a utils::loading_cache except for prepared statements don't need to be reloaded after
they are created.
Patches starting from PATCH5 are dealing with adding the utils::loading_cache the missing functionality (like making the "reloading"
conditional and adding the synchronous methods like find(key)) and then transitioning the CQL and Thrift prepared statements
caches to utils::loading_cache.
This also fixes #2474."
* 'evict_unused_prepared-v5' of https://github.com/vladzcloudius/scylla:
tests: loading_cache_test: initial commit
cql3::query_processor: implement CQL and Thrift prepared statements caches using cql3::prepared_statements_cache
cql3: prepared statements cache on top of loading_cache
utils::loading_cache: make the size limitation more strict
utils::loading_cache: added static_asserts for checking the callbacks signatures
utils::loading_cache: add a bunch of standard synchronous methods
utils::loading_cache: add the ability to create a cache that would not reload the values
utils::loading_cache: add the ability to work with not-copy-constructable values
utils::loading_cache: add EntrySize template parameter
utils::loading_cache: rework on top of utils::loading_shared_values
sstables::shared_index_list: use utils::loading_shared_values
utils: introduce loading_shared_values
Forward-declare untyped_result_set and untyped_result_set_row, and remove
the include from query_processor.hh.
Message-Id: <20170916170859.27612-3-avi@scylladb.com>
- Transition the prepared statements caches for both CQL and Trhift to the cql3::prepared_statements_cache class.
- Add the corresponding metrics to the query_processor:
- Evictions count.
- Current entries count.
- Current memory footprint.
Fixes#2474
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Usually, internal queries are used for short queries. Sometimes though,
like in the case of get compaction history, there could be a large
amount of results. Without paging it will overload the system.
This patch adds the ability to use paging internally.
Using paging will be done explicitely, all the relevant information
would be store in an internal_query_state, that would hold both the
paging state but also the query so consecutive calls can be made.
To use paging use the query method with a function.
The function gets beside a statement and its parameters a function that
will be used for each of the returned rows.
For example if qp is a query_processor:
qp.query("SELECT * from system.compaction_history", [] (const cql3::untyped_result_set::row& row) {
....
// do something with row
...
return stop_iteration::no; // keep on reading
});
Will run the function on each of the compaction history table rows.
To stop the iteration, the function can return stop_iteration::yes.
- introcduced "seastarx.hh" header, which does a "using namespace seastar";
- 'net' namespace conflicts with seastar::net, renamed to 'netw'.
- 'transport' namespace conflicts with seastar::transport, renamed to
cql_transport.
- "logger" global variables now conflict with logger global type, renamed
to xlogger.
- other minor changes
Use seastar::checked_ptr<weak_ptr<pepared_statement>> instead of shared_ptr for passing prepared statements around.
This allows an easy tracking and handling of statements invalidation.
This implementation will throw an exception every time an invalidated
statement reference is dereferenced.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
This patch adds support for thrift prepared statements. It specializes
the result_message::prepared into two types:
result_message::prepared::cql and result_message::prepared::thrift, as
their identifiers have different types.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Add the statistics counter for a number of unprepared statements
executions and expose it with collectd.
Since in our implementation a number of unprepared statements executions
equals to a number of executions of prepare() function we may simply
increment the new statistics counter every time query_processor::get_statement()
is called.
Fixes#1068
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Message-Id: <1461503492-32228-1-git-send-email-vladz@cloudius-systems.com>
Replicates https://issues.apache.org/jira/browse/CASSANDRA-7910 :
"Prepare a statement with a wildcard in the select clause.
2. Alter the table - add a column
3. execute the prepared statement
Expected result - get all the columns including the new column
Actual result - get the columns except the new column"
We use boost::any to convert to and from database values (stored in
serlialized form) and native C++ values. boost::any captures information
about the data type (how to copy/move/delete etc.) and stores it inside
the boost::any instance. We later retrieve the real value using
boost::any_cast.
However, data_value (which has a boost::any member) already has type
information as a data_type instance. By teaching data_type intances about
the corresponding native type, we can elimiante the use of boost::any.
While boost::any is evil and eliminating it improves efficiency somewhat,
the real goal is growing native type support in data_type. We will use that
later to store native types in the cache, enabling O(log n) access to
collections, O(1) access to tuples, and more efficient large blob support.
It is common for some operations, like system table updates, to try and guarantee
some particular ordering of operations.
The way Origin does it, is by simply incrementing one to the current timestamp.
Our calls, however, are being dispatched through our internal query processor, which
has a builtin client_state.
Our client_state has a mechanism to guarantee monotonicity, by adding 1 if needed
to operations that happen subsequentially. By using a clock that is not wired up
to this mechanism, we can't really guarantee that if other operations happened to
get in between.
If we expose this mechanism through the query_processor, we will be able to guarantee
that.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
There's a CQL version constant in query processor. Use it when
advertising CQL version to clients.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
This patch adds initial support for PREPARE and EXECUTE requests which
are used by the CQL binary protocol for prepared statements. The use of
prepared statement gives a nice 2.5x single core performance boost for
Urchin:
$ ./build/release/seastar --data data --smp 1
$ ./tools/bin/cassandra-stress write -mode cql3 simplenative -rate threads=32
Results:
op rate : 31728
partition rate : 31728
row rate : 31728
latency mean : 1.0
latency median : 0.9
latency 95th percentile : 1.8
latency 99th percentile : 1.8
latency 99.9th percentile : 5.6
latency max : 181.7
Total operation time : 00:00:30
END
$ ./tools/bin/cassandra-stress write -mode cql3 simplenative prepared -rate threads=32
Results:
op rate : 75033
partition rate : 75033
row rate : 75033
latency mean : 0.4
latency median : 0.4
latency 95th percentile : 0.7
latency 99th percentile : 0.8
latency 99.9th percentile : 3.4
latency max : 205.0
Total operation time : 00:00:30
END
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
There's a cyclic dependency between cql3 and transport namespaces when
we introduce prepared statements. The latter pulls
parsed_statement::prepared from the former. Break the cycle by using
a forward delcaration in cql_statement.hh.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Query processor needs to store prepared statements as part of a client
session for PREPARE and EXECUTE requests. Switch from unique_ptr to
shared_ptr in preparation for that.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
s/database/distributed<database>/ everywhere.
Use simple distribution rules: writes are broadcast, reads are local.
This causes tremendous data duplication, but will change soon.
