Both the Prometheus and the API servers are used for maintenance
operations, similarly to streaming. Run them under the streaming
scheduling group to prevent them from impacting normal operations,
and rename the streaming scheduling group to reflect the more
generic role.
This helps to prevent spikes from Prometheus or API requests from
interfering with the normal workload. Using an existing group is
preferable to creating a new group because in the worst case, all
the non-main-workload groups compete with the main workload.
Consolidating them allows us to give them significant shares in
total without increasing competition in the worst case.
The group's label is unchanged to preserve compatibility with
dashboards.
A nice side effect is that repair, which is initiated by API calls,
gets placed into the maintenance group naturally. Compaction tasks
which are run by compaction manager are not changed.
Message-Id: <20180714160723.23655-1-avi@scylladb.com>
Most queries run without tracing (and those that run with tracing
are not sensitive to a few cycles), so mark the tracing paths as
cold.
Message-Id: <20180723133000.30482-1-avi@scylladb.com>
This will allow continuous integration to use the optimal number
of compiler jobs, without having to resort to complex calculations
from its scripting environment.
Message-Id: <20180722172050.13148-1-avi@scylladb.com>
"
This series adds some optimisations to the paging logic, that attempt to
close the performance gap between paged and not paged queries. The
former are more complex so always are going to be slower, but the
performance loss was unacceptably large.
Fixes#3619.
Performance with paging:
./perf_paging_before ./perf_paging_after diff
read 271246.13 312815.49 15.3%
Without paging:
./perf_nopaging_before ./perf_nopaging_after diff
read 343732.17 342575.77 -0.3%
Tests: unit(release), dtests(paging_test.py, paging_additional_test.py)
"
* tag 'optimise-paging/v1' of https://github.com/pdziepak/scylla:
cql3: select statement: don't copy metadata if not needed
cql3: query_options: make simple getter inlineable
cql3: metadata: avoid copying column information
query_pager: avoid visiting result_view if not needed
query::result_view: add get_last_partition_and_clustering_key()
query::result_reader: fix const correctness
tests/uuid: add more tests including make_randm_uuid()
utils: uuid: don't use std::random_device()
std::random_device() uses the relatively slow /dev/urandom, and we rarely if
ever intend to use it directly - we normally want to use it to seed a faster
random_engine (a pseudo-random number generator).
In many places in the code, we first created a random_device variable, and then
using it created a random_engine variable. However, this practice created the
risk of a programmer accidentally using the random_device object, instead of the
random_engine object, because both have the same API; This hurts performance.
This risk materialized in just two places in the code, utils/uuid.cc and
gms/gossiper.cc. A patch for to uuid.cc was sent previously by Pawel and is
not included in this patch, and the fix for gossiper.{cc,hh} is included here.
To avoid risking the same mistake in the future, this patch switches across the
code to an idiom where the random_device object is not *named*, so cannot be
accidentally used. We use the following idiom:
std::default_random_engine _engine{std::random_device{}()};
Here std::random_device{}() creates the random device (/dev/urandom) and pulls
a random integer from it. It then uses this seed to create the random_engine
(the pseudo-random number generator). The std::random_device{} object is
temporary and unnamed, and cannot be unintentionally used directly.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20180726154958.4405-1-nyh@scylladb.com>
The column-related metadata is shared by all requests done with the same
perpared query. However, metadata class contains also some additional
flags and paging state which may differ. This patch allows sharing
column information among multiple instances of the metadata class.
query::result_visitor provides get_last_partition_and_clustering_key()
which allows getting those without iterating through the whole result.
Moreover, row count may be precomputed in the result, if it isn't there
is query::result_view::count_partitions_and_rows() for getting it.
Paging needs to get last partition and clustering key (if the latter
exists). Previously, this was done by result_view visitor but that is
suboptimal. Let's add a direct getter for those.
mock outputs files owned by root. This causes attempts
by scripts that want to junk the working directory (typically
continuous integration) to fail on permission errors.
Fixup those permissions after the fact.
Message-Id: <20180719163553.5186-1-avi@scylladb.com>
Count operations which were started on one shard and
were performed on another, due to non-shard-aware driver
and/or RPC.
Message-Id: <20180723155118.8545-1-avi@scylladb.com>
"
This mini-series covers a regression caused by newest versions
of jsoncpp library, which changed the way of quoting UTF-8 strings.
Tests: unit (release)
"
* 'add_json_quoting_3' of https://github.com/psarna/scylla:
tests: add JSON unit test
types: use value_to_quoted_string in JSON quoting
json: add value_to_quoted_string helper function
Ref #3622.
Reviewed-by: Nadav Har'El <nyh@scylladb.com>
After open-source-parsers/jsoncpp@42a161f commit jsoncpp's version
of valueToQuotedString no longer fits our needs, because too many
UTF-8 characters are unnecessarily escaped. To remedy that,
this commit provides our own string quoting implementation.
Reported-by: Nadav Har'El <nyh@scylladb.com>
Refs #3622
Previously CQL grammar wrongfully required INSERT JSON queries
to provide a list of columns, even though they are already
present in JSON itself.
Unfortunately, tests were written with this false assumption as well,
so they're are updated.
Message-Id: <33b496cba523f0f27b6cbf5539a90b6feb20269e.1532514111.git.sarna@scylladb.com>
Compactions start and end all the time, especially with many shards,
and don't contribute much to understanding what is going on these
days. Compaction throughput is available through the metrics and
other information is available via the compaction history table.
Demote compaction start and end messages to DEBUG level to keep
the log clean. Cleaning and resharding compactions are kept as
INFO, at least for now, since they are manual operations and
therefore rarer.
Message-Id: <20180724132859.14109-1-avi@scylladb.com>
"
This series follows up ALLOW FILTERING support series and depends on
this one: https://groups.google.com/d/msg/scylladb-dev/Qxt3_MP03jI/5ZhRTJ3gBwAJ
The following optimizations regarding clustering key prefix and filtering are
applied:
* if clustering key restrictions require filtering, but they still
contain any part of the prefix, this prefix can be used to narrow
down the query by using it in computing clustering bounds
* if an indexed query has partition key restrictions and any clustering
key restrictions that form a prefix, then from now on this prefix
will be used to narrow down the index query
"
Ref #3611.
* 'use_prefix_with_filtering_and_si_4' of https://github.com/psarna/scylla:
tests: add prefix cases to indexed filtered queries tests
cql3: use ck prefix in filtered queries
cql3: use clustering key prefix in index queries
cql3: add conversion to ck longest prefix restrictions
cql3: add prefix_size method to ck restrictions
If an indexed query has partition+clustering key restrictions as well
and at least some of these restrictions create a prefix, this prefix
is used in the index query to narrow down the number of rows read.
Refs #3611
sstable close is an asychronous operation launched in the background,
so we can't wait for it. If the test ends before all operations are
complete, the background operations are detected as leaks.
We need either a proper close(), or maybe a sstables::quiesce() that
waits until there are no sstables alive on the shard, but until then,
a hack.
"
This patchset authored by Piotr fixes ck filtering and fast forwarding in SSTables 3.x.
For now only clustering rows are supported and range tombstones will come next.
Test: unit {release}
"
* 'projects/sstables-30/filtering/v5' of https://github.com/argenet/scylla:
sstables: Minor clean-up and renaming to clustering_ranges_walker.
sstables: Add test for filtering and forwarding
sstables: Fix schema for static row tests
sstables: Fix ck filtering and fast forwarding
sstables: Introduce mutation_fragment_filter
"
This series changes the native CQL3 protocl layer so that it works with
fragmented buffers instead of a single temporary_buffer per request.
The main part is fragmented_temporary_buffer which represents a
fragmented buffer consisting of multiple temporary_buffers. It provides
helpers for reading fragmented buffer from an input_stream, interpreting
the data in the fragmented buffer as well as view that satisfy
FragmentRange concept.
There are still situations where a fragmented buffer is linearised. That
includes decompressing client requests (this uses reusable buffers in a
similar way to the code that sends compressed responses), CQL statement
restrictions and values that are hard-coded in prepared statements
(hopefully, the values in those cases will be small), value validation
in some cases (blobs are not validated, irrelevant for many fixed-size
small types, but may be a problem for large text cells) as well as
operations on collections.
Tests: unit(release), dtests(cql_prepared_test.py, cql_tests.py, cql_additional_tests.py)
"
* tag 'fragmented-cql3-receive/v1' of https://github.com/pdziepak/scylla: (23 commits)
types: bytes_view: override fragmented validate()
cql3: value_view: switch to fragmented_temporary_buffer::view
types: add validate that accepts fragmented_temporary_buffer::view
cql3 query_options: add linearize()
cql3: query_options: use bytes_ostream for temporaries
cql3: operation: make make_cell accept fragmented_temporary_buffer::view
atomic_cell: accept fragmented_temporary_buffer::view values
cql3: avoid ambiguity in a call to update_parameters::make_cell()
transport: switch to fragmented_temporary_buffer
transport: extract compression buffers from response class
tests/reusable_buffer: test fragmented_temporary_buffer support
utils: reusable_buffer: support fragmented_temporary_buffer
tests: add test for fragmented_temporary_buffer
util fragment_range: add general linearisation functions
utils: add fragmented_temporary_buffer
tests: add basic test for transport requests and responses
tests/random-utils: print seed
tests/random-utils: generate sstrings
cql3: add value_view printer and equality comparison
transport: move response outside of cql_server class
...
"
This patchset adds support for reading Index.db files written in
SSTables 3.x ('mc') format.
Note that the offsets map introduced in SSTables 3.x is neither used nor
read yet. It is located last in promoted index and so current parsers
just ignore it for the time being.
Later it should be used to perform binary search of a desired promoted
index block in large partition, thus reducing the complexity from linear
to logarithmic.
Tests: unit {release}
"
* 'projects/sstables-30/index_reader/v5' of https://github.com/argenet/scylla:
sstables: Add getter for end_open_marker to index_reader.
tests: Add test reading index for a partition comprised of RT markers of boundary types.
tests: Add test for reading index of a partition comprised of only range tombstones.
tests: Use std::adjacent_find in index_reader_assertions::has_monotonic_positions()
tests: Read rows only index
sstables: Do not seek through the promoted index for static row positions.
sstables: Read promoted index stored in SSTables 3.x ('mc') format.
sstables: Make promoted_index_block support clustering keys for both ka/la and mc formats.
utils: Add overloaded_functor helper.
position_in_partition: Add a constructor from range_tag_t{}, bound_kind and clustering_key_prefix.
sstables: Support reading signed vints in continuous_data_consumer.
sstables: Factor out the code building a vector of fixed clustering values lengths.
sstables: Remove unused includes from index_entry.hh
tests: Add test for reading SSTables 3.x index file with empty promoted index.
tests: Rename sstable_assertions.hh -> tests/index_reader_assertions.hh
sstables: Support parsing index entries from SSTables 3.x format.
sstables: move bound_kind_m to header
- Renamed _current to _current_range to better reflect its nature as
there are other similarly named members (_current_start and
_current_end).
- Don't use a temporary variable for incrementing the change counter.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
This class encapsulates the logic related to
clustering key filtering and fast forwarding.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
Not only this is easier to read and understand, but it also doesn't
force the promoted_index_block class to support copying which is
heavyweight and otherwise not needed.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
