Function messaging_service::get_rpc_client() suppose to either return
existing client or create one and return it. The function is suppose to
be atomic, so after checking that requested client does not exist it is
safe to assume emplace() will succeed. But we saw bugs that made the
function to not be atomic. Lets add an assert that will help to catch
such bugs easier if they will happen in the future.
Message-Id: <20190326115741.GX26144@scylladb.com>
"
Fixes#4348
v2 changes:
* added a unit test
This miniseries fixes decimal/varint serialization - it did not update
output iterator in all cases, which may lead to overwriting decimal data
if any other value follows them directly in the same buffer (e.g. in a tuple).
It also comes with a reproducing unit test covering both decimals and varints.
Tests: unit (dev)
dtest: json_test.FromJsonUpdateTests.complex_data_types_test
json_test.FromJsonInsertTests.complex_data_types_test
json_test.ToJsonSelectTests.complex_data_types_test
"
* 'fix_varint_serialization_2' of https://github.com/psarna/scylla:
tests: add test for unpacking decimals
types: fix varint and decimal serialization
Varint and decimal types serialization did not update the output
iterator after generating a value, which may lead to corrupted
sstables - variable-length integers were properly serialized,
but if anything followed them directly in the buffer (e.g. in a tuple),
their value will be overwritten.
Fixes#4348
Tests: unit (dev)
dtest: json_test.FromJsonUpdateTests.complex_data_types_test
json_test.FromJsonInsertTests.complex_data_types_test
json_test.ToJsonSelectTests.complex_data_types_test
Note that dtests still do not succeed 100% due to formatting differences
in compared results (e.g. 1.0e+07 vs 1.0E7, but it's no longer a query
correctness issue.
I noticed a test failure with
Mutation inequality is not symmetric for ...
And the difference between the two mutations was that one atomic_cell
was live and the other wasn't.
Looking at the code I found a few cases where the comparison was not
symmetrical. This patch fixes them.
This patch will not fix the test, as it will now fail with a
"Mutations differ" error, but that is probably an independent issue.
Ref #3975.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20190325194647.54950-1-espindola@scylladb.com>
When we call perftune.py in order to get a particular mode's cpu set
(e.g. mode=sq_split) it may fail and print an error message to stderr because
there are too few CPUs for a particular configuration mode (e.g. when
there are only 2 CPUs and the mode is sq_split).
We already treat these situations correctly however we let the
corresponding perftune.py error message get out into the syslog.
This is definitely confusing, stressful and annoying.
Let's not let these messages out.
Fixes#4211
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Message-Id: <20190325220018.22824-1-vladz@scylladb.com>
Compilation fails with fmt release 5.3.0 when we print a bytes_view
using "{}" formatter.
Compiler's complain is: "error: static assertion failed: mismatch between char-types of context and argument"
Resolve this by explicitly using the operator<<() across the whole
operator<<(std::ostream& os, const result_message::rows& msg) function.
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Message-Id: <20190325203628.5902-1-vladz@scylladb.com>
has_scylla_component is always false before loading the sstable.
Also, return exception future rather than throwing.
Hit with the following dtests:
counter_tests.TestCounters.upgrade_test
counter_tests.TestCountersOnMultipleNodes.counter_consistency_node_*_test
resharding_test.ReshardingTest_nodes?_with_*CompactionStrategy.resharding_counter_test
update_cluster_layout_tests.TestUpdateClusterLayout.increment_decrement_counters_in_threads_nodes_restarted_test
Fixes#4306
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20190326084151.18848-1-bhalevy@scylladb.com>
"
This series removes the usage of the static gossiper object in init.cc
and storage_service.
Follow up series will remove more in other components. This is the
effort to clean up the component dependencies and have better shutdown
procedure.
Tests: tests/gossip_test, tests/cql_query_test, tests/sstable_mutation_test, dtests.
"
* tag 'asias/storage_service_gossiper_dep_v5' of github.com:cloudius-systems/seastar-dev:
storage_service: Do not use the global gms::get_local_gossiper()
storage_service: Pass gossiper object to storage_service
gms: Remove i_failure_detector.hh
gossip: Get rid of the gms::get_local_failure_detector static object
dht: Do not use failure_detector::is_alive in failure_detector_source_filter
tests: Fix stop snitch in gossip_test.cc
gossiper: Do not use value_factory from storage_service object
gossiper: Use cfg options from _cfg instead of get_local_storage_service
gossiper: Pass db::config object to gossiper class
init: Pass gossiper object to init_ms_fd_gossiper
* seastar 33baf62...caa98f8 (8):
> Merge "Add file_accessible and file_stat methods" from Benny
> future::then: use std::terminate instead of abort
> build: Allow cooked dependencies with configure.py
> tests: Show a test's output when it fails
> posix_file_impl: Bypass flush() call iff opened with O_DSYNC
> posix_file_impl: Propagate and keep open_flags
> open_flags: Add O_DSYNC value
> build: Forward variables to CMake correctly
"
Both cql3_type and abstract_type are normally used inside
shared_ptr. This creates a problem when an abstract_type needs to refer
to a cql3_type as that creates a cycle.
To avoid warnings from asan, we were using a std::unordered_map to
store one of the edges of the cycle. This avoids the warning, but
wastes even more memory.
Even before this series cql3_type was a fairly light weight
structure. This patch pushes in that direction and now cql3_type is a
struct with a single member variable, a data_type.
This avoids the reference cycle and is easier to understand IMHO.
The one corner case is varchar. In the old system cql3_type::varchar
and cql3_type::text don't compare equal, but they both map to the same
data_type.
In the new system they would compare equal, so we avoid the confusion
by just removing the cql3_type::varchar variable.
Tests: unit (dev)
"
* 'espindola/merge-cq3-type-and-type-v3' of https://github.com/espindola/scylla:
Turn cql3_type into a trivial wrapper over data_type
Delete cql3_type::varchar
Simplify db::cql_type_parser::parse
Add a test for the varchar column representation
The crash observed in issue #4335 happens because
delete_large_data_entries is passed a deleted name.
Normally we don't get a crash, but a garbage name and we fail to
delete entries from system.large_*.
Adding a test for the fix found another issue that the second patch
is this series fixes.
Tests: unit (dev)
Fixes#4335.
* https://github.com/espindola/scylla guthub/fix-use-after-free-v4:
large_data_handler: Fix a use after destruction
large_data_handler: Make a variable non static
Allow large_data_handler to be stopped twice
Allow table to be stopped twice
Test that large data entries are deleted
"
Validate the to-be-loaded sstables in the open_sstable phase and handle any exceptions before calling cf.get_row_cache().invalidate.
Currently if exception is thrown from distributed_loader::open_sstable cf._sstables_opened_but_not_loaded may be left partially populated.
Fixes#4306
Tests: unit (dev)
- next-gating dtests (dev)
- migration_test:TestMigration_with_2_1_x.migrate_sstable_with_counter_test
migration_test:TestMigration_with_2_1_x.migrate_sstable_with_counter_test_expect_fail
- with bypassing exception in distributed_loader::flush_upload_dir
to trigger the exception in table::open_sstable
"
* 'issues/4306/v3' of https://github.com/bhalevy/scylla:
table: move sstable counters validation from load_sstable to open_sstable
distributed_loader::load_new_sstables: handle exceptions in open_sstable
"
Aligned way to build relocatable rpm with existing relocatable packages.
"
* 'relocatable-python3-fix-v3' of https://github.com/syuu1228/scylla:
reloc: allow specify rpmbuild dir
reloc/python3: archive package version number on build_reloc.sh
reloc/python3: archive rpm build script in the relocatable package, build rpm using the script
relloc/python3: fix PyYAML package name
reloc: rename python3 relocatable package filename to align same style with other packages
reloc: move relocatable python build scripts to reloc/python3 and dist/redhat/python3
When a view replica becomes unavailable, updates to it are stored as
hints at the paired based replica. This on-disk queue of pending view
updates grows as long as there are view updated and the view replica
remains unavailable. Currently, we take that relative queue size into
account when calculating the delay for new base writes, in the context
of the backpressure algorithm for materialized views.
However, the way we're calculating that on-disk backlog is wrong,
since we calculate it per-device and then feed it to all the hints
managers for that device. This means that normal hints will show up as
backlog for the view hints manager, which in turn introduces delays.
This can make the view backpressure mechanism kick-in even if the
cluster uses no materialized views.
There's yet another way in which considering the view hints backlog is
wrong: a view replica that is unavailable for some period of time can
cause the backlog to grow to a point where all base writes are applied
the maximum delay of 1 second. This turns a single-node failure into
cluster unavailability.
The fix to both issues is to simply not take this on-disk backlog into
account for the backpressure algorithm.
Fixes#4351Fixes#4352
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Reviewed-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20190321170418.25953-1-duarte@scylladb.com>
Since we archive rpm/deb build script on relocatable package and build
rpm/deb using the script, so align python relocatable package too.
Also added SCYLLA-RELOCATABLE-FILE, SCYLLA-RELEASE-FILE and SCYLLA-VERSION-FILE
since these files are required for relocatable package.
Debugging continuations is challenging. There is no support from gdb for
finding out which continuation was this continuation called from, nor
what other continuations are attached to it. GDB's `bt` command is of
limited use, at best a handful of continuations will appear in the
backtrace, those that were ready. This series attempts to fill part of
this void and provides a command that answers the latter question: what
continuations are attached to this one?
`scylla fiber` allows for walking a continuation chain, printing each
continuation. It is supposed to be the seastar equivalent of `bt`.
The continuation chain is walked starting from an arbitrary task,
specified by the user. The command will print all continuations attached
to the specified task.
This series also contains some loosely related cleanup of existing
commands and code in `scylla-gdb.py`.
* https://github.com/denesb/scylla.git scylla-fiber-gdb-command/v4:
scylla-gdb.py: fix static_vector
scylla-gdb.py: std_unique_ptr: add get() method
scylla-gdb.py: fix existing documentation
scylla-gdb.py: fix tasks and task-stats commands
scylla-gdb.py: resolve(): add cache parameter
scylla-gdb.py: scylla_ptr: move actual logic into analyze()
scylla-gdb.py: scylla_ptr: make analyze() usable for outside code
scylla-gdb.py: scylla_ptr: accept any valid gdb expression as input
scylla-gdb.py: add scylla fiber command
Store the failure_detector object inside gossiper object.
- No more the global object sharded<failure_detector>
- No need to initialize sharded<failure_detector> manually which
simplifies the code in tests/cql_test_env.cc and init.cc.
Switch failure_detector_source_filter to use get_local_gossiper::is_alive
directly since we are going to remove the static
gms::get_local_failure_detector object soon.
Pass the nodes that are down to the filter direclty, to avoid the
range_streamer to depends on gossiper at all.
This area is hard to test since we only issue deletes during
compaction and we wait for deletes only during shutdown.
That is probably worth it, seeing that two independent bugs would have
been found by this test.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
"
This series adds moving sstables uploaded via `nodetool refresh` to
staging/ directory if they require generating view updates from them.
Previous behavior (leaving these sstables in upload/ directory until
view updates are generated) might have caused sstables with
conflicting names to be mistakenly overwritten by the user.
Fixes#4047
Tests: unit (dev)
dtest: backup_restore_tests.py + backup_restore_tests.py modified with
having materialized view definitions
"
* 'use_staging_directory_for_uploaded_sstables_awaiting_view_updates' of https://github.com/psarna/scylla:
sstables: simplify requires_view_building
loader: move uploaded view pending sstables to staging
Current code captures a reference to rpc::client in a continuation, but
there is no guaranty that the reference will be valid when continuation runs.
Capture shared pointer to rpc::client instead.
Fixes#4350.
Message-Id: <20190314135538.GC21521@scylladb.com>
When we're populating a partition range and the population range ends
with a partition key (not a token) which is present in sstables and
there was a concurrent memtable flush, we would abort on the following
assert in cache::autoupdating_underlying_reader:
utils::phased_barrier::phase_type creation_phase() const {
assert(_reader);
return _reader_creation_phase;
}
That's because autoupdating_underlying_reader::move_to_next_partition()
clears the _reader field when it tries to recreate a reader but it finds
the new range to be empty:
if (!_reader || _reader_creation_phase != phase) {
if (_last_key) {
auto cmp = dht::ring_position_comparator(*_cache._schema);
auto&& new_range = _range.split_after(*_last_key, cmp);
if (!new_range) {
_reader = {};
return make_ready_future<mutation_fragment_opt>();
}
Fix by not asserting on _reader. creation_phase() will now be
meaningful even after we clear the _reader. The meaning of
creation_phase() is now "the phase in which the reader was last
created or 0", which makes it valid in more cases than before.
If the reader was never created we will return 0, which is smaller
than any phase returned by cache::phase_of(), since cache starts from
phase 1. This shouldn't affect current behavior, since we'd abort() if
called for this case, it just makes the value more appropriate for the
new semantics.
Tests:
- unit.row_cache_test (debug)
Fixes#4236
Message-Id: <1553107389-16214-1-git-send-email-tgrabiec@scylladb.com>
In commit 71bf757b2c, we call
wait_for_gossip_to_settle() which takes some time to complete in
storage_service::prepare_to_join().
In tests/cql_query_test calls init_server with do_bind == false which in
turn calls storage_service::prepare_to_join(). Since in the test, there
is only one node, there is no point to wait for gossip to settle.
To make the cql_query_test fast again, do not call
wait_for_gossip_to_settle if do_bind is false.
Before this patch, cql_query_test takes forever to complete.
After it takes 10s.
Tests: tests/cql_query_test
Message-Id: <3ae509e0a011ae30eef3f383c6a107e194e0e243.1553147332.git.asias@scylladb.com>
"
This series adds support for local indexing, i.e. when the index table
resides on the same partition as base data.
It addresses the performance issue of having an indexed query
that also specifies a partition key - index will be queried
locally.
"
* 'add_local_indexing_11' of https://github.com/psarna/scylla: (30 commits)
tests: add cases for local index prefix optimization
tests: add create/drop local index test case
tests: add non-standard names cases to local index tests
tests: add multi pk case for local index tests
tests: add test for malformed local index definitions
tests: add local index paging test
tests: add local indexing test
cql3: add CREATE INDEX syntax for local indexes
cql3: use serialization function to create index target string
index: add serialization function for index targets
index: use proper local index target when adding index
index: add parsing target column name from local index targets
db: add checking for local index in schema tables
index: add checking if serialized target implies local index
index: enable parsing multi-key targets
index: move target parser code to .cc file
json: add non-throwing overload for to_json_value
cql3: add checking for local indexes in has_supporting_index()
cql3: move finding index restrictions to prepare stage
cql3: add picking an index by score
...
When a materialized view was created, the verification code artificially
forbade creating a view without a clustering key column. However, there
is no real reason to forbid this. In the trivial case, the original base
table might not have had a clustering key, and the view might want to use
the exact same key. In a more complex case, a view may want to have all the
primary key columns as *partition* key columns, and that should be fine.
The patch also includes a regression test, which failed before this patch,
and succeeds with it (we test that we can create materialized views in both
aforementioned scenarios, and these materialized views work as expected).
Duarte raised the opinion that the "trivial" case of a view table with
a key identical to that of the base should be disallowed. However, this
should be done, if at all (I think it shouldn't), in a follow-up patch,
which will implement the non-triviality requirement consistently (e.g.,
require view primary key to be different from base's, regardless of
the existance or non-existance of clustering columns).
Fixes#4340.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20190320122925.10108-1-nyh@scylladb.com>
When we are tracing requests, we would like to know everything that
happened to a query that can contribute to it having increased
latencies.
We insert some of those latencies explicitly due to throttling, but we
do not log that into tracing.
In the case of storage proxy, we do have a log message at trace level
but that is rarely used: trace messages are too heavy of a hammer, there
is no way to specify specific queries, etc.
The correct place for that is CQL tracing. This patch moves that message
to CQL tracing. We also add a matching tracepoint assuring us that no
delay happened if that's the case.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <20190320163350.15075-1-glauber@scylladb.com>
"
Static compact tables are tables with compact storage and no
clustering columns.
Before this patch, Scylla was writing rows of static compact tables as
clustered rows instead of as static rows. That's because in our in-memory
model such tables have regular rows and no static row. In Cassandra's
schema (since 3.x), those tables have columns which are marked as
static and there are no regular columns.
This worked fine as long as Scylla was writing and reading those
sstables. But when importing sstables from Cassandra, our reader was
skipping the static row, since it's not present in our schema, and
returning no rows as a result. Also, Cassandra, and Scylla tools,
would have problems reading those sstables.
Fix this by writing rows for such tables the same way as Cassandra
does. In order to support rolling downgrade, we do that only when all
nodes are upgraded.
Fixes#4139.
Tests:
- unit (dev)
"
* tag 'static-compact-mc-fix-v3.1' of github.com:tgrabiec/scylla:
tests: sstables: Test reading of static compact sstable generated by Cassandra
tests: sstables: Add test for writing and reading of static compact tables
sstables: mc: Write static compact tables the same way as Cassandra
sstable: mc: writer: Set _static_row_written inside write_static_row()
sstables: Add sstable::features()
sstables: mc: writer: Prepare write_static_row() for working with any column_kind
storage_service: Introduce the CORRECT_STATIC_COMPACT feature flag
sstables: mc: writer: Build indexed_columns together with serialization_header
sstables: mc: writer: De-optimize make_serialization_header()
sstable: mc: writer: Move attaching of mc-specific components out of generic code
Both cql3_type and abstract_type are normally used inside
shared_ptr. This creates a problem when an abstract_type needs to refer
to a cql3_type as that creates a cycle.
To avoid warnings from asan, we were using a std::unordered_map to
store one of the edges of the cycle. This avoids the warning, but
wastes even more memory.
Even before this patch cql3_type was a fairly light weight
structure. This patch pushes in that direction and now cql3_type is a
struct with a single member variable, a data_type.
This avoids the reference cycle and is easier to understand IMHO.
Tests: unit (dev)
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
varchar is just an alias for text. Handle that conversion directly in
the parser and delete the cql3_type::varchar variable.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Since its first version, db::cql_type_parser::parse had special cases
for native and user defined types.
Those are not necessary, as the general parser has no problem handling
them.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
