There was nothing keeping the verify lambda alive after the return. It
worked most of the time since the only state kept by the lambda was
a pointer to cql_test_env.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20190426203823.15562-1-espindola@scylladb.com>
"
Previously we weren't validating elements of collections so it
was possible to add non-UTF-8 string to a column with type
list<text>.
Tests: unit(release)
Fixes#4009
"
* 'haaawk/4009/v5' of github.com:scylladb/seastar-dev:
types: Test correct map validation
types: Test correct in clause validation
types: Test correct tuple validation
types: Test correct set validation
types: Test correct list validation
types: Add test_tuple_elements_validation
types: Add test_in_clause_validation
types: Add test_map_elements_validation
types: Add test_set_elements_validation
types: Add test_list_elements_validation
types: Validate input when tuples
types: Validate input when parsing a set
types: Validate input when parsing a map
types: Validate input when parsing a list
types: Implement validation for tuple
types: Implement validation for set
types: Implement validation for map
types: Implement validation for list
types: Add cql_serialization_format parameter to validate
1. All nodes in the cluster have to support MC_SSTABLE_FEATURE
2. When a node observes that whole cluster supports MC_SSTABLE_FEATURE
then it should start using MC format.
3. Once all shards start to use MC then a node should broadcast that
unbounded range tombstones are now supported by the cluster.
4. Once whole cluster supports unbounded range tombstones we can
start accepting them on CQL level.
tests: unit(release)
Fixes#4205Fixes#4113
* seastar-dev.git dev/haaawk/enable_mc/v11:
system_keyspace: Add scylla_local
system_keyspace: add accessors for SCYLLA_LOCAL
storage_service: add _sstables_format field
feature: add when_enabled callbacks
system_keyspace: add storage_service param to setup
Add sstable format helper methods
Register feature listeners in storage_service
Add service::read_sstables_format
Use read_sstables_format in main.cc
Use _sstables_format to determine current format
Add _unbounded_range_tombstones_feature
Update supported features on format change
Currently, each instanciation of `random_mutation_generator::impl` will
generate a new random seed for itself. Altough these are printed,
mapping back all the printed seeds to the exact source location where it
has to be substituted in is non-trivial. This makes reproducing random
test failures very hard. To solve this problem, use
`tests::random::get_int()` to produce the random seed of the
`random_mutation_generator::impl` instances. This way the seed of all
the mutation generator will be derived from a single "master" seed that
is easily replaced after a test failure, hopefully also leading to
easily reproducible random test failures.
I checked that after substituting in a previously generated master
random seed, all derived seeds were exactly the same.
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <0471415938fc27485975ef9213d37d94bff20fd5.1555329062.git.bdenes@scylladb.com>
"
These are patches I wrote while working on UDF/UDA, but IMHO they are
independent improvements and are ready for review.
Tests: unit (debug) dtest (release)
I checked that all tests in
nosetests -v user_types_test.py sstabledump_test.py cqlsh_tests/cqlsh_tests.py
now pass.
"
* 'espindola/udf-uda-refactoring-v3' of https://github.com/espindola/scylla:
Refactor user type merging
cql_type_parser::raw_builder: Allow building types incrementally
cql3: delete dead code
Include missing header
return a const reference from return_type
delete unused var
Add a test on nested user types.
Currently the partition header will always be reported as different when
comparing two mutations. This is because they are prepended with the
"expected: " and "... but got: " texts. This generates unnecessary
noise. Inject a new line between the prefix and the partition-header
proper. This way the partition header will only show up in the diff when
there is an actual difference. The "expected: " and "... but got: "
phrases are still shown as different on the top of the diff but this is
fine as one can immediately see that they are not part of the data and
additionaly they help the reader in determining which part of the diff
is the expected one and which is the actual one.
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <29e0f413d248048d7db032224a3fd4180bf1b319.1554909144.git.bdenes@scylladb.com>
The problem happens after a schema change because we fail to properly
remove ongoing compaction, which stopped being tracked, from list that
is used to calculate backlog, so it may happen that a compaction read
monitor (ceases to exist after compaction ends) is used after freed.
Fixes#4410.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20190409024936.23775-1-raphaelsc@scylladb.com>
The comparison of tables before and after mutation is now done by a
generic diff_rows function. The same function will be used for user
defined functions and user defined aggregates.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
There were many calls to read_keyspace_mutation. One in each function
that prepares a mutation for some other schema change.
With this patch they are all moved to a single location.
Tests: unit (dev, debug)
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20190328024440.26201-1-espindola@scylladb.com>
"
File must be either owned by the process uid
or have both read and write access to it,
so it could be (hard) linked when sysctl
fs.protected_hardlinks is enabled.
Fixes#3117
"
* 'projects/valid_owner_and_mode/v3-rebased' of https://github.com/bhalevy/scylla:
storage_service: handle load_new_sstables exception
init: validate file ownership and mode.
treewide: use std::filesystem
Rather than {std::experimental,boost,seastar::compat}::filesystem
On Sat, 2019-03-23 at 01:44 +0200, Avi Kivity wrote:
> The intent for seastar::compat was to allow the application to choose
> the C++ dialect and have seastar follow, rather than have seastar choose
> the types and have the application follow (as in your patch).
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
"
Variable length integers are used are used extensively by SSTables mc
format. The current deserialisation routine is quite naive in a way that
it reads each byte separately. Since, those vints usually appear inside
much larger buffers, we optimise for such cases, read 8-bytes at once
and then mask out the unneeded parts (as well as fix their order because
big-endian).
Tests: unit(dev).
perf_vint (average time per element when deserializing 1000 vints):
before:
vint.deserialize 69442000 14.400ns 0.000ns 14.399ns 14.400ns
after:
vint.deserialize 241502000 4.140ns 0.000ns 4.140ns 4.140ns
perf_fast_forward (data on /tmp):
large-partition-single-key-slice on dataset large-part-ds1:
before:
range time (s) iterations frags frag/s mad f/s max f/s min f/s aio (KiB) blocked dropped idx hit idx miss idx blk c hit c miss c blk cpu
-> [0, 1] 0.000278 8792 2 7190 119 7367 1960 3 104 2 0 0 1 1 0 0 1 100.0%
-> [1, 100) 0.000344 96 99 288100 4335 307689 193809 2 108 2 0 0 1 1 0 0 1 100.0%
-> (100, 200] 0.000339 13254 100 295263 2824 301734 222725 2 108 2 0 0 1 1 0 0 1 100.0%
after:
range time (s) iterations frags frag/s mad f/s max f/s min f/s aio (KiB) blocked dropped idx hit idx miss idx blk c hit c miss c blk cpu
-> [0, 1] 0.000236 10001 2 8461 59 8718 2261 3 104 2 0 0 1 1 0 0 1 100.0%
-> [1, 100) 0.000285 89 99 347500 2441 355826 215745 2 108 2 0 0 1 1 0 0 1 100.0%
-> (100, 200] 0.000293 14369 100 341302 1512 350123 222049 2 108 2 0 0 1 1 0 0 1 100.0%
"
* tag 'optimise-vint/v2' of https://github.com/pdziepak/scylla:
sstable: pass full length of buffer to vint deserialiser
vint: optimise deserialisation routine
vint: drop deserialize_type structure
tests/vint: reduce test dependencies
tests/perf: add performance test for vint serialisation
"
This series introduce a rudimentary sstables manager
that will be used for making and deleting sstables, and tracking
of thereof.
The motivation for having a sstables manager is detailed in
https://github.com/scylladb/scylla/issues/4149.
The gist of it is that we need a proper way to manage the life
cycle of sstables to solve potential races between compaction
and various consumers of sstables, so they don't get deleted by
compaction while being used.
In addition, we plan to add global statistics methods like returning
the total capacity used by all sstables.
This patchset changes the way class sstable gets the large_data_handler.
Rather than passing it separately for writing the sstable and when deleting
sstables, we provide the large_data_handler when the sstable object is
constructed and then use it when needed.
Refs #4149
"
* 'projects/sstables_manager/v3' of https://github.com/bhalevy/scylla:
sstables: provide large_data_handler to constructor
sstables_manager: default_sstable_buffer_size need not be a function
sstables: introduce sstables_manager
sstables: move shareable_components def to its own header
tests: use global nop_lp_handler in test_services
sstables: compress.hh: add missing include
sstables: reorder entry_descriptor constructor params
sstables: entry_descriptor: get rid of unused ctor
sstables: make load_shared_components a method of sstable
sstables: remove default params from sstable constructor
database: add table::make_sstable helper
distributed_loader: pass column_family to load_sstables_with_open_info
distributed_loader: no need for forward declaration of load_sstables_with_open_info
distributed_loader: reshard: use default params for make_sstable
The goal of the sstables manager is to track and manage sstables life-cycle.
There is a sstable manager instance per database and it is passed to each column-family
(and test environment) on construction.
All sstables created, loaded, and deleted pass through the sstables manager.
The manager will make sure consumers of sstables are in sync so that sstables
will not be deleted while in use.
Refs #4149
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
"
Fixes#4348
v2 changes:
* added a unit test
This miniseries fixes decimal/varint serialization - it did not update
output iterator in all cases, which may lead to overwriting decimal data
if any other value follows them directly in the same buffer (e.g. in a tuple).
It also comes with a reproducing unit test covering both decimals and varints.
Tests: unit (dev)
dtest: json_test.FromJsonUpdateTests.complex_data_types_test
json_test.FromJsonInsertTests.complex_data_types_test
json_test.ToJsonSelectTests.complex_data_types_test
"
* 'fix_varint_serialization_2' of https://github.com/psarna/scylla:
tests: add test for unpacking decimals
types: fix varint and decimal serialization
I noticed a test failure with
Mutation inequality is not symmetric for ...
And the difference between the two mutations was that one atomic_cell
was live and the other wasn't.
Looking at the code I found a few cases where the comparison was not
symmetrical. This patch fixes them.
This patch will not fix the test, as it will now fail with a
"Mutations differ" error, but that is probably an independent issue.
Ref #3975.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20190325194647.54950-1-espindola@scylladb.com>
"
This series removes the usage of the static gossiper object in init.cc
and storage_service.
Follow up series will remove more in other components. This is the
effort to clean up the component dependencies and have better shutdown
procedure.
Tests: tests/gossip_test, tests/cql_query_test, tests/sstable_mutation_test, dtests.
"
* tag 'asias/storage_service_gossiper_dep_v5' of github.com:cloudius-systems/seastar-dev:
storage_service: Do not use the global gms::get_local_gossiper()
storage_service: Pass gossiper object to storage_service
gms: Remove i_failure_detector.hh
gossip: Get rid of the gms::get_local_failure_detector static object
dht: Do not use failure_detector::is_alive in failure_detector_source_filter
tests: Fix stop snitch in gossip_test.cc
gossiper: Do not use value_factory from storage_service object
gossiper: Use cfg options from _cfg instead of get_local_storage_service
gossiper: Pass db::config object to gossiper class
init: Pass gossiper object to init_ms_fd_gossiper
"
Both cql3_type and abstract_type are normally used inside
shared_ptr. This creates a problem when an abstract_type needs to refer
to a cql3_type as that creates a cycle.
To avoid warnings from asan, we were using a std::unordered_map to
store one of the edges of the cycle. This avoids the warning, but
wastes even more memory.
Even before this series cql3_type was a fairly light weight
structure. This patch pushes in that direction and now cql3_type is a
struct with a single member variable, a data_type.
This avoids the reference cycle and is easier to understand IMHO.
The one corner case is varchar. In the old system cql3_type::varchar
and cql3_type::text don't compare equal, but they both map to the same
data_type.
In the new system they would compare equal, so we avoid the confusion
by just removing the cql3_type::varchar variable.
Tests: unit (dev)
"
* 'espindola/merge-cq3-type-and-type-v3' of https://github.com/espindola/scylla:
Turn cql3_type into a trivial wrapper over data_type
Delete cql3_type::varchar
Simplify db::cql_type_parser::parse
Add a test for the varchar column representation
The crash observed in issue #4335 happens because
delete_large_data_entries is passed a deleted name.
Normally we don't get a crash, but a garbage name and we fail to
delete entries from system.large_*.
Adding a test for the fix found another issue that the second patch
is this series fixes.
Tests: unit (dev)
Fixes#4335.
* https://github.com/espindola/scylla guthub/fix-use-after-free-v4:
large_data_handler: Fix a use after destruction
large_data_handler: Make a variable non static
Allow large_data_handler to be stopped twice
Allow table to be stopped twice
Test that large data entries are deleted
Store the failure_detector object inside gossiper object.
- No more the global object sharded<failure_detector>
- No need to initialize sharded<failure_detector> manually which
simplifies the code in tests/cql_test_env.cc and init.cc.
This area is hard to test since we only issue deletes during
compaction and we wait for deletes only during shutdown.
That is probably worth it, seeing that two independent bugs would have
been found by this test.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
