Use an internally create instance of random engine. Passing a readily
seeded engine from the outside is pointless now that we have a mechanism
to seed entire test suites with a command line algorithm: the internal
engine is seeded from tests::random, so the seed of the test suite
determines the internal seed as well.
Update the sole user of this method (mutation_writer_test.cc) to not
generate local seeds anymore.
Introduce consumer in mutation compactor that will only consume
data that is purged away from regular consumer. The goal is to
allow compaction implementation to do whatever it wants with
the garbage collected data, like saving it for preventing
data resurrection from ever happening, like described in
issue #4531.
noop_compacted_fragments_consumer is made available for users that
don't need this capability.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
The new collector parameter is a pointer to a
`compaction_garbage_collector` implementation. This collector is passed
the row_marker when it expired and would be discarded.
The collector param is optional and defaults to nullptr.
The new collector parameter is a pointer to a
`compaction_garbage_collector` implementation. This collector is passed
all atoms that are expired and can would be discarded. The body of
`compact_and_expire()` was changed so that it checks cells' tombstone
coverage before it checks their expiry, so that cells that are both
covered by a tombstone and also expired are not passed to the collector.
The collector is forwarded to
`collection_type_impl::mutation::compact_and_expire()` as well.
The collector param is optional and defaults to nullptr
The new collector parameter is a pointer to a
`compaction_garbage_collector` implementation. This collector is passed
all atoms that are expired and would be discarded. The body of
`compact_and_expire()` was changed so that it checks cells' tombstone
coverage before it checks their expiry, so that cells that are both
covered by a tombstone and also expired are not passed to the collector.
The collector param is optional and defaults to nullptr. To accommodate
the collector, which needs to know the column id, a new `column_id`
parameter was added as well.
This interface can be used to implement a garbage collector that
collects atoms that are purged due to expiry during compaction.
The intended usage is collecting purged atoms for safekeeping until the
compaction process finishes safely, to be dropped only at the end when
the compaction is known to have finished successfully.
This fixes a possible cause of #4614.
From the backtrace in that issue, it looks like a file is being closed
twice. The first point in the backtrace where that seems likely is in
the MC writer.
My first idea was to add a writer::close and make it the responsibility
of the code using the writer to call it. That way we would move work
out of the destructor.
That is a bit hard since the writer is destroyed from
flat_mutation_reader::impl::~consumer_adapter and that would need to
get a close function too.
This patch instead just fixes an exception safety issue. If
_index_writer->close() throws, _index_writer is still valid and
~writer will try to close it again.
If the exception was thrown after _completed.set_value(), that would
explain the assert about _completed.set_value() being called twice.
With this patch the path outside of the destructor now moves the
writer to a local variable before trying to close it.
Fixes#4614
Message-Id: <20190710171747.27337-1-espindola@scylladb.com>
In debug mode the LSA needs objects to be 8-byte aligned in order to
maximise coverage from the AddressSanitizer.
Usually `close_active()` creates a dummy objects that covers the end of
the segment being closed. However, it the last real objects ends in the
last eight bytes of the segment then that dummy won't be created because
of the alignment requirements. This broke exit conditions on loops
trying to read all objects in the segment and caused them to attempt to
dereference address at the end of the segment. This patch fixes that.
Fixes#4653.
"
If the user creates a keyspace with the 'SimpleStrategy' replication class
in a multi-datacenter environment, they will receive a warning in the CQL shell
and in the server logs.
Resolves#4481 and #4651.
"
* 'multidc' of https://github.com/kbr-/scylla:
Warn user about using SimpleStrategy with Multi DC deployment
Add warning support to the CQL binary protocol implementation
"
Fixes#2027
Modifies inet address type in scylla to use seastar::net::inet_address,
and removes explicit use of ipv4_addr in various network code in favour
of socket_address. Thus capable of resolving and binding to ipv6.
Adds config option to enable/disable ipv6 (default enabled), so
upgrading cluster can continue to work while running mixed version
nodes (since gossip message address serialization becomes different).
"
* 'calle/ipv6' of https://github.com/elcallio/scylla:
test-serialization: Add small roundtrip test for inet address (v4 + v6)
inet_address/init: Make ipv6 default enabled
db::config: Add enable ipv6 switch (default off)
gms::inet_address: Make serialization ipv6 aware
Remove usage of inet_address::raw_addr()
Replace use of "ipv4_addr" with socket_address
inet_address: Add optional family to lookup
gms::inet_address: Change inet_address to wrap actual seastar::net::inet_address
types: Add ipv6_address support
Post commit b3adabda2d
(Reduce logalloc differences between debug and release)
logalloc_test's memory footprint has grown, in particular
in test_region_groups, and it triggers the oom killer on
our test automation machines.
This patch scales down this test case so it requires less memory.
Fixes#4669
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
This adds a '--repeat N' command line option to test.py, which can be
used to execute the tests N times. This is useful for finding flakey
tests, for example.
Message-Id: <20190710092115.15960-1-penberg@scylladb.com>
Currently the handling of partition tombstones is broken in multiple
ways:
* The partition-tombstone is lost when the bucket is calculated for its
timestamp (due to a misplaced `std::exchange()`).
* When the `partition_start` fragment (containing the partition
tombstone) is actually written to the bucket we emit another
`partition_start` fragment before it because the bucket has not seen
that partition before and we fail to notice that we are actually writing
the partition header.
This bug was allowed to fly under the radar because the unit test was
accidentally not creating partition tombstones in the generated data
(due to a mistake). It was discovered while working on unit tests for
another test and fixing the data generation function to actually
generate partition tombstones.
This patch fixes both problems in the handling of partition tombstones
but it doesn't yet fixes the test. That is deferred until the patch
series which uncovered this bug is merged to avoid merge conflicts.
The other series mentioned here is: [PATCH v6 00/15] compaction: allow
collecting purged data
Fixes: #4683
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20190710092427.122623-1-bdenes@scylladb.com>
The get_value method returns a pointer to the value that is used by the
value_to_json method.
The assumption is that the void pointer points to the actual value.
Fixes#4678
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Since commit bb56653 (repair: Sync schema from follower nodes before
repair), the behaviour of handling down node during repair has been
changed. That is, if a repair follower is down, it will fail to sync
schema with it and the repair of the range will be skipped. This means
a range can not be repaired unless all the nodes for the replicas are up.
To fix, we filter out the nodes that is down and mark the repair is
partial and repair with the nodes that are still up.
Tests: repair_additional_test:RepairAdditionalTest.repair_with_down_nodes_2b_test
Fixes: #4616
Backports: 3.1
Message-Id: <621572af40335cf5ad222c149345281e669f7116.1562568434.git.asias@scylladb.com>
Because inet_address was initially hardcoded to
ipv4, its wire format is not very forward compatible.
Since we potentially need to communicate with older version nodes, we
manually define the new serial format for inet_address to be:
ipv4: 4 bytes address
ipv6: 4 bytes marker 0xffffffff (invalid address)
16 bytes data -> address
Simliar to commit 9785754e0d
lister::guarantee_type needs to check the entry's type,
not the symlink it may point to.
Fixes#4606
The nodetool_refresh_with_wrong_upload_modes_test dtest creates a broken
symlink and following it fails, as it should, with the default follow_symlink::yes
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20190626110734.4558-1-bhalevy@scylladb.com>
"
This miniseries expands big_decimal interface with convenience operators
(-=, +, -), provides test cases for it and makes one of the constructors
explicit.
Tests: unit(dev)
"
* 'expand_big_decimal_interface' of https://github.com/psarna/scylla:
utils: make string-based big decimal constructor explicit
tests: add more operators to big decimal tests
utils: add operators to big_decimal
"
Implement LIKE parsing, intermediate representation, and query processing. Add tests
for this implementation (leaving the LIKE functionality tests in
tests/like_matcher_test.cc).
Refs #4477.
"
* 'finish-like' of https://github.com/dekimir/scylla:
cql3: Add LIKE operator to CQL grammar
cql3: Ensure LIKE filtering for partition columns
cql3: Add LIKE restriction
cql3: Add LIKE relation
If the user creates a keyspace with the 'SimpleStrategy' replication class
in a multi-datacenter environment, they will receive a warning in the CQL shell
and in the server logs.
Resolves#4481.
Signed-off-by: Kamil Braun <kbraun@scylladb.com>
Partition columns are implicitly filtered whenever possible, avoiding
expensive post-processing. But there are exceptions, eg, when
partition key is only partially restricted, or for CONTAINS
expressions. Here we add LIKE to this list of exceptions.
Also fix compute_bounds() to punt on LIKE restrictions, which cannot
be translated into meaningful bounds.
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
This restriction leverages like_matcher to perform filtering.
Make single_column_relation::new_LIKE_restriction() return this new
restriction.
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
Add a new type of relation with operator LIKE. Handle it in
relation::to_restriction by introducing a new virtual method for it.
The temporary implementation of this method returns null; that will be
replaced in a subsequent patch.
Add abstract_type::is_string() to recognize string columns and
disallow LIKE operator on non-string columns.
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
Command line arguments are parsed twice in Scylla: once in main and once in Seastar's app_template::run.
The first parse is there to check if the "--version" flag is present --- in this case the version is printed
and the program exists. The second parsing is correct; however, most of the arguments were improperly treated
as positional arguments during the first parsing (e.g., "--network host" would treat "host" as a positional argument).
This happened because the arguments weren't known to the command line parser.
This commit fixes the issue by moving the parsing code until after the arguments are registered.
Resolves#4141.
Signed-off-by: Kamil Braun <kbraun@scylladb.com>
"
The put_row_diff, get_row_dif and get_full_row_hashes verbs are switched
to use rpc stream instead of rpc verb. They are the verbs that could
send big rpc messages. The rpc stream sink and source are created per
repair follower for each of the above 3 verbs. The sink and source are
shared for multiple requests during the entire repair operation for a
given range, so there is no overhead to setup rpc stream.
The row buffer is now increased to 32MiB from 256KiB, giving better
bandwidth in high latency links. The downside of bigger row buffer is
reduced possibility that all the rows inside a row buffer are identical.
This causes more full hashes to be exchanged. To address this issue, the
plan is to add better set reconciliation algorithm in addition to the
current send full hashes.
I compared rebuild using regular stream plan with repair using rpc
stream. With 2 nodes, 1 smp, 8M rows, delete all data on one of the
node before repair or rebuild.
repair using seastar rpc verb
Time to complete: 82.17s
rebuild using regular streaming which uses seastar rpc stream
Time to complete: 63.87s
repair using seastar rpc stream
Time to complete: 68.48s
For 1) and 3), the improvement is 16.6% (repair using rpc verb v.s. repair using rpc stream)
For 2) and 3), the difference is 7.2% (repair v.s. stream)
The result is promising for the future repair-based bootstrap/replace node operations.
NOTE: We do not actually enable rpc stream in row level repair for now. We
will enable it after we fix the the stall issues caused by handling
bigger row buffers.
Fixes#4581
"
* 'repair_switch_to_rpc_stream_v9' of https://github.com/asias/scylla: (45 commits)
docs: Add RPC stream doc for row level repair
repair: Mark some of the helper functions static
repair: Increase max row buf size
repair: Hook rpc stream version of verbs in row level repair
repair: Add use_rpc_stream to repair_meta
repair: Add is_rpc_stream_supported
repair: Add needs_all_rows flag to put_row_diff
repair: Optimize get_row_diff
repair: Register repair_get_full_row_hashes_with_rpc_strea
repair: Register repair_put_row_diff_with_rpc_stream
repair: Register repair_get_row_diff_with_rpc_stream
repair: Add repair_get_full_row_hashes_with_rpc_stream_handler
repair: Add repair_put_row_diff_with_rpc_stream_handler
repair: Add repair_get_row_diff_with_rpc_stream_handler
repair: Add repair_get_full_row_hashes_with_rpc_stream_process_op
repair: Add repair_put_row_diff_with_rpc_stream_process_op
repair: Add repair_get_row_diff_with_rpc_stream_process_op
repair: Add put_row_diff_with_rpc_stream
repair: Add put_row_diff_sink_op
repair: Add put_row_diff_source_op
...
