"
This series contains minor improvements to commitlog log messages that
have helped investigating #4231, but are not specific to that bug.
"
* tag 'improve-commitlog-logs/v1' of https://github.com/pdziepak/scylla:
commitlog: use consistent chunk offsets in logs
commitlog: provide more information in logs
commitlog: remove unnecessary comment
Logs in commitlog writer use offset in the file of the chunk header to
identify chunks. However, the replayer is using offset after the header
for the same purpose. This causes unnecessary confusion suggesting that
the replayer is reading at the wrong position.
This patch changes the replayer so that it reports chunk header offsets.
This commits adds some more information to the logs. Motivated, by
experiences with investigating #4231.
* size of each write
* position of each write
* log message for final write
"
This series fixes a problem in the commitlog cycle() function that
confused in-memory and on-disk size of chunks it wrote to disk. The
former was used to decide how much data needs to be actually written,
and the latter was used to compute the offset of the next chunk. If two
chunk writes happened concurrently one the one positioned earlier in
the file could corrupt the header of the next one.
Fixes#4231.
Tests: unit(dev), dtest(commitlog_test.py:TestCommitLog.test_commitlog_replay_on_startup,test_commitlog_replay_with_alter_table)
"
* tag 'fix-commitlog-cycle/v1' of https://github.com/pdziepak/scylla:
commitlog: write the correct buffer size
utils/fragmented_temporary_buffer_view: add remove suffix
Introduced in 2a437ab427.
regular_compaction::select_sstable_writer() creates the sstable writer
when the first partition is consumed from the combined mutation
fragment stream. It gets the schema directly from the table
object. That may be a different schema than the one used by the
readers if there was a concurrent schema alter duringthat small time
window. As a result, the writing consumer attached to readers will
interpret fragments using the wrong version of the schema.
One effect of this is storing values of some columns under a different
column.
This patch replaces all column_family::schema() accesses with accesses
to the _schema memeber which is obtained once per compaction and is
the same schema which readers use.
Fixes#4304.
Tests:
- manual tests with hard-coded schema change injection to reproduce the bug
- build/dev/scylla boot
- tests/sstable_mutation_test
Message-Id: <1551698056-23386-1-git-send-email-tgrabiec@scylladb.com>
Commitlog files contain multiple chunks. Each chunk starts as a single
(possibly, fragmented buffer). The size of that buffer in memory may be
larger than the size in the file.
cycle() was incorrectly using the in-memory size to write the whole
buffer to the file. That sometimes caused data corruption, since a
smaller on-file size was used to compute the offset of the next chunk
and there could be multiple chunk writes happening at the same time.
This patch solves the issue by ensuring that only the actual on-file
size of the chunk is written.
This patch adds fragmented_temporary_buffer_view::remove_suffix(). It is
also necessary to adjust remove_prefix() since now the total size of all
fragments may be larger than the size of the view if both those
operations are performed.
"
This series heavily refactors `auth_test` in anticipation of
the last patch, which fixes a bug and which should be backported.
Branches: branch-3.0, branch-2.3
"
Fixes#4284
* 'jhk/check_can_login/v2' of https://github.com/hakuch/scylla:
auth: Reject logins from disallowed roles
tests: Restrict the scope of a variable
tests: Simplify boolean assertions in `auth_test`
tests: Abstract out repeated assertion checking
tests: Do not use the `auth` namespace
tests: Validate authentication correctly
tests: Ensure test roles are created and dropped
tests: Use `static` variables in `auth_test`
tests: Remove non-useful test
4 nodes in the cluster
n1, n2 in dc1
n3, n4 in dc2
dc1 RF=2, dc2 RF=2.
If we run
nodetool repair -hosts 127.0.0.1,127.0.03 -dc "dc1,dc2" multi
on n1.
The -hosts option will be ignored and only the -dc option
will be used to choose which hosts to repair. In this case, n1 to n4
will be repaired.
If user wants to select specific hosts to repair with, there is no need
to specify the -dc option. Use the -hosts option is enough.
Reject the combination and not to surprise the user.
In https://issues.apache.org/jira/browse/CASSANDRA-9876, the same logic
is introduced as well.
Refs #3836
Message-Id: <e95ac1099f98dd53bb9d6534316005ea3577e639.1551406529.git.asias@scylladb.com>
Scylla Manager communicates through SSH, so this patch adds SSH server
to Scylla's docker image in order for it to be configurable by Scylla
Manager.
Message-Id: <20190301161428.GA12148@shenzou.localdomain>
"
This series aims to fix inconsistencies in recent view update generation series (435447998).
First of all, it checks view row marker liveness instead of that of a base row marker
when deciding if optimizations can be applied or not.
Secondly, tests based on creating mutations directly are removed. Instead:
- dtest case which detected inconsistencies in previous series is ported to be a unit test
- the above case is also expanded to cover views with regular base column in their key
- additional test for TTL and timestamps is added and it's based on CQL
Tests: unit (dev)
dtest: materialized_views_test.TestMaterializedViews.test_no_base_column_in_view_pk_complex_timestamp_without_flush
Fixes: #4271
"
* 'fix_virtual_columns_liveness_checks_in_update_optimization_5' of https://github.com/psarna/scylla:
tests: add view update optimization case for TTL
database: add view_stats getter
tests: port complex timestamp view test from dtest
db,view: fix virtual columns liveness checks
tests: remove update generating test case
There are additional validation steps that the server executes in
addition to simply invoking the authenticator, so we adapt the tests to
also perform that validation.
We also eliminate lots of code duplication.
Since the role manager and authenticator work in tandem, the test cases
should use the wrapper for `auth::service` to create and drop users
instead of just doing it through the authenticator.
Password handling is verified in its own test suite, and this test not
only makes a number of assumptions about implementation details, but
also tries to verify a hashing scheme (bcrypt) which is not supported on
most Linux distributions.
These defines are global, so they can be in the mode-agnostic cxxflags
rather than the mode-specific cxxflags_{mode}.
Message-Id: <20190228081247.20116-1-avi@scylladb.com>
This test was useful in discovering corner cases for TTLs of virtual
columns, so it's ported to unit test suite from dtest.
The test is also extended with a mirrored case for base regular column
that *is* included in view pk.
When looking for optimization paths, columns selected in a view
are checked against multiple conditions - unfortunately virtual
columns were erroneously skipped from that check, which resulted
in ignoring their TTLs. That can lead to overoptimizing
and not including vital liveness info into view rows,
which can then result in row disappearing too early.
This test case should have been based on CQL instead of creating
artificial update scenarios. It also contains invalid cases
regarding base and view row marker, so it's removed here
and replaced with CQL-based test in this same series.
gnutls requires a configuration file, and the configuration file must match
the one used by the library. Since we ship our own version of the library with
the relocatable package, we must also ship the configuration file.
Luckily, it is possible to override the location of the configuration file via
an environment variable, so all we need to do is to copy the file to the archive
and provide the environment variable in the thunk that adjusts the library path.
Reviewed-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20190227110529.14146-1-avi@scylladb.com>
Currently, we only allocate memory for concurrent unit test runs. This can cause
CPU overcommit when running test.py on machines with a log of memory but few cores.
This overcommit can cause timeouts in tests that are time-sensitive (bad practice,
but can happen) and makes the desktop sluggish.
Improve by allocating at least one logical core per running test.
Reviewed-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20190227132516.22147-1-avi@scylladb.com>
Collect /etc/redhat-release as well as os-release from relevant
hosts. The problem with os-release is that it doesn't contain the
minor version of the EL OS family. Since this is only present in
Red Hat distributions and derivatives, it will not be collected
in Debian derivatives.
Another approach is to use lsb_release -a but it will not provide
anything more useful than os-release on Debian and lsb needs to be
installed on EL derivatives first.
Fixes#4093
Message-Id: <20190225204727.20805-4-dyasny@scylladb.com>
Hostname -i produces a garbled output on new systems with ipv6
enabled, better to use the clean hostname instead, for the file
names.
Message-Id: <20190225204727.20805-3-dyasny@scylladb.com>
The script relies on hostname -i for host address, which can be
wrong in some systems. This patch checks for where the defined
CQL_PORT is listening, and uses the correct IP address instead.
Message-Id: <20190225204727.20805-2-dyasny@scylladb.com>
"
This series restructures the SASL code that was previously internal
to the `password_authenticator` so that it can be used in other contexts.
"
* 'jhk/restructure_sasl/v1' of https://github.com/hakuch/scylla:
auth: Rename SASL challenge class for "PLAIN"
auth: Make a ctor `explicit`
auth: Move `sasl_challenge` to its own file
auth: Decouple SASL code from its parent class
"
This series adds a fuzzer-type unit test for range scans, which
generates a semi-random dataset and executes semi-random range scans
against it, validating the result.
This test aims to cover a wide range of corner cases with the help of
randomness. Data and queries against it are generated in such a way that
various corner cases and their combinations are likely to be covered.
The infrastructure under range-scans have gone under massive changes in
the last year, growing in complexity and scope. The correctness of range
scans is critical for the correct functioning of any Scylla cluster, and
while the current unit tests served well in detecting any major problems
(mostly while developing), they are too simplistic and can only be
relied on to check the correctness of the basic functionality. This test
aims to extend coverage drastically, testing cases that the author of
the range-scan code or that of the existing unit tests didn't even think
exists, by relying on some randomness.
Fixes: #3954 (deprecates really)
"
* 'more-extensive-range-scan-unit-tests/v2' of https://github.com/denesb/scylla:
tests/multishard_mutation_query_test: add fuzzy test
tests/multishard_mutation_query_test: refactor read_all_partitions_with_paged_scan()
tests/test_table: add advanced `create_test_table()` overload
tests/test_table: make `create_test_table()` customizable
query: add trim_clustering_row_ranges_to()
tests/test_table: add keyspace and table name params
tests/test_table: s/create_test_cf/create_test_table/
tests: move create_test_cf() to tests/test_table.{hh,cc}
tests/multishard_mutation_query_test: drop many partition test
tests/multishard_mutation_query_test: drop range tombstone test
"
This miniseries hides virtual columns's writetime and ttl
from the user.
Tests: unit (dev)
Fixes#4288
"
* 'hide_virtual_columns_writetime_and_ttl_2' of https://github.com/psarna/scylla:
tests: add test for hiding virtual columns from WRITETIME
cql3: hide virtual columns from WRITETIME() and TTL()
schema: add column_definition::is_hidden_from_cql
Virtual columns should not be visible to the user,
so they are now hidden not only from directly selecting them,
but also via WRITETIME() and TTL() keywords.
Fixes#4288
test_distributed_loader_with_pending_delete issues a dma write, but violates
the unwritten contract to temporary_buffer::aligned(), which requires that
size be a multiple of alignment. As a result the test fails spuriously.
Instead of playing with the alignment, rewrite that snippet to use the
easier-to-use make_file_output_stream().
Introduced in 1ba88b709f.
Branches: master.
Message-Id: <20190226181850.3074-1-avi@scylladb.com>
From the log it looks like these checks were added in 2014 because of
a broken clang.
Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
