Fixes a regression introduced in
74758c87cd, where tables started to be
created without compression by default (before they were created with
lz4 by default).
Fixes#3926
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
The build environment may not installed ninja-build before running
install-dependencies.sh, so do it after running the script.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20181031110737.17755-1-syuu@scylladb.com>
* seastar a44cedf...d59fcef (10):
> dns: Set tcp output stream buffer size to zero explicitly
> tests: add libc-ares to travis dependencies
> tests: add dns_test to test suite
> build: drop bundled c-ares package
> prometheus: replace the instance label with an optional one
> build: Refactor C++ dialect detection
> build: add libatomic to install-depenencies.sh
> core: use std::underlying_type for open_flags
> core: introduce open_flags::operator&
> core: Fix build for `gnu++14`
Currently, when advance_and_await() fails to allocate the new gate
object, it will throw bad_alloc and leave the phased_barrier object in
an invalid state. Calling advance_and_await() again on it will result
in undefined behavior (typically SIGSEGV) beacuse _gate will be
disengaged.
One place affected by this is table::seal_active_memtable(), which
calls _flush_barrier.advance_and_await(). If this throws, subsequent
flush attempts will SIGSEGV.
This patch rearranges the code so that advance_and_await() has strong
exception guarantees.
Message-Id: <1542645562-20932-1-git-send-email-tgrabiec@scylladb.com>
In (almost) all SSTable write paths, we need to inform the monitor that
the write has failed as well. The monitor will remove the SSTable from
controller's tracking at that point.
Except there is one place where we are not doing that: streaming of big
mutations. Streaming of big mutations is an interesting use case, in
which it is done in 2 parts: if the writing of the SSTable fails right
away, then we do the correct thing.
But the SSTables are not commited at that point and the monitors are
still kept around with the SSTables until a later time, when they are
finally committed. Between those two points in time, it is possible that
the streaming code will detect a failure and manually call
fail_streaming_mutations(), which marks the SSTable for deletions. At
that point we should propagate that information to the monitor as well,
but we don't.
Fixes#3732 (hopefully)
Tests: unit (release)
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <20181114213618.16789-1-glauber@scylladb.com>
In a homogeneous cluster this will reduce number of internal cross-shard hops
per request since RPC calls will arrive to correct shard.
Message-Id: <20181118150817.GF2062@scylladb.com>
In commit a33f0d6, we changed the way we handle arrays during the write
and parse code to avoid reactor stalls. Some potentially big loops were
transformed into futurized loops, and also some calls to vector resizes
were replaced by a reserve + push_back idiom.
The latter broke parsing of the estimated histogram. The reason being
that the vectors that are used here are already initialized internally
by the estimated_histogram object. Therefore, when we push_back, we
don't fill the array all the way from index 0, but end up with a zeroed
beginning and only push back some of the elements we need.
We could revert this array to a resize() call. After all, the reason we
are using reserve + push_back is to avoid calling the constructor member
for each element, but We don't really expect the integer specialization
to do any of that.
However, to avoid confusion with future developers that may feel tempted
to converted this as well for the sake of consistency, it is safer to
just make sure these arrays are zeroed.
Fixes#3918
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <20181116130853.10473-1-glauber@scylladb.com>
Remove the timeout argument to
db::view::view_builder::wait_until_built(), a test-only function to
wait until a given materialized view has finished building.
This change is motivated by the fact that some tests running on slow
environments will timeout. Instead of incrementally increasing the
timeout, remove it completely since tests are already run under an
exterior timeout.
Fixes#3920
Tests: unit release(view_build_test, view_schema_test)
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20181115173902.19048-1-duarte@scylladb.com>
map_reduce_column_families_locally iterate over all tables (column
family) in a shard.
If the number of tables is big it can cause latency spikes.
This patch replaces the current loop with a do_for_each allowing
preepmtion inside the loop.
Fixes#3886
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <20181115154825.23430-1-amnon@scylladb.com>
In a previous patch I fixed most TTLs in the view_complex_test.cc tests
from low numbers to 100 seconds. I missed one. This one never caused
problems in practice, but for good form, let's fix it too.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20181115160234.26478-1-nyh@scylladb.com>
After this patch, the Materialized Views and Secondary Index features
are considered generally-available and no longer require passing an
explicit "--experimental=on" flag to Scylla.
The "--experimental=on" flag and the db::config::check_experimental()
function remain unused, as we graduated the only two features which used
this flag. However, we leave the support for experimental features in
the code, to make it easier to add new experimental features in the future.
Another reason to leave the command-line parameter behind is so existing
scripts that still use it will not break.
Fixes#3917
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20181115144456.25518-1-nyh@scylladb.com>
"
This series enables filtering support for CONTAINS restriction.
"
* 'enable_filtering_for_contains_2' of https://github.com/psarna/scylla:
tests: add CONTAINS test case to filtering tests
cql3: enable filtering for CONTAINS restriction
cql3: add is_satisfied_by(bytes_view) for CONTAINS
Several of the tests in tests/view_complex_test.cc set a cell with a
TTL, and then skip time ahead artificially with forward_jump_clocks(),
to go past the TTL time and check the cell disappeared as expected.
The TTLs chosen for these tests were arbitrary numbers - some had 3 seconds,
some 5 seconds, and some 60 seconds. The actual number doesn't matter - it
is completely artificial (we move the clock with forward_jump_clocks() and
never really wait for that amount of time) and could very well be a million
seconds. But *low* numbers, like the 3 seconds, present a problem on extremely
overcomitted test machines. Our eventually() function already allows for
the possibility that things can hang for up to 8 seconds, but with a 3 second
TTL, we can find ourselves with data being expired and the test failing just
after 3 seconds of wall time have passed - while the test intended that the
dataq will expire only when we explicitly call forward_jump_clocks().
So this patch changes all the TTLs in this test to be the same high number -
100 seconds. This hopefully fixes#3918.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20181115125607.22647-1-nyh@scylladb.com>
With the use of Docker image, some extra options needed to be exposed
to provide extended functionality when starting the image. The flags
added by this commit are:
- cluster-name: name of the Scylla cluster. cluster_name option in
scylla.yaml.
- rpc-address: IP address for client connections (CQL). rpc_address
option in scylla.yaml.
- endpoint-snitch: The snitch used to discover the cluster topology.
endpoint_snitch option in scylla.yaml.
- replace-address-first-boot: Replace a Scylla node by its IP.
replace_address_first_boot option in scylla.yaml.
Signed-off-by: Yannis Zarkadas <yanniszarkadas@gmail.com>
[ penberg@scylladb.com: fix up merge conflicts ]
Message-Id: <20181108234212.19969-2-yanniszarkadas@gmail.com>
test.py:26:1: F401 'signal' imported but unused
test.py:27:1: F401 'shlex' imported but unused
test.py:28:1: F401 'threading' imported but unused
test.py:173:1: E305 expected 2 blank lines after class or function definition,
found 1
test.py:181:34: E241 multiple spaces after ','
test.py:183:34: E241 multiple spaces after ','
test.py:209:24: E222 multiple spaces after operator
test.py:240:5: E301 expected 1 blank line, found 0
test.py:249:23: W504 line break after binary operator
test.py:254:9: E306 expected 1 blank line before a nested definition, found 0
test.py:263:13: F841 local variable 'out' is assigned to but never used
test.py:264:33: E128 continuation line under-indented for visual indent
test.py:265:33: E128 continuation line under-indented for visual indent
test.py:266:33: E128 continuation line under-indented for visual indent
test.py:274:64: F821 undefined name 'e'
test.py:278:53: F821 undefined name 'e'
Signed-off-by: Alexys Jacob <ultrabug@gentoo.org>
Message-Id: <20181104115255.22547-1-ultrabug@gentoo.org>
gen_segmented_compress_params.py:52:47: E226 missing whitespace around
arithmetic operator
gen_segmented_compress_params.py:56:64: E226 missing whitespace around
arithmetic operator
gen_segmented_compress_params.py:60:36: E226 missing whitespace around
arithmetic operator
gen_segmented_compress_params.py:60:48: E226 missing whitespace around
arithmetic operator
gen_segmented_compress_params.py:70:35: E226 missing whitespace around
arithmetic operator
gen_segmented_compress_params.py:70:48: E226 missing whitespace around
arithmetic operator
gen_segmented_compress_params.py:99:43: E226 missing whitespace around
arithmetic operator
gen_segmented_compress_params.py:106:18: E225 missing whitespace around
operator
gen_segmented_compress_params.py:120:5: E303 too many blank lines (2)
gen_segmented_compress_params.py:200:30: E261 at least two spaces before
inline comment
gen_segmented_compress_params.py:200:31: E262 inline comment should start with
'# '
gen_segmented_compress_params.py:218:76: E261 at least two spaces before
inline comment
gen_segmented_compress_params.py:219:59: E703 statement ends with a semicolon
gen_segmented_compress_params.py:219:60: E261 at least two spaces before
inline comment
Signed-off-by: Alexys Jacob <ultrabug@gentoo.org>
Message-Id: <20181104115753.4701-1-ultrabug@gentoo.org>
fix_system_distributed_tables.py:28:20: E203 whitespace before ':'
fix_system_distributed_tables.py:29:20: E203 whitespace before ':'
fix_system_distributed_tables.py:30:20: E203 whitespace before ':'
fix_system_distributed_tables.py:31:20: E203 whitespace before ':'
fix_system_distributed_tables.py:33:20: E203 whitespace before ':'
fix_system_distributed_tables.py:34:23: E203 whitespace before ':'
fix_system_distributed_tables.py:35:23: E203 whitespace before ':'
fix_system_distributed_tables.py:39:20: E203 whitespace before ':'
fix_system_distributed_tables.py:40:20: E203 whitespace before ':'
fix_system_distributed_tables.py:41:20: E203 whitespace before ':'
fix_system_distributed_tables.py:42:20: E203 whitespace before ':'
fix_system_distributed_tables.py:43:20: E203 whitespace before ':'
fix_system_distributed_tables.py:44:20: E203 whitespace before ':'
fix_system_distributed_tables.py:45:20: E203 whitespace before ':'
fix_system_distributed_tables.py:46:20: E203 whitespace before ':'
fix_system_distributed_tables.py:47:20: E203 whitespace before ':'
fix_system_distributed_tables.py:48:20: E203 whitespace before ':'
fix_system_distributed_tables.py:52:20: E203 whitespace before ':'
fix_system_distributed_tables.py:53:20: E203 whitespace before ':'
fix_system_distributed_tables.py:54:20: E203 whitespace before ':'
fix_system_distributed_tables.py:55:20: E203 whitespace before ':'
fix_system_distributed_tables.py:56:20: E203 whitespace before ':'
fix_system_distributed_tables.py:57:20: E203 whitespace before ':'
fix_system_distributed_tables.py:58:20: E203 whitespace before ':'
fix_system_distributed_tables.py:59:20: E203 whitespace before ':'
fix_system_distributed_tables.py:60:20: E203 whitespace before ':'
fix_system_distributed_tables.py:61:20: E203 whitespace before ':'
fix_system_distributed_tables.py:62:20: E203 whitespace before ':'
fix_system_distributed_tables.py:66:19: E203 whitespace before ':'
fix_system_distributed_tables.py:67:19: E203 whitespace before ':'
fix_system_distributed_tables.py:72:19: E203 whitespace before ':'
fix_system_distributed_tables.py:73:19: E203 whitespace before ':'
fix_system_distributed_tables.py:74:19: E203 whitespace before ':'
fix_system_distributed_tables.py:78:19: E203 whitespace before ':'
fix_system_distributed_tables.py:79:19: E203 whitespace before ':'
fix_system_distributed_tables.py:80:19: E203 whitespace before ':'
fix_system_distributed_tables.py:84:19: E203 whitespace before ':'
fix_system_distributed_tables.py:85:19: E203 whitespace before ':'
fix_system_distributed_tables.py:89:19: E203 whitespace before ':'
fix_system_distributed_tables.py:90:19: E203 whitespace before ':'
fix_system_distributed_tables.py:91:19: E203 whitespace before ':'
fix_system_distributed_tables.py:95:22: E203 whitespace before ':'
fix_system_distributed_tables.py:96:22: E203 whitespace before ':'
fix_system_distributed_tables.py:99:1: E302 expected 2 blank lines, found 0
fix_system_distributed_tables.py:103:72: E201 whitespace after '['
fix_system_distributed_tables.py:103:82: E202 whitespace before ']'
fix_system_distributed_tables.py:105:43: E201 whitespace after '['
fix_system_distributed_tables.py:105:53: E202 whitespace before ']'
fix_system_distributed_tables.py:111:16: E713 test for membership should be
'not in'
fix_system_distributed_tables.py:118:20: E713 test for membership should be
'not in'
fix_system_distributed_tables.py:135:25: E722 do not use bare 'except'
fix_system_distributed_tables.py:138:5: E722 do not use bare 'except'
fix_system_distributed_tables.py:144:1: E305 expected 2 blank lines after
class or function definition, found 0
fix_system_distributed_tables.py:145:47: E251 unexpected spaces around keyword
/ parameter equals
fix_system_distributed_tables.py:145:49: E251 unexpected spaces around keyword
/ parameter equals
fix_system_distributed_tables.py:160:1: W391 blank line at end of file
Signed-off-by: Alexys Jacob <ultrabug@gentoo.org>
Message-Id: <20181104113001.22783-1-ultrabug@gentoo.org>
dist/docker/redhat/docker-entrypoint.py:20:1: E722 do not use bare 'except'
dist/docker/redhat/commandlineparser.py:13:13: E128 continuation line
under-indented for visual indent
Signed-off-by: Alexys Jacob <ultrabug@gentoo.org>
Message-Id: <20181104120134.9598-1-ultrabug@gentoo.org>
With the number of unit tests approaching one hundred, the output of
test.py becomes more challenging to read.
If some test fails, we will only get the details after all the tests
complete, but some tests take way longer than others.
With the coloured status, it is much simpler to immediately locate
failing tests. Developer can cancel others and repeat the failing
ones.
Signed-off-by: Vladimir Krivopalov <vladimir@scylladb.com>
Message-Id: <63a99a2fb70fdc33fd6eeb8e18fee977a47bd278.1541541184.git.vladimir@scylladb.com>
"
This series adds DEFAULT UNSET and DEFAULT NULL keyword support
to INSERT JSON statement, as stated in #3909.
Tests: unit (release)
"
* 'add_json_default_unset_2' of https://github.com/psarna/scylla:
tests: add DEFAULT UNSET case to JSON cql tests
tests: split JSON part of cql query test
cql3: add DEFAULT UNSET to INSERT JSON
When inserting a JSON, additional DEFAULT UNSET or DEFAULT NULL
keywords can be appended.
With DEFAULT UNSET, values omitted in JSON will not be changed
at all. With DEFAULT NULL (default), omitted values will be
treated as having a 'null' value.
Fixes#3909
"
Fix for #3897
"Ec2MultiRegionSnitch: prints a cryptic error when a Public IP is not
available"
Ec2MultiRegionSnitch naturally requires a Public IP to be available and
therefore it's expected to refuse to work without it.
However the error message that is printed today is a total disaster and
has to be fixed ASAP to be something much more human readable.
This series adds a human readable preabmle that will let a poor user
understand what should he/she do.
"
* 'improve-ec2-multi-region-snitch-error-message-when-pulic-address-is-not-available-v2' of https://github.com/vladzcloudius/scylla:
locator: ec2_multi_region_snitch::start(): print a human readable error if Public IP may not be retrieved
locator: ec2_multi_region_snitch::start(): rework on top of seastar::thread
During streaming, there are cases when we should invoke the view write
path. In particular, if we're streaming because of repair or if a view
has not yet finished building and we're bootstrapping a new node.
The design constraints are:
1) The streamed writes should be visible to new writes, but the
sstable should not participate in compaction, or we would lose the
ability to exclude the streamed writes on a restart;
2) The streamed writes must not be considered when generating view
updates for them;
3) Resilient to node restarts;
4) Resilient to concurrent stream sessions, possibly streaming mutations for overlapping ranges.
We achieve this by writing the streamed writes to an sstable in a
different folder, call it "staging". We achieve 1) by publishing the
sstable to the column family sstable set, but excluding it from
compactions. We do these steps upon boot, by looking at the staging
directory, thus achieving 3).
Fixes#3275
* 'streaming_view_to_staging_sstables_9' of https://github.com/psarna/scylla: (29 commits)
tests: add materialized views test
tests: add view update generator to cql test env
main: add registering staging sstables read from disk
database: add a check if loaded sstable is already staging
database: add get_staging_sstable method
streaming: stream tables with views through staging sstables
streaming: add system distributed keyspace ref to streaming
streaming: add view update generator reference to streaming
main: add generating missed mv updates from staging sstables
storage_service: move initializing sys_dist_ks before bootstrap
db/view: add view_update_from_staging_generator service
db/view: add view updating consumer
table: add stream_view_replica_updates
table: split push_view_replica_updates
table: add as_mutation_source_excluding
table: move push_view_replica_updates to table.cc
database: add populating tables with staging sstables
database: add creating /staging directory for sstables
database: add sstable-excluding reader
table: add move_sstable_from_staging_in_thread function
...
Right now materialized_views_test.cc contains view updating tests,
but the intention is to move mv-related tests from cql_query_test
here and use it for all future unit testing of MV.
Staging sstables are loaded before regular ones. If the process
fails midway, an sstable can be linked both in the regular directory
and in staging directory. In such cases, the sstable remains
in staging and will be moved to the regular directory
by view update streamer service.
This method can be used to check if sstable is staging,
i.e. it shouldn't be compacted and it will not be used
for generating view updates from other staging tables,
and return proper shared_sstable pointer if it is.
While streaming to a table with paired views, staging sstables
are used. After the table is written to disk, it's used to generate
all required view updates. It's also resistant to restarts as it's
stored on a hard drive in staging/ directory.
Refs #3275
