"
This patch series changes the build system to build all tarballs to
build/dist/<mode>/tar directory. For example, running:
./tools/toolchain/dbuild ./configure.py --mode=dev && ./tools/toolchain/dbuild ninja-build dist-tar
produces the following tarballs in build/dist/dev/tar:
$ ls -1 build/dist/dev/tar/
scylla-jmx-package.tar.gz
scylla-package.tar.gz
scylla-python3-package.tar.gz
scylla-tools-package.tar.gz
This makes it easy to locate release tarballs for humans and scripts. To
preserve backward compatibility, the tarballs are also retained in their
original locations. Once release engineering infrastructure has been
adjusted to use the new locations, we can drop the duplicate copies.
"
* 'penberg/build-dist-tar/v1' of github.com:penberg/scylla:
configure.py: Copy tarballs to build/dist/<mode>/tar directory
configure.py: Add "dist-<component>-tar" targets
reloc/python3: Add "--builddir" to build_deb.sh
configure.py: Use copy-on-write copies when possible
"
Contains several fixes which improve debuggability in situations where
too large column ids are passed to column definition loop methods.
"
* 'schema-range-check-fix' of github.com:tgrabiec/scylla:
schema: Add table name and schema version to error messages
schema: Use on_internal_error() for range check errors
schema: Fix off-by-one in column range check
schema: Make range checks for regular and static columns the same as for clustering columns
select() is too generic for the method that retrieve sstable runs,
and it has a completely different meaning that the former select
method used to select sstables based on token range.
let's give it a more descriptive name.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20200811193401.22749-1-raphaelsc@scylladb.com>
regression caused by 55cf219c97.
remove_by_toc_name() must work both for a sealed sstable with toc,
and also a partial sstable with tmp toc.
so dirname() should be called conditionally on the condition of
the sstable.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20200813160612.101117-1-raphaelsc@scylladb.com>
LCS can have its overlapping invariant broken after operations that can
proceed in parallel to regular compaction like cleanup. That's because
there could be two compactions in parallel placing data in overlapping
token ranges of a given level > 0.
After reshape, the whole table will be rewritten, on restart, if a
given level has more than (fan_out*2)=20 overlaps.
That may sound like enough, but that's not taking into account the
exponential growth in # of SSTables per level, so 20 overlaps may
sound like a lot for level 2 which can afford 100 sstables, but it's
only 2% of level 3, and 0.2% of level 4. So let's change the
overlapping tolerance from the constant of fan_out*2 to 10% of level
limit on # of SSTables, or fan_out, whichever is higher.
Refs #6938.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20200810154510.32794-1-raphaelsc@scylladb.com>
After 8014c7124, cleanup can potentially pick a compacting SSTable.
Upgrade and scrub can also pick a compacting SSTable.
The problem is that table::candidates_for_compaction() was badly named.
It misleads the user into thinking that the SSTables returned are perfect
candidates for compaction, but manager still need to filter out the
compacting SSTables from the returned set. So it's being renamed.
When the same SSTable is compacted in parallel, the strategy invariant
can be broken like overlapping being introduced in LCS, and also
some deletion failures as more than one compaction process would try
to delete the same files.
Let's fix scrub, cleanup and ugprade by calling the manager function
which gets the correct candidates for compaction.
Fixes#6938.
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20200811200135.25421-1-raphaelsc@scylladb.com>
Before this patch, modifying cdc/cdc_options.hh required recompiling 264
source files. This is because this header file was included by a couple
other header files - most notably schema.hh, where a forward declaration
would have been enough. Only the handful of source files which really
need to access the CDC options should include "cdc/cdc_options.hh" directly.
After this patch, modifying cdc/cdc_options.hh requires only 6 source files
to be recompiled.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20200813070631.180192-1-nyh@scylladb.com>
C++17 introduced try_emplace for maps to replace a pattern:
if(element not in a map) {
map.emplace(...)
}
try_emplace is more efficient and results in a more concise code.
This commit introduces usage of try_emplace when it's appropriate.
Tests: unit(dev)
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <4970091ed770e233884633bf6d46111369e7d2dd.1597327358.git.piotr@scylladb.com>
This testcase was temporarily commented out in 37ebe52, because it
relied on buggy (#6369) behaviour fixed by that commit. Specifically,
it expected a NULL comparison to match a NULL cell value. We now
bring it back, with corrected result expectation.
Tests: unit (dev)
Signed-off-by: Dejan Mircevski <dejan@scylladb.com>
The _clients is std::vector, it doesn't have _M_elems.
Luckily there's std_vector() class for it.
The seastar::rpc::server::_conns is unordered_map, not
unordered_set.
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Message-Id: <20200814070858.32383-1-xemul@scylladb.com>
It is needed to print the boost::intrusive::list which is used
by repair_meta_for_masters in repair.
Fixes#7037
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Signed-off-by: Asias He <asias@scylladb.com>
C++20 introduced `contains` member functions for maps and sets for
checking whether an element is present in the collection. Previously
`count` function was often used in various ways.
`contains` does not only express the intend of the code better but also
does it in more unified way.
This commit replaces all the occurences of the `count` with the
`contains`.
Tests: unit(dev)
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <b4ef3b4bc24f49abe04a2aba0ddd946009c9fcb2.1597314640.git.piotr@scylladb.com>
1. The node1 is shutdown
2. The node1 sends shutdown message to node2
3. The node2 receives gossip shutdown message but the handler yields
4. The node1 is restarted
5. The node1 sends new gossip endpoint_state to node2, node2 applies the state
in apply_state_locally and calls gossiper::handle_major_state_change
and then calls gossiper::mark_alive
6. The shutdown message handler in step 3 resumes and sets status of node1 to SHUTDOWN
7. The gossiper::mark_alive fiber in step 5 resumes and calls gossiper::real_mark_alive,
node2 will skip to mark node1 as alive because the status of node1 is
SHUTDOWN. As a result, node1 is alive but it is not marked as UP by node2.
To fix, we serialize the two operations.
Fixes#7032
Merged pull request https://github.com/scylladb/scylla/pull/7028
By Calle Wilund:
Changes the "preimage" option from binary true/false to on/off/full (accepting true/false, and using old style notation for normal to string - for upgrade reasons), where "full" will force us to include all columns in pre image log rows.
Adds small test (just adding the case to preimage test).
Uses the feature in alternator
Fixes#7030
alternator: Set "preimage" to "full" for streams
cdc_test: Do small test of "full"
cdc: Make pre image optionally "full" (include all columns)
Fixes#7030
Dynamo/alternator streams old image data is supposed to
contain the full old value blob (all keys/values).
Setting preimage=full ensures we get even those properties
that have separate columns if they are not part of an actual
modification.
Makes the "preimage" option for cdc non-binary, i.e. it can now
be "true"/"on", "false"/"off" or "full. The two former behaving like
previously, the latter obviously including all columns in pre image.
Merged pull request https://github.com/scylladb/scylla/pull/7018
by Piotr Sarna:
This series addresses various issues with metrics and semaphores - it mainly adds missing metrics, which makes it possible to see the length of the queues attached to the semaphores. In case of view building and view update generation, metrics was not present in these services at all, so a first, basic implementation is added.
More precise semaphore metrics would ease the testing and development of load shedding and admission control.
view_builder: add metrics
db, view: add view update generator metrics
hints: track resource_manager sending queue length
hints: add drain queue length to metrics
table: add metrics for sstable deletion semaphore
database: remove unused semaphore
The `shard` parameter of `find_db()` is optional and is defaulted to
`None`. When missing, the current shard's database instance is returned.
The problem is that the if condition checking this uses `not shard`,
which also evaluates to `True` if `shard == 0`, resulting in returning
the current shard's database instance for shard 0. Change the condition
to `shard is None` to avoid this.
Fixes: #7016
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20200812091546.1704016-1-bdenes@scylladb.com>
"
This series adds progress metrics for the node operations. Metrics for bootstrap and rebuild progress are added as a starter. I will add more for the remaining operations after getting feedback.
With this the Scylla Monitor and Scylla Manager can know the progress of the bootstrap and other node operations. E.g.,
scylla_node_ops_bootstrap_nr_ranges_finished{shard="0",type="derive"} 50
scylla_node_ops_bootstrap_nr_ranges_total{shard="0",type="derive"} 1040
Fixes#1244, #6733
"
* 'repair_progress_metrics_v3' of github.com:asias/scylla:
repair: Add progress metrics for repair ops
repair: Add progress metrics for rebuild ops
repair: Add progress metrics for bootstrap ops
"
It is pretty hard to find the repair_meta object when debugging a core.
This patch makes it is easier by putting repair_meta object created by
both repair follower and master into a map.
Fixes#7009
"
* asias-repair_make_debug_eaiser_track_all_repair_metas:
repair: Add repair_meta_tracker to track repair_meta for followers and masters
repair: Move thread local object _repair_metas out of the function
* 'espindola/move-out-of-line' of https://github.com/espindola/scylla:
test: Move code in sstable_run_based_compaction_strategy_for_tests.hh out of line
test: Drop ifdef now that we always use c++20
test: Move sstable_run_based_compaction_strategy_for_tests.hh to test/lib
It is pretty hard to find the repair_meta object when debugging a core.
This patch makes it is easier by putting repair_meta object created by
both repair follower and master into boost intrusive list.
Fixes#7009
test is currently flaky since system reads can happen
in the background and disturb the global row cache stats.
Use the table's row_cache stats instead.
Fixes#6773
Test: cql_query_test.test_cache_bypass(dev, debug)
Credit-to: Botond Dénes <bdenes@scylladb.com>
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20200811140521.421813-1-bhalevy@scylladb.com>
make all required allocations in advance to merging sstables
into _compacting_sstables so it should not throw
after registering some sstables, but not all.
Test: database_test(dev)
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20200811132440.416945-1-bhalevy@scylladb.com>
Currently when running against a debug build, our integration test suite
suffers from a ton of timeout related error logs, caused by auth queries
timing out. This causes spurious test failures due to the unexpected
error messages in the log.
This patch increases the timeout for internal distributed auth queries
in debug mode, to give the slow debug builds more headroom to meet the
timeout.
Refs: #6548
Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <20200811145757.1593350-1-bdenes@scylladb.com>
"
Let users disable the unencrypted native transport too by setting the port to
zero in the scylla.yaml configuration file.
Fixes#6997
"
* penberg-penberg/native-transport-disable:
docs/protocol: Document CQL protocol port configuration options
transport: Allow user to disable unencrypted native transport
