The gossiper checks if features should be enabled from its timer
callback when it detects that endpoint_state_map changed, that is
different than shadow_endpoint_state_map.
shadow_endpoint_state_map is also assigned from endpoint_state_map in
storage_service::replicate_tm_and_ep_map(), called from
storage_service::on_change()
Call gossiper:maybe_enable_features() in replicate_tm_and_ep_map so
that we won't miss gossip feature update.
Fixes#2824
* git@github.com:scylladb/seastar-dev asias/gossip_miss_feature_update_v1:
gossip: Move the _features_condvar signal code to
maybe_enable_features
gossip: Make maybe_enable_features public
storage_service: Check gossip feature update in
replicate_tm_and_ep_map
(cherry picked from commit 02d41864af)
This reverts commit b56ba02335.
After commit 8fa35d6ddf (messaging_service: Get rid of timeout and retry
logic for streaming verb), streaming verb in rpc does not check if a
node is in gossip memebership since all the retry logic is removed.
Remove the extra wait before removing the joining node from gossip
membership.
Message-Id: <a416a735bb8aad533bbee190e3324e6b16799415.1504063598.git.asias@scylladb.com>
(cherry picked from commit cc18da5640)
It specifies the maximum gossip shadow round time. It can be used to
reduce the gossip feature check time during node boot up.
For instance, when the first node in the cluster, which listed both
itself and other node as seed in the yaml config, boots up, it will try
to talk to other seed nodes which are not started yet. The gossip shadow
round will be used to fetch the feature info of the cluster. Since there
is no other seed node in the cluster, the shadow round will fail. User
can reduce the default shadow_round_ms option to reduce the boot time.
Fixes#2615
Message-Id: <10916ce9059f3c7f1a1fb465919ae57de3b67d59.1500540297.git.asias@scylladb.com>
(cherry picked from commit cf6f4a5185)
* seastar 7ebbb26...124467d (4):
> peering_sharded_service: prevent over-run the container
> sharded: fix move constructor for peering_sharded_service services
> sharded: improve support for cooperating sharded<> services
> sharded: support for peer services
Includes change to batchlog_manager constructor to adapt it to
seastar::sharded::start() change.
Needed for gossip backport.
* seastar e4fcb6c...7ebbb26:
Warn: seastar doesn't contain commit e4fcb6c27cc5dce70d44472522166abe1af29af6
The submodule link pointed nowhere, point it at the tip of scylla-seastar/branch-2.0.
We don't pull schema during rolling upgrade, that is until
schema_tables_v3 feature is enabled on all nodes.
Because features are enabled from gossiper timer, there is a race
between feature enablement and processing of endpoint states which may
trigger schema pull. It can happen that we first try to pull, but
only later enable the feature. In that case the schema pull will not
happen until the next schema change.
The fix is to ensure that pulls abandoned due to feature not being enabled
will be retried when it is enabled.
Fixes sporadic failure in dtest:
repair_additional_test.py:RepairAdditionalTest.repair_schema_test
Message-Id: <1506428715-8182-2-git-send-email-tgrabiec@scylladb.com>
(cherry picked from commit b704710954)
"Fixes the problem of concurrent populations of clustering row ranges
leading to some readers skipping over some of the rows.
Spotted during code review.
Fixes #2834."
* tag 'tgrabiec/fix-cache-reader-skipping-rows-v2' of github.com:scylladb/seastar-dev:
tests: mvcc: Add test for partition_snapshot_row_cursor
tests: row_cache: Add test for concurrent population
tests: row_cache: Make populate_range() accept partition_range
tests: Add simple_schema::make_ckey_range()
cache_streamed_mutation: Add missing _next_row.maybe_refresh() call
mvcc: partition_snapshot_row_cursor: Fix cursor skipping over rows added after its position
mvcc: partition_snapshot_row_cursor: Rename up_to_date() to iterators_valid()
mvcc: Keep track of all iterators in partition_snapshot_row_cursor
mvcc: Make partition_snapshot_row_cursor printable
(cherry picked from commit af1976bc30)
[tgrabiec: resolved conflicts]
Needed in tests to limit amount of prefetching done by readers, so
that it's easier to test interleaving of various events.
(cherry picked from commit cb16b038ef)
Since commit 8378fe190, we disable schema sync in a mixed cluster.
The detection is done using gossiper features. We need to make sure
the features are registerred, and thus can be enabled, before the
bootstrapping of a non-seed node happens. Otherwise the bootstrap will
hang waiting on schema sync which will not happen.
Message-Id: <1505893837-27876-2-git-send-email-tgrabiec@scylladb.com>
(cherry picked from commit 8e46d15f91)
"On bad_alloc the section is retried. If the exception happened inside
fast_forward_to() on the underlying reader, that call will be
retried. However, the reader should not be used after exception is
thrown, since it is in unspecified state. Also, calling
fast_forward_to() with cache region locked increases the chances of it
failing to allocate.
We shouldn't call fast_forward_to() with the cache region locked.
Fixes #2791."
* 'tgrabiec/dont-ffwd-in-alloc-section' of github.com:scylladb/seastar-dev:
cache_streamed_mutation: De-futurize cursor movement
cache_streamed_mutation: Call fast_forward_to() outside allocating section
cache_streamed_mutation: Switch from flags to explicit state machine
(cherry picked from commit 5b0cb28af9)
[tgrabiec: resolved minor conflicts]
"This series tries to improve the bootstrap of a node in a large cluster by
improving how gossip applies the gossip node state. In #2404, the joining node
failed to bootstrap, because it did not see the seed node when
storage_service::bootstrap ran. After this series, we apply the whole gossip
state contained in the gossip ack/ack2 message before applying the next one,
and we apply the state of the seed node earlier than non-seed node so we can
have the seed node's state faster. We also add some randomness to the order of
applying gossip node state to prevent some of the nodes' state are always
applied earlier than the others.
This series improves apply_state_locally for large cluster:
- Tune the order of applying endpoint_state
- Serialize apply_state_locally
- Avoid copying of the gossip state map
Fixes#2404"
* tag 'asias/gossip_issue_2404_v2' of github.com:scylladb/seastar-dev:
gossip: Avoid copying with apply_state_locally
gossip: Serialize apply_state_locally
gossip: Tune the order of applying endpoint_state in apply_state_locally
gossip: Introduce is_seed helper
gossip: Pass const endpoint_state& in notify_failure_detector
gossip: Pass reference in notify_failure_detector
(cherry picked from commit d2632ddf1d)
gossiper::apply_state_locally() calls handle_major_state_change() for
each endpoint, in a seastar thread, which calls mark_alive() for new
nodes, which calls ms().send_gossip_echo(id).get(). So it synchronously
waits for each node to respond before it moves on to the next entry. As
a result it may take a while before whole state is processed.
Apache (tm) Cassandra (tm) sends echos in the background.
In a large cluster, we see at the time the joining node starts
streaming, it hasn't managed to apply all the endpoint_state for peer
nodes, so the joining node does not know some of the nodes yet, which
results in the joining node ingores to stream from some of the existing
nodes.
Fixes#2787Fixes#2797
Message-Id: <3760da2bef1a83f1b6a27702a67ca4170e74b92c.1505719669.git.asias@scylladb.com>
(cherry picked from commit 8f8273969d)
"When there are at least 2 nodes upgraded to 2.0, and the two exchanged schema
for some reason, reads or writes which involve both 1.7 and 2.0 nodes may
start to fail with the following error logged:
storage_proxy - Exception when communicating with 127.0.0.3: Failed to load schema version 58fc9b89-74ab-37ca-8640-8b38a1204f8d
The situation should heal after whole cluster is upgraded.
Table schema versions are calculated by 2.0 nodes differently than 1.7 nodes
due to change in the schema tables format. Mismatch is meant to be avoided by
having 2.0 nodes calculate the old digest on schema migration during upgrade,
and use that version until next time the table is altered. It is thus not
allowed to alter tables during the rolling upgrade.
Two 2.0 nodes may exchange schema, if they detect through gossip that their
schema versions don't match. They may not match temporarily during boot, until
the upgraded node completes the bootstrap and propagates its new schema
through gossip. One source of such temporary mismatch is construction of new
tracing tables, which didn't exist on 1.7. Such schema pull will result in a
schema merge, which cause all tables to be altered and their schema version to
be recalculated. The new schema will not match the one used by 1.7 nodes,
causing reads and writes to fail, because schema requesting won't work during
rolling upgrade from 1.7 to 2.0.
The main fix employed here is to hold schema pulls, even among 2.0 nodes,
until rolling upgrade is complete."
Fixes#2802.
* 'tgrabiec/fix-schema-mismatch' of github.com:scylladb/seastar-dev:
tests: schema_change_test: Add test_merging_does_not_alter_tables_which_didnt_change test case
tests: cql_test_env: Enable all features in tests
schema_tables: Make make_scylla_tables_mutation() visible
migration_manager: Disable pulls during rolling upgrade from 1.7
storage_service: Introduce SCHEMA_TABLES_V3 feature
schema_tables: Don't alter tables which differ only in version
schema_mutations: Use mutation_opt instead of stdx::optional<mutation>
(cherry picked from commit 8378fe190a)
"These patches make Scylla refuse to load counter sstables that may
contain unsupported counter shards. They are recognised by the lack of
the Scylla component.
Fixes #2766."
* tag 'reject-non-scylla-counter-sstables/v1' of https://github.com/pdziepak/scylla:
db: reject non-Scylla counter sstables in flush_upload_dir
db: disallow loading non-Scylla counter sstables
sstable: add has_scylla_component()
(cherry picked from commit fe019ad84d)
"Scylla 1.7.4 and older use incorrect ordering of counter shards, this
was fixed in 0d87f3dd7d ("utils::UUID:
operator< should behave as comparison of hex strings/bytes"). However,
that patch was not backported to 1.7 branch until very recently. This
means that versions 1.7.4 and older emit counter shards in an incorrect
order and expect them to be so. This is particularly bad when dealing
with imported correct sstables in which case some shards may become
duplicated.
The solution implemented in this patch is to allow any order of counter
shards and automaticly merge all duplicates. The code is written in a
way so that the correct ordering is expected in the fast path in order
not to excessively punish unaffected deployments.
A new feature flag CORRECT_COUNTER_ORDER is introduced to allow seamless
upgrade from 1.7.4 to later Scylla versions. If that feature is not
available Scylla still writes sstables and sends on-wire counters using
the old ordering so that it can be correctly understood by 1.7.4, once
the flag becomes available Scylla switches to the correct order.
Fixes #2752."
* tag 'fix-upgrade-with-counters/v2' of https://github.com/pdziepak/scylla:
tests/counter: verify counter_id ordering
counter: check that utils::UUID uses int64_t
mutation_partition_serializer: use old counter ordering if necessary
mutation_partition_view: do not expect counter shards to be sorted
sstables: write counter shards in the order expected by the cluster
tests/sstables: add storage_service_for_tests to counter write test
tests/sstables: add test for reading wrong-order counter cells
sstables: do not expect counter shards to be sorted
storage_service: introduce CORRECT_COUNTER_ORDER feature
tests/counter: test 1.7.4 compatible shard ordering
counters: add helper for retrieving shards in 1.7.4 order
tests/counter: add tests for 1.7.4 counter shard order
counters: add counter id comparator compatible with Scylla 1.7.4
tests/counter: verify order of counter shards
tests/counter: add test for sorting and deduplicating shards
counters: add function for sorting and deduplicating counter cells
counters: add counter_id::operator>
(cherry picked from commit 31706ba989)
* dist/ami/files/scylla-ami b41e5eb...5ffa449 (3):
> amzn-main.repo: stick to Amazon Linux 2017.03 kernel (4.9.x)
> Prevent dependency error on 'yum update'
> scylla_create_devices: don't raise error when no disks found
Fixes#2751.
Still tracking master branch.
Initialize the system_auth and system_traces keyspaces and their tables after
the Node joins the token ring because as a part of system_auth initialization
there are going to be issues SELECT and possible INSERT CQL statements.
This patch effectively reverts the d3b8b67 patch and brings the initialization order
to how it was before that patch.
Fixes#2273
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Message-Id: <1500417217-16677-1-git-send-email-vladz@scylladb.com>
(cherry picked from commit e98adb13d5)
"Fixes #2734."
* 'tgrabiec/make-sstable-reader-work-with-empty-range-set' of github.com:scylladb/seastar-dev:
tests: Introduce clustering_ranges_walker_test
tests: simple_schema: Add missing include
sstables: reader: Make clustering_ranges_walker work with empty range set
clustering_ranges_walker: Make adjacency more accurate
(cherry picked from commit 5224ab9c92)
- Removed text from Report's "PURPOSE" section, which was referring to the "MANUAL CHECK LIST" (not needed anymore).
- Removed curl command (no longer using the api_address), instead using scylla --version
- Added -v flag in iptables command, for more verbosity
- Added support to for OEL (Oracle Enterprise Linux) - minor fix
- Some text changes - minor
- OEL support indentation fix + collecting all files under /etc/scylla
- Added line seperation under cp output message
Signed-off-by: Tomer Sandler <tomer@scylladb.com>
Message-Id: <20170828131429.4212-1-tomer@scylladb.com>
(cherry picked from commit f1eb6a8de3)
Ref #2733.
* 'tgrabiec/fix-fast-forwarding' of github.com:scylladb/seastar-dev:
tests: mutation_source_test: Add more tests for fast forwarding across partitions
sstables: Fix abort in mutation reader for certain skip pattern
sstables: Fix reader returning partition past the query range in some cases
sstables: Introduce data_consume_context::eof()
(cherry picked from commit 4e67bc9573)
"Preserve the networking configuration mode during the upgrade by generating the /etc/scylla.d/perftune.yaml
file and using it."
Fixes#2725.
* 'dist_respect_cpuset_conf-v3' of https://github.com/vladzcloudius/scylla:
scylla_prepare: respect the cpuset.conf when configuring the networking
scylla_cpuset_setup: rm perftune.yaml
scylla_cpuset_setup: add a missing "include" of scylla_lib.sh
(cherry picked from commit 40aeb00151)
* seastar 2993cae...06790c0 (3):
> scripts: posix_net_conf.sh: allow passing a perftune.py configuration file as a parameter
> scripts: perftune.py: add the possibility to pass the parameters in a configuration file and print the YAML file with the current configuration
> scripts: perftune.py: actually use the number of Rx queues when comparing to the number of CPU threads
