Single-node clusters can agree on schema

At some points while bootstrapping [1], new non-seed Scylla nodes wait
for schema agreement among all known endpoints in the cluster.

The check for schema agreement was in
`service::migration_manager::is_ready_for_bootstrap`. This function
would return `true` if, at the time of its invocation, the node was
aware of at least one `UP` peer (not itself) and that all `UP` peers had
the same schema version as the node.

We wish to re-use this check in the `auth` sub-system to ensure that
the schema for internal system tables used for access-control have
propagated to the entire cluster.

Unlike in `service/storage_service.cc`, where `is_ready_for_bootstrap`
was only invoked for seed nodes, we wish to wait for schema agreement
for all nodes regardless of whether or not they are seeds.

For a single-node cluster with itself as a seed,
`is_ready_for_bootstrap` would always return `false`.

We therefore change the conditions for schema agreement. Schema
agreement is now reached when there are no known peers (so the endpoint
map of the gossiper consists only of ourselves), or when there is at
least one `UP` peer and all `UP` peers have the same schema version as
us.

This change should not impact any bootstrap behavior in
`storage_service` because seed nodes do not invoke the function and
non-seed nodes wait for peer visibility before checking for schema
agreement.

Since this function is no longer checking for schema agreement only in
the context of bootstrapping non-seed nodes, we rename it to reflect its
generality.

[1] http://thelastpickle.com/blog/2017/05/23/auto-bootstrapping-part1.html

service/migration_manager.cc

@@ -132,10 +132,15 @@ future<> migration_manager::schedule_schema_pull(const gms::inet_address& endpoi
     return make_ready_future<>();
 }
 
-bool migration_manager::is_ready_for_bootstrap() {
+bool migration_manager::have_schema_agreement() {
+    const auto known_endpoints = gms::get_local_gossiper().endpoint_state_map;
+    if (known_endpoints.size() == 1) {
+        // Us.
+        return true;
+    }
     auto our_version = get_local_storage_proxy().get_db().local().get_version();
     bool match = false;
-    for (auto& x : gms::get_local_gossiper().endpoint_state_map) {
+    for (auto& x : known_endpoints) {
         auto& endpoint = x.first;
         auto& eps = x.second;
         if (endpoint == utils::fb_utilities::get_broadcast_address() || !eps.is_alive()) {

service/migration_manager.hh

@@ -144,7 +144,10 @@ public:
 
     future<> stop();
 
-    bool is_ready_for_bootstrap();
+    /**
+     * Known peers in the cluster have the same schema version as us.
+     */
+    bool have_schema_agreement();
 
     void init_messaging_service();
 private:

service/storage_service.cc

@@ -413,7 +413,7 @@ void storage_service::join_token_ring(int delay) {
         }
         // if our schema hasn't matched yet, keep sleeping until it does
         // (post CASSANDRA-1391 we don't expect this to be necessary very often, but it doesn't hurt to be careful)
-        while (!get_local_migration_manager().is_ready_for_bootstrap()) {
+        while (!get_local_migration_manager().have_schema_agreement()) {
             set_mode(mode::JOINING, "waiting for schema information to complete", true);
             sleep(std::chrono::seconds(1)).get();
         }
@@ -442,7 +442,7 @@ void storage_service::join_token_ring(int delay) {
             }
 
             // Check the schema and pending range again
-            while (!get_local_migration_manager().is_ready_for_bootstrap()) {
+            while (!get_local_migration_manager().have_schema_agreement()) {
                 set_mode(mode::JOINING, "waiting for schema information to complete", true);
                 sleep(std::chrono::seconds(1)).get();
             }