Files
scylladb/frozen_mutation.hh
Tomasz Grabiec f893516e55 Merge "lwt: store column_mapping's for each table schema version upon a DDL change" from Pavel Solodovnikov
This patch introduces a new system table: `system.scylla_table_schema_history`,
which is used to keep track of column mappings for obsolete table
schema versions (i.e. schema becomes obsolete when it's being changed
by means of `CREATE TABLE` or `ALTER TABLE` DDL operations).

It is populated automatically when a new schema version is being
pulled from a remote in get_schema_definition() at migration_manager.cc
and also when schema change is being propagated to system schema tables
in do_merge_schema() at schema_tables.cc.

The data referring to the most recent table schema version is always
present. Other entries are garbage-collected when the corresponding
table schema version is obsoleted (they will be updated with a TTL equal
to `DEFAULT_GC_GRACE_SECONDS` on `ALTER TABLE`).

In case we failed to persist column mapping after a schema change,
missing entries will be recreated on node boot.

Later, the information from this table is used in `paxos_state::learn`
callback in case we have a mismatch between the most recent schema
version and the one that is stored inside the `frozen_mutation`
for the accepted proposal.

Such situation may arise under following circumstances:
 1. The previous LWT operation crashed on the "accept" stage,
    leaving behind a stale accepted proposal, which waits to be
    repaired.
 2. The table affected by LWT operation is being altered, so that
    schema version is now different. Stored proposal now references
    obsolete schema.
 3. LWT query is retried, so that Scylla tries to repair the
    unfinished Paxos round and apply the mutation in the learn stage.

When such mismatch happens, prior to that patch the stored
`frozen_mutation` is able to be applied only if we are lucky enough
and column_mapping in the mutation is "compatible" with the new
table schema.

It wouldn't work if, for example, the columns are reordered, or
some columns, which are referenced by an LWT query, are dropped.

With this patch we try to look up the column mapping for
the obsolete schema version, then upgrade the stored mutation
using obtained column mapping and apply an upgraded mutation instead.

* git@github.com:ManManson/scylla.git feature/table_schema_history_v7:
  lwt: add column_mapping history persistence tests
  schema: add equality operator for `column_mapping` class
  lwt: store column_mapping's for each table schema version upon a DDL change
  schema_tables: extract `fill_column_info` helper
  frozen_mutation: introduce `unfreeze_upgrading` method
2020-10-15 20:48:29 +02:00

136 lines
4.6 KiB
C++

/*
* Copyright (C) 2015 ScyllaDB
*/
/*
* This file is part of Scylla.
*
* Scylla is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* Scylla is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with Scylla. If not, see <http://www.gnu.org/licenses/>.
*/
#pragma once
#include "dht/i_partitioner.hh"
#include "atomic_cell.hh"
#include "database_fwd.hh"
#include "mutation_partition_view.hh"
#include "mutation_fragment.hh"
class mutation;
class flat_mutation_reader;
namespace ser {
class mutation_view;
}
// Immutable, compact form of mutation.
//
// This form is primarily destined to be sent over the network channel.
// Regular mutation can't be deserialized because its complex data structures
// need schema reference at the time object is constructed. We can't lookup
// schema before we deserialize column family ID. Another problem is that even
// if we had the ID somehow, low level RPC layer doesn't know how to lookup
// the schema. Data can be wrapped in frozen_mutation without schema
// information, the schema is only needed to access some of the fields.
//
class frozen_mutation final {
private:
bytes_ostream _bytes;
partition_key _pk;
private:
partition_key deserialize_key() const;
ser::mutation_view mutation_view() const;
public:
explicit frozen_mutation(const mutation& m);
explicit frozen_mutation(bytes_ostream&& b);
frozen_mutation(bytes_ostream&& b, partition_key key);
frozen_mutation(frozen_mutation&& m) = default;
frozen_mutation(const frozen_mutation& m) = default;
frozen_mutation& operator=(frozen_mutation&&) = default;
frozen_mutation& operator=(const frozen_mutation&) = default;
const bytes_ostream& representation() const { return _bytes; }
utils::UUID column_family_id() const;
utils::UUID schema_version() const; // FIXME: Should replace column_family_id()
partition_key_view key() const;
dht::decorated_key decorated_key(const schema& s) const;
mutation_partition_view partition() const;
// The supplied schema must be of the same version as the schema of
// the mutation which was used to create this instance.
// throws schema_mismatch_error otherwise.
mutation unfreeze(schema_ptr s) const;
// Automatically upgrades the stored mutation to the supplied schema with custom column mapping.
mutation unfreeze_upgrading(schema_ptr schema, const column_mapping& cm) const;
struct printer {
const frozen_mutation& self;
schema_ptr schema;
friend std::ostream& operator<<(std::ostream&, const printer&);
};
// Same requirements about the schema as unfreeze().
printer pretty_printer(schema_ptr) const;
};
frozen_mutation freeze(const mutation& m);
struct frozen_mutation_and_schema {
frozen_mutation fm;
schema_ptr s;
};
// Can receive streamed_mutation in reversed order.
class streamed_mutation_freezer {
const schema& _schema;
partition_key _key;
bool _reversed;
tombstone _partition_tombstone;
std::optional<static_row> _sr;
std::deque<clustering_row> _crs;
range_tombstone_list _rts;
public:
streamed_mutation_freezer(const schema& s, const partition_key& key, bool reversed = false)
: _schema(s), _key(key), _reversed(reversed), _rts(s) { }
stop_iteration consume(tombstone pt);
stop_iteration consume(static_row&& sr);
stop_iteration consume(clustering_row&& cr);
stop_iteration consume(range_tombstone&& rt);
frozen_mutation consume_end_of_stream();
};
static constexpr size_t default_frozen_fragment_size = 128 * 1024;
using frozen_mutation_consumer_fn = std::function<future<stop_iteration>(frozen_mutation, bool)>;
future<> fragment_and_freeze(flat_mutation_reader mr, frozen_mutation_consumer_fn c,
size_t fragment_size = default_frozen_fragment_size);
class reader_permit;
class frozen_mutation_fragment {
bytes_ostream _bytes;
public:
explicit frozen_mutation_fragment(bytes_ostream bytes) : _bytes(std::move(bytes)) { }
const bytes_ostream& representation() const { return _bytes; }
mutation_fragment unfreeze(const schema& s, reader_permit permit);
};
frozen_mutation_fragment freeze(const schema& s, const mutation_fragment& mf);