"This series changes buffering of mutation fragments in streamed mutations
so that the size of the fragments is taken into account.
The original implementation buffered up to 16 fragments which was pretty
much meaningless since it could be far too much if the fargments were
large or not nearly enough in case they were small
Fixes #2036.."
* 'pdziepak/buffer-mfs-by-size/v1' of github.com:cloudius-systems/seastar-dev:
streamed_mutation: size-based mutation_fragment buffer limit
mutation_fragment: cache size in memory
mutation_fragment: make write access more explicit
Currently, streamed mutations buffer up to 16 mutation fragments. This
may be too much, not enough or a perfect choice depending on the
mutation fragment size.
This patch makes streamed mutation choose how much mutation fragments to
keep in the buffer depending on their size, so that we avoid using too
much memory in case of large mutation fragments and are able to buffer a
lot of fragments if they are small.
mutation_fragments are going to be caching their size in memory. In
order to be able to invalidate that correctly, they need to know when
that size may change (but avoid invalidation when it is not necessary).
"This series uses the newly added histogram and label support to add metrics to
the storage_proxy and to the column_family.
This would add latency and histogram and the missing metrics from column family."
* 'amnon/histogram_metrics' of github.com:cloudius-systems/seastar-dev:
database: add metrics registration for the coloumn family
storage_proxy: add read and write latency histogram
estimated_histogram: returns a metrics histogram
"This series makes sure that schemas containing both counter and non-counter
regular or static columns are not allowed."
* 'pdziepak/disallow-mixed-schemas/v1' of github.com:cloudius-systems/seastar-dev:
schema: verify that there are no both counter and non-counter columns
test/mutation_source: specify whether to generate counter mutations
tests/canonical_mutation: don't try to upgrade incompatible schemas
Tests using random mutation generator should be provided with bot
counter and non-counter mutations to ensure that both cases are
sufficiently covered. However, mixed schemas (with both counter and
non-counter columns) are not allowed so the RMG has to be explicitly
told whether to use counter or non-counter schema.
Test case test_reading_with_different_schemas uses randomly generated
pairs of mutations and tries to upgrade one to the schema of the other.
However, there are cases when one schema cannot be upgraded to another,
for example, counter and non-counter schemas.
This patch adds a metrics registration to the column_family.
Using label each column metrics is label with its keyspace and column
family name.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
The metrics histogram is a struct that describe a histogram.
This patch adds a getter method that lets the estimated_histogram return
a metrics::histogram, this will allow to register it as a histogram
metrics.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
"This patchset adds bits of the MV write-path, enough to support
new entries to be added. Note that this is still limited, as only
adding new rows to a base table will work correctly."
* 'materialized-views/insert-path/v4' of https://github.com/duarten/scylla: (30 commits)
database: Apply mutation to views
column_family: Push view replica update
materialized views: partial mutate_MV
materialized views: function to send a mutation to endpoint
materialized views: add VIEW write type
database: Ensure new write_type is correctly printed
materialized views: match base and view replicas
column_family: Generate view updates
column_family: Adds affected_views() function
view: Add view_update_builder class
range_tombstone_accumulator: Expose current tombstone
range_tombstone_accumulator: apply() takes value
view_updates: Generate updates
view_updates: Adds function to replace row
view_updates: Update view entry
view_updates: Delete old view entry
mutation_partition: Introduce shadowable tombstone
view_updates: Create view entry
view_updates: Compute row marker
view: Introduce view_updates class
...
This patch changes the database apply path so that it also
generates the mutations for the column family's views and
sends them to the paired view replicas.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This adds a function mutate_MV() which takes view mutations and sends
them to the appropriate nodes (this may be the current node, or a
remote node).
This is only a partial implementation - we still don't do the local
batch log (to survive reboots and failures) and some other stuff which
is left commented out.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Add a function for sending one mutation to one remote replica owning
this mutation. This is needed for materialized views, where each
base replica sends each view mutation to one particular view replica.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
This adds to the "write_type" enum also the "VIEW" write type.
To be honest, I don't understand why the "write_type" distinction
is important.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
By removing the default case in the switch statement over a write_type
variable, we ensure the compiler warns us about lack of exhaustiveness
in case we add a value to the enum but forget to change the
corresponding operator<<().
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
A function to find the appropriate replica to send a view update to.
This patch creates a new source file db/view/view.cc. We should
eventually move a lot more of the materialized views code there.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
This patch adds the generate_view_updates() function to the
column_family class, which will use the view_update_builder to
generate updates to the column_family's materialized views.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch adds the view_update_builder class, which is responsible
for calculating the mutations to apply to a column family's
materialized views, given a streamed_mutation representing an update
to the base table and a streamed_mutation representing the
pre-existing rows which the update covers.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
range_tombstone_accumulator::apply() now takes a value so the caller
can decide whether to move or copy the argument.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch adds the view_updates::generate_update() function to
generate view updates given a base row update and the corresponding,
pre-existing row. This function will decide which of the previously
introduced functions to call based on whether there is a pre-existing
row and whether there exists a regular base column that's part of the
view's PK.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch adds a function to replace a view row given a base
table update and the pre-existing row, which simply deletes the old
view entry and adds a new one.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch introduces the view_updates::update_entry function,
which creates the updates to apply to the existing view entry given
the base table row before and after the update.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch introduces the view_updates::delete_old_entry function,
which creates a view row mutation to delete an entry given an updated
base table row.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch introduces shadowable row tombstones. A shadowable row
tombstone is valid only if the row has no live marker. In other words,
the row tombstone is only valid as long as no newer insert is done
(thus setting a live row marker; note that if the row timestamp set
is lower than the tombstone's, then the tombstone remains in effect
as usual).
If a row has a shadowable tombstone with timestamp Ti and that row
is updated with a timestamp Tj, such that Tj > Ti (and that update
sets the row marker), then the shadowable tombstone is shadowed by
that update. A concrete consequence is that if the update has cells
with timestamp lower than Ti, then those cells are preserved (since
the deletion is removed), and this is contrary to a regular,
non-shadowable row tombstone where the tombstone is preserved and
such cells are removed.
Currently, only Materialized Views require shadowable row tombstones,
which solve a problem with view row deletions. Consider a base row with
columns p, v1, v2, PRIMARY KEY (p) denormalized into a view row consisting
of columns p, v1, v2 PRIMARY KEY (p, v1), and the following operations:
1) INSERT INTO base (p, v1, v2) VALUES (0, 0, 1) USING TIMESTAMP 0;
2) UPDATE base SET v1 = 1 USING TIMESTAMP 1 WHERE p = 0;
3) UPDATE base SET v1 = 0 USING TIMESTAMP 2 WHERE p = 0;
Without shadowable tombstones, the view contains:
At 1), pk = (0, 0), row_marker@T0, v2=1@T0
At 2), pk = (0, 0), row_marker@T0, row_tombstone@T1, v2=1@T0
pk = (0, 1), row_marker@T1, v2=1@T0
At 3), pk = (0, 0), row_marker@T2, row_tombstone@T1, v2=1@T0
pk = (0, 1), row_marker@T1, row_tombstone@T2, v2=1@T0
Notice how, if we read row (0, 0), the value of v2 will be shadowed by
the row tombstone we previously inserted.
With a view's row tombstone becoming shadowable, at 3) the row (0, 0)
will look like pk = (0, 0), row_marker@T2, shadowable_tombstone@T1, v2=1@T0,
which is equivalent to pk = (0, 0), row_marker@T2, v2=1@T0.
Since the shadowable tombstone is shadowed by the new row marker (T0 <
T2), now v2 would be taken into account.
Finally, note that this patch doesn't generalize the idea of
shadowable tombstone, instead taking advantage of the fact that they
are only needed by Materialized Views. This saves changing the
tombstone representation to account for an extra flag, the bits such
representation would require, and also avoids changes to the storage
format.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch introduces the view_updates::create_entry function, which
creates a view row mutation given a new base table row.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch adds a function to compute the row marker of a view row
given the base row. There are two cases to consider when building the
row marker: 1) there is a column C that is a regular base column but
is in the view PK; and 2) the columns for the base and the view PKs
are the same.
For 1), the view row marker timestamp will be the biggest between the
base's row marker and C. The TTL will be that of C. This means that if
C expires, the view row maker will expire as well (and the row, if no
other column is keeping it alive). Note that if the base row marker
expires but not C, then the base row will still be live due to C and
we shouldn't expire the view row.
For 2), the view row timestamp will be the same as the base row
timestamp. The TTL should be set in such a way that both base and view
rows live for the same time. We thus set the view row TTL to be the
max of any other TTL in the base row. This is particularly important
in the case where the base row marker has a TTL, but a column *absent*
from the view holds a greater one.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch introduces the view_updates class, which is responsible
for generating and storing updates to a particular materialized view.
The updates will be generated from an updated base row and the
pre-existing one (if any), in later patches.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch introduces the collection_type_impl::for_each_cell()
function, which allows the caller to iterate over the cells of a
particular collection_mutation_view.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
To help calculate the view mutations from a base update, we store in
the view class the column that's part of the view's primary key but
not part of the base's, if such column exists.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch adds the matches_view_filter() function which specifies
whether a given base row matches the view filter. Unlike
may_be_affected_by(), this function has no false positives.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch adds the is_satisfied_by() function to
single_column_restriction, which given a clustering row returns
whether the restrictions applies or not.
This is useful for secondary indexing such as materialized views,
where filters on regular columns precisely select which base table
rows to denormalize.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch exposes the non-primary key column restrictions in a given
select statement, exposing them as single_column_restrictions.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch adds the is_set() and is_list() functions to
collection_type_impl, which identify the concrete collection
type.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch renames abstract_restriction::uses_function() to
term_uses_function(), as it was previously hiding a function with the
same name in the restriction base class.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch adds the may_be_affected_by() function to the view class,
which is responsible to determine whether an update to a base class
affects one of its views.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Instead of storing the view in the column_family's map of materialized
views, store a lw_shared_ptr so that the view can be removed while it
is being updated.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
This patch fixes a use-after-free error in
rename_column_in_where_clause(), where we were creating a boost
adaptor on an rvalue.
Signed-off-by: Duarte Nunes <duarte@scylladb.com>
"Fixes #1531
Adds lookup to gms::inet_address and uses it in (hopefully all) the
salient places where configured symbolic names are interpreted.
Removes the dummy dns modula in scylla in favour of the seastar one."
* 'calle/use-dns' of github.com:cloudius-systems/seastar-dev:
remove scylla dns code
service::storage_service: Remove depedency on scylla dns
main.cc: remove scylla dns dependency
main/init: Lookup inet addresses from config by dns lookup
db::system_keyspace: Find rpc_address by lookup
gms::inet_address: Add lookup functionality.
scylla tls: Add option support for client auth and tls opts
