storage_proxy is never deinitialized, so it may have still used cdc_service
after its destructor was called.
This fixes the problem by cdc_service inheriting from
async_sharded_service and storage_proxy calling shared_from_this on
the service whenever it uses it.
cdc_service inherits from async_sharded_service and not simply from
enable_shared_from_this, because there might be other services that
cdc_service depends on. Assuming that these services are
deinitialized after cdc_service (as they should), i.e. after stop() is
called on cdc_service, making cdc_service async_sharded_service will
keep their deinitialization code from being called until all references
to cdc_service disappear (async_sharded_service keeps stop() from
returning until this happens).
Some more improvements should be possible through some refactoring:
1. Make augment_mutation_call a free function, not a member of
cdc_service: it doesn't need any state that cdc_service has.
db_context can be passed down from storage_proxy when it calls the
function.
2. Remove the storage_proxy -> cdc_service reference. storage_proxy
only needs augment_mutation_call, which would not be a part of the
service. This would also get rid of the proxy -> cdc -> proxy
reference cycle that we have now, and would allow storage_proxy to be
safely deinitialized after cdc_service.
3. Maybe we could even remove the cdc_service -> storage_proxy
reference. Is it really needed?
CL of LOCAL_QUORUM used to be hardcoded into CDC preimage query
and led to an error when number of replicas was lower than CL
would require. The solution here is to link the CLs of writes
to base table with the CLs of CDC reads, so the client will get
the (limited) control over the consistency of preimage SELECTs
(instead of getting error every time).
The algorithm is as follows:
1. If write that caused CDC activity was done with CL = ANY,
then do preimage read with CL = ONE.
2. If write that caused CDC activity was done with CL = ALL,
then do preimage read with CL = QUORUM.
3. SERIAL and LOCAL_SERIAL writes cause preimage read with QUORUM
and LOCAL_QUORUM, respectively.
4. In other cases do preimage read with the same CL as base write.
The `result_callback` was a callback returned by `augment_mutation_call`
that was supposed to be used in the CDC postimage implementation.
Because CDC postimage was implemented without using this callback, and
currently a no-op function is always returned, this callback can safely
be removed.
Merged pull request https://github.com/scylladb/scylla/pull/5996 from
Calle Wilund:
Fixes#4992
Implements post-image support by synthesizing it from
pre-image + delta.
Post-image data differs from the delta data in two ways:
1.) It merges non-atomics into an actual result value
2.) It contains all columns of the row, not just
those affected by the update.
For a non-atomic field, the post-image value of a column
is either the pre-image or the delta (maybe null)
Tested by adding post-image checks to pre-image test
and collection/udt tests
Fixes#4992
Implements post-image support by synthesizing it from
pre-image + delta.
Post-image data differs from the delta data in two ways:
1.) It merges non-atomics into an actual result value
2.) It contains _all_ columns of the row, not just
those affected by the update.
For a non-atomic field, the post-image value of a column
is either the pre-image or the delta (maybe null)
Tested by adding post-image checks to pre-image test
and collection/udt tests
If a batch update is performed with a sequence of changes with a single
timestamp, they will now show up in CDC with a single timeuuid in the
`time` column, distinguished by different `batch_seq_no` values.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
According to "new" spec:
Data column is now pure frozen original type.
If column is deleted (set to null), a metadata column
cdc$deleted_<name> is set to true, to distinguish
null column == not involved in row operation
For non-atomic collections, a cdc$deleted_elements_<name>
column is added, and when removing elements from collection
this is where they are shown.
For non-atomic assign, the "cdc$deleted_<name>" is true,
and <name> is set to new value.
column_op removed.
When incoming mutation contains live row marker the `operation` is
described as "insert", not as an "update".
Also, I extended the test case "test_row_delete" with one insert,
which is expected to log different value of `operation` than update
or delete. Renamed the test case accordingly.
Test cases that relied on "update" being the same as "insert" are
updated accordingly (`test_pre_image_logging`, `test_cdc_across_shards`,
`test_add_columns`).
Fixes#5723
This patch causes inclusive and exclusive range deletes to be
distinguished in cdc log. Previously, operations `range_delete_start`
and `range_delete_end` were used for both inclusive and exclusive bounds
in range deletes. Now, old operations were renamed to
`range_delete_*_inclusive`, and for exclusive deletes, new operations
`range_delete_*_exclusive` are used.
Tests: unit(dev)
To increase modularity, making it easier to find what is where and
maintain.
The 'log' module (cdc/log.{hh,cc}) is responsible for updating CDC log
tables when base table writes are performed.
The 'generation' module (cdc/generation.{hh,cc}) handles stream
generation changes in response to topology change events.
cdc/metadata.{hh,cc} contains a helper class which holds the currently
used generation of streams. It is used by both aforementioned modules:
'log' queries it, while 'generation' updates it.
