Commit Graph

18855 Commits

Author SHA1 Message Date
Asias He
e8c13444ba repair: Increase max row buf size
If the cluster supports row level repair with rpc stream interface, we
can use bigger row buf size to have better repair bandwidth in high
latency links.
2019-07-03 08:01:37 +08:00
Asias He
7d08a8d223 repair: Hook rpc stream version of verbs in row level repair
If rpc stream is supported, use the rpc stream version of the
get_row_diff, put_row_diff, get_full_row_hashes.
2019-07-03 08:01:37 +08:00
Asias He
fccaa0324f repair: Add use_rpc_stream to repair_meta
Determine if rpc stream should be used.
2019-07-03 08:01:37 +08:00
Asias He
7bf0c646be repair: Add is_rpc_stream_supported
Given a row_level_diff_detect_algorithm, return if this algo supports
rpc stream interface.
2019-07-03 08:01:04 +08:00
Asias He
1c92643f02 repair: Add needs_all_rows flag to put_row_diff
So we can avoid copy _working_row_buf in get_row_diff on master node if
there is only one follower node and all repair rows are needed by
follower node.
2019-07-03 07:56:22 +08:00
Asias He
6595417567 repair: Optimize get_row_diff
Move _working_row_buf instead of copy if it is follower node or
it is master node with only one follow. In these cases, the
_working_row_buf will not be used after this function, so we can move
it.
2019-07-03 07:56:22 +08:00
Asias He
c4eb0ee361 repair: Register repair_get_full_row_hashes_with_rpc_strea
Register the get_full_row_hashes rpc stream verb.
2019-07-03 07:56:22 +08:00
Asias He
b56cced5b8 repair: Register repair_put_row_diff_with_rpc_stream
Register the put_row_diff rpc stream verb.
2019-07-03 07:56:22 +08:00
Asias He
67130031b1 repair: Register repair_get_row_diff_with_rpc_stream
Register the get_row_diff rpc stream verb.
2019-07-03 07:56:22 +08:00
Asias He
f255f902bd repair: Add repair_get_full_row_hashes_with_rpc_stream_handler
It is the handler for the get_full_row_hashes rpc stream verb on the
receiving side.
2019-07-03 07:56:17 +08:00
Asias He
e3267ad98c repair: Add repair_put_row_diff_with_rpc_stream_handler
It is the handler for the put_row_diff rpc stream verb on the
receiving side.
2019-07-03 07:55:24 +08:00
Asias He
06ac014261 repair: Add repair_get_row_diff_with_rpc_stream_handler
It is the handler for the get_row_diff rpc stream verb on the receiving
side.
2019-07-03 07:54:43 +08:00
Asias He
5f25969da3 repair: Add repair_get_full_row_hashes_with_rpc_stream_process_op
It is the helper for the get_full_row_hashes rpc stream verb handler.
2019-07-03 07:54:03 +08:00
Asias He
39d5a9446e repair: Add repair_put_row_diff_with_rpc_stream_process_op
It is the helper for the put_row_diff rpc stream verb handler.
2019-07-03 07:53:21 +08:00
Asias He
049e793fe5 repair: Add repair_get_row_diff_with_rpc_stream_process_op
It is the helper for the get_row_diff rpc stream verb handler.
2019-07-03 07:52:12 +08:00
Asias He
b1188f299e repair: Add put_row_diff_with_rpc_stream
It is rpc stream version of put_row_diff. It uses rpc stream instead of
rpc verb to put the repair rows to follower nodes.
2019-07-02 21:22:41 +08:00
Asias He
31b30486a7 repair: Add put_row_diff_sink_op
It is a helper that works on the sink() of the put_row_diff
rpc stream verb.
2019-07-02 21:22:41 +08:00
Asias He
dbe035649b repair: Add put_row_diff_source_op
It is a helper that works on the source() of the put_row_diff rpc
stream verb.
2019-07-02 21:22:41 +08:00
Asias He
72d3563da1 repair: Add get_row_diff_with_rpc_stream
It is rpc stream version of get_row_diff. It uses rpc stream instead of
rpc verb to get the repair rows from follower nodes.
2019-07-02 21:22:41 +08:00
Asias He
4cb44baa08 repair: Add get_row_diff_sink_op
It is a helper that works on the sink() of the get_row_diff rpc
stream verb.
2019-07-02 21:22:41 +08:00
Asias He
a1e19514f9 repair: Add get_row_diff_source_op
It is a helper that works on the source() of the
get_row_diff rpc stream verb.
2019-07-02 21:22:41 +08:00
Asias He
473bd7599c repair: Add get_full_row_hashes_with_rpc_stream
It is rpc stream version of get_full_row_hashes. It uses rpc stream
instead of rpc verb to get the repair hashes data from follower nodes.
2019-07-02 21:22:41 +08:00
Asias He
1e2a598fe7 repair: Add get_full_row_hashes_sink_op
It is a helper that works on the sink() of the get_full_row_hashes
rpc stream verb.
2019-07-02 21:22:41 +08:00
Asias He
149c54b000 repair: Add get_full_row_hashes_source_op
It is a helper that works on the source() of the
get_full_row_hashes rpc stream verb.
2019-07-02 21:22:41 +08:00
Asias He
b3e7299032 repair: Add sink and source object into repair_meta
They will soon be used to sync repair hashes and repair rows bewteen
master and follower nodes.
2019-07-02 21:22:41 +08:00
Asias He
acd40fd529 repair: Add sink_source_for_put_row_diff
Use sink_source_for_repair to define sink_source_for_put_row_diff
with sink = repair_row_on_wire_with_cmd, source = repair_stream_cmd
for REPAIR_PUT_ROW_DIFF_WITH_RPC_STREAM rpc stream verb.
2019-07-02 21:22:41 +08:00
Asias He
4405f7a6ff repair: Add sink_source_for_get_row_diff
Use sink_source_for_repair to define sink_source_for_get_row_diff with
sink = repair_hash_with_cmd, source = repair_row_on_wire_with_cmd for
REPAIR_GET_ROW_DIFF_WITH_RPC_STREAM rpc stream verb.
2019-07-02 21:22:41 +08:00
Asias He
0bffd07e7e repair: Add sink_source_for_get_full_row_hashes
Use the sink_source_for_repair to define
sink_source_for_get_full_row_hashes with sink = repair_stream_cmd,
source = repair_hash_with_cmd for
REPAIR_GET_FULL_ROW_HASHES_WITH_RPC_STREAM rpc stream verb.
2019-07-02 21:22:41 +08:00
Asias He
8400dafa12 repair: Add sink_source_for_repair helper class
It is used to store the sink and source objects for the rpc stream verbs
used by row level repair.
2019-07-02 21:22:41 +08:00
Asias He
37b3de4ea0 messaging_service: Add REPAIR_GET_FULL_ROW_HASHES_WITH_RPC_STREAM support
It is used by row level repair.
2019-07-02 21:18:55 +08:00
Asias He
a7c7ba9765 messaging_service: Add REPAIR_PUT_ROW_DIFF_WITH_RPC_STREAM support
It is used by row level repair.
2019-07-02 21:18:55 +08:00
Asias He
dc92bda93b messaging_service: Add REPAIR_GET_ROW_DIFF_WITH_RPC_STREAM support 2019-07-02 21:18:55 +08:00
Asias He
f312c95b74 messaging_service: Add do_make_sink_source helper
It is used by the row level repair rpc stream verbs to make sink and
source object.
2019-07-02 21:18:55 +08:00
Asias He
bc295a00a6 messaging_service: Add rpc stream verb for row level repair
- REPAIR_GET_ROW_DIFF_WITH_RPC_STREAM

Get repair rows from follower nodes

- REPAIR_PUT_ROW_DIFF_WITH_RPC_STREAM

Put repair rows to follower nodes

- REPAIR_GET_FULL_ROW_HASHES_WITH_RPC_STREAM:

Get full hashes from follower nodes
2019-07-02 21:18:55 +08:00
Asias He
c93113f3a5 idl: Add repair_row_on_wire_with_cmd 2019-07-02 21:18:54 +08:00
Asias He
a90fb24efc idl: Add repair_hash_with_cmd 2019-07-02 21:18:37 +08:00
Asias He
599d40fbe9 idl: Add repair_stream_cmd 2019-07-02 21:18:15 +08:00
Asias He
672c24f6b0 idl: Add send_full_set_rpc_stream for row_level_diff_detect_algorithm 2019-07-02 21:17:36 +08:00
Asias He
fb3f0125ee repair: Add default construct for partition_key_and_mutation_fragments
This is useful when we want to add an empty
partition_key_and_mutation_fragments.
2019-06-26 09:12:55 +08:00
Asias He
3fc53a6b72 repair: Add send_full_set_rpc_stream in row_level_diff_detect_algorithm
It is used to negotiate if the master can use the rpc stream interface
to transfer data.
2019-06-26 09:12:55 +08:00
Asias He
6054a56333 repair: Add repair_row_on_wire_with_cmd
It is used to contain both a repair cmd and repair_row_on_wire object.
2019-06-26 09:12:55 +08:00
Asias He
9f36d775dc repair: Add repair_hash_with_cmd
It is a wrapper contains both a repair cmd and repair_hash object.
2019-06-26 09:12:55 +08:00
Asias He
6b59279e26 repair: Add repair_stream_cmd
It is used by row level repair to add small protocol on top of the rpc stream
interface.
2019-06-26 09:12:55 +08:00
Rafael Ávila de Espíndola
94d2194c77 dht: token: Simplify operator<
While this is a strict weak ordering, it is not obvious and duplicates
a bit of logic. This ptach simplifies it by using tri_compare.

Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20190621204820.37874-1-espindola@scylladb.com>
2019-06-25 19:06:30 +03:00
Tomasz Grabiec
269e65a8db Merge "Sync schema before repair" from Asias
This series makes sure new schema is propagated to repair master and
follower nodes before repair.

Fixes #4575

* dev.git asias/repair_pull_schema_v2:
  migration_manager: Add sync_schema
  repair: Sync schema from follower nodes before repair
2019-06-25 19:05:29 +03:00
Amos Kong
f0cd589a75 dist: suppress the yaml load warning
YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated,
as the default Loader is unsafe. Please read https://msg.pyyaml.org/load
for full details.

Fix it by use new safe interface - yaml.safe_load()

Signed-off-by: Amos Kong <amos@scylladb.com>
Cc: Vlad Zolotarov <vladz@scylladb.com>
Message-Id: <9b68601845117274573474ede0341cc81f80efa6.1561156205.git.amos@scylladb.com>
2019-06-25 19:05:29 +03:00
Avi Kivity
fc629bb14f Merge "cql3: lift infinite bound check" from Benny & Piotr
"
If the database supports infinite bound range deletions,
CQL layer will no longer throw an error indicating that both ranges
need to be specified.

Fixes #432

Update test_range_deletion_scenarios unit test accordingly.
"

* 'cql3-lift-infinite-bound-check' of https://github.com/bhalevy/scylla:
  cql3: lift infinite bound check if it's supported
  service: enable infinite bound range deletions with mc
  database: add flag for infinite bound range deletions
2019-06-25 19:05:29 +03:00
Nadav Har'El
a88c9ca5a5 Merge branch 'add_proper_aggregation_for_paged_indexing_2' of git://github.com/psarna/scylla into next
Piotr Sarna says:

Fixes #4540
This series adds proper handling of aggregation for paged indexed queries.
Before this series returned results were presented to the user in per-page
partial manner, while they should have been returned as a single aggregated
value.

Tests: unit(dev)

Piotr Sarna (8):
  cql3: split execute_base_query implementation
  cql3: enable explicit copying of query_options
  cql3: add a query options constructor with explicit page size
  cql3: add proper aggregation to paged indexing
  cql3: make DEFAULT_COUNT_PAGE_SIZE constant public
  tests: add query_options to cquery_nofail
  tests: add indexing + paging + aggregation test case
  tests: add indexing+paging test case for clustering keys
2019-06-25 19:05:29 +03:00
Avi Kivity
7195f75fb2 Update seastar submodule
* seastar ded50bd8a4...b629d5ef7a (9):
  > sharded: no_sharded_instance_exception: fix grammar
  > core,net: output_stream: remove redundant std::move()
  > perftune: make sure that ethtool -K has a chance of succeeding
  > net/dpdk: upgrade to dpdk-19.05
  > perftune.py: Fix a few more places where we use deprecated pyudev.Device ones
  > reactor: provide an uptime function
  > rpc: add sink::flush() to streaming api
  > Use a table to document the various build modes
  > foreign_ptr: Fix compilation error due to unused variable
2019-06-25 19:05:29 +03:00
Avi Kivity
9d21341733 review-checklist.md: add common checks
- code style
 - naming
 - micro-performance
 - concurrency
 - unit-testing
 - templates and type erasure
 - singletons
2019-06-25 19:05:29 +03:00