Commit Graph

8243 Commits

Author SHA1 Message Date
Asias He
bc4ac5004e streaming: Kill stream_result_future::create_and_register
The helper is used only once in init_sending_side and in
init_receiving_side we do not use create_and_register to create
stream_result_future. Kill the trivial helper to make the code more
consistent.

In addition, rename variables "future" and "f" to sr (streaming_result).
2016-01-25 11:38:13 +08:00
Asias He
face74a8f2 streaming: Rename stream_result_future::init to ::init_sending_side
So we have:

- init_sending_side
  called when the node initiates a stream_session

- init_receiving_side
  called when the node is a receiver of a stream_session initiated by a peer
2016-01-25 11:38:13 +08:00
Asias He
dc94c5e42e streaming: Rename get_or_create_next_session to get_or_create_session
There is only one session for each peer in stream_coordinator.
2016-01-25 11:38:13 +08:00
Asias He
e46d4166f2 streaming: Refactor host_streaming_data
In scylla, in each stream_coordinator, there will be only one
stream_session for each remote peer. Drop the code supporting multiple
stream_sessions in host_streaming_data.

We now have

   shared_ptr<stream_session> _stream_session

instead of

   std::map<int, shared_ptr<stream_session>> _stream_sessions
2016-01-25 11:38:13 +08:00
Asias He
8a4b563729 streaming: Drop the get_or_create_session_by_id interafce
The session index will always be 0 in stream_coordinator. Drop the api for it.
2016-01-25 11:38:13 +08:00
Asias He
9a346d56b9 streaming: Drop unnecessary parameters in stream_init_message
- from
  We can get it form the rpc::client_info

- session_index
  There will always be one session in stream_coordinator::host_streaming_data with a peer.

- is_for_outgoing
  In cassandra, it initiates two tcp connections, one for incoming stream and one for outgoing stream.
  logger.debug("[Stream #{}] Sending stream init for incoming stream", session.planId());
  logger.debug("[Stream #{}] Sending stream init for outgoing stream", session.planId());
  In scylla, it only initiates one "connection" for sending, the peer initiates another "connection" for receiving.
  So, is_for_outgoing will also be true in scylla, we can drop it.

- keep_ss_table_level
  In scylla, again, we stream mutations instead of sstable file. It is
  not relevant to us.
2016-01-25 11:38:13 +08:00
Asias He
1bc5cd1b22 streaming: Drop streaming/messages/session_failed_message
It is not used.
2016-01-25 11:38:13 +08:00
Asias He
2a04e8d70e streaming: Drop streaming/messages/incoming_file_message
It is not used.
2016-01-25 11:38:13 +08:00
Asias He
26ba21949e streaming: Drop streaming/messages/retry_message
It is not used.
2016-01-25 11:38:13 +08:00
Asias He
4b4363b62d streaming: Drop streaming/messages/received_message
It is not used.
2016-01-25 11:38:13 +08:00
Asias He
b3e00472ed streaming: Drop streaming/streaming.cc
It is used in the early stage of development to make sure things compile.
2016-01-25 11:38:13 +08:00
Asias He
5a0bf10a0b streaming: Drop streaming/messages/complete_message
It is not used.
2016-01-25 11:38:13 +08:00
Asias He
bdd6a69af7 streaming: Drop unused parameters
- int connections_per_host

Scylla does not create connections per stream_session, instead it uses
rpc, thus connections_per_host is not relevant to scylla.

- bool keep_ss_table_level
- int repaired_at

Scylla does not stream sstable files. They are not relevant to scylla.
2016-01-25 11:38:13 +08:00
Takuya ASADA
0f0d1c7aed dist: don't depends on libvirtd, since we are not using it
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1453470970-31036-1-git-send-email-syuu@scylladb.com>
2016-01-24 17:15:13 +02:00
Avi Kivity
65a140481c Merge " streaming COMPLETE_MESSAGE failure and message retry logic fix" from Asias
"This series:

- Add more debug info to stream session
- Fail session if we fail to send COMPLETE_MESSAGE
- Handle message retry logic for verbs used by streaming

See commit log for details."
2016-01-24 16:41:06 +02:00
Avi Kivity
6135e0ae78 Merge "Move read/write mutation path to use IDL" from Gleb 2016-01-24 13:35:04 +02:00
Avi Kivity
b415f87324 Merge "Serializer Deserializer code generation" from Amnon
"The series do the following:
It adds the code generation
Perform the needed changes in the current classes so each would have getter for
each of its serializable value and a constructor from the serialized values.
It adds a schema definition that cover gossip_diget_ack
It changes the messaging_service to use the generated code.

An overall explanation of the solution with a description of the schema IDL can
be found on the wiki page:

https://github.com/scylladb/scylla/wiki/Serializer-Deserializer-Code-generation
"
2016-01-24 12:56:42 +02:00
Gleb Natapov
b9b6f703c3 Remove old serializer for frozen_mutation and reconcilable_result 2016-01-24 12:45:41 +02:00
Gleb Natapov
067bdb23cd Move reconcilable_result and frozen_mutation to idl 2016-01-24 12:45:41 +02:00
Gleb Natapov
18dff5ebc8 Move smart pointer serialization helpers to .cc file.
They are not used outside of the .cc file, so should not be in the
header.
2016-01-24 12:45:41 +02:00
Gleb Natapov
93da9b2725 Remove redundant vector serialization code.
IDL serializer has the code to serialize vectors, so use it instead.
2016-01-24 12:45:41 +02:00
Gleb Natapov
ab6703f9bc Remove old query::result serializer 2016-01-24 12:45:41 +02:00
Gleb Natapov
afc407c6e5 Move query::result to use idl. 2016-01-24 12:45:41 +02:00
Gleb Natapov
be4e68adbf Add bytes_ostream serializer. 2016-01-24 12:45:41 +02:00
Gleb Natapov
043d132ba9 Remove no longer used serializers. 2016-01-24 12:45:41 +02:00
Gleb Natapov
4ae906b204 Add serializer overload for query::partition_range.
From now on query::partition_range will use generated code.
2016-01-24 12:45:41 +02:00
Gleb Natapov
2d1b2765e6 Add serializer overload for query::read_command.
From now on query::read_command will use generated code.
2016-01-24 12:45:41 +02:00
Gleb Natapov
49ce2b83df Add ring_position constructor needed by serializer. 2016-01-24 12:45:41 +02:00
Gleb Natapov
6cc5b15a9c Fix read_command constructor to not copy parameters. 2016-01-24 12:45:41 +02:00
Gleb Natapov
4384c7fe85 un-nest range::bound class.
Serializer does not support nested classes yet, so move bound outside.
2016-01-24 12:45:41 +02:00
Gleb Natapov
7357b1ddfe Move specific_ranges to .hh and un-nest it.
Serializer requires class to be defined, so it has to be in .h file. It
also does not support nested types yet, so move it outside of containing
class.
2016-01-24 12:45:41 +02:00
Gleb Natapov
9ae7dc70da Prepare partition_slice to be used by serializer.
Add missing _specific_ranges getter and setter.
2016-01-24 12:45:41 +02:00
Gleb Natapov
48ab0bd613 Make constructor from bytes for partition_key and clustering_key_prefix public
Make constructor from bytes public since serializer will use it.
2016-01-24 12:45:41 +02:00
Gleb Natapov
8deb5e424c Add idl files for more types.
Add idl for uuid/range/read_command/token/ring_position/clustering_key_prefix/partition_key.
2016-01-24 12:45:41 +02:00
Gleb Natapov
11299aa3db Add serializers for more basic types.
We will need them in following patches.
2016-01-24 12:45:41 +02:00
Gleb Natapov
a643f3d61f Reorder bool and uint8_t serializers
bool serializer uses uint8_t one so should be defined after it.
2016-01-24 12:45:41 +02:00
Gleb Natapov
cba31eb4f8 cleanup gossip_digest.idl
Remove uuid class, nonexistent application states and add ';'.
2016-01-24 12:45:37 +02:00
Avi Kivity
a3efecb8fe Merge seastar upstream
* seastar 5c2660b...97f418a (4):
  > io_queues: register individual classes with collectd
  > reactor: destroy the I/O queues explicitly
  > rpc_impl: add pragma once
  > rpc: add skip to simple_input_stream
2016-01-24 12:35:24 +02:00
Takuya ASADA
b5029dae7e dist: remove abrt from AMI, since it's not able to work with Scylla
New CentOS Base Image contains abrt by default, so remove it.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1453416479-28553-2-git-send-email-syuu@scylladb.com>
2016-01-24 12:31:50 +02:00
Amnon Heiman
0006f236a6 Add an IDL definition file
This adds the IDL definition file.
It is also cover in the wiki page:
https://github.com/scylladb/scylla/wiki/Serializer-Deserializer-Code-generation

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2016-01-24 12:29:21 +02:00
Amnon Heiman
f266c2ed42 README.md: Add dependency for pyparsing python3
python3 needs to install pyparsing excplicitely. This adds the
installation of python3-pyparsing to the require dependencies in the
README.md

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2016-01-24 12:29:21 +02:00
Amnon Heiman
577ce0d231 Adding a sepcific template initialization in messaging_service to use
the serializer

This patch adds a specific template initialization so that the rpc would
use the serializer and deserializer that are auto-generated.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2016-01-24 12:29:21 +02:00
Amnon Heiman
b625363072 Adding the serializer decleration and implementation files.
This patch adds the serializer and serializer_imple files. They holds
the functions that are not auto generated: primitives and templates (map
and vector) It also holds the include to the auto-generated code.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2016-01-24 12:29:20 +02:00
Amnon Heiman
451cf2692c configure.py Add serializer code generation from schema
This patch adds rules and the idl schema to configure, which will call
the code generation to create the serialization and deserialization
functions.

There is also a rule to create the header file that include the auto
generated header files.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2016-01-24 12:29:20 +02:00
Amnon Heiman
0715dcd6ba A schema definition for gossip_digest_ack
This is a definition example for gossip_digest_ack with all its sub
classes.

It can be used by the code generator to create the serializer and
deserializer functions.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2016-01-24 12:29:14 +02:00
Amnon Heiman
d27734b9be Add a constructor to inet_address from uint32_t
inet_address uses uint32_t to store the ip address, but its constructor
is int32_t.
So this patch adds such a constructor.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2016-01-24 12:13:01 +02:00
Amnon Heiman
8a4d211a99 Changes the versioned_value to make serializeble
This patch contains two changes, it make the constructor with parameters
public. And it removes the dependency in messaging_service.hh from the
header file by moving some of the code to the .cc file.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2016-01-24 12:13:01 +02:00
Amnon Heiman
ddc3fe1328 endpoint_state adds a constructor for all serialized parameters
An external deserialize function needs a constructor with all the
serialized parameters.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2016-01-24 12:13:01 +02:00
Amnon Heiman
4a34ed82a3 Add code generation for serializer and deserializer
The code generation takes a schema file and create two files from it,
one with a dist.hh extension containing the forward declarations and a second
with dist.impl.hh with the actual implementation.

Because the rpc uses templating for the input and output streams. The
generated functions are templates.

For each class, struct or enum, two functions are created:
serialize - that gets the output buffer as template parameter and
serialize the object to it. There must be a public way to get to each of
the parameters in the class (either a getter or the parameter should be
public)

deserialize - that gets an input buffer, and return the deserialize
object (and by reference the number of char it read).
To create the return object, the class must have a public constructor
with all of its parameters.

The solution description can be found here:
https://github.com/scylladb/scylla/wiki/Serializer-Deserializer-Code-generation

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2016-01-24 12:12:51 +02:00
Takuya ASADA
aef1e67a9b dist: remove mdadm,xfsprogs from dependencies, install it when constructing RAID by scylla_raid_setup
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1453422886-26297-2-git-send-email-syuu@scylladb.com>
2016-01-24 12:10:41 +02:00