mirrors/scylladb
1
0
Fork 0
mirror of https://github.com/scylladb/scylladb.git synced 2026-05-12 19:02:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
5,120 Commits 245 Branches 546 Tags
4e72b2a6b1697267cdad4e7252daf69595a82f5e
Commit Graph

214 Commits

Author SHA1 Message Date
Asias He
a216f258a2 streaming: Serialize query::range<token> in stream_request 2015-07-23 09:08:15 +08:00
Asias He
54d482afe4 streaming: Test both pushing and pulling of data 
stream_plan.transfer_ranges() sends data from local to remote node.

stream_plan.request_ranges() asks remote to send data to local.

After streaming, both nodes contains all the keys.

$ cat /tmp/out1|grep "\[Stream"
[Stream #9fd8c3c0-3023-11e5-b450-000000000000] Executing streaming plan for MYPLAN
[Stream #9fd8c3c0-3023-11e5-b450-000000000000] Starting streaming to 127.0.0.2
[Stream #9fd8c3c0-3023-11e5-b450-000000000000] Sending stream init for incoming stream
[Stream #9fd8c3c0-3023-11e5-b450-000000000000 ID#0] Beginning stream session with 127.0.0.2
[Stream #9fd8c3c0-3023-11e5-b450-000000000000 ID#0] Prepare completed.  Receiving 1 files(105553124400080 bytes), sending 1 files(105553124104160 bytes)
[Stream #9fd8c3c0-3023-11e5-b450-000000000000] Session with 127.0.0.2 is complete
[Stream #9fd8c3c0-3023-11e5-b450-000000000000] All sessions completed

$ cat /tmp/out2|grep "\[Stream"
[Stream #9fd8c3c0-3023-11e5-b450-000000000000 ID#0] Creating new streaming plan for MYPLAN
[Stream #9fd8c3c0-3023-11e5-b450-000000000000 ID#0] Received streaming plan for MYPLAN
[Stream #9fd8c3c0-3023-11e5-b450-000000000000 ID#0] Prepare completed.  Receiving 1 files(105553124104160 bytes), sending 1 files(105553124400080 bytes)
[Stream #9fd8c3c0-3023-11e5-b450-000000000000] Session with 127.0.0.1 is complete
[Stream #9fd8c3c0-3023-11e5-b450-000000000000] All sessions completed

Node 1
$ sstable2json tmp/1/ks/*/la-1-big-Data.db | grep key | sort
{"key": "1",
{"key": "2",
{"key": "3",
{"key": "4",
{"key": "5",
{"key": "6",

Node 2
$ sstable2json tmp/2/ks/*/la-1-big-Data.db | grep key | sort
{"key": "1",
{"key": "2",
{"key": "3",
{"key": "4",
{"key": "5",
{"key": "6",
 2015-07-22 11:49:30 +08:00
Asias He
96049a99cf streaming: Add more debug print for stream_session::prepare 2015-07-22 11:49:30 +08:00
Asias He
1c60844727 streaming: Always start_streaming_files upon receiving of PREPARE_MESSAGE reply 2015-07-22 11:49:30 +08:00
Asias He
736c0bc08f streaming: Improve query::range<token> deserialization a bit 
query::range<token> is not serialized ATM. Always use full range for
now.
 2015-07-22 11:49:30 +08:00
Asias He
d39dd0f9d1 streaming: Add operator<< for stream_request 2015-07-22 11:49:30 +08:00
Asias He
d934b2c761 streaming: Fix prepare_message and stream_request deserialization 
vector(size_type count) constructs the container with count
default-inserted instances of T. So, current code will end up with 2*num
elements which is wrong.
 2015-07-22 11:49:30 +08:00
Asias He
f83d0bf7c4 streaming: Fix info printout format 2015-07-22 11:49:30 +08:00
Asias He
19f46fdbe1 streaming: Remove redundant debug log info 2015-07-21 16:12:54 +08:00
Asias He
93f17024a8 streaming: Fix a logger printout 
Before:

[Stream #136c5310-2f7a-11e5-87df-000000000000 ID#0]
[Stream #136c5310-2f7a-11e5-87df-000000000000, ID#0]

After:

[Stream #136c5310-2f7a-11e5-87df-000000000000 ID#0]
[Stream #136c5310-2f7a-11e5-87df-000000000000 ID#0]
 2015-07-21 16:12:54 +08:00
Asias He
0e5fa35bd2 streaming: Register to stream_manager in create_and_register 2015-07-21 16:12:54 +08:00
Asias He
fc718dc87d streaming: Set up dst_cpu_id in PREPARE_MESSAGE hanlder 2015-07-21 16:12:54 +08:00
Asias He
ad86f5ea6e streaming: Fix get_or_create_host_data 
It is always creating a new host_streaming_data which is incorrect.
 2015-07-21 16:12:54 +08:00
Asias He
a010829f0c streaming: Add src_cpu_id parameter for PREPARE_MESSAGE verb 
We need it to setup dst_cpu_id for the session of the follower.
 2015-07-21 16:12:54 +08:00
Asias He
6712e9404e streaming: Implement session completion logic 2015-07-21 16:12:54 +08:00
Asias He
f9109c33ba streaming: Implement stream_transfer_task completion logic 2015-07-21 16:12:54 +08:00
Asias He
f2960a7cb0 streaming: Send plan_id for STREAM_MUTATION 
We need this to find session associated with this frozen_mutation.
 2015-07-21 16:12:54 +08:00
Asias He
ccb32ceec5 streaming: Add stream_transfer_task::complete 2015-07-21 16:12:54 +08:00
Asias He
34e33a9afe streaming: Wire up handle_session_complete and friends 2015-07-21 16:12:54 +08:00
Asias He
1f5feee5f2 streaming: Implement stream_result_future::handle_progress 2015-07-21 16:12:54 +08:00
Asias He
d74e414d09 streaming: Implement stream_result_future::handle_session_complete 2015-07-21 16:12:54 +08:00
Asias He
5e6f84cba8 streaming: Implement stream_result_future::handle_session_prepared 
Instead of playing the game of casting between stream_event and derived
class. We overload handle_stream_event with derived stream_event class.

virtual void handle_stream_event(session_complete_event event) {}
virtual void handle_stream_event(progress_event event) {}
virtual void handle_stream_event(session_prepared_event event) {}

Also, make the virtual function non pure virtual, so user can override
the interested event only without defining all of the three.
 2015-07-21 16:12:54 +08:00
Asias He
5ff5abbe2a streaming: handle_stream_event should be public 2015-07-21 16:12:54 +08:00
Asias He
d574e6e3c8 streaming: Rename getTotalFilesToSend to get_total_files_to_send 2015-07-21 16:12:54 +08:00
Asias He
f20b281e1c streaming: Update comments in stream_session::prepare 
To make things a bit clear. Note: In Origin, prepare() can be called by
both initiator and follower.
 2015-07-21 16:12:54 +08:00
Asias He
9b8d542b35 streaming: Improve handler of prepare_message from follower 
Upon receiving of the prepare_message message from the follower,
msg.requests can only be empty. Don't bother look at the requests.
 2015-07-21 16:12:54 +08:00
Asias He
d9b7cb4142 streaming: Enable logger in stream_result_future 2015-07-21 16:12:54 +08:00
Asias He
2e8b34dd9c streaming: Enable logger in stream_coordinator 
Move connect_all_stream_sessions to source first.
 2015-07-21 16:12:54 +08:00
Asias He
2dac3306a0 streaming: Enable more loggers in stream_session 2015-07-21 16:12:54 +08:00
Asias He
8561315cf2 streaming: de-thread_local-ize logger 2015-07-21 16:12:54 +08:00
Asias He
521df14240 streaming: Rename sstable_details to stream_details 2015-07-21 16:12:54 +08:00
Asias He
b53b8dfec8 streaming: Kill stream_reader 
It is used by IncomingFileMessage's deserialize to read from network and
write int sstable. In urchin, we use messaging service, incoming
mutation is handler within the STREAM_MUTATION handler. No need for
stream_reader.
 2015-07-21 16:12:54 +08:00
Asias He
8acf335f15 streaming: Kill stream_writer 
In Origin, it is used by OutgoingFileMessage's serialize function to
write given section of the SSTable to network. In urchin, we send
mutaion directly in stream_transfer_task::start(). We can kill
stream_write now.
 2015-07-21 16:12:54 +08:00
Asias He
7e365cb440 streaming: Handle cf_id does not exist case in STREAM_MUTATION handler 2015-07-21 16:12:54 +08:00
Asias He
69ce554a0f streaming: Init session when follower receives a PREPARE_MESSAGE 2015-07-21 16:12:54 +08:00
Asias He
9794fa1f97 streaming: Improve the test 
Instead of streaming system.local table, create and stream user created
table.
 2015-07-21 16:12:54 +08:00
Asias He
fa2aee57ac utils: Move util/serialization.hh to utils/serialization.hh 
Now we will not have the ugly utils and util directories, only utils.
 2015-07-21 16:12:54 +08:00
Asias He
292ddd68c2 streaming: Delay adding wrappers for some stream verbs 
It might be changed or be dropped since the upcoming change of streaming
code. It is not used at the moment.
 2015-07-16 17:22:53 +08:00
Asias He
857fa5ccbb messaging_service: Add wrapper for STREAM_MUTATION verb 2015-07-16 17:19:51 +08:00
Asias He
1f6a76f2a0 messaging_service: Add wrapper for PREPARE_MESSAGE verb 2015-07-16 17:19:51 +08:00
Glauber Costa
9c464aff9b database: clean up various APIs 
In much of our column_families APIs, we need to pass a pointer to the database.
The only reason we do that, is so we can properly handle the commit log entries
after we seal the current memtables into sstables.

Now that we store a pointer to the commit log in the CF itself at the time it
is created, we no longer have to do it. As a result, the APIs are a lot
cleaner, with no gratuitous parameters.

My motivation for this was the flush method, but as a result, apply() also gets
cleaner.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
 2015-07-15 10:24:20 -04:00
Asias He
6191fcb62e messaging_service: Add wrapper for STREAM_INIT_MESSAGE verb 
This patch serves as an example of how we can add wrappers for
ms.send_message and ms.register_handler.

When we convert all the users of them, we can make messaging_service.hh
do not include rpc.hh.
 2015-07-15 14:17:04 +03:00
Asias He
0313be256c streaming: Handle non-exist cf_name in get_column_family_stores 2015-07-14 20:56:37 +08:00
Asias He
d720dadf7b streaming: Switch to use logger class 2015-07-14 20:56:28 +08:00
Asias He
e82bdf2995 streaming: Swith to use shared_ptr from std::shared_ptr 
Since our shared_ptr works with incomplete types now, switch to it.
 2015-07-14 20:41:14 +08:00
Asias He
8fd8f39d63 streaming: Add more debug info for message exchange 2015-07-14 20:41:14 +08:00
Asias He
ca7f5ca5c9 streaming: Set proper dst_cpu_id in shard_id for PREPARE_MESSAGE and STREAM_MUTATION 2015-07-14 20:41:14 +08:00
Asias He
845de56614 streaming: Fix handler of PREPARE_MESSAGE 
With this patch, the receiver of PREPARE_MESSAGE can create a session
correctly now.
 2015-07-14 20:41:14 +08:00
Asias He
0282d4d6d0 streaming: Start stream_manager on startup 2015-07-14 20:41:14 +08:00
Asias He
8734213890 streaming: Lookup session 2015-07-14 20:41:14 +08:00