mirrors/scylladb
1
0
Fork 0
mirror of https://github.com/scylladb/scylladb.git synced 2026-06-03 13:37:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
8,282 Commits 257 Branches 558 Tags
91b57c7e20fdff297fa50e6550efe0f8f72b6b43
Commit Graph

8282 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Avi Kivity
91b57c7e20 Merge "Move streaming to use IDL" from Asias 2016-01-25 17:10:22 +02:00
Asias He
f027a9babe streaming: Drop unused serialization code 2016-01-25 22:39:13 +08:00
Asias He
ad80916905 messaging_service: Add streaming implementation for idl 
- stream_request
- stream_summary
- prepare_message

 Please enter the commit message for your changes. Lines starting
 2016-01-25 22:36:58 +08:00
Asias He
b299cc3bee idl: Add streaming.idl.hh 
- stream_request
- stream_summary
- prepare_message
 2016-01-25 22:29:25 +08:00
Asias He
5e100b3426 streaming: Drop unused repaired_at in stream_request 2016-01-25 22:28:48 +08:00
Avi Kivity
6fade0501b Update test/message.cc for MESSAGE verb rename 2016-01-25 14:47:55 +02:00
Nadav Har'El
db19a43d98 repair: try harder to repair, even when some nodes are unreachable 
In the existing code, when we fail to reach one of the replicas of some
range being repaired, we would give up, and not continue to repair the
living replicas of this range. The thinking behind this was since the
repair should be considered failed anyway, there's no point in trying
to do a half-job better.

However, in a discussion I had with Shlomi, he raised the following
alternative thinking, which convinced me: In a large cluster, having
one node or another temporarily dead has a high probability. In that
case, even if the if the repair is doomed to be considered "failed",
we want it at least to do as much as it possibly can to repair the
data on the living part of the cluster. This is what this patch does:
If we can only reach some of the replicas of a given range, the repair
will be considered failed (as before), but we will still repair the
reachable replicas of this range, if they have different checksums.

Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <1453724443-29320-1-git-send-email-nyh@scylladb.com>
 2016-01-25 14:37:39 +02:00
Amnon Heiman
039e627b32 idl-compiler: Fix an issue with default values 
This patch fix an issue when a parameter with a version attribute had a
default value.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1453723251-9797-1-git-send-email-amnon@scylladb.com>
 2016-01-25 14:32:00 +02:00
Takuya ASADA
e9fdb426b6 dist: add pyparsing on CentOS build time dependency CentOS 
Porting pyparsing from Fedora23, build it for python34 which provided by epel.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1453720780-21105-1-git-send-email-syuu@scylladb.com>
 2016-01-25 13:26:58 +02:00
Takuya ASADA
b8b0ff0482 dist: add pyparsing on Ubuntu build time dependency 
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1453720081-15979-1-git-send-email-syuu@scylladb.com>
 2016-01-25 13:08:48 +02:00
Avi Kivity
5c5207f122 Merge "Another round of streaming cleanup" from Asias 
"- Merge stream_init_message and stream_parepare_message
- Drop  session_index / keep_ss_table_level / file_message_header"
 2016-01-25 12:54:30 +02:00
Asias He
77684a5d4c messaging_service: Drop STREAM_INIT_MESSAGE 
The verb is not used anymore.
Message-Id: <1453719054-29584-1-git-send-email-asias@scylladb.com>
 2016-01-25 12:53:08 +02:00
Asias He
53c6cd7808 gossip: Rename echo verb to gossip_echo 
It is used by gossip only. I really could not allow myself to get along
this inconsistence. Change before we still can.
Message-Id: <1453719054-29584-2-git-send-email-asias@scylladb.com>
 2016-01-25 12:53:07 +02:00
Takuya ASADA
67d2aa677e dist: add pyparsing on Fedora build time dependency 
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1453715594-32675-1-git-send-email-syuu@scylladb.com>
 2016-01-25 11:59:32 +02:00
Takuya ASADA
78d107ccaa dist: add missing dependencies for scylla-gdb 
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1453715339-32296-1-git-send-email-syuu@scylladb.com>
 2016-01-25 11:59:31 +02:00
Asias He
51fa717b8e streaming: Get rid of file_message_header 
Again, we do not send sstable files, thus neither header info for
sstables files.

TODO: Estimate mutation size we sent.
 2016-01-25 17:56:43 +08:00
Asias He
eba9820b22 streaming: Remove stream_session::file_sent 
It is the callback after sending file_message_header. In scylla, we do
not sent the file_message_header. Drop it.
 2016-01-25 17:25:34 +08:00
Asias He
592683650a streaming: Remove unused serialization code for file_message_header 2016-01-25 17:16:57 +08:00
Asias He
fa4e94aa27 streaming: Get rid of keep_ss_table_level 
We stream mutation instead of files, so keep_ss_table_level is not
relevant for us.
 2016-01-25 16:58:57 +08:00
Asias He
2cc31ac977 streaming: Get rid of the stream_index 
It is always zero.
 2016-01-25 16:58:57 +08:00
Asias He
6b30f08a38 streaming: Always return zero for session_index in api/stream_manager 
We will remove session_index soon. It will always be zero. Do not drop
it in api so that the api will be compatible with c*.
 2016-01-25 16:58:51 +08:00
Asias He
ad4a096b80 streaming: Get rid of stream_init_message 
Unlike streaming in c*, scylla does not need to open tcp connections in
streaming service for both incoming and outgoing messages, seastar::rpc
does the work. There is no need for a standalone stream_init_message
message in the streaming negotiation stage, we can merge the
stream_init_message into stream_prepare_message.
 2016-01-25 16:24:16 +08:00
Asias He
048965ea02 streaming: Do not print session_index in handle_session_prepared 
session_index is always 0. It will be removed soon.
 2016-01-25 16:24:16 +08:00
Avi Kivity
449b81f5d3 Merge "streaming cleanup" from Asias 
"No mercy to the unused parameters and messages.
This will help the upcoming IDL serialize/deserialize work."
 2016-01-25 10:21:16 +02:00
Avi Kivity
9ebd3f8098 Merge "Move gossip to use IDL" from Asias 
"This changes gossip to use IDL based serialization code."
 2016-01-25 10:18:34 +02:00
Asias He
20496ed9a8 tests: Stop gossip during shutdown in cql_test_env 
Fixes the heap-use-after-free error in build/debug/tests/auth_test

==1415==ERROR: AddressSanitizer: heap-use-after-free on address
0x62200032cfa8 at pc 0x00000350701d bp 0x7fec96df8d40 sp
0x7fec96df8d30
READ of size 8 at 0x62200032cfa8 thread T1
    #0 0x350701c in
_ZZN3gms8gossiper3runEvENKUlOT_E0_clI6futureIJEEEEDaS2_
(/home/penberg/scylla/build/debug/tests/auth_test_g+0x350701c)
    #1 0x35795b1 in apply<gms::gossiper::run()::<lambda(auto:40&&)>,
future<> > /home/penberg/scylla/seastar/core/future.hh:1203
    #2 0x369103d in
_ZZN6futureIJEE12then_wrappedIZN3gms8gossiper3runEvEUlOT_E0_S0_EET0_S5_ENUlS5_E_clI12future_stateIJEEEEDaS5_
(/home/penberg/scylla/build/debug/tests/auth_test_g+0x369103d)
    #3 0x369182a in run /home/penberg/scylla/seastar/core/future.hh:399
    #4 0x435f24 in
reactor::run_tasks(circular_buffer<std::unique_ptr<task,
std::default_delete<task> >, std::allocator<std::unique_ptr<task,
std::default_delete<task> > > >&) core/reactor.cc:1368
    #5 0x43a44f in reactor::run() core/reactor.cc:1672
    #6 0x952e4b in app_template::run_deprecated(int, char**,
std::function<void ()>&&) core/app-template.cc:123
    #7 0x58dc79d in test_runner::start(int,
char**)::{lambda()#1}::operator()()
(/home/penberg/scylla/build/debug/tests/auth_test_g+0x58dc79d)
    #8 0x58e6cd6 in _M_invoke /usr/include/c++/5.3.1/functional:1871
    #9 0x688639 in std::function<void ()>::operator()() const
/usr/include/c++/5.3.1/functional:2271
    #10 0x8d939c in posix_thread::start_routine(void*) core/posix.cc:51
    #11 0x7feca02a4609 in start_thread (/lib64/libpthread.so.0+0x7609)
    #12 0x7fec9ffdea4c in clone (/lib64/libc.so.6+0x102a4c)

0x62200032cfa8 is located 5800 bytes inside of 5808-byte region
[0x62200032b900,0x62200032cfb0)
freed by thread T1 here:
    #0 0x7feca4f76472 in operator delete(void*, unsigned long)
(/lib64/libasan.so.2+0x9a472)
    #1 0x3740772 in gms::gossiper::~gossiper()
(/home/penberg/scylla/build/debug/tests/auth_test_g+0x3740772)
    #2 0x2588ba1 in shared_ptr<gms::gossiper>::~shared_ptr()
seastar/core/shared_ptr.hh:389
    #3 0x4fc908c in
seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned
int)#1}::operator()(unsigned
int)::{lambda()#1}::operator()()::{lambda()#1}::~stop()
(/home/penberg/scylla/build/debug/tests/auth_test_g+0x4fc908c)
    #4 0x4ff722a in future<>
future<>::then<seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned
int)#1}::operator()(unsigned
int)::{lambda()#1}::operator()()::{lambda()#1}, future<>
>(seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned
int)#1}::operator()(unsigned
int)::{lambda()#1}::operator()()::{lambda()#1}&&)::{lambda(seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned
int)#1}::operator()(unsigned
int)::{lambda()#1}::operator()()::{lambda()#1})#1}::~then()
(/home/penberg/scylla/build/debug/tests/auth_test_g+0x4ff722a)
    #5 0x509a28c in continuation<future<>
future<>::then<seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned
int)#1}::operator()(unsigned
int)::{lambda()#1}::operator()()::{lambda()#1}, future<>
>(seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned
int)#1}::operator()(unsigned
int)::{lambda()#1}::operator()()::{lambda()#1}&&)::{lambda(seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned
int)#1}::operator()(unsigned
int)::{lambda()#1}::operator()()::{lambda()#1})#1}>::~continuation()
seastar/core/future.hh:395
    #6 0x509a40d in continuation<future<>
Message-Id: <f8f1c92c1eb88687ab0534f5e7874d53050a5b93.1453446350.git.asias@scylladb.com>
 2016-01-25 08:19:18 +02:00
Asias He
bc4ac5004e streaming: Kill stream_result_future::create_and_register 
The helper is used only once in init_sending_side and in
init_receiving_side we do not use create_and_register to create
stream_result_future. Kill the trivial helper to make the code more
consistent.

In addition, rename variables "future" and "f" to sr (streaming_result).
 2016-01-25 11:38:13 +08:00
Asias He
face74a8f2 streaming: Rename stream_result_future::init to ::init_sending_side 
So we have:

- init_sending_side
  called when the node initiates a stream_session

- init_receiving_side
  called when the node is a receiver of a stream_session initiated by a peer
 2016-01-25 11:38:13 +08:00
Asias He
dc94c5e42e streaming: Rename get_or_create_next_session to get_or_create_session 
There is only one session for each peer in stream_coordinator.
 2016-01-25 11:38:13 +08:00
Asias He
e46d4166f2 streaming: Refactor host_streaming_data 
In scylla, in each stream_coordinator, there will be only one
stream_session for each remote peer. Drop the code supporting multiple
stream_sessions in host_streaming_data.

We now have

   shared_ptr<stream_session> _stream_session

instead of

   std::map<int, shared_ptr<stream_session>> _stream_sessions
 2016-01-25 11:38:13 +08:00
Asias He
8a4b563729 streaming: Drop the get_or_create_session_by_id interafce 
The session index will always be 0 in stream_coordinator. Drop the api for it.
 2016-01-25 11:38:13 +08:00
Asias He
9a346d56b9 streaming: Drop unnecessary parameters in stream_init_message 
- from
  We can get it form the rpc::client_info

- session_index
  There will always be one session in stream_coordinator::host_streaming_data with a peer.

- is_for_outgoing
  In cassandra, it initiates two tcp connections, one for incoming stream and one for outgoing stream.
  logger.debug("[Stream #{}] Sending stream init for incoming stream", session.planId());
  logger.debug("[Stream #{}] Sending stream init for outgoing stream", session.planId());
  In scylla, it only initiates one "connection" for sending, the peer initiates another "connection" for receiving.
  So, is_for_outgoing will also be true in scylla, we can drop it.

- keep_ss_table_level
  In scylla, again, we stream mutations instead of sstable file. It is
  not relevant to us.
 2016-01-25 11:38:13 +08:00
Asias He
1bc5cd1b22 streaming: Drop streaming/messages/session_failed_message 
It is not used.
 2016-01-25 11:38:13 +08:00
Asias He
2a04e8d70e streaming: Drop streaming/messages/incoming_file_message 
It is not used.
 2016-01-25 11:38:13 +08:00
Asias He
26ba21949e streaming: Drop streaming/messages/retry_message 
It is not used.
 2016-01-25 11:38:13 +08:00
Asias He
4b4363b62d streaming: Drop streaming/messages/received_message 
It is not used.
 2016-01-25 11:38:13 +08:00
Asias He
b3e00472ed streaming: Drop streaming/streaming.cc 
It is used in the early stage of development to make sure things compile.
 2016-01-25 11:38:13 +08:00
Asias He
5a0bf10a0b streaming: Drop streaming/messages/complete_message 
It is not used.
 2016-01-25 11:38:13 +08:00
Asias He
bdd6a69af7 streaming: Drop unused parameters 
- int connections_per_host

Scylla does not create connections per stream_session, instead it uses
rpc, thus connections_per_host is not relevant to scylla.

- bool keep_ss_table_level
- int repaired_at

Scylla does not stream sstable files. They are not relevant to scylla.
 2016-01-25 11:38:13 +08:00
Asias He
7b633ad127 gossip: Drop unused serialization code 
- heart_beat_state
 2016-01-25 11:28:29 +08:00
Asias He
4ce08ff251 messaging_service: Add heart_beat_state implementation 2016-01-25 11:28:29 +08:00
Asias He
d7c7994f37 gossip: Drop unused serialization code 
- versioned_value
 2016-01-25 11:28:29 +08:00
Asias He
8098ba10b7 gossip: Drop unused serialization code 
- endpoint_state
 2016-01-25 11:28:29 +08:00
Asias He
ecca969adf messaging_service: Add gossip::endpoint_state implementation 2016-01-25 11:28:29 +08:00
Asias He
2a0b6589dd messaging_service: Add versioned_value implementation 2016-01-25 11:28:29 +08:00
Asias He
6660658742 gossip: Drop unused serialization code 
- gossip_digest_serialization_helper
- gossip_digest
 2016-01-25 11:28:29 +08:00
Asias He
15f2b353b9 messaging_service: Add gossip_digest implementation 2016-01-25 11:28:29 +08:00
Asias He
736d21a912 gossip: Drop unused serialization code 
- gossip_digest_syn
- gossip_digest_ack
- gossip_digest_ack2
 2016-01-25 11:28:29 +08:00
Asias He
d81fc12af3 messaging_service: Add gossip_digest_ack2 implementation 2016-01-25 11:28:29 +08:00
Asias He
e67cecaee1 messaging_service: Add gossip_digest_syn implementation 2016-01-25 11:28:29 +08:00