Commit Graph

3698 Commits

Author SHA1 Message Date
Tomasz Grabiec
4548bd1a65 cql_test_env: Work-around infinite recursion in boost::any
cql_query_test segfaults in debug mode due to infinite recursion (see
trace below). The problem started to appear after Avi's always-defer
change.

It looks like the following boost::any() constructor template is
instantiated when boost::any is copied into a lambda:

        // Perfect forwarding of ValueType
        template<typename ValueType>
        any(ValueType&& value, typename boost::disable_if<boost::is_same<any&, ValueType> >::type* = 0)
          : content(new holder< typename remove_reference<ValueType>::type >(static_cast<ValueType&&>(value)))
        {
        }

It checks for any& to disable itself on forwarding, but it doesn't
check for const any&. There are (non-template) constructors which take
const any&, but I guess some C++ rule favors the template
version. This results in infinite recursion of constructor invocations
between any() and holder().

Workaround is to make the lambda mutable, so that the argument type is
any& and template doesn't kick in.

Trace:

CU 0x4debed5, DIE 0x4f176e6>, this=0x602000044550) at /usr/include/boost/any.hpp:175
ql_query_test, CU 0x4debed5, DIE 0x4f6c509>, this=0x602000044518) at /usr/include/boost/any.hpp:72
CU 0x4debed5, DIE 0x4f176e6>, this=0x602000044510) at /usr/include/boost/any.hpp:175
ql_query_test, CU 0x4debed5, DIE 0x4f6c509>, this=0x6020000444e8) at /usr/include/boost/any.hpp:72
CU 0x4debed5, DIE 0x4f176e6>, this=0x6020000444e0) at /usr/include/boost/any.hpp:175
oost::any const>, void>::type*) (this=0x7ffff2ffa6f0, value=<unknown type in /home/tgrabiec/src/urchin2/build
/release/tests/urchin/cql_query_test, CU 0x4debed5, DIE 0x4f6c509>) at /usr/include/boost/any.hpp:72
5u> const&, std::vector<boost::any, std::allocator<boost::any> >, std::vector<boost::any, std::allocator<boos
t::any> >, basic_sstring<char, unsigned int, 15u> const&, boost::any)::{lambda(database&)#1}::operator()(data
base&) const::{lambda(std::unique_ptr<mutation_partition const, std::default_delete<mutation_partition> >)#1}
::unique_ptr(std::unique_ptr<mutation_partition const, std::default_delete<mutation_partition> >&&) (this=0x7
ffff2ffa6c0) at tests/urchin/cql_test_env.cc:126
2015-06-05 14:02:58 +02:00
Pekka Enberg
56a790cdc4 service/storage_proxy: Fix query_local() to respect given key
We want query_local() to actually respect the key we pass to it. Fixes
an issue in keyspace merging code where we returned multiple rows for a
keyspace.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-06-05 13:15:12 +02:00
Tomasz Grabiec
1908a019f2 Merge branch 'penberg/schema-merging-fixes' from seastar-dev.git
From Pekka:

"This series fixes two issues in schema merging code that caused
cql_query_test to sometimes think that two identical keyspaces are
different. This forced the code to use the alter keyspace paths which
are not implemented and just throw an exception."
2015-06-05 12:45:50 +02:00
Tomasz Grabiec
c777886028 Merge branch 'penberg/optimize-adler32-checksum/v3' from seastar-dev.git
From Pekka:

"Checksum calculation is in the sstable writeout fastpath so
micro-optimize it a bit."
2015-06-05 12:12:00 +02:00
Pekka Enberg
ee3dbcd294 query-result-set: Add operator<< for result sets
Add operator<< for result sets to make debugging schema merging code
issues easier.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-06-05 12:59:57 +03:00
Pekka Enberg
647d95d170 types: Fix operator== for data_value
We must compare types with ->equal(). Fixes keyspace merging issues
where the merging code mistakenly thinks two keyspaces are different and
calls the alter keyspace path which is not implemented.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-06-05 12:58:52 +03:00
Pekka Enberg
1cac0c7b31 db/legacy_schema_tables: Fix read_schema_for_keyspaces()
Fix the check in read_schema_for_keyspaces() to not insert empty
result sets in the return value. There's no functional change as the
merge algorithms already deal with the case. However, the code is now
closer to what origin does.

Spotted while reading the code.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-06-05 12:58:46 +03:00
Asias He
1aac08b8ab Revert "storage_service: Remove ad-hoc token_metadata creation"
This reverts commit a19d2171eb.

This commit breaks cql_query_test.

   [asias@hjpc urchin]$ ./cql_query_test
   Running 1 test case...
   WARNING: Not implemented: COMPACT_TABLES
   WARNING: Not implemented: METRICS
   WARNING: Not implemented: PERMISSIONS
   cql_query_test: core/distributed.hh:290: Service&
   distributed<Service>::local() [with Service =
   service::storage_service]: Assertion `local_is_initialized()' failed.
   unknown location(0): fatal error in "test_create_keyspace_statement":
   signal: SIGABRT (application abort requested)
   tests/test-utils.cc(31): last checkpoint

   *** 1 failure detected in test suite "tests/urchin/cql_query_test.cc"
   (gdb) bt
   #0  0x00000032930348d7 in __GI_raise (sig=sig@entry=6) at
   ../sysdeps/unix/sysv/linux/raise.c:55
   #1  0x000000329303653a in __GI_abort () at abort.c:89
   #2  0x000000329302d47d in __assert_fail_base (fmt=0x3293186cb8
   "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
   assertion=assertion@entry=0x8ec10a "local_is_initialized()",
   file=file@entry=0x92508d "core/distributed.hh",
       line=line@entry=290, function=function@entry=0x8ed440
   <distributed<service::storage_service>::local()::__PRETTY_FUNCTION__>
   "Service& distributed<Service>::local() [with Service =
   service::storage_service]")
       at assert.c:92
   #3  0x000000329302d532 in __GI___assert_fail (assertion=0x8ec10a
   "local_is_initialized()", file=0x92508d "core/distributed.hh",
   line=290,
       function=0x8ed440
   <distributed<service::storage_service>::local()::__PRETTY_FUNCTION__>
   "Service& distributed<Service>::local() [with Service =
   service::storage_service]") at assert.c:101
   #4  0x0000000000430f19 in local (this=<optimized out>) at
   core/distributed.hh:290
   #5  get_local_storage_service () at service/storage_service.hh:3326
   #6  keyspace::create_replication_strategy (this=0x7ffff6bf8350) at
   database.cc:690
   #7  0x000000000061537a in
   _ZZZN2db20legacy_schema_tables15merge_keyspacesERN7service13storage_proxyEOSt3mapI13basic_sstringIcjLj15EE13lw_shared_ptrIN5query10result_setEESt4lessIS6_ESaISt4pairIKS6_SA_EEESI_ENKUlRT_E0_clISt6ve
   ctorISF_SG_EEEDaSK_ENKUlR8databaseE_clESQ_ () at
   db/legacy_schema_tables.cc:584
   #8  0x0000000000617d19 in operator() (__closure=0x7ffff6bf8650) at
   ./core/distributed.hh:284

In the test, storage_service and other services are not stared.

Let's revert it and figure out a way to run cql_query_test with the
needed services started properly and then bring the "storage_service:
Remove ad-hoc token_metadata creation" change back.
2015-06-05 08:21:59 +03:00
Pekka Enberg
a9d08438cd sstable: Inline adler32 checksum functions
They're called in the fast-path so inline the functions to avoid an
extra function call.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-06-04 15:48:37 +03:00
Pekka Enberg
31e9381be3 sstables: Defer full checksum calculation
Optimize checksum calculation by deferring "full checksum" update until
we've computed a full per-chunk checksum.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-06-04 15:48:37 +03:00
Avi Kivity
6f821a84d2 Merge seastar upstream 2015-06-04 14:44:07 +03:00
Vlad Zolotarov
db73024f0c core: distributed::local(): check the vector boundaries before trying to access the element
assert() if we are out of bounds or if the element is not initialized.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2015-06-04 14:34:58 +03:00
Avi Kivity
d3cd8ec24a Merge "token bootstrap"
From Asias:

"With this series, simple replication strategy is supposed to work.

Start two nodes, after bootstrap, uuid and token are spread correctly
through gossip:

ep=127.0.0.2, eps=EndpointState: HeartBeatState = generation = 1433298647, version = 156, AppStateMap =
{ 0 : Value(NORMAL,TOKENS,30) }  { 5 : Value(urchin_1_0,4) }  { 8 : Value(,3) }  { 11 : Value(ms_1_0,1) }
{ 12 : Value(6127dd57-fb40-4aca-9046-c3509eca4d1e,2) }  { 13 : Value(acd66a3f2e5bb1af;d0046f49663e2d9b;43dcb3bd8dc7397b,29) }
ep=127.0.0.1, eps=EndpointState: HeartBeatState = generation = 1433298640, version = 161, AppStateMap =
{ 0 : Value(NORMAL,TOKENS,11) }  { 5 : Value(urchin_1_0,4) }  { 8 : Value(,3) }  { 11 : Value(ms_1_0,1) }
{ 12 : Value(a6d2ac36-2f0e-492f-8676-198bbbc42dd1,2) }  { 13 : Value(71e4bd2878ec2446;2170ecc473cd6240;e0f2574988bb909e,9) }

Endpoint -> Token
inet_address=127.0.0.2, token=ac d6 6a 3f 2e 5b b1 af
inet_address=127.0.0.2, token=d0 04 6f 49 66 3e 2d 9b
inet_address=127.0.0.1, token=e0 f2 57 49 88 bb 90 9e
inet_address=127.0.0.1, token=21 70 ec c4 73 cd 62 40
inet_address=127.0.0.2, token=43 dc b3 bd 8d c7 39 7b
inet_address=127.0.0.1, token=71 e4 bd 28 78 ec 24 46

Endpoint -> UUID
inet_address=127.0.0.2, uuid=6127dd57-fb40-4aca-9046-c3509eca4d1e
inet_address=127.0.0.1, uuid=a6d2ac36-2f0e-492f-8676-198bbbc42dd1

Sorted Token
token=ac d6 6a 3f 2e 5b b1 af
token=d0 04 6f 49 66 3e 2d 9b
token=e0 f2 57 49 88 bb 90 9e
token=21 70 ec c4 73 cd 62 40
token=43 dc b3 bd 8d c7 39 7b
token=71 e4 bd 28 78 ec 24 46"
2015-06-04 12:46:08 +03:00
Asias He
c95364fe31 failure_detector: Start on all cpus
Code calls failure_detector::is_alive on all cpus, so we start
failure_detector on all cpus. However, the internal data of failure_detector
is modified on cpu zero and it is not replicated to non-zero cpus.
This is fine since the user of failure_detector (the gossiper) accesses
it on cpu0 only.
2015-06-04 17:25:20 +08:00
Asias He
26cd039005 gossip: Add is_alive helper
failure_detector::is_alive asks gossiper if a node is up or down.
2015-06-04 17:16:58 +08:00
Asias He
77e8f361bb storage_service: Reduce time for non-seed node to join the ring
Waiting for 30 seconds is way too long for testing. Reduce it to 5
seconds.

When we have a proper config system, we can specify in cmdline.
2015-06-04 17:16:50 +08:00
Asias He
a19d2171eb storage_service: Remove ad-hoc token_metadata creation
Use token_metadata from storage_service when creating a
replication_strategy in keyspace::create_replication_strategy.
2015-06-04 17:16:50 +08:00
Asias He
f1ed0cdc7e storage_service: Start on all cpus and replicate _token_metadata
_token_metadata is needed by replication strategy code on all cpus.
Changes to _token_metadata are done on cpu 0. Replicate it to all cpus.

We may copy only if _token_metadata actually changes. As a starter, we
always copy in gossip modification callbacks.
2015-06-04 17:16:50 +08:00
Asias He
8467db11aa token_metadata: Print _sorted_tokens in debug
_sorted_tokens is used by replication code.
2015-06-04 17:12:10 +08:00
Asias He
cae9d65e9d storage_service: Move more code to source file 2015-06-04 17:12:10 +08:00
Asias He
4311662828 storage_service: Implement update_peer_info 2015-06-04 17:12:10 +08:00
Asias He
a85cee6afe storage_service: Rename isSurveyMode to _is_survey_mode 2015-06-04 17:12:10 +08:00
Asias He
db527c1a81 storage_service: Move joinRing to source file 2015-06-04 17:12:10 +08:00
Asias He
4dc4e54e50 storage_service: Add is_joined 2015-06-04 17:12:10 +08:00
Asias He
ca2e151c03 storage_service: Rename initialized to _initialized 2015-06-04 17:12:10 +08:00
Asias He
e5c653939b storage_service: Add is_bootstrap_mode and finish_bootstrapping 2015-06-04 17:12:10 +08:00
Asias He
c87f950aff storage_service: Implement handle_state_normal
Start two nodes, after bootstrap, uuid and token are spread correctly
through gossip:

----------- endpoint_state_map:  -----------
ep=127.0.0.1, eps=EndpointState: HeartBeatState = generation =
1433172216, version = 66, AppStateMap =  { 0 : Value(NORMAL,TOKENS,11) }
{ 5 : Value(urchin_1_0,4) }  { 8 : Value(,3) }  { 11 : Value(ms_1_0,1) }
{ 12 : Value(06eb49d2-a092-483a-a89a-f774cff2c3e5,2) }  { 13 :
Value(0b20137e213f697b;c39a029ad9dd2948;0003be0eeb569d5a,9) }

ep=127.0.0.2, eps=EndpointState: HeartBeatState = generation =
1433172229, version = 56, AppStateMap =  { 0 : Value(NORMAL,TOKENS,51) }
{ 5 : Value(urchin_1_0,4) }  { 8 : Value(,3) }  { 11 : Value(ms_1_0,1) }
{ 12 : Value(adc8eb9f-7c1f-4695-905c-c1c4fdeea4d8,2) }  { 13 :
Value(6f5607a9b4cbadf0;eb7d976656cafad1;a225d312b9f42e5b,50) }

----------- token_metadata:  -----------
Endpoint -> Token
inet_address=127.0.0.2, token=a2 25 d3 12 b9 f4 2e 5b
inet_address=127.0.0.1, token=c3 9a 02 9a d9 dd 29 48
inet_address=127.0.0.2, token=eb 7d 97 66 56 ca fa d1
inet_address=127.0.0.1, token=00 03 be 0e eb 56 9d 5a
inet_address=127.0.0.1, token=0b 20 13 7e 21 3f 69 7b
inet_address=127.0.0.2, token=6f 56 07 a9 b4 cb ad f0
Endpoint -> UUID
inet_address=127.0.0.1, uuid=06eb49d2-a092-483a-a89a-f774cff2c3e5
inet_address=127.0.0.2, uuid=adc8eb9f-7c1f-4695-905c-c1c4fdeea4d8
2015-06-04 17:12:10 +08:00
Asias He
9649414011 dht: Align token print
Before:
token=df 96 79 87 21 b2 ed 80
token=5c 98 e a0 4f 5e 28 6b

After:
token=df 96 79 87 21 b2 ed 80
token=5c 98 0e a0 4f 5e 28 6b
2015-06-04 17:12:10 +08:00
Asias He
abad1520ad gossip: Fix get_host_id
Return a real UUID.
2015-06-04 17:12:10 +08:00
Asias He
8a578a1364 utils: Add UUID(const sstring& uuid_string) constructor
Construct a UUID from a UUID string.
2015-06-04 17:12:10 +08:00
Asias He
1ed5d01cd2 storage_service: Fix STATUS in set_tokens
Here, we should set STATUS to NORMAL.
2015-06-04 17:12:10 +08:00
Asias He
68f671a8b7 storage_service: Move gossip callback to source file 2015-06-04 17:12:09 +08:00
Asias He
6917a904c3 storage_service: Implement handle_state_bootstrap 2015-06-04 17:12:09 +08:00
Asias He
0c98a4413f token_metadata: Add _leaving_endpoints and _moving_endpoints 2015-06-04 17:12:09 +08:00
Asias He
34b3d679ab dht: Do move in token constructor 2015-06-04 17:12:09 +08:00
Asias He
9dc7a60b4a storage_service: Move handle_state_bootstrap and friends to source file 2015-06-04 17:12:09 +08:00
Asias He
42a2f24c77 token_metadata: Add add_bootstrap_tokens and remove_bootstrap_tokens 2015-06-04 17:12:09 +08:00
Asias He
9c5cd2bca8 storage_service: Switch to use unordered_set for tokens
We do not care about the order of the tokens.

Also, in token_metadata, we use unordered_set for tokens as well, e.g.
update_normal_tokens. Unify the usage.
2015-06-04 17:12:09 +08:00
Asias He
6b263e46eb token_metadata: Add is_member 2015-06-04 17:12:09 +08:00
Asias He
06a792d6be token_metadata: Add get_host_id and friends 2015-06-04 17:12:09 +08:00
Asias He
edee90550c database: Fix boost::find compile error
boost::find confuses compiling when both <boost/algorithm/string/find.hpp> and
<boost/range/algorithm/find.hpp> are included.
2015-06-04 17:12:09 +08:00
Avi Kivity
7fa17d9880 Merge "range query read path"
Conflicts:
	database.cc
2015-06-04 10:21:48 +03:00
Avi Kivity
8d0bd44b08 do_with: doxygenize 2015-06-03 20:44:48 +03:00
Avi Kivity
5333c54958 Networking module documentation 2015-06-03 20:27:36 +03:00
Pekka Enberg
fcd6f147fc db/legacy_schema_tables.cc: Use schema_result::value_type instead of std::pair
Switch to schema_result::value_type instead of the open-coded std::pair
so that the actual types are defined in one place.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-06-03 19:14:39 +02:00
Avi Kivity
32a7e3b21f Merge "failure detector REST API"
From Amnon:

"This series adds the failure detector API. The API definition is based on the
FailureDetectorMBean API."
2015-06-03 19:23:07 +03:00
Amnon Heiman
ba8365d95a Adding the Failure detector API implementation
This series adds the implementation for teh Failure detector API.
After this patch the following APIs will be supported:
/failure_detector/endpoints
/failure_detector/count/endpoint/up
/failure_detector/count/endpoint/down
/failure_detector/phi
POST:/failure_detector/phi
/failure_detector/simple_states
/failure_detector/endpoints/states

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-06-03 19:13:03 +03:00
Amnon Heiman
a75376e8e3 API: Add a helper function from map to key value list
When using swagger definition file, returning a map, needs to be in a
key, value list. To handle this common case in the API, a helper
function was added that gets an unorder_map and return a vector of key,
value mapping.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-06-03 19:13:03 +03:00
Amnon Heiman
711fe64208 Expose the failure_detector functionality
The failure detector runs on CPU 0, for external usage, this is an
implementation detail which is unrelevant.

This adds a wrapper functions for the functions that are defined in
FailureDetectorMBean which would map the request to the correct CPU.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-06-03 19:13:03 +03:00
Amnon Heiman
71bfd07d69 API Adding the failure detector swagger definition
This adds the failure detector definition that is based on the
FailureDetectorMBean

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-06-03 19:12:57 +03:00