Commit Graph

117 Commits

Author SHA1 Message Date
Gleb Natapov
fc6f6634fa support query of multiple singular ranges 2015-06-11 15:18:07 +03:00
Gleb Natapov
b7155ad862 pass partitions_ranges separately from from read_command
partitions_ranges will be manipulated upon to be split for different
destination, so provide it separately from read_command to not copy the
later for each destination.
2015-06-11 15:18:07 +03:00
Vlad Zolotarov
73278798a9 added missing methods (stubs) required for snitch implementation
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>

New in v2:
   - storage_service: add a non-const version of get_token_metadata().
   - get_broadcast_address(): check if net::get_messaging_service().local_is_initialized()
     before calling net::get_local_messaging_service().listen_address().
   - get_broadcast_address(): return an inet_address by value.
   - system_keyspace: introduce db::system_keyspace::endpoint_dc_rack
   - fb_utilities: use listen_address as broadcast_address for now
2015-06-09 15:33:29 +03:00
Pekka Enberg
56a790cdc4 service/storage_proxy: Fix query_local() to respect given key
We want query_local() to actually respect the key we pass to it. Fixes
an issue in keyspace merging code where we returned multiple rows for a
keyspace.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-06-05 13:15:12 +02:00
Asias He
1aac08b8ab Revert "storage_service: Remove ad-hoc token_metadata creation"
This reverts commit a19d2171eb.

This commit breaks cql_query_test.

   [asias@hjpc urchin]$ ./cql_query_test
   Running 1 test case...
   WARNING: Not implemented: COMPACT_TABLES
   WARNING: Not implemented: METRICS
   WARNING: Not implemented: PERMISSIONS
   cql_query_test: core/distributed.hh:290: Service&
   distributed<Service>::local() [with Service =
   service::storage_service]: Assertion `local_is_initialized()' failed.
   unknown location(0): fatal error in "test_create_keyspace_statement":
   signal: SIGABRT (application abort requested)
   tests/test-utils.cc(31): last checkpoint

   *** 1 failure detected in test suite "tests/urchin/cql_query_test.cc"
   (gdb) bt
   #0  0x00000032930348d7 in __GI_raise (sig=sig@entry=6) at
   ../sysdeps/unix/sysv/linux/raise.c:55
   #1  0x000000329303653a in __GI_abort () at abort.c:89
   #2  0x000000329302d47d in __assert_fail_base (fmt=0x3293186cb8
   "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
   assertion=assertion@entry=0x8ec10a "local_is_initialized()",
   file=file@entry=0x92508d "core/distributed.hh",
       line=line@entry=290, function=function@entry=0x8ed440
   <distributed<service::storage_service>::local()::__PRETTY_FUNCTION__>
   "Service& distributed<Service>::local() [with Service =
   service::storage_service]")
       at assert.c:92
   #3  0x000000329302d532 in __GI___assert_fail (assertion=0x8ec10a
   "local_is_initialized()", file=0x92508d "core/distributed.hh",
   line=290,
       function=0x8ed440
   <distributed<service::storage_service>::local()::__PRETTY_FUNCTION__>
   "Service& distributed<Service>::local() [with Service =
   service::storage_service]") at assert.c:101
   #4  0x0000000000430f19 in local (this=<optimized out>) at
   core/distributed.hh:290
   #5  get_local_storage_service () at service/storage_service.hh:3326
   #6  keyspace::create_replication_strategy (this=0x7ffff6bf8350) at
   database.cc:690
   #7  0x000000000061537a in
   _ZZZN2db20legacy_schema_tables15merge_keyspacesERN7service13storage_proxyEOSt3mapI13basic_sstringIcjLj15EE13lw_shared_ptrIN5query10result_setEESt4lessIS6_ESaISt4pairIKS6_SA_EEESI_ENKUlRT_E0_clISt6ve
   ctorISF_SG_EEEDaSK_ENKUlR8databaseE_clESQ_ () at
   db/legacy_schema_tables.cc:584
   #8  0x0000000000617d19 in operator() (__closure=0x7ffff6bf8650) at
   ./core/distributed.hh:284

In the test, storage_service and other services are not stared.

Let's revert it and figure out a way to run cql_query_test with the
needed services started properly and then bring the "storage_service:
Remove ad-hoc token_metadata creation" change back.
2015-06-05 08:21:59 +03:00
Asias He
77e8f361bb storage_service: Reduce time for non-seed node to join the ring
Waiting for 30 seconds is way too long for testing. Reduce it to 5
seconds.

When we have a proper config system, we can specify in cmdline.
2015-06-04 17:16:50 +08:00
Asias He
a19d2171eb storage_service: Remove ad-hoc token_metadata creation
Use token_metadata from storage_service when creating a
replication_strategy in keyspace::create_replication_strategy.
2015-06-04 17:16:50 +08:00
Asias He
f1ed0cdc7e storage_service: Start on all cpus and replicate _token_metadata
_token_metadata is needed by replication strategy code on all cpus.
Changes to _token_metadata are done on cpu 0. Replicate it to all cpus.

We may copy only if _token_metadata actually changes. As a starter, we
always copy in gossip modification callbacks.
2015-06-04 17:16:50 +08:00
Asias He
cae9d65e9d storage_service: Move more code to source file 2015-06-04 17:12:10 +08:00
Asias He
4311662828 storage_service: Implement update_peer_info 2015-06-04 17:12:10 +08:00
Asias He
a85cee6afe storage_service: Rename isSurveyMode to _is_survey_mode 2015-06-04 17:12:10 +08:00
Asias He
db527c1a81 storage_service: Move joinRing to source file 2015-06-04 17:12:10 +08:00
Asias He
4dc4e54e50 storage_service: Add is_joined 2015-06-04 17:12:10 +08:00
Asias He
ca2e151c03 storage_service: Rename initialized to _initialized 2015-06-04 17:12:10 +08:00
Asias He
e5c653939b storage_service: Add is_bootstrap_mode and finish_bootstrapping 2015-06-04 17:12:10 +08:00
Asias He
c87f950aff storage_service: Implement handle_state_normal
Start two nodes, after bootstrap, uuid and token are spread correctly
through gossip:

----------- endpoint_state_map:  -----------
ep=127.0.0.1, eps=EndpointState: HeartBeatState = generation =
1433172216, version = 66, AppStateMap =  { 0 : Value(NORMAL,TOKENS,11) }
{ 5 : Value(urchin_1_0,4) }  { 8 : Value(,3) }  { 11 : Value(ms_1_0,1) }
{ 12 : Value(06eb49d2-a092-483a-a89a-f774cff2c3e5,2) }  { 13 :
Value(0b20137e213f697b;c39a029ad9dd2948;0003be0eeb569d5a,9) }

ep=127.0.0.2, eps=EndpointState: HeartBeatState = generation =
1433172229, version = 56, AppStateMap =  { 0 : Value(NORMAL,TOKENS,51) }
{ 5 : Value(urchin_1_0,4) }  { 8 : Value(,3) }  { 11 : Value(ms_1_0,1) }
{ 12 : Value(adc8eb9f-7c1f-4695-905c-c1c4fdeea4d8,2) }  { 13 :
Value(6f5607a9b4cbadf0;eb7d976656cafad1;a225d312b9f42e5b,50) }

----------- token_metadata:  -----------
Endpoint -> Token
inet_address=127.0.0.2, token=a2 25 d3 12 b9 f4 2e 5b
inet_address=127.0.0.1, token=c3 9a 02 9a d9 dd 29 48
inet_address=127.0.0.2, token=eb 7d 97 66 56 ca fa d1
inet_address=127.0.0.1, token=00 03 be 0e eb 56 9d 5a
inet_address=127.0.0.1, token=0b 20 13 7e 21 3f 69 7b
inet_address=127.0.0.2, token=6f 56 07 a9 b4 cb ad f0
Endpoint -> UUID
inet_address=127.0.0.1, uuid=06eb49d2-a092-483a-a89a-f774cff2c3e5
inet_address=127.0.0.2, uuid=adc8eb9f-7c1f-4695-905c-c1c4fdeea4d8
2015-06-04 17:12:10 +08:00
Asias He
1ed5d01cd2 storage_service: Fix STATUS in set_tokens
Here, we should set STATUS to NORMAL.
2015-06-04 17:12:10 +08:00
Asias He
68f671a8b7 storage_service: Move gossip callback to source file 2015-06-04 17:12:09 +08:00
Asias He
6917a904c3 storage_service: Implement handle_state_bootstrap 2015-06-04 17:12:09 +08:00
Asias He
9dc7a60b4a storage_service: Move handle_state_bootstrap and friends to source file 2015-06-04 17:12:09 +08:00
Asias He
9c5cd2bca8 storage_service: Switch to use unordered_set for tokens
We do not care about the order of the tokens.

Also, in token_metadata, we use unordered_set for tokens as well, e.g.
update_normal_tokens. Unify the usage.
2015-06-04 17:12:09 +08:00
Calle Wilund
5d32364e34 Migration manager: add placeholder method for announce_column_family_update 2015-06-03 10:13:53 +02:00
Avi Kivity
b2ca316516 storage_proxy: fix shared_ptr misuse in query_local()
Instead of using do_with(), we open-code it and do it badly, dropping
the reference count on the remote shard.

Fix by dropping the reference count on the local core.
2015-06-01 17:18:35 +02:00
Asias He
40bf00bdb1 storage_service: Move some code to source file 2015-06-01 11:24:39 +08:00
Asias He
4c23121b47 token_metadata: Add helper to print debug info for _token_metadata 2015-06-01 11:24:39 +08:00
Asias He
7d39ca0b4a storage_service: Insert local host_id to _token_metadata.
This is how host_id is generated for each and every node.
We will learn other host_id of other node through gossip.
2015-06-01 11:24:39 +08:00
Asias He
6bffee5e11 token_metadata: Add update_host_id
Use std::unordered_map instead of boost:bimap. std::unordered_map is
much much easier to use. It is a premature optimization to user bimap.
We can iterate the map to check if host_id is unique. Modification of
host_id is not a frequent or performance sensitive operation anyway.
2015-06-01 11:24:38 +08:00
Asias He
291690d745 storage_service: Rename getRingDelay to get_ring_delay 2015-06-01 11:24:38 +08:00
Asias He
0acf306ccf storage_service: Implement on_change 2015-06-01 11:24:38 +08:00
Asias He
4829965496 storage_service: Enable handleStateBootstrap and friends
They will be used in on_change functions.
2015-06-01 11:24:38 +08:00
Asias He
904e584c14 storage_service: Implement on_join 2015-06-01 11:24:38 +08:00
Asias He
8be4226a36 storage_service: Update _token_metadata with update_normal_token 2015-06-01 11:24:38 +08:00
Asias He
74fcb3d9aa storage_service: Add get_broadcast_address helper 2015-06-01 11:24:38 +08:00
Asias He
cbbaecf85c storage_service: Drop messaging_service handler registration code
We register the messaging_service handler within the subsystem who uses
it, e.g., we register gossip related in the gossiper class.
2015-06-01 11:24:38 +08:00
Asias He
3dc03738bc storage_service: Add debug info in gossip state change callback
Now, we can verify that get_tokens_for(ip) get the correct tokens
through gossip.

Start the seed node first:
$./seastar --listen-address 127.0.0.1 --seed-provider-parameters 127.0.0.1

Then start a normal node:
$./seastar --listen-address 127.0.0.2 --seed-provider-parameters 127.0.0.1

On the second node:

SS::on_join endpoint=127.0.0.1
endpoint=127.0.0.1,
tokens_string=8768e8a2d360bce8;d99d50eae7e514a2;f937ba030e4f1fa2
t=87 68 e8 a2 d3 60 bc e8
t=d9 9d 50 ea e7 e5 14 a2
t=f9 37 ba 03 0e 4f 1f a2

ep=127.0.0.1, eps=EndpointState: HeartBeatState = generation =
1432868747, version = 52, AppStateMap =  { 0 : Value(BOOT,TOKENS,11) }
{ 5 : Value(urchin_1_0,4) }  { 8 : Value(,3) }  { 11 : Value(ms_1_0,1) }
{ 12 : Value(9a6e8c60-05af-11e5-b9b8-000000000000,2) }  { 13 :
Value(8768e8a2d360bce8;d99d50eae7e514a2;f937ba030e4f1fa2,9) }
2015-06-01 11:24:38 +08:00
Asias He
e1b9b2e09a storage_service: Set tokens on startup 2015-06-01 11:24:38 +08:00
Asias He
03b526f06b storage_service: Implement get_tokens_for 2015-06-01 11:24:38 +08:00
Asias He
322b27c12b storage_service: Implement set_tokens 2015-06-01 11:24:38 +08:00
Asias He
d28503374d storage_service: Add debug print helper 2015-06-01 11:24:38 +08:00
Asias He
8d7aff89e3 storage_service: Reduce default num_tokens to 3
The default value 256 in Origin is too big for debug. We can set it back
when storage_service is maturer.
2015-06-01 11:24:38 +08:00
Asias He
487768732f storage_service: Rename bootstrapTokens to _bootstrap_tokens 2015-06-01 11:24:38 +08:00
Asias He
e6f5618d97 storage_service: Rename tokenMetadata to _token_metadata 2015-06-01 11:24:37 +08:00
Avi Kivity
ff8769fdf7 Merge "Extend storage_service API support"
From Amnon:

"This seriese adds functionality to the storage_service API. It focus on tokens
information, that are partially supported in the storage_service and the call
to the API will return an empty list."
2015-05-31 17:57:30 +03:00
Pekka Enberg
27749769e8 migration_manager: Convert announce_new_column_family() to C++
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-05-27 16:22:13 +03:00
Amnon Heiman
041fd4c617 storage_service: expose the storage functionality
The storage service runs on CPU 0, this adds a wrapper function that
would expose the relevent storage_service function but will hide the
cpu this is actually be run on.

The implemented methods are the one needed by the API and are equivelent
to the ones defined in StorageServiceMBean

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-05-27 15:29:49 +03:00
Asias He
64a9935206 storage_service: Implement join_token_ring 2015-05-27 15:05:12 +08:00
Asias He
de169ca9e7 storage_service: Call boot_strapper::get_random_tokens 2015-05-27 13:06:33 +08:00
Asias He
7c85b8763a storage_service: Enable bootstrapTokens
We need BootStrapper.getBootstrapTokens and BootStrapper.getRandomTokens
to calculate it.
2015-05-26 17:45:29 +08:00
Asias He
9736dc6d9c storage_service: Convert should_bootstrap 2015-05-26 17:45:29 +08:00
Asias He
e98ec068ca storage_service: Start gossiper inside prepare_to_join 2015-05-26 17:45:29 +08:00