Commit Graph

7140 Commits

Author SHA1 Message Date
Takuya ASADA
5aab69e72f dist: do not remove build/ dir when scylla-server ubuntu package building
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-10-27 18:23:09 +02:00
Avi Kivity
7f88db8625 Merge "storage_service and gossip update" from Asias 2015-10-27 18:14:43 +02:00
Lucas Meneghel Rodrigues
a9a33d5a99 test.py: PEP8 Fixes
Fix some PEP8 problems found in the tester code:

 * Wrong spacing around operators
 * Lines between class and function definitions
 * Fixed some of the larger than 80 column statements
 * Removed an unused import

Signed-off-by: Lucas Meneghel Rodrigues <lmr@scylladb.com>
2015-10-27 17:54:54 +02:00
Tomasz Grabiec
491eae58a4 Merge branch 'gleb/read-repair' from seastar-dev.git
From Gleb:

Now that we can calculate mutation diffs do so on digest mismatch and send
them out.
2015-10-27 16:36:05 +01:00
Asias He
7e6e90dc52 storage_service: Serialize decommission
Prevent two operations happen simultaneously.
2015-10-27 21:48:38 +08:00
Asias He
4b75815306 storage_service: Introduce run_with_write_api_lock helper
It is useful to run code on cpu zero with the api lock.
2015-10-27 21:48:38 +08:00
Asias He
62228726a3 storage_service: Introduce a rwlock to serialize management operations 2015-10-27 21:48:38 +08:00
Asias He
1bbc1920d2 range_streamer: Start to use get_preferred_ip
It is available now.
2015-10-27 21:48:37 +08:00
Asias He
306bab9ead storage_service: Use get_preferred_ip
Now that it is available, use it.
2015-10-27 21:48:37 +08:00
Asias He
55b76a8963 api: No cpu zero trick for remove_node
storage_service::remove_node is guaranteed to run on cpu zero only.
2015-10-27 21:48:37 +08:00
Asias He
b3c7305d25 storage_service: Make remove_node runs on cpu 0 only
We need to serialize nodetool operations to avoid two operations
happening simultaneously. Running on cpu 0 is one step toward this
goal.
2015-10-27 21:48:37 +08:00
Asias He
6c6b1c4ba7 storage_service: Make decommission runs on cpu 0 only
We need to serialize nodetool operations to avoid two operations
happening simultaneously. Running on cpu 0 is one step toward this goal.
2015-10-27 21:48:37 +08:00
Asias He
49daba2599 storage_service: Do not ignore future in decommission
gossiper::stop returns a future which we can not ignore.
2015-10-27 21:48:37 +08:00
Asias He
00311817bd storage_service: Implement shutdown_client_servers 2015-10-27 21:48:37 +08:00
Asias He
83eb36796f storage_service: Kill FIXME for LoadBroadcaster.BROADCAST_INTERVAL
It is available now.
2015-10-27 21:48:37 +08:00
Asias He
1469cec5bf gossiper: Kill free function helper to get heart version and generation number
They can only be executed on cpu 0. Make the gossiper member
functions for them to do so.
2015-10-27 21:48:37 +08:00
Asias He
f573059698 gossiper: Kill free function helper for {unsafe_,}assassinate_endpoint
They can only be executed on cpu 0. Make the gossiper member functions
for them to do so.
2015-10-27 21:48:37 +08:00
Asias He
c5f377eb8b gossip: Simplify get_endpoint_downtime
_unreachable_endpoints is replicated to call cores. No need to query
on core 0.
2015-10-27 21:48:37 +08:00
Asias He
6f1db4fb72 gossip: Simplify get_unreachable_members
_unreachable_endpoints is replicated to call cores. No need to query on
core 0.

This also fixes a bug in storage_proxy::truncate_blocking
which might access _unreachable_endpoints on non-zero cores.
2015-10-27 21:48:37 +08:00
Asias He
a9f96d1f5a gossip: Replicate _unreachable_endpoints to all cores 2015-10-27 21:48:37 +08:00
Asias He
2439a2a982 gossip: Simplify get_live_members
_live_endpoints is replicated to call cores. No need to query on core 0.
2015-10-27 21:48:37 +08:00
Asias He
a28ba9cde8 api: Simplify get_tokens get_node_tokens and get_token_endpoint
token_metadata is replicated to all cores. No need to query on core 0.
2015-10-27 21:48:37 +08:00
Asias He
4e886f0399 storage_service: Implement is_rpc_server_running 2015-10-27 21:48:37 +08:00
Asias He
7f0634b429 storage_service: Implement stop_rpc_server 2015-10-27 21:48:37 +08:00
Asias He
ca66bea619 storage_service: Implement is_native_transport_running 2015-10-27 21:48:37 +08:00
Asias He
7bc49a1efe storage_service: Implement stop_native_transport 2015-10-27 21:48:37 +08:00
Asias He
8218ab7922 storage_service: Implement start_native_transport and start_rpc_server
They are used for APIs. Share the code in main.cc as well.
2015-10-27 21:48:37 +08:00
Lucas Meneghel Rodrigues
42c0acfc44 test.py: Return test output only if subprocess succeeded
The current code will try to print the output of a
subprocess.Popen().communicate() call even if that
call raised an exception and that output is None.

Let's fix this problem by only printing the output
if it's not None.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@scylladb.com>
2015-10-27 15:17:05 +02:00
Asias He
3c47844e8c storage_service: Complete check_for_endpoint_collision
Part of it was stubbed.
2015-10-27 21:17:02 +08:00
Asias He
77a87cb2b6 storage_service: Implement prepare_replacement_info
Needed by replace node operation.
2015-10-27 21:17:02 +08:00
Vlad Zolotarov
5cdbc3701a tests: set broadcast address
Since commit 5613979a85
broadcast address has to be set before it's used for the first
time.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2015-10-27 15:16:13 +02:00
Avi Kivity
345f739f28 Merge seastar upstream
* seastar 501e4cb...9d8913a (3):
  > Add mutable to with_lock and do_with
  > app-template: disable collectd by default
  > reactor: use fdatasync() instead of fsync()
2015-10-27 15:13:36 +02:00
Avi Kivity
42a5c8b92e Merge "CQL request load balancing" from Pekka
"Currently, CQL requests are processed on the same CPU core where the
connection lives in. This series adds infrastructure for migrating CQL
processing to other cores and implements a round-robin load balancing
algorithm that can be enabled with the "--load-balance=round-robin"
command line option. Load balancing is not enabled by default because we
need to first run performance tests to determine if the simple
round-robin algorithm is sufficient, or wheter we need to implement more
sophisticated dynamic load balancing."
2015-10-27 15:04:44 +02:00
Gleb Natapov
58154333e8 storage_proxy: send out mutation diffs to each destination 2015-10-27 14:58:35 +02:00
Pekka Enberg
a772938e73 transport/server: Round-robin CQL request load balancing
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-10-27 13:24:58 +02:00
Pekka Enberg
ed0607c10e transport/server: Merge client state changes in process_request()
In preparation for processing queries on shards other than where the
connection lives in, merge client state changes in process_request().

Signed-off-by: Pekka Enberg <penberg@scylladb.com>
2015-10-27 13:24:58 +02:00
Pekka Enberg
4641dfff24 service: Copy client state to query state
In preparation for processing CQL requests on different core than where
the connection lives in, copy client state to query state for
processing and merge back the results after we're done.

Signed-off-by: Pekka Enberg <penberg@scylladb.com>
2015-10-27 13:24:58 +02:00
Pekka Enberg
27c678b2a5 transport/server: Use bytes_view instead of moving temporary buffer around
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-10-27 13:13:25 +02:00
Pekka Enberg
4482406aee transport/server: Write response from process_request()
In preparation for spreading request processing to multiple cores, make
sure CQL response is written out on the connection shard.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-10-27 13:13:25 +02:00
Pekka Enberg
3b6eba1344 transport/server: Remove _query_states from connection
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-10-27 13:13:25 +02:00
Pekka Enberg
3884f1c8b6 transport/server.cc: Fix formatting glitch
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-10-27 13:13:25 +02:00
Gleb Natapov
122f03e689 storage_proxy: calculate mutation diffs during reconcile. 2015-10-27 13:02:15 +02:00
Gleb Natapov
8bbe4bdbd4 storage_proxy: rename reconciliate to reconcile
'reconciliate' does not seams to be a world.
2015-10-27 12:57:02 +02:00
Vlad Zolotarov
3a3867588a tests::cql_test_env: set broadcast address
cql_query_test hasn't configured Broadcast address before
it was used for the first time.

Broadcast address is an essential Node's configuration.

There is an assert in utils::fb_utils::get_broadcast_address()
that ensures that broadcast address has been properly configured
before it's used for the first time and it is triggered without
this patch.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2015-10-27 12:38:56 +02:00
Avi Kivity
0754e9a34d Merge "Even more commitlog fixes" from Calle
"Fixes for commitlog (debug) test failures related to shutdowns.
Note that most the fixes here are only really related to the tests
failing, not really real scylla runs. However, at some point we'll
have real shutdown in scylla as well (not just hard exit), at which
point this becomes more relevant there as well.

Main issue was post-flush continuation chains for stats update
remaining unexecuted, due to task reordering, once the commitlog
object itself had been destroyed. This could have been handled by just
making the stats object a shared pointer, but in general it seems more
prudent to enforce having all tasks completed after shutdown.

* Change commitlog shutdown to use gate+wait for all outstanding ops
  (flush, write, timer). Thus we can ensure everything is finished
  when returning from "shutdown".
* Fix bug with "commitlog::clear" (test method) not doing the intended deed
* Most importantly, fix the tests themselves, cleaning up old crud, and
  fixing invalid assumptions (CL behaviour changed quite a bit since tests
  were created), and remove races.

Disclaimer: I've _never_ managed to reproduce the debug tests failing
like in jenkins locally (though I managed to provoke other failures),
but at least jenkins runs with this series have been clean. Knock knock."
2015-10-27 12:16:20 +02:00
Paweł Dziepak
f46cba7bc8 sstable: simplify key reader
Now that #475 is solved an read_indexes() guarantees to return disjont
sets of keys sstable key reader can be simplified, namely, only two key
lookups are needed (the first and the last one) and there is no need for
range splitting.

Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-10-27 10:33:02 +02:00
Avi Kivity
fa0f00e9d2 Merge "Add EC2MultiRegionSnitch and Co." from Vlad
"This series add the mighty EC2MultiRegionSnitch and some missing
multi-DC related functionality:
   - Use the proper Broadcast Address: either the one from the
    .yaml configuration (if present) or the one configured by some
    scylla component (e.g. snitch).
   - Introduce the ability to switch to internal IPs when connecting
     to Nodes in the same data center.
   - Store the known internal IPs in system.peers table and
     load then immediately during boot.

This series also contains some related fixes done on the way."
2015-10-26 18:38:57 +02:00
Gleb Natapov
5bc532261f load_broadcaster: add missing header file protector 2015-10-26 18:38:01 +02:00
Avi Kivity
91bde5e2e8 Merge "sstable improvements" from Raphael 2015-10-26 17:25:58 +02:00
Asias He
6f08c4facb dns: Move gethostbyname to source file
Fix multiple definition of gethostbyname.
2015-10-26 15:59:58 +02:00