Commit Graph

8393 Commits

Author SHA1 Message Date
Avi Kivity
1f245e3bcb mutation_partition: fix use of boost::intrusive::set<>::comp()
Seems like boost::intrusive::set<>::comp() is not accessible on some
versions of boost.  Replace by the equivalent
boost::intrusive::set<>::key_comp().

Fixes #858.
Message-Id: <1454326483-29780-1-git-send-email-avi@scylladb.com>
2016-02-01 13:54:52 +01:00
Calle Wilund
159dbe3a64 sstable_datafile_tests: Replace '---' with auto
Fixes compilation issues on some g++.
Message-Id: <1454323749-21933-1-git-send-email-calle@scylladb.com>
2016-02-01 12:58:33 +02:00
Avi Kivity
2b84bd3b75 Merge "standalone tcp connection for streaming" from Asias
"Make the streaming use standalone tcp connection and send more mutations in
parallel.

It is supposed to help: "Decommission not fully utilizing hardware #849""
2016-02-01 09:54:11 +02:00
Asias He
c618c699b3 streaming: Increase mutation_send_limiter
The idea behind the current 10 stream_mutations per core limitation is
to avoid streaming overwhelms the TCP connection and starves normal cql
verbs if the streaming mutations are big and takes long time to
complete.

Now that we use a standalone connection for streaming verbs, we can
increase the limitation.

Hopefully, this will fix #849.
2016-02-01 11:01:56 +08:00
Asias He
fbf796b812 messaging_service: Use standalone connection for stream verbs
In streaming, the amount of data needs to be streamed to peer nodes
might be large.

In order to avoid the streaming overwhelms the TCP connection used by
user CQL verbs and starves the user CQL queries, we use a standalone TCP
connection for streaming verbs.
2016-02-01 11:01:56 +08:00
Avi Kivity
1146e3796d Merge "streaming refactor" from Asias
"- Wire up session progress
- Refactor stream_coordinator::host_streaming_data
- Introduce get_session helper to simplfy verb handling
- Remove unused code

Tested with streaming in update_cluster_layout_tests.py"
2016-01-31 20:17:53 +02:00
Tomasz Grabiec
945ae5d1ea Move std::hash<range<T>> definition to range.hh
Message-Id: <1454008052-5152-1-git-send-email-tgrabiec@scylladb.com>
2016-01-31 20:11:30 +02:00
Avi Kivity
f6e7dbf080 Merge seastar upstream
* seastar 6623379...f8beab9 (2):
  > json_base_element: do not assign the element name
  > io_queue: change visibility of internal function
2016-01-31 16:34:39 +02:00
Raphael S. Carvalho
a46aa47ab1 make sstables::compact_sstables return list of created sstables
Now, sstables::compact_sstables() receives as input a list of sstables
to be compacted, and outputs a list of sstables generated by compaction.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <0d8397f0395ce560a7c83cccf6e897a7f464d030.1454110234.git.raphaelsc@scylladb.com>
2016-01-31 12:39:20 +02:00
Raphael S. Carvalho
ee84f310d9 move deletion of sstables generated by interrupted compaction
This deletion should be handled by sstables::compact_sstables, which
is the responsible for creation of new sstables.
It also simplifies the code.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <541206be2e910ab4edb1500b098eb5ebf29c6509.1454110234.git.raphaelsc@scylladb.com>
2016-01-31 12:39:20 +02:00
Glauber Costa
7214649b8a sstables: const where const is due
Some SSTable methods are not marked as const. But they should be.

Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <72cd3ef0157eb38e7fd48d0c989f2342cbc42f3c.1454103008.git.glauber@scylladb.com>
2016-01-31 12:36:36 +02:00
Avi Kivity
3434b8e7c6 Merge seastar upstream
* seastar fbd9b30...6623379 (1):
  > fstream: improve make_file_input_stream() for a subrange of a file
2016-01-31 12:01:46 +02:00
Avi Kivity
a3fa123070 Update scylla-ami submodule
* dist/ami/files/scylla-ami e284bcd...b2724be (2):
  > Revert "Run scylla.yaml construction only once"
  > Move AMI dependent part of scylla_prepare to scylla-ami-setup.service
2016-01-31 12:01:15 +02:00
Avi Kivity
f08f5858a8 Merge "Introduce scylla-ami-setup.service, fix bugs" from Takuya
"This moves AMI dependent part of scylla_prepare to scylla-ami repo, make it scylla-ami-setup.service which is independent systemd unit.
Also, it stopped calling scylla_sysconfig_setup on scylla_setup (called on AMI creation time), call it from scylla-ami-setup instead."
2016-01-31 12:00:32 +02:00
Takuya ASADA
111dc19942 dist: construct scylla.yaml on first startup of AMI instance, not AMI image creation time
Install scylla-ami-setup.service, stop calling scylla_sysconfig_setup on AMI.
scylla-ami-setup.service will call it instead.
Only works with scylla-ami fix.
Fixes #857

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-01-30 15:48:45 -05:00
Takuya ASADA
71a26e1412 dist: don't need AMI_KEEP_VERSION anymore, since we fixed the issue that 'yum update' mistakenly replaces scylla development version with release version
It actually doesn't called unconditionally now, (since $LOCAL_PKG is always empty) so we can safely remove this.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-01-30 15:47:05 -05:00
Takuya ASADA
f9d32346ef dist: scylla_sysconfig_setup uses current sysconfig values as default value
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-01-30 15:46:21 -05:00
Takuya ASADA
4d5baef3e3 dist: keep original SCYLLA_ARGS when updating sysconfig
Since we dropped scylla_run, now default SCYLLA_ARGS parameter is not empty.
So we need to support it.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-01-30 15:44:03 -05:00
Asias He
f07cd30c81 streaming: Remove unused create_message_for_retry 2016-01-29 16:31:07 +08:00
Asias He
cb92fe75e6 streaming: Introduce get_session helper
To simplify streaming verb handler.

- Use get_session instead of open coded logic to get get_coordinator and
  stream_session in all the verb handlers

- Use throw instead of assert for error handling

- init_receiving_side now returns a shared_ptr<stream_result_future>
2016-01-29 16:31:07 +08:00
Asias He
360df6089c streaming: Remove unused stream_session::retry 2016-01-29 16:31:07 +08:00
Asias He
2f48d402e2 streaming: Remove unused commented code 2016-01-29 16:31:07 +08:00
Asias He
ed3da7b04c streaming: Drop flush_tables option for add_transfer_ranges
We do not stream sstable files. No need to flush it.
2016-01-29 16:31:07 +08:00
Asias He
aa69d5ffb2 streaming: Drop update_progress in stream_coordinator
Since we have session_info inside stream_session now, we can call
update_progress directly in stream_session.
2016-01-29 16:31:07 +08:00
Asias He
30c745f11a streaming: Get rid of stream_coordinator::host_streaming_data
Now host_streaming_data only holds shared_ptr<stream_session>, we can
get rid of it and put shared_ptr<stream_session> inside _peer_sessions.
2016-01-29 16:31:07 +08:00
Asias He
46bec5980b streaming: Put session_info inside stream_session
It is 1:1 mapping between session_info and stream_session. Putting
session_info inside stream_session, we can get rid of the
stream_coordinator::host_streaming_data class.
2016-01-29 16:31:07 +08:00
Asias He
91e245edac streaming: Initialize total_size in stream_transfer_task
Also rename the private member to _total_size and _files
2016-01-29 16:31:07 +08:00
Asias He
c4bdb6f782 streaming: Wire up session progress
The progress info is needed by JMX api.
2016-01-29 16:31:07 +08:00
Avi Kivity
3e4ce609ee Merge seastar upstream
* seastar ec468ba...fbd9b30 (9):
  > Add implementation of count_leading_zeros<LL>
  > Fix htonl usage for clang
  > Fix gnutls_error_category ctor for clang
  > Add header files required for libc++
  > Add clang warning suppressions
  > Switch to correct usage of std::abs
  > Fix the do_marshall(sic) function to init_list
  > Corrected sockaddr_in initialization
  > Remove unused const char misc_strings
2016-01-28 18:24:46 +02:00
Raphael S. Carvalho
3b7970baff compaction: delete generated sstables in event of an interrupt
Generated sstables may imply either fully or partially written.
Compaction is interrupted if it was deriberately asked to stop (stop API)
or it was forced to do so in event of a failure, ex: out of disk space.
There is a need to explicitly delete sstables generated by a compaction
that was interrupted. Otherwise, such sstables will waste disk space and
even worsen read performance, which degrades as number of generations
to look at increases.

Fixes #852.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <49212dbf485598ae839c8e174e28299f7127f63e.1453912119.git.raphaelsc@scylladb.com>
2016-01-28 14:05:57 +02:00
Pekka Enberg
3c3c819280 Merge "api: Fix stream_manager" from Asias
"Fix the metrics for bytes sent and received"
2016-01-28 13:57:59 +02:00
Raphael S. Carvalho
ba4260ea8f api: print proper compaction type
There are several compaction types, and we should print the correct
one when listing ongoing compaction. Currently, we only support
compaction types: COMPACTION and CLEANUP.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <c96b1508a8216bf5405b1a0b0f8489d5cc4be844.1453851299.git.raphaelsc@scylladb.com>
2016-01-28 13:47:00 +02:00
Tomasz Grabiec
9fa62af96b database: Move implementation to .cc
Message-Id: <1453980679-27226-1-git-send-email-tgrabiec@scylladb.com>
2016-01-28 13:35:33 +02:00
Tomasz Grabiec
ca6bafbb56 canonical_mutation: Remove commented out junk 2016-01-28 12:29:20 +01:00
Tomasz Grabiec
41dc98bb79 Merge branch 'cleanup_improvements' from git@github.com:raphaelsc/scylla.git
Compaction cleanup improvements from Raphael.
2016-01-27 18:30:46 +01:00
Avi Kivity
873deb5808 Merge "move paging_state to use idl" from Gleb 2016-01-27 19:06:04 +02:00
Takuya ASADA
03caacaad0 dist: enable collectd client by default
Fixes #838

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1453910311-24928-2-git-send-email-syuu@scylladb.com>
2016-01-27 18:45:45 +02:00
Takuya ASADA
f33656ef03 dist: eliminate startup script
Fixes #373

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1453910311-24928-1-git-send-email-syuu@scylladb.com>
2016-01-27 18:45:35 +02:00
Gleb Natapov
b065e2003f Move paging_state to use idl 2016-01-27 18:39:43 +02:00
Raphael S. Carvalho
45c446d6eb compaction: pass dht::token by reference
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2016-01-27 13:25:41 -02:00
Raphael S. Carvalho
fc541e2f08 compaction: remove code to sort local ranges
storage_service::get_local_ranges returns sorted ranges, which are
not overlapping nor wrap-around. As a result, there is no need for
the consumer to do anything.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2016-01-27 13:15:36 -02:00
Avi Kivity
c75d1c4eeb Merge "Ubuntu 'expect stop' related fixes" from Takuya 2016-01-27 17:00:23 +02:00
Gleb Natapov
65bd429a0b Add serialization helper to use outside of rpc. 2016-01-27 16:43:06 +02:00
Takuya ASADA
4162fb158c main: raise SIGSTOP only when scylla become ready
supervisor_notify() calls periodically, to log message on systemd.
So raise(SIGSTOP) will called multiple times, upstart doesn't expected that.
We need to call it just one time.

Fixes #846

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-01-27 23:30:26 +09:00
Takuya ASADA
851951d32d dist: run upstart job as 'scylla' user
Don't use sudo when launching scylla, run directly from upstart.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-01-27 23:30:02 +09:00
Takuya ASADA
89f0fc89b4 dist: set ulimit in upstart job
Upstart job able to specify ulimit like systemd, so drop ubuntu's scylla_run and merge with redhat one.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-01-27 23:29:52 +09:00
Takuya ASADA
b4accd8904 main: autodetect systemd/upstart
We can autodetect systemd/upstart by environment variables, don't need program argument.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2016-01-27 23:29:32 +09:00
Takuya ASADA
559f913494 dist: use nightly for prebuilt 3rdparty packages (CentOS)
Developers probably wants to use latest dependency packages, so switch to nightly.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1453904521-2716-1-git-send-email-syuu@scylladb.com>
2016-01-27 16:24:49 +02:00
Gleb Natapov
19c55693fd idl: add missing header to serializer.hh 2016-01-27 15:49:29 +02:00
Amnon Heiman
7b53b99968 idl-compiler: split the idl list
Not all the idls are used by the messaging service, this patch removes
the auto-generated single include file that holds all the files and
replaes it with individual include of the generated fiels.
The patch does the following:
* It removes from the auto-generated inc file and clean the configure.py
  from it.
* It places an explicit include for each generated file in
  messaging_serivce.
* It add dependency of the generated code in the idl-compiler, so a
change in the compiler will trigger recreation of the generated files.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1453900241-13053-1-git-send-email-amnon@scylladb.com>
2016-01-27 15:23:00 +02:00