Commit Graph

53948 Commits

Author SHA1 Message Date
Calle Wilund
6d25b8d496 query_pagers: Handle partition with no data but statics
If we get a partition with no row data, but statics, we should treat this as
a row (include in count), but also make sure we skip to next partition
if our page ends here.

The "end partition" with zero rows but static data can also happen if we
happen to resume paging by giving a column range exluding all data. In this
case we should _not_ include it, since we have already provided the
data in question in previous page.

Fixes #556
2015-11-17 17:49:14 +02:00
Calle Wilund
5368f1a998 query_pagers: Fix aggregate query bug
1.) Should not reset to input query state if run repeatedly
2.) And if run repeatedly without input state, likewise keep
    the internal one active

Fixes #560
2015-11-17 17:48:58 +02:00
Avi Kivity
7d8c72d24f Merge "move scylla.yaml to /etc/scylla" from Takuya
"To keep compatibility with scylla-tools-java, it links /etc/scylla to /var/lib/scylla/conf.

Problem on this patchset is, I added SCYLLA_HOME and SCYLLA_CONF on /etc/sysconfig/scylla-server.
However, the file is marked as config file, it won't be automatically upgrade.
If user doesn't upgrade the file manually, scylla-server still able to run with /var/lib/scylla/conf because we have symlink, but never switches to /etc/scylla."
2015-11-17 17:02:14 +02:00
Takuya ASADA
60efcd791a dist: don't specify scylla.yaml path directly, use SCYLLA_CONF instead 2015-11-17 02:37:37 +09:00
Takuya ASADA
6972d78e93 dist: specify SCYLLA_CONF as /etc/scylla
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-17 02:37:37 +09:00
Takuya ASADA
aeb076a6c0 dist: change scylla.yaml install path to /etc/scylla (Ubuntu)
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-17 02:37:37 +09:00
Takuya ASADA
b5bd34c0c2 dist: change scylla.yaml install path to /etc/scylla (Fedora/CentOS)
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-17 02:37:37 +09:00
Takuya ASADA
51813dc8ce dist: merge redhat/limits.d and debian/limits.d to common/limits.d
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-17 02:37:37 +09:00
Takuya ASADA
863eea2bb3 dist: do not overwrite config files on 'yum update'
Fixes a problem on AMI which recently reported by Glauber

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-17 02:37:37 +09:00
Asias He
6ac54a27dc streaming: Skip non-exist cf for stream_transfer_task
Skip sending the mutation if the cf is dropped after we call
make_local_reader in stream_session::add_transfer_ranges().

Fix #550.
2015-11-16 16:48:35 +01:00
Paweł Dziepak
c37afcfdee lsa: account for size of objects too big for LSA
While the objects above max_manage_object_size aren't stored in the
LSA segments they are still considered to be belonging to the LSA
region and are evictable using that region evictor.

Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-11-16 12:22:12 +01:00
Avi Kivity
e04a6ea80a Merge "Add the natural_endpoints API" from Amnon
"This series adds the natural_endpoints API. It adds the implementation to the storage_service and to the storage_service API.

After this series the noodtool command getendpoints should work.

example:
$ bin/nodetool getendpoints keyspace1 standard1 0x5032394c323239385030127.0.0.2
127.0.0.2"
2015-11-16 13:16:42 +02:00
Amnon Heiman
c2051c200e API: messaging service add timeout and dropped support
This patch adds the API for timeout messages and dropped messages.

For dropped messages, origin has two APIs one for messages and one for
command.

droped messages return the number of messages per ver, so our API was
rename to reflect that.

For dropped messages (command) we currently do not have this logic of
throwing messages before sending, so the API will always return 0.

The total timeout API was removed and will be done on the jmx proxy
level.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-16 13:15:59 +02:00
Gleb Natapov
79ab5e68f4 enable broadcast_address configuration option
Reviewed-by: Asias He <asias@scylladb.com>
2015-11-16 13:14:07 +02:00
Asias He
358648a8ee init: Check seeds address if listen_address != broadcast_address
If listen_address is different than broadcast_address, we should use
broadcast_address for the seeds list. Check and ask user to fix the
configuration, e.g.,

$ scylla --rpc-address 127.0.0.1 --listen-address 127.0.0.1 --broadcast-address 192.168.1.100 --seed-provider-parameters seeds=127.0.0.1
Use broadcast_address instead of listen_address for seeds list: seeds={127.0.0.1}, listen_address=127.0.0.1, broadcast_address=192.168.1.100
Exiting on unhandled exception of type 'std::runtime_error': Use broadcast_address for seeds list
2015-11-16 13:11:44 +02:00
Asias He
c973d2ea65 to_string: Support std::set and std::unordered_set for to_string 2015-11-16 13:11:43 +02:00
Avi Kivity
9575a7b2ea Merge "Gossiper const correctness" from Pekka
"Fix various const correctness issues in gossiper code found while
working on inter-node protocol."
2015-11-16 13:08:53 +02:00
Asias He
b5cc3ac81c transport: Fix cql server stop
Abort accept and kill existing connections and wait for them.

Fix tests in debug mode:

==29352==ERROR: AddressSanitizer: heap-use-after-free on address 0x60c000012758
at pc 0x00000211e371 bp 0x7ffe27369e10 sp 0x7ffe27369e00
READ of size 8 at 0x60c000012758 thread T0
    #0 0x211e370 in transport::cql_server::do_accepts(int)::{
    lambda(connected_socket, socket_address)#1}::operator()(connected_socket,
    socket_address)::{lambda(future<>)#1}::operator()(future) const
    (/home/asias/.dtest/dtest-AOBJua/test/node2/bin/scylla+0x211e370)
    #1 0x21a8090 in do_void_futurize_apply<transport::cql_server::do_accepts(int)::<
    lambda(connected_socket, socket_address)> mutable::<lambda(future<>)>, future<> >
    /home/asias/src/cloudius-systems/scylla/seastar/core/future.hh:1078
    #2 0x217e861 in apply<transport::cql_server::do_accepts(int)::<
    lambda(connected_socket, socket_address)> mutable::<lambda(future<>)>, future<> >
    /home/asias/src/cloudius-systems/scylla/seastar/core/future.hh:1126
    #3 0x223deb9 in _ZZN6futureIJEE12then_wrappedIZZN9transport10cql_server10do_accepts
    EiENUl16connected_socket14socket_addressE_clES4_S5_EUlS0_E_S0_EET0_OT_ENUlSA_E_
    clI12future_stateIJEEEEDaSA_
    (/home/asias/.dtest/dtest-AOBJua/test/node2/bin/scylla+0x223deb9)
2015-11-16 13:06:20 +02:00
Gleb Natapov
eb220507ce storage_proxy: use correct endpoint address for mutation acks processing
Write handler keeps track of all endpoints that not yet acked mutation
verb. It uses broadcast address as an enpoint id, but if local address
is different from broadcast address for local enpoints acknowledgements
will come from different address, so socket address cannot be used as
an acknowledgement source. Origin solves this by sending "from" in each
message, it looks like an overhead, solve this by providing endpoint's
broadcast address in rpc client_info and use that instead.
2015-11-16 10:29:47 +01:00
Tomasz Grabiec
a6e73d40d7 Move seastar.git HEAD
Gleb Natapov (2):
      rpc: allow non const client_info to be passed to rpc handler
      rpc: allow auxiliary data to be stored in client_info
2015-11-16 10:29:47 +01:00
Pekka Enberg
3c72ea9f96 gms: Fix gossiper::handle_major_state_change() restart logic
The restart logic is wrong because C* had a bug in
bf599fb5b062cbcc652da78b7d699e7a01b949ad and they fixed later and we
translated the broken version. We must check if there is an existing
endpoint state and call on_restart() hooks on that, not the newly
available endpoint state.

Spotted while inspecting the code.

Acked-by: Asias He <asias@scylladb.com>
2015-11-16 11:25:48 +02:00
Tomasz Grabiec
7e0f99cc3b Merge tag 'native-preparatory/v1' from https://github.com/avikivity/scylla.git
Assorted patches that pave the way for native storage (while not
committing us in any way).
2015-11-16 10:01:38 +01:00
Tomasz Grabiec
b6e7312c07 Merge 'memtable-allocating_section/v2' from https://github.com/avikivity/scylla.git
From Avi:

Memtables do not use an allocating_section to guard against allocation
failure, and hence can fail an allocation.  Reproducible by changing
perf_mutation to use an allocating type (bytes_type with a nontrivial
size) and making the loop longer.

Fix by using an allocating_section.
2015-11-16 10:01:05 +01:00
Avi Kivity
a40a62d840 memtable: use allocating_section to guard allocations
Without this, an allocation can fail, and we may not be able to reclaim
memory.
2015-11-16 10:56:06 +02:00
Pekka Enberg
5f48f4aa7c gms/heart_beat_state: Make get_heart_beat_version() const
Signed-off-by: Pekka Enberg <penberg@scylladb.com>
2015-11-16 08:12:25 +02:00
Pekka Enberg
dc6eaa3a44 gms/gossip_digest: Make some members const
Signed-off-by: Pekka Enberg <penberg@scylladb.com>
2015-11-16 08:12:16 +02:00
Pekka Enberg
5815bde695 gms/gossip_digest_ack: Make some members const
Signed-off-by: Pekka Enberg <penberg@scylladb.com>
2015-11-16 08:12:16 +02:00
Avi Kivity
1c425d6b50 logalloc: allow allocating_section code blocks to return references 2015-11-15 19:10:24 +02:00
Tomasz Grabiec
6626d1a100 storage_proxy: Stop latency_counter before throwing 2015-11-15 10:34:55 +02:00
Glauber Costa
0989f80c97 provide cf_stats where one is needed
Recently, I have introduced cf_stats into the database, propagating all the way
back to the column family. The problem, however, is that some tests create a
column family config themselves instead of going through make_column_family.

That is ultimately ok if those tests are not expected to flush memtables. But
if they are, the cf_stats pointer will be null and we will crash. Although
there are many solutions to this, the one that is in tune with our current
practices is to have the test that requires it provide an empty cf_stats storage
area that can be written to. That's already how we handle the disk directory and
other things like compaction properties.

With this patch, test.py passes again.

Signed-off-by: Glauber Costa <glommer@scylladb.com>
2015-11-15 10:31:32 +02:00
Glauber Costa
fe2b928a3f change defaults for commitlog maximum size
arbitrary 8G -> all_memory.

Signed-off-by: Glauber Costa <glommer@scylladb.com>
2015-11-15 10:29:23 +02:00
Glauber Costa
00c12319f1 config: change type for commitlog maximum size config option
This patch substitutes uint64_t for uint32_t as the type for
commitlog_total_space_in_mb.  Moving to 64 is not strictly needed, since even a
signed 32-bit type would allow us to easily handle 2TB. But since we store that
in the commitlog as a 64-bit value, let's match it.

Moving from unsigned to signed, however, allow us to represent negative
numbers.  With that in place, we can change the semantics of the value
slightly, so to allow a negative number to mean "all memory".

The reason behind this, is that the default value "8GB", is an artifact of the
JVM.  We don't need that, and in many-shards configuration, each shard flushes
the commitlog way too often, since 8GB / many_shards = small_number.

8GB also happens to be a popular heap size for C* in the JVM. For us, we would
like to equate that (at least) with the amount of memory. The problem is how to
do that without introducing new options or changing the semantics of existing
options too radically.

The proposed solution will allow us to still parse C* yaml files, since those
will always have positive numbers, while introducing our own defaults.

Signed-off-by: Glauber Costa <glommer@scylladb.com>
2015-11-15 10:29:23 +02:00
Avi Kivity
baf3b2ff4f Merge "Eliminate errors/warnings on building ubuntu package" from Takuya
"Fixes part of #493 (few warnings still remains)"
2015-11-15 10:26:19 +02:00
Takuya ASADA
df9e7d542f dist: ignore bogus 'dist/ubuntu/debian/source/lintian-overrides' on ubuntu package building 2015-11-15 15:03:02 +09:00
Takuya ASADA
d40ce51a01 dist: fix 'init.d-script-missing-dependency-on-remote_fs' message on building ubuntu package 2015-11-15 15:03:02 +09:00
Takuya ASADA
87ed97c387 dist: fix 'init.d-script-not-included-in-package' message on building ubuntu package 2015-11-15 15:03:02 +09:00
Takuya ASADA
100c22cfa5 dist: place 'default' file by debian way 2015-11-15 15:03:02 +09:00
Takuya ASADA
7c4c182268 dist: move redhat/sysconfig to common/sysconfig
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-15 15:03:02 +09:00
Takuya ASADA
0c9f837d04 dist: fix 'source-is-missing' message on building ubuntu package 2015-11-15 15:03:02 +09:00
Takuya ASADA
580ca26092 dist: fix 'extended-description-line-too-long' message on building ubuntu package 2015-11-15 15:03:02 +09:00
Takuya ASADA
ee45201b5a dist: make ubuntu package as 'debian non-native package'
Debian package system has two types of package, 'native' and 'non-native'.
'native' is the package just for Debian, it contains debian/ directory source tar.gz, doesn't have debian.tar.gz.
'non-native' has orig.tar.gz which is upstream source code tar ball, then it has debian.tar.gz which contains debian/ directory.
Scylla is 'native' now but should be 'non-native' since this is not just for Debian, so move debian/ to dist/ubuntu/, make orig.tar.gz using git-archive-all, copy dist/ubuntu/debian/ to debian/ then generate debian.tar.gz.
2015-11-15 15:03:02 +09:00
Takuya ASADA
c9c9a86195 dist: fix 'maintainer-script-needs-depends-on-adduser postinst' message on building ubuntu package
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-15 15:03:02 +09:00
Takuya ASADA
922f747122 dist: ignore some files under debian/ which are created on building time
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-15 15:03:02 +09:00
Takuya ASADA
10a7bd2d4a dist: fix 'wrong-section-according-to-package-name' and 'debug-package-should-be-priority-extra' message on building ubuntu package
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-15 15:03:02 +09:00
Takuya ASADA
63e262e507 correct permission of cassandra-rackdc.properties
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-15 15:03:02 +09:00
Takuya ASADA
91bd7cf71a dist: fix 'maintainer-script-lacks-debhelper-token' message on building ubuntu package
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-15 15:03:02 +09:00
Takuya ASADA
89fe283506 dist: fix 'ancient-standards-version 3.9.2' message on building ubuntu package
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-15 15:03:02 +09:00
Takuya ASADA
c09226fda8 dist: fix 'missing-license-paragraph-in-dep5-copyright' message on building ubuntu package
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
2015-11-15 15:03:02 +09:00
Avi Kivity
36994a5d08 managed_bytes: add a constructor from std::initializer_list<>
Not actually used in the patchset now, but nice.
2015-11-13 17:13:07 +02:00
Avi Kivity
f3afe3e876 allocation_strategy: constify migrate_fn
Since abstract_type will be providing our migrate_fn, they must be const,
and indeed a migration does not change the migration function.
2015-11-13 17:13:07 +02:00