Commit Graph

11716 Commits

Author SHA1 Message Date
Takuya ASADA
b9cb91e934 dist: stop ntpd before running ntpdate
New CentOS Base Image runs ntpd by default, so shutdown it before running ntpdate.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1453241256-23338-3-git-send-email-syuu@scylladb.com>
2016-01-20 09:40:35 +02:00
Takuya ASADA
98e61a93ef dist: disable SELinux only when it enabled
New CentOS7 Base Image disabled SELinux by default, and running 'setenforce 0' on the image causes error, we won't able to build AMI.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1453241256-23338-2-git-send-email-syuu@scylladb.com>
2016-01-20 09:40:01 +02:00
Raphael S. Carvalho
c318f3baa3 sstables: fix sstable::data_stream_at
After 63967db8, offset is ignored when creating a input stream.
Found the problem after sstable_test failed recently.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <56ece21ff6e043e224eb2a6e76cdd422b94821b0.1453232689.git.raphaelsc@scylladb.com>
2016-01-20 09:35:57 +02:00
Raphael S. Carvalho
ff9b1694fe api: implement stop_compaction
stop_compaction is implemented by calling stop_compaction() of
compaction manager for each database.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2016-01-19 23:15:18 -02:00
Raphael S. Carvalho
5cceb7d249 api: fix paramType of parameter of stop_compaction
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2016-01-19 23:15:18 -02:00
Raphael S. Carvalho
3bd240d9e8 compaction: add ability to stop an ongoing compaction
That's needed for nodetool stop, which is called to stop all ongoing
compaction. The implementation is about informing an ongoing compaction
that it was asked to stop, so the compaction itself will trigger an
exception. Compaction manager will catch this exception and re-schedule
the compaction.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2016-01-19 23:15:18 -02:00
Raphael S. Carvalho
ec4c73d451 compaction: rename compaction_stats to compaction_info
compaction_info makes more sense because this structure doesn't
only store stats about ongoing compaction. Soon, we will add
information to it about whether or not an user asked to stop the
respective ongoing compaction.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
2016-01-19 23:15:18 -02:00
Tomasz Grabiec
bd34adcf22 tests: memory_footprint: Show canonnical_mutation size
Message-Id: <1453227147-21918-1-git-send-email-tgrabiec@scylladb.com>
2016-01-19 20:22:59 +02:00
Tomasz Grabiec
b8c3fa4d46 cql3: Print only column name in error message
Printing column_definition prints all fields of the struct, we want
only name here.
Message-Id: <1453207531-16589-1-git-send-email-tgrabiec@scylladb.com>
2016-01-19 20:22:37 +02:00
Tomasz Grabiec
0596455dc2 Merge branch 'pdziepak/date-timestamp-fixes/v2'
From Paweł:

These patches contain fixes for date and timestamp types:
 - date and timestamp are considered compatible types
 - date type is added to abstract_type::parse_type()
2016-01-19 18:35:09 +01:00
Glauber Costa
63967db8bf sstables: always use a file_*_stream_options in our readers and writes
Instead of using the APIs that explicitly pass things like buffer_size,
always use the options instance instead.

This will make it easier to pass extra options in the future.

Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <5b04e60ab469c319a17a522694e5bedf806702fe.1453219530.git.glauber@scylladb.com>
2016-01-19 18:26:37 +02:00
Glauber Costa
c3ac5257b5 sstables: don't repeat file_writer creation all the time
When this code was originally written, we used to operate on a generic
output_stream. We created a file output stream, and then moved it into
the generic object.

Many patches and reworks later, we now have a file_writer object, but
that pattern was never reworked.

So in a couple of places we have something like this:

    f = file_object acquired by open_file_dma
    auto out = file_writer(std::move(f), 4096);
    auto w = make_shared<file_writer>(std::move(out));

The last statement is just totally redundant. make_shared can create
an object from its parameters without trouble, so we can just pass
the parameter list directly to it.

Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <c01801a1fdf37f8ea9a3e5c52cd424e35ba0a80d.1453219530.git.glauber@scylladb.com>
2016-01-19 18:26:36 +02:00
Calle Wilund
59bf54d59a commitlog_replayer: Modify logging to more match origin
* Match origin log messages
  - Demote per-file printouts to "debug" level.
* Print an all-files stat summary for whole replay (begin/summary)
  - At info level, like origin

Prompted by dtest that expects origin log output.

Message-Id: <1453216558-18359-1-git-send-email-calle@scylladb.com>
2016-01-19 17:19:52 +02:00
Avi Kivity
07e0f0a31f Merge "Support schema changes in batchlog manager" from Tomasz
"We need to be able to replay mutations created using older versions of
the table's schema. frozen_mutation can be only read using the version
it was serialized with, and there is no guarantee that the node will
know this version at the time of replay. Currently versions are kept
in-memory so a node forgets all past versions when it restarts. This
was not implemented yet, replay would fail with exception if the
version is unknown."
2016-01-19 17:17:47 +02:00
Calle Wilund
3f4c8d9eea commitlog_replayer: Modify logging to more match origin
* Match origin log messages
  - Demote per-file printouts to "debug" level.
* Print an all-files stat summary for whole replay (begin/summary)
  - At info level, like origin

Prompted by dtest that expects origin log output.

v2:
* Fixed broken + operator
* Use map_reduce instead of easily readable code
2016-01-19 15:14:21 +00:00
Paweł Dziepak
db30ac8d2d tests/types: add test for timestamp and date compatibility
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2016-01-19 15:34:45 +01:00
Avi Kivity
a2953833dc Merge seastar upstream
* seastar e93cd9d...0516ed0 (9):
  > http: use default file input stream options in file_handler
  > linecount: use default file input stream options
  > fstream: do not pass offset as part options member
  > net: move posix network stack registration to reactor.cc
  > net: throw a human-readable error if using an unregistered network stack
  > io_queue: remove pending_io counter
  > Revert "Merge "Improve rpc server-side statistics""
  > tests: corrections regarding Boost.Test 1.59 compilation failures
  > Merge "Improve rpc server-side statistics"
2016-01-19 16:33:35 +02:00
Calle Wilund
1b4b7aeb66 Merge branch 'master' of https://github.com/scylladb/scylla 2016-01-19 13:51:00 +00:00
Paweł Dziepak
900f5338e7 types: make timestamp_type and date_type compatible
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2016-01-19 14:03:15 +01:00
Tomasz Grabiec
ec12b75426 batchlog_manager: Store canonical_mutations
We need to be able to replay mutations created using older versions of
the table's schema. frozen_mutation can be only read using the version
it was serialized with, and there is no guarantee that the node will
know this version at the time of replay. Currently versions kept
in-memory so a node forgets all past versions when it restarts.

To solve this, let's store canonical_mutations which, like data in
sstables, can be read using any later schema version of given table.
2016-01-19 13:46:28 +01:00
Tomasz Grabiec
e21049328f batchlog_manager: Add more debug logging 2016-01-19 13:46:28 +01:00
Tomasz Grabiec
608b606434 canonical_mutation: Introduce column_family_id() getter 2016-01-19 13:46:28 +01:00
Tomasz Grabiec
06d1f4b584 database: Print table name when printing mutation 2016-01-19 13:46:28 +01:00
Tomasz Grabiec
52073d619c database: Add trace-level logging of applied mutations 2016-01-19 13:46:28 +01:00
Paweł Dziepak
a6171d3e99 types: add date type to parse_type()
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2016-01-19 13:43:36 +01:00
Paweł Dziepak
f77ab67809 types: use correct name for date_type
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2016-01-19 13:42:53 +01:00
Tomasz Grabiec
d7cb88e0af Merge branch 'pdziepak/fixes-for-alter-table/v1'
From Paweł:

"This series contains some more fixes for issues related to alter table,
namely: incorrect parsing of collection information in comparator, missing
schema::_raw._collections in equality check, missing compatibility
information for utf8->blob, ascii->blob and ascii->utf8 casts."
2016-01-19 13:22:10 +01:00
Calle Wilund
de9f9308a5 auth_test: workaround ASan false error
test_password_authenticator_operations causes ASan failures, in a way
that I am 99% sure is fully false positive, caused by a combo of
seastar threads, exception throwing and externals.

In lieu of actually identifying what ASan flaw causes this and
potentially cure it, for now, lets just re-write the test in question
to not use seastar::async, but normal continuation. Less easy to read,
but passes ASan.
Message-Id: <1453205136-10308-1-git-send-email-calle@scylladb.com>
2016-01-19 13:11:20 +01:00
Calle Wilund
79a5f7b19d auth_test: workaround ASan false error
test_password_authenticator_operations causes ASan failures, in a way
that I am 99% sure is fully false positive, caused by a combo of
seastar threads, exception throwing and externals.

In lieu of actually identifying what ASan flaw causes this and
potentially cure it, for now, lets just re-write the test in question
to not use seastar::async, but normal continuation. Less easy to read,
but passes ASan.
2016-01-19 12:02:50 +00:00
Raphael S. Carvalho
0c67b1d22b compaction: filter out mutation that doesn't belong to shard
When compacting sstable, mutation that doesn't belong to current shard
should be filtered out. Otherwise, mutation would be duplicated in
all shards that share the sstable being compacted.
sstable_test will now run with -c1 because arbitrary keys are chosen
for sstables to be compacted, so test could fail because of mutations
being filtered out.

fixes #527.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <1acc2e8b9c66fb9c0c601b05e3ae4353e514ead5.1453140657.git.raphaelsc@scylladb.com>
2016-01-19 10:16:41 +01:00
Vlad Zolotarov
922eb218b1 locator::reconnectable_snitch_helper: don't check messaging_service version
Don't demand the messaging_service version to be the same on both
sides of the connection in order to use internal addresses.

Upstream has a similar change for CASSANDRA-6702 in commit a7cae32 ("Fix
ReconnectableSnitch reconnecting to peers during upgrade").

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Message-Id: <1452686729-32629-1-git-send-email-vladz@cloudius-systems.com>
2016-01-19 11:04:37 +02:00
Paweł Dziepak
7c9708953e tests/cql3: add tests for ALTER TABLE with multiple collections
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2016-01-19 09:39:24 +01:00
Paweł Dziepak
e249d4eab5 tests/type: add test for simple type compatibility
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2016-01-19 09:39:20 +01:00
Paweł Dziepak
440b6d058e types: fix compatibility for text types
bytes_type is_compatible_with utf8_type and ascii_type
utf8_type is_compatible_with ascii_type

Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2016-01-19 09:39:16 +01:00
Paweł Dziepak
17ca7e06f3 schema: print collection info
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2016-01-19 09:39:12 +01:00
Paweł Dziepak
2e2de35dfb schema: add _raw._collections check to operator==()
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2016-01-19 09:39:08 +01:00
Paweł Dziepak
92dc95b73b schema: fix comparator parsing
The correct format of collection information in comparator is:

o.a.c.db.m.ColumnToCollection(<name1>:<type1>, <name2>:<type2>, ...)

not:

o.a.c.db.m.ColumnToCollection(<name1>:<type1>),
o.a.c.db.m.ColumnToCollection(<name2>:<type2>) ...

Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2016-01-19 09:39:05 +01:00
Amnon Heiman
9be42bfd7b API: Add version to application state in failure_detection
The upstream of origin adds the version to the application_state in the
get_endpoints in the failure detector.

In our implementation we return an object to the jmx proxy and the proxy
do the string formatting.

This patch adds the version to the return object which is both useful as
an API and will allow the jmx proxy to add it to its output when we move
forward with the jmx version.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1448962889-19611-1-git-send-email-amnon@scylladb.com>
2016-01-19 10:23:56 +02:00
Tomasz Grabiec
5a1587353f tests: Don't depend on partition_key representation
Representation format is an implementation detail of
partition_key. Code which compares a value to representation makes
assumptions about key's representation. Compare keys to keys instead.
Message-Id: <1453136316-18125-1-git-send-email-tgrabiec@scylladb.com>
2016-01-18 19:01:56 +02:00
Pekka Enberg
2ca8606b4e streaming/stream_session: Don't stop stream manager
We cannot stop the stream manager because it's accessible via the API
server during shutdown, for example, which can cause a SIGSEGV.

Spotted by ASan.
Message-Id: <1453130811-22540-1-git-send-email-penberg@scylladb.com>
2016-01-18 16:34:19 +01:00
Pekka Enberg
422cff5e00 api/messaging_service: Fix heap-buffer-overflows in set_messaging_service()
Fix various issues in set_messaging_service() that caused
heap-buffer-overflows when JMX proxy connects to Scylla API:

  - Off-by-one error in 'num_verb' definition

  - Call to initializer list std::vector constructor variant that caused
    the vector to be two elements long.

  - Missing verb definitions from the Swagger definition that caused
    response vector to be too small.

Spotted by ASan.
Message-Id: <1453125439-16703-1-git-send-email-penberg@scylladb.com>
2016-01-18 15:43:29 +01:00
Pekka Enberg
3723beb302 service/storage_service: Fix typos in logger messages
Message-Id: <1453128076-18613-1-git-send-email-penberg@scylladb.com>
2016-01-18 15:43:04 +01:00
Takuya ASADA
d5d5857b62 dist: extend coredump size limit
16GB is not enough for some larger machines, so extend it.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1453115792-21989-2-git-send-email-syuu@scylladb.com>
2016-01-18 13:38:43 +02:00
Takuya ASADA
023c6dc620 dist: preserve environment variable when running scylla_prepare on sudo
sysconfig parameters are passed via environment variables, but sudo resets it by default.
Need to keep them beyond sudo.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1453115792-21989-1-git-send-email-syuu@scylladb.com>
2016-01-18 13:22:14 +02:00
Gleb Natapov
dde2e80a20 storage_proxy: remove batchlog synchronously
Wait for batchlog removal before completing a query otherwise batchlog
removal queries may accumulate. Still ignore an error if it happens
since it is not critical, but log it.

Message-Id: <20160118095642.GB6705@scylladb.com>
2016-01-18 12:38:12 +02:00
Avi Kivity
221ef4536c messaging service: limit rpc server resources
Otherwise, a slow node can be overwhelmed by other nodes and run out of
memory.

Fixes #596.
Message-Id: <1452776394-13682-1-git-send-email-avi@scylladb.com>
2016-01-18 11:16:45 +02:00
Avi Kivity
a881e596fa Merge "Ubuntu dependency packages fix" from Takuya 2016-01-18 11:13:18 +02:00
Gleb Natapov
f97eed0c94 fix batch size checking
warn_threshold is in kbytes v.size is in bytes size is in kbytes.

Message-Id: <20160118090620.GZ6705@scylladb.com>
2016-01-18 11:08:13 +02:00
Avi Kivity
5313a28044 Merge "Fix re-addinig collections" from Paweł
"This series makes sure that Scylla rejects adding a collections if
its column name is the same as a collection that existed before and
their types are incompatible.

Fixes #782"
2016-01-18 10:58:40 +02:00
Tomasz Grabiec
237819c31f logalloc: Excluded zones' free segments in lsa/byres-non_lsa_used_space
Historically the purpose of the metric is to show how much memory is
in standard allocations. After zones were introduced, this would also
include free space in lsa zones, which is almost all memory, and thus
the metric lost its original meaning. This change brings it back to
its original meaning.

Message-Id: <1452865125-4033-1-git-send-email-tgrabiec@scylladb.com>
2016-01-18 10:48:14 +02:00