Commit Graph

11716 Commits

Author SHA1 Message Date
Tomasz Grabiec
0798ea22c8 tests: mutation_source_tests: Verify monotonicty of positions 2017-02-23 18:50:54 +01:00
Tomasz Grabiec
d0421ba545 tests: random_mutation_generator: Spread the keys more
The deviation was very low so most ranges were very close. Spread them
to test more cases.
2017-02-23 18:50:54 +01:00
Tomasz Grabiec
27ff169b6b tests: mutation_source_test: Make blobs more easily distinguishable
It's easier to compare them if they differ only by a few most
significant bits, than by all bits.
2017-02-23 18:50:53 +01:00
Tomasz Grabiec
182e3f981b tests: streamed_mutation: Test that merged stream passes mutation source tests 2017-02-23 18:50:53 +01:00
Tomasz Grabiec
122562c1cc tests: mutation_source_test: Add tests for forwarding of streamed_mutation 2017-02-23 18:50:53 +01:00
Tomasz Grabiec
1d7e84f770 tests: streamed_mutation_assertions: Add methods for navigating the stream 2017-02-23 18:50:53 +01:00
Tomasz Grabiec
f2feb54fb0 tests: Add range generators to random_mutation_generator 2017-02-23 18:50:53 +01:00
Tomasz Grabiec
f56308597c partition_slice_builder: Add with_ranges() 2017-02-23 18:50:53 +01:00
Tomasz Grabiec
0073df30aa query: Introduce full_clustering_range 2017-02-23 18:50:53 +01:00
Tomasz Grabiec
cbf4601e31 streamed_mutation: Add non-owning variant of mutation_from_streamed_mutation() 2017-02-23 18:50:53 +01:00
Tomasz Grabiec
892d4a2165 db: Enable creating forwardable readers via mutation_source
Right now all mutation source implementations will use
make_forwardable() wrapper.
2017-02-23 18:50:44 +01:00
Tomasz Grabiec
b1d1091906 mutation_source: Document liveness requirements 2017-02-23 18:23:52 +01:00
Tomasz Grabiec
15db80188b mutation_source: Cleanup
- combines telescopic overloads into one method with default paramters.
 - Introduce func_type for a full handler to avoid some duplication.
2017-02-23 18:23:52 +01:00
Tomasz Grabiec
586dbaa8d3 db: Replace virtual_reader_type with mutation_source_opt
Virtual reader is a mutation_source.
2017-02-23 18:23:52 +01:00
Tomasz Grabiec
acfad565f0 partition_version: Refactor make_partition_snapshot_reader() overloads
So that streamed_mutation is created in only one of the overloads and
others delegate to that one. Later there will be common logic added to
the construction and doing this will help avoid a duplication.
2017-02-23 18:23:52 +01:00
Tomasz Grabiec
f46ae8128d database: Fix mutation_source created by as_mutation_source() to not ignore trace_state_ptr
It was using the state passed via as_mutation_source() instead. Let's
respect mutation_source contract instead, and use the state passed via
mutation_source invocation.

Technically just a cleanup. Alse prerequisite for more cleanup.
2017-02-23 18:23:52 +01:00
Tomasz Grabiec
2cc27f72ca memtable: Accept all mutation_source parameters 2017-02-23 18:23:52 +01:00
Tomasz Grabiec
53b1a257cc streamed_mutation: Implement fast_forward_to() in stream merger 2017-02-23 18:23:52 +01:00
Tomasz Grabiec
e0a7ed48b0 streamed_mutation: Add generic implementation of forwardable streamed_mutation
Generic but not very efficient wrapper which simply drops
fragments from the original stream.
2017-02-23 18:23:51 +01:00
Tomasz Grabiec
301cd4912b streamed_mutation: Add fast_forward_to() API 2017-02-23 18:23:28 +01:00
Gleb Natapov
2dc56013f8 commitlog: handle cycle() error
Do not ignore a future<> retuned by cycle() since it will produce a
warning in case of an error. Log it instead.

Message-Id: <20170219151811.GN11471@scylladb.com>
2017-02-22 19:15:14 +01:00
Calle Wilund
d5f57bd047 messaging_service: Move log printout to actual listen start
Fixes  #1845
Log printout was before we actually had evaluated endpoint
to create, thus never included SSL info.
Message-Id: <1487766738-27797-1-git-send-email-calle@scylladb.com>
2017-02-22 17:08:21 +01:00
Avi Kivity
9b113ffd3e config: enable new sharding algorithm for new deployments
Set murmur3_partitioner_ignore_msb_bits to 12 (enabling the new sharding
algorithm), but do this in scylla.yaml rather than the built-in defaults.
This avoids changing the configuration for existing clusters, as their
scylla.yaml file will not be updated during the upgrade.
Message-Id: <20170214123253.3933-1-avi@scylladb.com>
2017-02-22 11:23:12 +01:00
Calle Wilund
0a4edca756 counters/cql: allow wormholing actual counter values (with shards) via cql
Adds yet another magic function "SCYLLA_COUNTER_SHARD_LIST", indicating that
argument value, which must be a list of tuples <int, UUID, long, long>,
should be inserted as an actual counter value, not update.

This of course to allow counters to be read from sstable loader.

Note that we also need to allow timestamps for counter mutations,
as well as convince the counter code itself to treat the data as
already baked. So ugly wormhole galore.

v2:
* Changed flag names
* More explicit wormholing, bypassing normal counter path, to
  avoid read-before-write etc
* throw exceptions on unhandled shard types in marshalling
v3:
* Added counter id ordering check
* Added batch statement check for mixing normal and raw counter updates
Message-Id: <1487683665-23426-2-git-send-email-calle@scylladb.com>
2017-02-22 09:19:46 +00:00
Calle Wilund
0d87f3dd7d utils::UUID: operator< should behave as comparison of hex strings/bytes
I.e. need to be unsigned comparison.
Message-Id: <1487683665-23426-1-git-send-email-calle@scylladb.com>
2017-02-22 09:19:22 +00:00
Tomasz Grabiec
2b2d5c4c7a Update seastar submodule
* seastar 5088065...4d4a58d (3):
  > reactor utilization should return the utilization in 0-1 range
  > collectd should ignore type label in name creation
  > fix append_challenged_posix_file_impl::process_queue() to handle recursion
2017-02-22 09:40:25 +01:00
Calle Wilund
e20b804a65 commitlog/database: Add "release" method to ensure we free segments
On database stop, we do flush memtables and clean up commit log segment usage.
However, since we never actually destroy the distributed<database>, we
don't actually free the commitlog either, and thus never clear out
the remaining (clean) segments. Thus we leave perfectly clean segments
on disk.

This just adds a "release" method to commitlog, and calls it from
database::stop, after flushing CF:s.
Message-Id: <1485784950-17387-1-git-send-email-calle@scylladb.com>
2017-02-21 18:17:47 +01:00
Gleb Natapov
0977f4fdf8 sstable: close sstable_writer's file if writing of sstable fails.
Failing to close a file properly before destroying file's object causes
crashes.

[tgrabiec: fixed typo]

Message-Id: <20170221144858.GG11471@scylladb.com>
2017-02-21 18:17:47 +01:00
Tomasz Grabiec
8fd19a71ff position_in_partition: Introduce position_range 2017-02-21 16:49:36 +01:00
Tomasz Grabiec
78c563ea6a position_in_partition: Introduce position constructor for right after the static row 2017-02-21 16:43:09 +01:00
Tomasz Grabiec
ce58706b50 streamed_mutation: Make cast to view non-explicit 2017-02-21 16:43:09 +01:00
Paweł Dziepak
274bcd415a tests/cql_test_env: wait for storage service initialization
Message-Id: <20170221121130.14064-1-pdziepak@scylladb.com>
2017-02-21 17:05:45 +02:00
Paweł Dziepak
359c617821 db: restore call to check_valid_rp()
5a0955e89d "db: add operations for
applying counter updates" merged two column_family::apply() overloads
into do_apply() in order to reduce code duplication. Unfortunately,
a call to check_valid_rp() didn't survive that change.
Message-Id: <20170221133800.30411-1-pdziepak@scylladb.com>
2017-02-21 15:26:04 +01:00
Tomasz Grabiec
b4fd3a08e6 streamed_mutation: Make schema() getter non-copying 2017-02-21 14:18:57 +01:00
Duarte Nunes
65b21e3a99 schema_registry: Don't leak schemas
When loading a schema asynchronously, we're leaving a strong
reference to the loaded schema in the entry's shared future. This
patch fixed this by storing a shared_promised, which is reset when the
schema is loaded.

Signed-off-by: Duarte Nunes <duarte@scylladb.com>
Message-Id: <20170220193654.17439-1-duarte@scylladb.com>
2017-02-21 09:56:21 +01:00
Tomasz Grabiec
33457cc9a9 sstables: Fix detection of repeated tombstones
The check was not catching range tombstone repeated immediately after
itself.
Message-Id: <1487596098-17409-1-git-send-email-tgrabiec@scylladb.com>
2017-02-20 15:35:15 +00:00
Tomasz Grabiec
cc439df542 Revert "sstables: Simplify sstable_streamed_mutation::read_next()"
This reverts commit 1e2c01ff49.

We do not detect repeated tombstone if it follows an in-range
tombstone following a skipped clustering row, because _in_progress
will be disengaged after such tombstone is emitted.
Message-Id: <1487596080-21480-1-git-send-email-tgrabiec@scylladb.com>
2017-02-20 15:34:58 +00:00
Vlad Zolotarov
978241d473 database: move lister class into separate files
Move lister class away from database.cc.
This is a preparation for moving it to the seastar library.

Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
2017-02-17 17:50:40 -05:00
Vlad Zolotarov
34cafa71c3 database: make 'clearsnapshot' to delete the snapshots of deleted keyspaces if requested
The current implementation of 'nodetool clearsnapshot' command only deletes the snapshots
of the keyspaces that are alive at the time the command is issued (issue #2045).

This, besides not implementing the spec, prevents users from being able to clear
the disk space occupied by snapshots of deleted keyspaces that are no longer needed
(e.g. snapshots created when KS is deleted).

This patch fixes this issue by making the database::clear_snapshot() scan the data directories
looking for the snapshots to be deleted instead of relying on in-memory data structures.

This patch makes column_family::clear_snapshot() method not needed any more.

Fixes #2045

Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
2017-02-17 17:50:40 -05:00
Vlad Zolotarov
e1ee669aff database: lister: add the rmdir() static method
Removes the directory with all its contents (like 'rm -rf <dir name>' shell command).

Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
2017-02-17 17:50:40 -05:00
Vlad Zolotarov
53532ba5ff database: lister: pass the parent path object to callbacks
Pass a parent directory boost::filesystem::path object
to the walker and filter callbacks.

Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
2017-02-17 17:50:37 -05:00
Vlad Zolotarov
b4c970dfc6 database: lister: make the "filter" callback receive directory_entry instead of sstring
Filter should get all information that the caller has in hand that may be used for filtering.

directory_entry has the following information:
   - Type of the entry
   - Its name

For the code that used lister filters so far this would be enough, however it's not
hard to imagine a filter that may need the parent directory as well.

We will add the parent directory path in the follow up patches to make the interface
complete.

Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
2017-02-17 17:46:59 -05:00
Vlad Zolotarov
6f9f0e1b3f database: lister: add "show_hidden" parameter
If show_hidden parameter is set to show_hidden::yes - list hidden entries, otherwise skip them.
By default set to show_hidden::no.

This patch also completely removes default parameters in lister::scan_dir() and replaces them
with a few lister::scan_dir() overloads that ensure that lambdas are always going to
be the last parameter in the parameters list.

Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
2017-02-17 17:46:58 -05:00
Vlad Zolotarov
9aedb191f6 database: lister: if entries' types set is empty - list everything
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
2017-02-17 17:46:58 -05:00
Vlad Zolotarov
cb614f9be4 database: lister::guarantee_type: handle the case when entry type may not be read
There is a possibility that the type of the given entry may not be available
that would manifest in the ENOENT or ENOTDIR value set in the errno
by the fstat() call for this entry. In this case engine().file_type() will return
a not engaged optional<directory_entry_type> value.

Return the future with the std::runtime_error exception in this case.
This will prevent any further usage of the not engaged optional value by the code
in the normal flow.

The exception is going to be propagated to the caller and it's the caller's responsibility
to handle it.

Fixes #2071

Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
2017-02-17 17:46:55 -05:00
Tomasz Grabiec
9f63e172fb tests: compaction_manager_test: Fix abort on exception
Message-Id: <1487343901-12745-1-git-send-email-tgrabiec@scylladb.com>
2017-02-17 15:53:55 +00:00
Avi Kivity
113ed9e963 Merge seastar upstream
* seastar 28a143a...5088065 (8):
  > configure.py: switch cmake to build c-ares to do out-of-source-tree build
  > iotune: make sure help is working
  > collectd: send double correctly for gauge
  > tls: make shutdown/close do "clean" handshake shutdown in background
  > tls: Make sink/source (i.e. streams) first class channel owners
  > native-stack: Make sink/source (i.e. streams) first class channel owners
  > posix-stack: Make sink/source (i.e. streams) first class channel owners
  > Merge "Detector for tasks blocked" from Glauber

Fixes #2085.

Packaging updated to require cmake, drop libtool and automake.
2017-02-16 19:34:28 +02:00
Vlad Zolotarov
25502149cf database: lister::scan_dir(): std::move() all that needs to be moved
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
2017-02-16 11:56:44 -05:00
Raphael S. Carvalho
53d9008052 sstables/deletion_manager: kill dead code
Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <2b24d9e622238030a737fbbe12b8439853d5d075.1487095059.git.raphaelsc@scylladb.com>
2017-02-16 18:38:54 +02:00
Vlad Zolotarov
f2e4629254 main.cc: expose scylla version as a gauge metrics
Add a new metric that exposes the current ScyllaDB version as a
gauge metrics.

The version is exposed as a label with the "version" key.

Fixes #1979

Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Message-Id: <1487083703-27929-1-git-send-email-vladz@scylladb.com>
2017-02-16 16:57:55 +02:00