Commit Graph

5010 Commits

Author SHA1 Message Date
Tomasz Grabiec
440962dbbf tests: Run mutation source tests on memtable 2015-07-22 13:14:33 +02:00
Tomasz Grabiec
f9da612581 memtable: Implement range queries 2015-07-22 13:14:33 +02:00
Tomasz Grabiec
9c4956c5dc memtable: Use boost::intrusive_set<> to store partition entries
So that we can use heterogenous comparators. For range queries we will
need to compare keys with ring_position.
2015-07-22 13:14:33 +02:00
Tomasz Grabiec
0f3588708e tests: Extract range query tests from sstable_mutation_test into mutation_source_test
The idea is to reuse the same testing code on any mutation_source, for
example on memtable.

The range query test cases are now part of a generic mutation_source
test suite.
2015-07-22 13:14:33 +02:00
Tomasz Grabiec
33bcf9e8d3 tests: Make reader_assertions::produces() work on any range 2015-07-22 13:14:10 +02:00
Tomasz Grabiec
c8f3471055 sstable_mutation_tests: Add tests for inclusive/exclusive bounds with keys and tokens 2015-07-22 13:14:07 +02:00
Tomasz Grabiec
59b91fe350 mutation: Introduce slice() helper 2015-07-22 13:13:38 +02:00
Tomasz Grabiec
4248d87544 Introduce mutation_decorated_key_less_comparator 2015-07-22 13:13:38 +02:00
Tomasz Grabiec
152582a869 sstables: Add read_range_rows() variant which takes a partition_range 2015-07-22 13:13:38 +02:00
Tomasz Grabiec
6373987704 partition_range: Introduce is_wrap_around() helper 2015-07-22 13:13:38 +02:00
Tomasz Grabiec
5fe7c1093f sstables: Make mutation_reader::impl unmovable and uncopyable 2015-07-22 13:13:38 +02:00
Tomasz Grabiec
7aea858108 sstables: Make data_consume_rows(0, 0) return no rows
data_consume_rows(0, 0) was returning all partitions instead of no
partitions, because -1 was passed as count in such case, which was
then casted to uint64_t.

Special-casing it that way is problematic for code which calculates
the bounds, and when the key is not found we simple end up with 0 as
upper bound. Instead of convoluting the range lookup code to special
case for 0, let's simplify the interface so that (0, 0) returns no
rows, same as (1, 1). There is a new overload of data_consume_rows()
without bounds, which returns all data.
2015-07-22 13:10:01 +02:00
Tomasz Grabiec
3de682ebeb tests: sstable_data_file_test: Fix use-after-free on reader 2015-07-22 13:10:00 +02:00
Tomasz Grabiec
f68b771927 sstables: Use lower_bound() and upper_bound() to search the partition index
I will need those abstractions later to handle
inclusiveness/exclusiveness of both staring and ending bounds.

They're also familiar abstractions, so the code is hopefully easier to
comprehend now.
2015-07-22 10:27:48 +02:00
Tomasz Grabiec
ff0308104c sstables: Add data_end_position() on summary page level 2015-07-22 10:27:48 +02:00
Tomasz Grabiec
e9a050da78 sstables: Obtain the key from entries using get_key() rather than casting to bytes_view
The entry contains not only the key, but other stuff like
position. Why would casting to bytes_view give the view on just the
key and not the whole entry. Better to be explicit.
2015-07-22 10:27:48 +02:00
Tomasz Grabiec
0b5f908a0b sstables: Make key_view comparable with partition_key_view 2015-07-22 10:27:48 +02:00
Tomasz Grabiec
73ccd51cc5 sstables: Add key_view::tri_compare() 2015-07-22 10:27:48 +02:00
Tomasz Grabiec
6882e6ca50 storage_proxy: Fix range splitting when split point is exlcluded by lower bound
If we had a range (x; ...] then x is excluded, but token iterator was
initialized with x. The splitting loop would exit prematurely because
it would detect that the token is outside the range.

The fix is to teach ring_range() to recognize this and always give
tokens which are not smaller than the range's lower bound.
2015-07-22 10:27:48 +02:00
Tomasz Grabiec
c219c3676b storage_proxy: Avoid unnecessary copy of a partition_range 2015-07-22 10:27:48 +02:00
Tomasz Grabiec
abdb36a0f6 storage_proxy: Fix ordering comparator used during merging and reconciliation 2015-07-22 10:27:48 +02:00
Tomasz Grabiec
bad7602299 keys: Introduce ring_order_tri_compare() 2015-07-22 10:27:48 +02:00
Tomasz Grabiec
9b52f5bf2b storage_proxy: Make range splitting never produce a wrap around range
Origin has no notion of a maximum token so a range without upper bound
is represented as (x; min]. The splitting code is supposed to produce
only non-wrapping ranges, but (x; min] looks like a wrapping range, so
database code which consumes it would have to special-case for it. A
simpler solution is to change the splitting code to never produce a
wrapping range.
2015-07-22 10:27:48 +02:00
Tomasz Grabiec
f557951554 storage_proxy: Encapsulate access to range tokens
The wrappers also take care of cases when each bound is undefined, and
return mimimum or maximum token respectively, which fixes undefined
behavior in get_restricted_ranges().

Alternative solution would be to make partition_range use a different
type of range, the one where bounds are always specified. However it's
not worth introducing a new range type just for those few users.
2015-07-22 10:27:48 +02:00
Tomasz Grabiec
5587d2ce9a storage_proxy: Simplify access to storage_proxy::response_id_type
storage_proxy::storage_proxy::response_id_type
 -> storage_proxy::response_id_type
2015-07-22 10:27:48 +02:00
Tomasz Grabiec
0b0ea04958 range: Remove start_value() and end_value()
It's easy to miss that they may be undefined. start() and end(), which
return optional<bound> const&, make it clear.
2015-07-22 10:27:47 +02:00
Tomasz Grabiec
4a18693a23 db: Remove dead code 2015-07-22 10:27:47 +02:00
Raphael S. Carvalho
89698b0d1c db: dont rethrow exceptions for termination of compaction fiber
broken_semaphore and seastar::gate_closed_exception exceptions are
used for regular termination of compaction fiber, which otherwise
would live forever. We shouldn't re-throw these exceptions, but
instead only print a log message.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
2015-07-22 11:23:58 +03:00
Asias He
547d9c347e storage_service: Simplfy do_update_system_peers_table
Using template helper.
2015-07-22 11:22:18 +03:00
Avi Kivity
f2370725f5 Merge "Enable logger for gossip and storage_service" from Asias 2015-07-22 11:12:36 +03:00
Avi Kivity
0c4430cffc Merge "streaming: Support both pushing and pulling of data"
"stream_plan.transfer_ranges() sends data from local to remote node.
stream_plan.request_ranges() asks remote to send data to local.

After streaming, both nodes contains all the keys."
2015-07-22 10:13:10 +03:00
Paweł Dziepak
fc44658fa7 transport: add varint to type_codec::type_id_to_type
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
2015-07-22 08:50:54 +03:00
Asias He
54d482afe4 streaming: Test both pushing and pulling of data
stream_plan.transfer_ranges() sends data from local to remote node.

stream_plan.request_ranges() asks remote to send data to local.

After streaming, both nodes contains all the keys.

$ cat /tmp/out1|grep "\[Stream"
[Stream #9fd8c3c0-3023-11e5-b450-000000000000] Executing streaming plan for MYPLAN
[Stream #9fd8c3c0-3023-11e5-b450-000000000000] Starting streaming to 127.0.0.2
[Stream #9fd8c3c0-3023-11e5-b450-000000000000] Sending stream init for incoming stream
[Stream #9fd8c3c0-3023-11e5-b450-000000000000 ID#0] Beginning stream session with 127.0.0.2
[Stream #9fd8c3c0-3023-11e5-b450-000000000000 ID#0] Prepare completed.  Receiving 1 files(105553124400080 bytes), sending 1 files(105553124104160 bytes)
[Stream #9fd8c3c0-3023-11e5-b450-000000000000] Session with 127.0.0.2 is complete
[Stream #9fd8c3c0-3023-11e5-b450-000000000000] All sessions completed

$ cat /tmp/out2|grep "\[Stream"
[Stream #9fd8c3c0-3023-11e5-b450-000000000000 ID#0] Creating new streaming plan for MYPLAN
[Stream #9fd8c3c0-3023-11e5-b450-000000000000 ID#0] Received streaming plan for MYPLAN
[Stream #9fd8c3c0-3023-11e5-b450-000000000000 ID#0] Prepare completed.  Receiving 1 files(105553124104160 bytes), sending 1 files(105553124400080 bytes)
[Stream #9fd8c3c0-3023-11e5-b450-000000000000] Session with 127.0.0.1 is complete
[Stream #9fd8c3c0-3023-11e5-b450-000000000000] All sessions completed

Node 1
$ sstable2json tmp/1/ks/*/la-1-big-Data.db | grep key | sort
{"key": "1",
{"key": "2",
{"key": "3",
{"key": "4",
{"key": "5",
{"key": "6",

Node 2
$ sstable2json tmp/2/ks/*/la-1-big-Data.db | grep key | sort
{"key": "1",
{"key": "2",
{"key": "3",
{"key": "4",
{"key": "5",
{"key": "6",
2015-07-22 11:49:30 +08:00
Asias He
96049a99cf streaming: Add more debug print for stream_session::prepare 2015-07-22 11:49:30 +08:00
Asias He
1c60844727 streaming: Always start_streaming_files upon receiving of PREPARE_MESSAGE reply 2015-07-22 11:49:30 +08:00
Asias He
736c0bc08f streaming: Improve query::range<token> deserialization a bit
query::range<token> is not serialized ATM. Always use full range for
now.
2015-07-22 11:49:30 +08:00
Asias He
d39dd0f9d1 streaming: Add operator<< for stream_request 2015-07-22 11:49:30 +08:00
Asias He
d934b2c761 streaming: Fix prepare_message and stream_request deserialization
vector(size_type count) constructs the container with count
default-inserted instances of T. So, current code will end up with 2*num
elements which is wrong.
2015-07-22 11:49:30 +08:00
Asias He
f83d0bf7c4 streaming: Fix info printout format 2015-07-22 11:49:30 +08:00
Avi Kivity
6d2278f345 build: make libseastar.a depend on seastar configuration
Should cause a rebuild if seastar configuration changes, for example due
to --enable-dpdk.
2015-07-21 19:22:48 +03:00
Avi Kivity
006473b45c tests: improve capture of stdout/stderr
Use a single pipe for both, instead of a temporary file.
2015-07-21 18:44:31 +03:00
Avi Kivity
675a1ee8ea Merge "Add varint type" from Paweł
"This patchset adds support for varint type."
2015-07-21 18:21:21 +03:00
Asias He
ff457b018a gossip: Enable logger in gossiper class 2015-07-21 22:56:24 +08:00
Asias He
7ee4ae2ff7 storage_service: Use logger.error instead of print 2015-07-21 22:56:24 +08:00
Asias He
94e34cde64 storage_service: Drop ss_debug 2015-07-21 22:56:24 +08:00
Asias He
72a8c27b56 storage_service: Enable logger 2015-07-21 22:56:24 +08:00
Glauber Costa
0ea8a27b47 system_keyspace: stop the local cache.
Apparently we need to register the stop function explicitly. That was not being
visible before, because the urchin binary was failing on exit before this for
some other reason. Once that was fixed, this one became apparent.

Scylla can correctly shutdown now.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
2015-07-21 17:21:53 +03:00
Avi Kivity
cbc44d19cb build: fix build with dpdk
Rip out the seastar-related dpdk configuration, replace with simply
forwarding dpdk configuration to seastar.
2015-07-21 17:19:01 +03:00
Avi Kivity
4991d505c3 Merge seastar upstream 2015-07-21 17:18:46 +03:00
Paweł Dziepak
e24aa67177 tests/cql3: add varint to test_types
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
2015-07-21 14:47:17 +02:00