Commit Graph

6948 Commits

Author SHA1 Message Date
Tomasz Grabiec
9dbd5a92d0 partition_slice_builder: Introduce reversed() 2015-10-22 10:32:08 +02:00
Calle Wilund
412c2a1e5b storage_proxy: mutate_atomically fix for consistency and BL removal
The change to use consistency_level::ONE in send_batchlog_mutation
sort of fixes #478, but is not 100% correct.
When doing async_remove_from_batchlog, the CL is actually supposed to
be ANY.

Also, we should _not_ remove the batch log mutation from any nodes
if the mutate fails, since having it there in case of failure is sort of
the whole point of it. I.e. async_remove_from_batchlog should not be
called from a "finally", but from a "then".

Refs #478
2015-10-21 16:48:27 +02:00
Tomasz Grabiec
764d913d84 Merge branch 'pdziepak/row-cache-range-query/v4' from seastar-dev.git
From Pawel:

This series enables row cache to serve range queries. In order to achieve
that row cache needs to know whether there are some other partitions in
the specified range that are not cached and need to be read from the sstables.
That information is provied by key_readers, which work very similarly to
mutation_readers, but return only the decorated key of partitions in
range. In case of sstables key_readers is implemented to use partition
index.

Approach like this has the disadvantage of needing to access the disk
even if all partitions in the range are cached. There are (at least) two
solutions ways of dealing with that problem:
 - cache partition index - that will also help in all other places where it
   is neededed
 - add a flag to cache_entry which, when set, indicates that the immediate
   successor of the partition is also in the cache. Such flag would be set
   by mutation reader and cleared during eviction. It will also allow
   newly created mutations from memtable to be moved to cache provided that
   both their successors and predecessors are already there.

The key_reader part of this patchsets adds a lot of new code that probably
won't be used in any other place, but the alternative would be to always
interleave reads from cache with reads from sstables and that would be
more heavy on partition index, which isn't cached.

Fixes #185.
2015-10-21 15:26:45 +02:00
Gleb Natapov
6a2a0d628b storage_proxy: use CL=ONE to write logged batch
This is a regression created by logged batch code rework.

Fixes #478.
2015-10-21 15:29:49 +03:00
Avi Kivity
c69c02c162 Merge 2015-10-21 15:17:32 +03:00
Avi Kivity
c49dd5c576 Merge "move dependencies to /opt/scylladb" from Takuya 2015-10-21 15:17:04 +03:00
Avi Kivity
16006949d0 logalloc: make migrator an object, not a function pointer
The migrator tells lsa how to move an object when it is compacted.
Currently it is a function pointer, which means we must know how to move
the object at compile time.  Making it an object allows us to build the
migration function at runtime, making it suitable for runtime-defined types
(such as tuples and user-defined types).

In the future, we may also store the size there for fixed-size types,
reducing lsa overhead.

C++ variable templates would have made this patch smaller, but unfortunately
they are only supported on gcc 5+.
2015-10-21 11:24:56 +02:00
Avi Kivity
e2cd40e3bc Merge "remove and decommission node support part 2" from Asias
"More preparatory patches for remove and decommission node support:

- stream hints and reanges
- unbootstrap
- replication finished notification"
2015-10-21 12:24:14 +03:00
Takuya ASADA
1bf18679bb dist: add more build dependency for binutils
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-10-21 09:02:40 +00:00
Takuya ASADA
613c7e8226 dist: use scylla-* dependency packages on scylla-server.spec
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-10-21 08:49:34 +00:00
Takuya ASADA
29847edd4e dist: move antlr3-C++-devel package to /opt/scylladb
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-10-21 08:49:34 +00:00
Takuya ASADA
ad80845a1d dist: move antlr3-tool package to /opt/scylladb
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-10-21 08:49:34 +00:00
Takuya ASADA
cb18a305db dist: move ragel package to /opt/scylladb
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-10-21 08:49:34 +00:00
Takuya ASADA
b4daf31fcf dist: move ninja-build package to /opt/scylladb, remove unused dependency for re2c
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-10-21 08:49:34 +00:00
Takuya ASADA
91005ba366 dist: move boost package to /opt/scylladb
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-10-21 08:49:34 +00:00
Takuya ASADA
2e379179d3 dist: move gcc package to /opt/scylladb
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-10-21 08:49:34 +00:00
Takuya ASADA
09099081e1 dist: move isl package to /opt/scylladb
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-10-21 08:49:34 +00:00
Takuya ASADA
18a6c25434 dist: move binutils package to /opt/scylladb
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-10-21 08:49:34 +00:00
Takuya ASADA
90666cf558 dist: add scylla-env package for CentOS, to use /opt/scylladb as prefix
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-10-21 08:49:34 +00:00
Takuya ASADA
659624a2ee dist: dependency fix for ninja-build
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-10-21 08:49:34 +00:00
Takuya ASADA
7340a5a83a dist: fix srpm not found error
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
2015-10-21 08:49:34 +00:00
Asias He
1271ad6894 storage_service: Implement send_replication_notification 2015-10-21 16:11:33 +08:00
Asias He
56e55cd272 storage_proxy: Register replication_finished verb handler 2015-10-21 16:11:33 +08:00
Asias He
ffce7a7af8 storage_service: Implement confirm_replication 2015-10-21 16:11:33 +08:00
Asias He
1965e8751b messaging_service: Add REPLICATION_FINISHED verb
It is used to send replication finished message by storage_service when
removing a node from a cluster.
2015-10-21 16:11:33 +08:00
Asias He
d903bd0dba storage_service: Implement on_leave_cluster in excise 2015-10-21 14:55:01 +08:00
Asias He
4f57f0cdae storage_service: Enable restore_replica_count in remove_node 2015-10-21 14:55:01 +08:00
Asias He
5b170d1ffe storage_service: Complete handle_state_removing
Implement #if 0'ed code.
2015-10-21 14:55:01 +08:00
Asias He
7d656fe127 storage_service: Enable add_leaving_endpoint in handle_state_leaving 2015-10-21 14:55:01 +08:00
Asias He
d3120b3c2f storage_service: Complete is_replacing logic in handle_state_normal 2015-10-21 14:55:01 +08:00
Asias He
434a9e211e storage_service: Kill one FIXME in handle_state_bootstrap
It is already fixed.
2015-10-21 14:55:01 +08:00
Asias He
6f8f4816a5 storage_service: Implement unbootstrap
All the missing functions for unbootstrap are ready, we can implement
unbootstrap now.
2015-10-21 14:55:01 +08:00
Asias He
8a9374b331 storage_service: Implement stream_hints
Needed by unbootstrap.
2015-10-21 14:55:01 +08:00
Asias He
3f5e9baa17 storage_service: Implement stream_ranges
Needed by unbootstrap.
2015-10-21 14:55:01 +08:00
Asias He
d1eaccd234 storage_service: Implement leave_ring
Needed by unbootstrap.
2015-10-21 14:55:01 +08:00
Asias He
2f86feb581 storage_service: Move send_replication_notification to source file 2015-10-21 14:55:01 +08:00
Avi Kivity
5453cfbab7 Merge "snapshots: take + clear" from Glauber
"This is the code for taking a snapshot, and clearing a snapshot."
2015-10-21 08:59:42 +03:00
Paweł Dziepak
2876c9ebc7 unimplemented: drop RANGE_QUERIES
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-10-20 20:27:53 +02:00
Paweł Dziepak
c2b53c5282 row_cache: add scanning_and_populating_reader
This reader enables range queries on row cache. An underlying key_reader
is used to obtain information about partitions that belong to the
specified range and if any of them isn't in the cache an underlying
mutation reader is used to read the missing data.

Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-10-20 20:27:53 +02:00
Paweł Dziepak
9abefbfa28 row_cache: add just_cache_scanning_reader
This mutation reader returns mutations from cache that are in a given
range. There may be other mutations in the system (e.g. in sstables)
that won't be returned, so this reader on its own cannot really satisfy
any query.

Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-10-20 20:27:53 +02:00
Paweł Dziepak
c765b38599 row_cache: add modification counter
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-10-20 20:27:53 +02:00
Paweł Dziepak
c1e95dd893 row_cache: pass underlying key_source to row_cache
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-10-20 20:27:53 +02:00
Paweł Dziepak
6b73d6bb36 row_cache: add cache_entry::ring_position_compare
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-10-20 20:27:53 +02:00
Paweł Dziepak
96a42a9c69 column_family: add sstables_as_key_source()
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-10-20 20:27:53 +02:00
Paweł Dziepak
b0edaa5bb7 memtable: add as_key_source()
Needed only for tests.

Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-10-20 20:27:53 +02:00
Paweł Dziepak
68e1b1f613 tests/sstables: test sstable key_reader
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-10-20 20:27:47 +02:00
Paweł Dziepak
7fe6d2dc07 sstables: provide key_reader
In order to get just partition keys stored in the sstable its summary
and index are used.

Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-10-20 20:27:47 +02:00
Paweł Dziepak
ca278e6b52 tests/key_reader: add tests for key_readers
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-10-20 20:27:47 +02:00
Paweł Dziepak
7c480f29c3 key_reader: add filtering key reader
Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-10-20 20:27:47 +02:00
Paweł Dziepak
e683d57768 key_reader: add key_from_mutation_reader
This is a wrapper that allows using mutation_reader as key_reader.
Useful mostly in tests.

Signed-off-by: Paweł Dziepak <pdziepak@scylladb.com>
2015-10-20 20:27:47 +02:00