Commit Graph

53948 Commits

Author SHA1 Message Date
Asias He
189b2515e0 streaming: Add is_empty to stream_plan 2015-06-24 16:13:30 +08:00
Asias He
ed27012503 streaming: Add has_active_sessions to stream_coordinator 2015-06-24 16:13:30 +08:00
Asias He
934c17fa6b streaming: Add set_state and get_state to stream_session()
Note: Origin uses name sate() for the function name. We can not since it
conflicts with the name of enum class stream_session::state.
2015-06-24 16:13:30 +08:00
Asias He
6619122b35 streaming: Add _coordinator to stream_plan 2015-06-24 16:13:30 +08:00
Asias He
573e96c4c2 streaming: Convert StreamState.java to C++ 2015-06-24 16:13:30 +08:00
Asias He
15da955766 streaming: Import StreamState.java 2015-06-24 16:13:30 +08:00
Asias He
bb40fed991 streaming: Add add_transfer_file to stream_transfer_task 2015-06-24 16:13:23 +08:00
Avi Kivity
27b0200912 tests: don't mix up stdout and the xml log
Can result in corrupted xml.
2015-06-24 09:33:55 +03:00
Asias He
5d4f533e14 streaming: Add constructor for outgoing_file_message 2015-06-24 14:18:11 +08:00
Asias He
8ecacca2a9 streaming: Add keep_ss_table_level to stream_session 2015-06-24 14:18:11 +08:00
Glauber Costa
fd6ec4a7ca database: fix sstables exception
This exception handling code is clearly bogus. That's old code, and it is not
the proper way to propagate it.

Fix it to use then_wrapped. Also include the filename in the message, so we have
a better clue about what happened.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
2015-06-24 06:16:54 +03:00
Glauber Costa
d6b14c967f db: flush system tables to disk
We flush the mutation we receive, and this is not complete yet.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
2015-06-23 15:23:37 -04:00
Glauber Costa
a6ef9815e9 database: futurize seal_active_memtables
By doing this, it is possible to synchronously wait for the seal to complete by
waiting on this future. This is useful in situations where we want to
synchronously flush data to disk.

Existing callers will not be patched, and this keeps their current behavior,
alas, asynchronously initiating a write, is preserved.

TODO: A better interface would guarantee that all writes before this one are
also complete

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
2015-06-23 15:23:37 -04:00
Avi Kivity
b35d1412c4 Merge "Compaction metadata" from Raphael 2015-06-23 21:22:54 +03:00
Glauber Costa
83d4a47a21 stop using get() / get0() arbitrarily
Being able to just get() a future inside a thread is a very nice functionality,
but we should be more careful a to when we do it. In particular, we are doing this
in a function inside db::legacy_schema_tables, because that function is supposed to
be used - so far - only inside a threaded function.

That approach simplify things, but at a great reusability cost. That means that this
function can now only be reused inside thread code.

This patch turns the offender into a normal future function, and moves the get0() to
code that is directly under seastar::async.

As a rule of thumb, we should stay away from ever doing get / get0 in functions that
are not used directly by async code.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
2015-06-23 21:16:56 +03:00
Avi Kivity
1a515ca807 Merge "CQL3 validation: keyspace and table creation" from Paweł
"This patchset adds validation of 'create table' and 'create keyspace'
statements and mostly contains simple Java to C++ translation.
There are still few things missing, especially more detailed validation of
arguments passed to certain strategies (e.g. replication strategy, compaction,
compression). Some of them can be relatively easy to add (replication strategy,
maybe compression) while some probably need to wait a bit (compaction)."
2015-06-23 21:07:56 +03:00
Raphael S. Carvalho
0f95066be4 sstables: fix a bug which invalidated generated sstables
Column stats min_timestamp, max_timestamp and max_local_deletion_time
were being update incorrectly.

max_local_deletion_time should be std::numeric_limits<int>::max() by
default, and then keep track of max local deletion time, if any.

This bug prevented a sstable generated by us from being compacted by
c* because max_local_deletion_time was storing std::numeric_limits<int>
::min(), and thus the sstable would be considered fully expired.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
2015-06-23 12:13:26 -03:00
Raphael S. Carvalho
f831d1bce9 sstables: add support to generate compaction metadata
compaction metadata is composed of ancestors and cardinality.

ancestors data is generated via compaction process, so it will be
empty by the time being.

cardinality data is generated by hashing the keys, offering the
values to hyperloglog and retrieving a buffer with the data to be
stored.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
2015-06-23 12:13:26 -03:00
Raphael S. Carvalho
954688b327 hyperloglog: modify it according to our needs
The first change was to add the function get_bytes, which will create
a temporary buffer with the format expected by compaction metadata's
cardinality. For creating the format, I had to import write_unsigned_
var_int from stream-lib.
write_unsigned_var_int is about using fewer bytes to encode smaller
integer values, but will use slighly more bytes to larger values.

The last change was to add the function offer_hashed, which receives
a 64-bit hashed value instead. Hash algorithm used by c* is murmur
hash - hash2_64.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
2015-06-23 12:13:25 -03:00
Raphael S. Carvalho
2c95fdd7b6 sstables: import hyperloglog implementation
Imported from: https://github.com/hideo55/cpp-HyperLogLog
HEAD: 3ff431b5af84aa28a6390430bbc60c4678bec763

hyperloglog is a cardinality estimator that will used by sstable
compaction metadata stored by Statistics file.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
2015-06-23 11:59:45 -03:00
Raphael S. Carvalho
0d345dfd32 sstables: extend metadata collector to add ancestors
Ancestors is a member of compaction metadata.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
2015-06-23 11:18:50 -03:00
Paweł Dziepak
3addb285cd tests/cql: test validation of keyspace and table creation requests
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
2015-06-23 16:17:45 +02:00
Paweł Dziepak
26ff34060d tests/cql: add assert_that_failed()
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
2015-06-23 16:17:45 +02:00
Paweł Dziepak
67551832c0 cql3: implement displayRecogintionError for lexer and parser
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
2015-06-23 16:17:45 +02:00
Paweł Dziepak
0369d169cc cql3: validate name and columns of the created table
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
2015-06-23 16:17:45 +02:00
Paweł Dziepak
3e760454d5 cql3: validate properties of newly created tables
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
2015-06-23 16:17:45 +02:00
Paweł Dziepak
3806f0e268 cql3: validate the name of the created keyspace
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
2015-06-23 16:17:45 +02:00
Paweł Dziepak
c5e617ea78 schema: add NAME_LENGTH constant
It's probably not the best place for this constant, but that's where
it is in origin.

Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
2015-06-23 16:17:45 +02:00
Paweł Dziepak
7de8f2cda8 cql3: validate properties at keyspace creation
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
2015-06-23 16:17:44 +02:00
Paweł Dziepak
cb8f8f84be cql3: make property_definitions::validate() take const references
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
2015-06-23 16:17:44 +02:00
Tomasz Grabiec
de23b54764 types: Implement to_string()/from_string() for boolean_type 2015-06-23 17:07:37 +03:00
Vlad Zolotarov
efe1696410 cql_test_env: Start the global snitch before storage service
This order is required since 5e1348e741
(storage_service: Use get_local_snitch_ptr in gossip_snitch_info).

This fixes the breakage in the cql_query_test.

Reported-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2015-06-23 16:36:27 +03:00
Avi Kivity
20b86ea839 Merge seastar upstream 2015-06-23 15:57:09 +03:00
Gleb Natapov
12f3d53372 storage_proxy: cleanup leftovers from timer consolidation 2015-06-23 15:43:59 +03:00
Gleb Natapov
2be9dfc242 storage_proxy: use fb_utilities::get_broadcast_address()
fixes some fixmes
2015-06-23 15:43:59 +03:00
Gleb Natapov
67ea1b0ec8 Revert "db: hold onto write response handler until timeout handler is executed"
This reverts commit 52aa0a3f91.

After c9909dd183 this is no longer needed since reference to a
handler is not used in abstract_write_response_handler::wait() continuation.

Conflicts:
	service/storage_proxy.cc
2015-06-23 15:43:59 +03:00
Avi Kivity
20e7e3576e Merge "Introduce simple cache" from Tomasz
"This introduces a very simple cache which caches whole partitions.

There is a reclaimer registerred which clears all caches upon memory pressure.
This is a temporary measure until we implement log-structured allocator and
incremental eviction.

I can see that for small data sets this series imporoves cassandra-stress read
throughput from 2k to 50k tps on muninn/huginn."
2015-06-23 15:30:21 +03:00
Tomasz Grabiec
d4e0e5957b db: Integrate cache with the read path 2015-06-23 13:49:25 +02:00
Tomasz Grabiec
d11773bc14 tests: Introduce row cache test 2015-06-23 13:49:24 +02:00
Tomasz Grabiec
2b5d9a917f tests: Convert mutation_reader_test to use seastar threads
Also extract assertions into header file.
2015-06-23 13:49:24 +02:00
Tomasz Grabiec
e40638823e db: Introduce mutation cache
row_cache class is meant to cache data for given table by wrapping
some underlying data source. It gives away a mutation_reader which
uses in-memory data if possible, or delegates to the underlying reader
and populates the cache on-the-fly.

Accesses to data in cache is tracked for eviction purposes by a
separate entity, the cache_tracker. There is one such tracker for the
whole shard.
2015-06-23 13:49:24 +02:00
Tomasz Grabiec
b9288d9fa7 db: Make column_family managed by lw_shared_ptr<>
It will be share-owned by readers.
2015-06-23 13:49:24 +02:00
Tomasz Grabiec
8fd466338d mutation_reader: Introduce helper for consuming all mutations 2015-06-23 13:49:23 +02:00
Tomasz Grabiec
83e7a21dfb mutation: Add apply() helper which works on mutation_opt 2015-06-23 13:49:23 +02:00
Tomasz Grabiec
bdd3fd5019 tests: Add missing blank line 2015-06-23 13:44:37 +02:00
Gleb Natapov
c9909dd183 cluster: consolidate mutation clustering timers
Currently mutation clustering uses two timers, one expires when wait for
cl timeouts and is canceled when cl is achieved, another expires if some
endpoints do not answer for a long time (cl may be already achieved at
this point and first timer will be canceled). This is too complicated
especially since both timers can expire simultaneously. Simplify it by
having only one timer and checking in a callback whether cl was achieved.
2015-06-23 14:42:56 +03:00
Tomasz Grabiec
14a8110d1f future: Avoid copying of the result in get0()
Even though we accept std::tuple<T...>&&, 'x' is an l-value reference
inside get0().
2015-06-23 14:28:01 +03:00
Asias He
5e1348e741 storage_service: Use get_local_snitch_ptr in gossip_snitch_info 2015-06-23 12:12:33 +03:00
Avi Kivity
51eb220c49 Merge "stream reader and writer" from Asias 2015-06-23 11:31:10 +03:00
Asias He
0f8c35a2fe streaming: Convert StreamReader.java to C++ 2015-06-23 16:06:28 +08:00