As Avi suggested, we should use size_t only for memory sizes, not disk
sizes, as some hypothetical 32-bit machine could have 32-bit size_t
but still support 64-bit file sizes.
So this patch changes a number of places we used size_t in sstables/
to use uint64_t instead. It doesn't change *all* uses of size_t: Where
the size_t refers to a size of an object in memory (or an object that
should fit into memory - like the summary file), I left size_t.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
The current sstable write interface only knows how to write a memtable.
For compaction, we also want it to be able to write the compaction's
output, which we can represent as a mutation_reader. So this patch
changes the sstable::write_components() method to accept a mutation_reader,
and whatever else is needed (a schema and the number of partitions in
the reader - or an estimate thereof).
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Similar to std::shared_mutex, allows shared and exclusive locking of a
resource (also known as a reader/writer lock). This implementation is
strictly FIFO.
In a couple of places in sstable_datafile_test.cc, we had
make_shared<memtable> instead of the usual make_lw_shared<memtable>.
It appears that since commit bc468f9a0e,
which made memtables inherit from enable_lw_shared_from_this (not the "lw"),
this no longer works correctly. I'm not sure I can really explain the
details of what's wrong, but with some refactoring I did on the sstable
writing code, it stopped working giving various strange crashes which appear
like the object protected by this shared_ptr got prematurely destructed.
Changing the test to use make_lw_shared() fixes the problems I was having.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
If last response comes after write timeout is triggered, but before
continuation, that suppose to handle it runs the handler can be removed
to earlier and be access from the continuation after deletion. Fix it by
making response handler to be shared pointer instead of unique and
holding to it in timeout continuation.
We need to do that in order to close the database cleanly, flushing all pending
data before we do.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
If compression is used, we should provide both uncompressed and
compressed length to metadata collector, so as for the ratio to
be computed. Stats metadata stores compression ratio.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
When an exceptional future is ignored, we print a message, and for an
std::exception we print its what(). However, it would be useful to also
see the exception's type. We already had such exception type printing
code for engine_exit(), and this patch extracts that code into a separate
function, and also uses it to print the warning message when an exceptional
future is ignored.
For example, in one test before this patch I see:
WARNING: exceptional future ignored: _Map_base::at
After this patch,
WARNING: exceptional future ignored of type 'std::out_of_range': _Map_base::at
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Our queue<T> is a convenient mechanism for passing data between a producer
fiber (a set of consecutive continuations) and a consumer fiber.
However, queue<T> is difficult to use *correctly*. The biggest problem is
how to handle premature stopping: What if one of the two fibers (the reader
or the writer) stops prematurely, and will never read or write any more?
When queue<T> is used naively, the other fiber will just hang indefinitely
while it waits to read from the empty queue, or write to the full queue.
The solution proposed in this patch is a new pipe mechanism, implemented
internally over a queue. pipe<T>() returns two separate objects - a pipe
reader, and a pipe writer. Typically each object is std::move()ed into a
different fiber. When a fiber stops and its captured variables are destroyed,
one end of the pipe is destroyed, and that causes the other end's operations
to return immediately (if the other end was already blocked, it will resume
immediately, and return an exceptions). This behavior is analogous to
Unix's EOF or broken-pipe behavior.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
