The file_input_stream interface was messy: it was not fiber safe (e.g., code
doing seek() in the middle of an ongoing read_exactly()), and went against
the PIMPL philosophy.
So this patch removes the file_input_stream class, and replaces it with a
completely different design:
We now have in fstream.hh a global function:
input_stream<char>
make_file_input_stream(
lw_shared_ptr<file> file, uint64_t offset = 0,
uint64_t buffer_size = 8192);
In other words, instead of "seeking" in an input stream, we just open a new
input stream object at a particular offset of the given file. Multiple input
streams might be concurrently active on the same file.
Note how make_file_input_stream now returns a regular "input_stream", not a
subtype, and it can be used just like any normal input_stream to read the stream
starting at the given position.
This patch makes "input_stream" a "final" type: we no longer subclass it in our
code, and we shouldn't in the future because it goes against the PIMPL design
(the subclass should be of the inner workings, like the data_source_impl, not
of input_stream).
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
We had an include loop, which can cause problems in some cases:
1. iostream.hh #includes scattered_message.hh #includes reactor.hh
2. reactor.hh #includes iostream.hh
This patch fixes the loop be removing an unnecessary include.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
reactor.hh includes forward declarations for input_stream, output_stream
template classes. These are unnecessary, because it #include's iostream.hh,
which contains the full definition of these classes.
These unnecessary forward declarations are harmless in the current code, but
they will become harmful if we change the definitions in iostream.hh (e.g., I
wanted to make input_stream a "final" class).
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
core/reactor.hh had two #include lines in the middle of the file, which
duplicate previous #include lines in the top of the file. So remove them.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
make_shared() has a special case for detecting a created class deriving
from enable_shared_from_this<>, so it can point the refcount pointer into
the object's data area instead of creating a shared_ptr_count_base for it.
The code, however, fails to detect a creating class deriving indirectly
from enable_shared_from_this:
struct base : enable_shared_from_this<base> {};
struct derived : base {};
make_shared<derived>(); // <- allocates independent refcount
The result is that the object reference counter lives in two locations.
Fix by detecting the derived class case as well.
This patch adds some of the common functionalities from std:string to
sstring.
It adds length (implement by size() )
It define the constant npos to indicate no possition.
It adds the at (reference and const reference)
It define the find char and find sstring methods
and the substr method
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
need merge sstring
1. Moved all signal handler functions/members into a wrapping class
2. Block all signals on desctuctor call (resetting of signal handler is
not enough since some signals will arive even if we reset signal handler
to SIG_DFL e.g. timer)
Fixes use-after-free when a signal is caught after the reactor is
destroyed.
Signed-off-by: Shlomi Livne <shlomi@cloudius-systems.com>
Add "finally" continuation overload for functions returning future type that
awaits finishing the continuation in question before continuing the "present"
one. I.e. a wrapper around "then_wrapped" to remove the need to deal with the
original return value.
Signed-off-by: Calle Wilund <calle@cloudius-systems.com>
Basic explanation of the reclaimer algorithm:
- Each slab page has a descriptor containing information about it, such as
refcnt, vector of free objects, link into LRU, etc.
- The LRU list will only contain slab pages which items are unused, so as
to make the reclaiming process faster and easier. Maintaining the LRU of slab
pages has a performance penalty of ~1.3%. Shlomi suggested an approach where
LRU would no longer exist and timestamp would be used instead to keep track of
recency. Reclaimer would then iterate through all slab pages checking for an
unused slab page with the lowest timestamp.
- Reclaimer will get the least-recently-used slab page from the LRU list,
do all the management stuff required, and iterate through the page erasing any
of the items there contained. Once reclaimer was called, it's likely that slab
memory usage is calibrated, thus slab pages shouldn't be allocated anymore.
- Reclaimer is enabled by default but can be disabled by specifying the slab
size using the application parameter --max-slab-size.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
input_stream's constructor had an unused buf_size parameter. Such a
parameter is not needed - whenever the input_stream needs more
data it calls the underlying data_source's get(), and thus only the
data_source gets to decide the buffer size. Moreover, in some
implementations, this read buffer size will be different each time -
e.g., in a chunk-compressed file, the uncompressed chunk's size will
be different each time.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
SFINAE only works for substituted template parameters, not any complication
error (or it would be called CEINAE); therefore hash<T> for enums will fail
to compile, given a non-enum, rather than being ignored.
It's not possible to specialize hash<> for enums, since the primary template
does not have en extra Enable template argument for use with enable_if. We
therefore rename it to enum_hash<> and require users to explicitly define
hash<MyEnum> as inheriting from it.
We currently only signal eof for consume() users. If one is calling
read_exactly, eof will never be signalled.
This might make some sense in continuous streams, but specially when
dealing with files, eof is a natural part of line that can and will
happen all the time. Every "read-until-finish" file-loop will at some
point rely on eof.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
boost::join() provided by boost/algorithm/string.hpp conflicts with
boost::join() from boost/range/join.hpp. It looks like a boost issue
but let's not pollute the namespace unnecesssarily.
Regarding the change in configure.py, it looks like scollectd.cc is
part of the 'core' package, but it needs 'net/api.hh', so I added
'net/net.cc' to core.
It is sometimes frustrating to use open_file_dma, because it has the hardcoded
behavior of always assuming O_CREAT. Sometimes this is not desirable, and it
would be nice to have the option not to do so.
Note that, by design, I am only including in the open_flags enum things that we
want the user of the API to control. Stuff like O_DIRECT should not be
optional, and therefore is not included in the visible interface.
Because of that I am changing the function signature to include a paramater
that specifies whether or not we should create the file if it does not exist.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
* Slab allocator resembles the one used by stock memcached, where
it's composed of slab classes and slab classes are composed of
chunks of the same size.
* Per-slab-class LRU is also available.
* Slab allocator exports stats to collectd.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
The ".or_terminate()" continuation is needed when one needs to exit the
application on an unhandled exception, instead of just letting the event
loop continue to spin forever.
or_terminate() currently calls std::terminate() which only incidentally
(as a gcc-specific implementation) prints out the exception's type; It
then calls abort(), which results in a painfully slow core dump. This
abort() is NOT helpful, because at that point the debugger can only find
where abort() was called, not where the original exception was thrown (see
issue #32).
So instead of calling std::terminate(), this patch switches to calling
engine().exit(), to cleanly shut down the application, without dumping
core. It also prints, like gcc's std::terminate(), the exception's type.
This printing requires non-standard gcc extensions.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
CPU may automatically prefetch next cache line, so if statistics that
are collected on different cpus resided on adjacent cache lines CPU may
erroneously prefetch cache line that is guarantied to be accessed by
another CPU. Fix it by putting a cache line between two structures.
- Allocate the data buffers instead of using the default inline rte_mbuf
layout.
- Implement an rx_gc() and add an _rx_gc_poller to call it: we will refill
the rx mbuf's when at least 64 free buffers.
This threshold has been chosen as a sane enough number.
- Introduce the mbuf_data_size == 4K. Allocate 4K buffers for a detached flow.
We are still going to allocate 2K data buffers for an inline case since 4K
buffers would require 2 pages per mbuf due to "mbuf_overhead".
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
When using print() to debug on smp, it is very annoying to get interleaved
output.
Fix by wrapping stdout with a fake stream that has a line buffer for each
thread.
