Commit Graph

43077 Commits

Author SHA1 Message Date
Tomasz Grabiec
e831884c13 tests: add zero copy UDP test
It listens for requests on port 10000 and sends responses comprised of
three chunks of data in one packet. The chunk sizes are specified via
the --chunk-size argument.

The reqest can be anything, its content is ignored.

You can switch to equivalent copying version by passing --copy
argument.
2014-12-04 13:51:35 +01:00
Tomasz Grabiec
c4335c49f6 core: convert output APIs to work on packets
This way zero-copy supporting code can put data directly to packet
object and pass it through all layers efficiently.
2014-12-04 13:51:26 +01:00
Tomasz Grabiec
ba0ac1c2b8 core: simplify write_all()
The only case when write_all() does not write all the data is when the
fiber fails at some point, in which case the resulting future is
failed too.
2014-12-04 13:37:36 +01:00
Tomasz Grabiec
cd3ba33ead core: introduce scattered_message
It's a builder class for creating messages comprised of multiple
fragments.
2014-12-04 13:37:35 +01:00
Tomasz Grabiec
a2ca556836 sstring: introduce release()
Releases owenrship of the data and gives it away as
temporary_buffer. This way we can avoid allocation when putting rvalue
sstring if it's already using external storage. Except we need to
allocate a deleter which uses delete[], but this can be fixed later.
2014-12-04 13:37:35 +01:00
Tomasz Grabiec
72b0794759 packet: add constructor for appending temporary_buffers 2014-12-04 13:37:35 +01:00
Tomasz Grabiec
3a2d74e3d3 packet: add reserve() method 2014-12-04 13:37:35 +01:00
Tomasz Grabiec
f3dada6f1d packet: add constructor for appending deleters
Deleters not always come with fragments. When multiple fragments share
a deleter, first fragments are appended and then one deleter for all
of them.
2014-12-04 13:37:35 +01:00
Tomasz Grabiec
8ffcdac455 packet: move lambdas rather than copy them
Some lambdas are not copyable.
2014-12-04 13:37:35 +01:00
Tomasz Grabiec
2650c68824 packet: add more constructor variants 2014-12-04 13:37:35 +01:00
Tomasz Grabiec
e720f53c22 deleter: add chaining make_free_deleter() overload 2014-12-04 13:37:35 +01:00
Tomasz Grabiec
8e38cb4159 deleter: introduce append()
It's to help chaining up deleters when appending a fragment with anq
already created deleter.
2014-12-04 13:37:35 +01:00
Tomasz Grabiec
7fd878b1ed tests: memcached: fix test_mutliple_keys_in_get()
Values may be reported in different order than the order of keys in
the request.
2014-12-04 13:37:35 +01:00
Tomasz Grabiec
411e6c1b02 tests: memcached: add test checking splitting of large responses 2014-12-04 13:37:35 +01:00
Tomasz Grabiec
b6cb2b7477 memcached: add missing return 2014-12-04 13:37:35 +01:00
Tomasz Grabiec
e4583699fd memcached: make sure datagrams are below the size limit 2014-12-04 13:37:35 +01:00
Tomasz Grabiec
71556de0e6 tests: add more test cases to output_stream_test 2014-12-04 13:37:35 +01:00
Tomasz Grabiec
11a501d884 tests: fix assertion failure at test program exit
Seastar's allocator has an assertion which checks that the memory
block is freed on the same CPU on which it was allocated. This is
reasonable because all allocations in seastar need to be CPU-local.

The problem is that boost libraries (program_options,
unit_testing_framework) make heavy use of lazily-allocated static
variables, for instance:

    const variable_value&
    variables_map::get(const std::string& name) const
    {
        static variable_value empty;
        // ...
    }

Such variable will be allocated in in the current threads but freed in
the main thread when exit handlers are executed.

This results in:

output_stream_test: core/memory.cc:415: void memory::cpu_pages::free(void*):
Assertion `((reinterpret_cast<uintptr_t>(ptr) >> cpu_id_shift) & 0xff) == cpu_id' failed.

This change works around the problem by forcing initialization in the
main thread.

See issue #10.
2014-12-04 13:37:02 +01:00
Tomasz Grabiec
a88ddcec25 sstring: overload to_sstring() for temporary_buffer<> 2014-12-04 13:35:33 +01:00
Tomasz Grabiec
bcea3a67ca output_stream: support for output packet trimming
For UDP memcached we cannot generate arbitrarily large chunks, we need
to trim to datagram size. It's most efficient to split in the
output_stream.
2014-12-03 20:02:21 +01:00
Tomasz Grabiec
4b7c42a5c7 output_stream: fix bug in write()
When coalescing large buffer with buffered data _end was not updated
so flush() would yield shorter packet.
2014-12-03 20:02:21 +01:00
Tomasz Grabiec
6ae5177c2c output_stream: do not allocate on flush()
In UDP memcached flush() is always the last operation on
outpout_stream, so that allocation is wasted.
2014-12-03 20:02:21 +01:00
Tomasz Grabiec
584139decd future-util: make do_for_each() propagate failure 2014-12-03 20:02:21 +01:00
Tomasz Grabiec
8d48c91a35 future-util: introduce now()
This function belongs to a group of functions for associating futures
with time points. Currently there's only now(), which servers as a shorthand
for make_ready_future<>().
2014-12-03 19:57:43 +01:00
Avi Kivity
3e4842a2a1 Merge branch 'asias/ip' of github.com:cloudius-systems/seastar-dev
IP fragment reassembly from Asias.
2014-12-03 16:03:18 +02:00
Asias He
59aa280f0d ip: Add IPv4 reassembly support
If a TCP or UDP IP datagram is fragmented, only the first fragment will
contain the port information. When a fragment without port information
is received, we have no idea which "stream" this fragment belongs to,
thus we no idea how to forward this packet.

To solve this problem, we use "forward twice" method. When IP datagram
which needs fragmentation is received, we forward it using the
frag_id(src_ip, dst_ip, identification, protocol) hash. When all the
fragments are received, we forward it using the connection_id(src_ip,
src_port, dst_ip, dst_port) hash.
2014-12-03 21:40:49 +08:00
Gleb Natapov
4fd3313e3e reactor: add "--poll" command line switch
If the switch is used reactor never goes idle.
2014-12-03 14:37:49 +02:00
Gleb Natapov
d151763967 reactor: move memory barrier to idle() accessors 2014-12-03 14:37:41 +02:00
Gleb Natapov
4d3b6497ea reactor: rework poll infrastructure
Move idle state management out from smp poller back to generic code. Each
poller returns if it did any useful work and generic code decided if it
should go idle based on that. If a poller requires constant polling it
should always return true.
2014-12-03 14:37:33 +02:00
Tomasz Grabiec
f556172619 temporary_buffer: make empty buffer don't need to malloc() 2014-12-03 13:15:09 +01:00
Tomasz Grabiec
1c49669f59 temporary_buffer: introduce operator bool()
It's used as a test for emptiness by convention.

Allows for things like:

  if (buf) {
     // not empty
  }
2014-12-03 13:15:09 +01:00
Tomasz Grabiec
cbe6169d36 test.py: speed up allocator test when running tests in fast mode 2014-12-03 13:15:09 +01:00
Tomasz Grabiec
76a8908b21 virtio: fix indentation 2014-12-03 13:15:09 +01:00
Asias He
2702af5e7d net: Add help packet_merger
This can be used for both TCP out-of-order and IP fragmentation merging.
2014-12-03 17:47:30 +08:00
Asias He
8335787268 net: Expose interface::forward
This can be used with ipv4 fragmentation.
2014-12-03 17:47:29 +08:00
Asias He
7ca33fdd72 ip: Add helper for fragmentation 2014-12-03 17:47:29 +08:00
Gleb Natapov
7dbc333da6 core: Allow forwarding from/to any cpu 2014-12-03 17:47:29 +08:00
Asias He
6c097fe2e9 tests: Make udp_server SMP aware 2014-12-03 11:03:00 +02:00
Avi Kivity
1e572a3248 app-template: add missing app-template.cc 2014-12-01 18:01:12 +02:00
Nadav Har'El
8827eb3b27 Clean up link line with DPDK (v2)
The command line linking with DPDK's libraries looked like a cross between
random character generator and black magic. Reading a bit on the DPDK
mailing list, it turns out there is method in this madness (flawed method,
but method nontheless):

1. Instead of using "-l..." they used "-Wl,-l..." everywhere. Turns out
   they did this ugliness to "hide" this option from libtool.

   We don't use libtool, and don't need to hide anything from it.

2. They used "--start-group ... --end-group" to avoid having to figure
   out the right link order.

   It was easy to figure out the right link order and avoid this option.

3. They used "--whole-archive" on all the DPDK libraries. Unfortunately,
   this option *is* needed, because the way DPDK is written, it is not
   suited to be compiled into an (non-shared) library: Each of the DPDK
   drivers ("librte_pmd_*") has a constructor function which needs to
   run to register itself. This works fine with shared libraries (whose
   constructors are run on load) but with a ".a" library, the whole
   library is left out because nothing from the outside refers to any
   of its symbols.

   So what we should do is to use --whole-archive only on the PMD drivers,
   and all will be fully compiled into the generated program. The rest of
   the DPDK libraries will be linked normally, and hopefully because we
   don't use large parts of DPDK, big chunks will not be compiled in.

   If we don't add this "--whole-archive", none of the drivers will be
   compiled into the program, the initialization will not be able to
   find any driver, and just complain there are no ethernet ports.

After this patch, Seastar with DPDK still compiles, and runs.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Reviewed-by: Vlad Zolotarov <vladz@cloudius-systems.com>
2014-12-01 18:00:14 +02:00
Avi Kivity
6b5973af70 app-template: don't alias boost::program_options as bpo in a header file
We only have one global namespace, let's work together to keep it free
of pollution.
2014-12-01 17:56:34 +02:00
Avi Kivity
78691fc72f app-template: move to a .cc file
Reduce compile loads.
2014-12-01 17:48:18 +02:00
Avi Kivity
1820c8eaf6 blkdiscard_test: add missing include
For keep_doing().
2014-12-01 17:47:57 +02:00
Avi Kivity
e1397038d4 future-util.hh: add missing include
'task_quota' needs reactor.hh
2014-12-01 17:47:28 +02:00
Avi Kivity
256d1823c6 app-template: warn on debug mode 2014-12-01 17:33:47 +02:00
Avi Kivity
7619c01941 Merge branch 'flashcache' of github.com:cloudius-systems/seastar-dev
Flashcache fixes from Raphael.
2014-12-01 14:53:22 +02:00
Avi Kivity
be0ae4f5dc memory: Un-hide standard allocator functions
With -fvisibility=hidden, all executable symbols are hidden from shared
objects, allowing more optimizations (especially with -flto).  However, hiding
the allocator symbols mean that memory allocated in the executable cannot
be freed in a library, since they will use different allocators.

Fix by exposing these symbols with default visibility.

Fixes crash loading some dpdk libraries.
2014-12-01 14:49:04 +02:00
Raphael S. Carvalho
0653a9f3f7 flashcache: fix _total_mem_disk accounting when erasing mem-disk items
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
2014-12-01 10:30:23 -02:00
Raphael S. Carvalho
9de9f34423 flashcache: fix erase on disk-based items
Fixed by adding missing break statement.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
2014-12-01 10:28:42 -02:00
Gleb Natapov
c90e56e4fb memory: dynamically search for memory level in a topology
Current code assumes that memory is at node level, but on non numa
machines there is no node level at all. Instead of assuming memory
location in a topology search for it dynamically.
2014-12-01 14:09:36 +02:00