This demonstrate how to use a swagger definition file to create an API.
The swagger file demo.json define one api called hello_world, it has
both query and path parameters and return an object as a result.
The handler implementation simply places the given parameters in the
return object.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
This patch makes compiling a file "hello.cc" using seastar as simple as:
g++ `pkg-config --cflags --libs build/release/seastar.pc` hello.cc
Our current build system makes it very hard for users to build a project
*using* Seastar - basically requiring them to integrate their project into
the Seastar tree and its Makefile into our "ninja" framework.
This patch simplifies things considerably by doing two things:
First it builds an archive of all Seastar objects - build/$mode/libseastar.a.
But this in itself is not enough - the user wouldn't know how to use it
(currently, -Wl,--whole-archive is needed, unfortunately, because of reliance
of C++ static constructors), and will also need a long list of other
libraries, as wall as various definitions and include paths during compilation.
The solution is to use pkg-config, which has become the de-facto standard
method on Linux for makefiles to figure out how to compile with a certain
library. With this patch, someone can do this for example, to compile
tests/fileiotests.cc:
g++ `pkg-config --cflags --libs build/release/seastar.pc` tests/fileiotest.cc
Note how we have a different ".pc" file for each mode, with the appropriate
compile and link flags. A more eleborate example with separate compile and
link stages will use "pkg-config --cflags" on the compilation stage, and
"--libs" on the linking stage.
Eventually, we should have a "ninja install", which will install libseastar.a
and seastar.pc to their system-wide default location, and then the above
command line will become as simple as:
g++ `pkg-config --cflags --libs seastar` tests/fileiotest.cc
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
This use the routes and the reqeuest found in the http directory and
move all files but main to the http directory
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
We would like to extend the httpd capabilities and use it for the API
implementation.
The first step is to make it a library with main that calls an instanse.
This break the implementation to a header file, implementation and main,
that simply calls the implementation.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
boost::join() provided by boost/algorithm/string.hpp conflicts with
boost::join() from boost/range/join.hpp. It looks like a boost issue
but let's not pollute the namespace unnecesssarily.
Regarding the change in configure.py, it looks like scollectd.cc is
part of the 'core' package, but it needs 'net/api.hh', so I added
'net/net.cc' to core.
Flashcached integration was done relying on certain similarities between
flashcached and memcached. Old assumptions no longer hold.
As a result, the current code is terrible to integrate the slab allocator
into it, while keeping flashcached alive.
This patch reverts flashcached from memcached. It should be re-integrated
in the future, but definitely in a better way.
When using print() to debug on smp, it is very annoying to get interleaved
output.
Fix by wrapping stdout with a fake stream that has a line buffer for each
thread.
dpdk zero-copy tx, from Vlad:
"This patch series introduces zero-copy Tx with DPDK networking backend:
- Split the dpdk_qp mempool into separate pools for Rx and Tx queues.
- Configure the dpdk_qp mempools to use external memory buffer when we
can ensure pinning and virt2phys translation (currently only when
running on top of hugetlbfs).
- Properly divide the memory between seastar and DPDK when running on
top of hugetlbfs.
- Tx zero-copy itself. See more details in the PATCH7 description."
Send packets without copying fragments data:
- Poll all the Tx descriptors and place them into a circular_buffer.
We will take them from there when we need to send new packets.
- PMD will return the completed buffers descriptors to the Tx mempool.
This way we are going to know that we may release the buffer.
- "move" the packet object into the last segment's descriptor's private data.
When this fragment is completed means the whole packet has been sent
and its memory may be released. So, we will do it by calling the packet's
destructor.
Exceptions:
- Copy if hugepages backend is not enabled.
- Copy when we failed to send in a zero-copy flow (e.g. when we failed
to translate a buffer virtual address).
- Copy if first frag requires fragmentation below 128 bytes level - this is
in order to avoid headers splitting.
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
New in v5:
- NULL -> nullptr across the board.
- Removed unused macros: MBUF_ZC_PRIVATE() and max_frags_zc.
- Improved the local variables localization according to Nadav's remarks.
- tx_buf class:
- Don't regress the whole packet to the copy-send if a single fragment failed to be sent
in a zero-copy manner (e.g. its data failed the virt2phys translation). Send only such a
fragment in a copy way and try to send the rest of the fragments in a zero-copy way.
- Make set_packet() receive packet&&.
- Fixed the comments in check_frag0(): we check first 128 bytes and not first 2KB.
starting from v2.
- Use assert() instead of rte_exit() in do_one_frag().
- Rename in set_one_data_buf() and in copy_one_data_buf(): l -> buf_len
- Improve the assert about the size of private data in the tx_buf class:
- Added two MARKER fields at the beginning and at the end of the private fields section
which are going to be allocated on the mbuf's private data section.
- Assert on the distance between these two markers.
- Replace the sanity_check() (checks that packet doesn't have a zero-length) in a
copy-flow by an assert() in a general function since this check
is relevant both for a copy and for a zero-copy flows.
- Make a sanity_check to be explicitly called frag0_check.
- Make from_packet() receive packet&&.
- In case frag0_check() fails - copy only the first fragment and
not the whole packet.
- tx_buf_factory class:
- Change the interface to work with tx_buf* instead of tx_buf&.
- Better utilize for-loop facilities in gc().
- Kill the extra if() in the init_factory().
- Use std::deque instead of circular_buffer for storing elements in tx_buf_factory.
- Optimize the tx_buf_factory::get():
- First take the completed buffers from the mempool and only if there
aren't any - take from the factory's cache.
- Make Tx mempools using cache: this significantly improves the performance despite the fact that it's
not the right mempool configuration for a single-producer+single-consumer mode.
- Remove empty() and size() methods.
- Add comments near the assert()s in the fast-path.
- Removed the not-needed "inline" qualifiers:
- There is no need to specify "inline" qualifier for in-class defined
methods INCLUDING static methods.
- Defining process_packets() and poll_rx_once() as inline degraded the
performance by about 1.5%.
- Added a _tx_gc_poller: it will call tx_buf_factory::gc().
- Don't check a pointer before calling free().
- alloc_mempool_xmem(): Use posix_memalign() instead of memalign().
New in v4:
- Improve the info messages.
- Simplified the mempool name creation code.
- configure.py: Opt-out the invalid-offsetof compilation warning.
New in v3:
- Add missing macros definitions dropped in v2 by mistake.
New in v2:
- Use Tx mbufs in a LIFO way for better cache utilization.
- Lower the frag0 non-split thresh to 128 bytes.
- Use new (iterators) semantics in circular_buffer.
- Use optional<packet> for storing the packing in the mbuf.
- Use rte_pktmbuf_alloc() instead of __rte_mbuf_raw_alloc().
- Introduce tx_buf class:
- Hide the private rte_mbuf area handling.
- Hide packet to rte_mbuf cluster translation handling.
- Introduce a "Tx buffers factory" class:
- Hide the rte_mbuf flow details:
mempool->circular_buffer->(PMD->)mempool
- Templatization:
- Make huge_pages_mem_backend a dpdk_qp class template parameter.
- Unite the from_packet_xxx() code into a single template function.
- Unite the translate_one_frag() and copy_one_frag() into a single
template function.
It is based on tcp_client and works with our httpd server.
1) timer based, to run the test for 10 seconds
$ http_client --server 192.168.66.100:10000 --conn 100 --duration 10 --smp 2
========== http_client ============
Server: 192.168.66.100:10000
Connections: 100
Requests/connection: dynamic (timer based)
Requests on cpu 0: 33400
Requests on cpu 1: 33368
Total cpus: 2
Total requests: 66768
Total time: 10.011478
Requests/sec: 6669.145442
========== done ============
2) nr of reqs per connection based, to run the test with 100 connections
each has to run 1000 reqs
$ http_client --server 192.168.66.100:10000 --conn 100 --reqs 1000 --smp 2
========== http_client ============
Server: 192.168.66.100:10000
Connections: 100
Requests/connection: 1000
Requests on cpu 0: 50000
Requests on cpu 1: 50000
Total cpus: 2
Total requests: 100000
Total time: 15.002731
Requests/sec: 6665.453192
========== done ============
This patch is based on Shlomi's initial version.
Signed-off-by: Shlomi Livne <shlomi@cloudius-systems.com>
Signed-off-by: Asias He <asias@cloudius-systems.com>
Directory listing support, using subscription<sstring> to represent the
stream of file names produced by the directory lister running in parallel
with the directory consumer.
Remove pre-poll-mode code, from Gleb:
"This series moves most of eventfd users to use other form of notification
which can be polled without entering the kernel and moving epoll in its own
poller which is enabled only if there is an fd that needs to be polled."
[avi: add -lrt to linker command line]
It listens for requests on port 10000 and sends responses comprised of
three chunks of data in one packet. The chunk sizes are specified via
the --chunk-size argument.
The reqest can be anything, its content is ignored.
You can switch to equivalent copying version by passing --copy
argument.
The command line linking with DPDK's libraries looked like a cross between
random character generator and black magic. Reading a bit on the DPDK
mailing list, it turns out there is method in this madness (flawed method,
but method nontheless):
1. Instead of using "-l..." they used "-Wl,-l..." everywhere. Turns out
they did this ugliness to "hide" this option from libtool.
We don't use libtool, and don't need to hide anything from it.
2. They used "--start-group ... --end-group" to avoid having to figure
out the right link order.
It was easy to figure out the right link order and avoid this option.
3. They used "--whole-archive" on all the DPDK libraries. Unfortunately,
this option *is* needed, because the way DPDK is written, it is not
suited to be compiled into an (non-shared) library: Each of the DPDK
drivers ("librte_pmd_*") has a constructor function which needs to
run to register itself. This works fine with shared libraries (whose
constructors are run on load) but with a ".a" library, the whole
library is left out because nothing from the outside refers to any
of its symbols.
So what we should do is to use --whole-archive only on the PMD drivers,
and all will be fully compiled into the generated program. The rest of
the DPDK libraries will be linked normally, and hopefully because we
don't use large parts of DPDK, big chunks will not be compiled in.
If we don't add this "--whole-archive", none of the drivers will be
compiled into the program, the initialization will not be able to
find any driver, and just complain there are no ethernet ports.
After this patch, Seastar with DPDK still compiles, and runs.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Reviewed-by: Vlad Zolotarov <vladz@cloudius-systems.com>
With gcc 4.9.2, build with DPDK enabled breaks with error like:
../dpdk-1.7.1/x86_64-native-linuxapp-gcc/include/rte_pci.h:99:37:
warning: invalid suffix on literal; C++11 requires a space between literal
and string macro [-Wliteral-suffix]
#define PCI_SHORT_PRI_FMT "%.2"PRIx8":%.2"PRIx8".%"PRIx8
The problem is that C++11 outlawed, breaking decades of proud C-preprocessor
tradition, using a macro if stuck to the end of a string. But this used
in DPDK's header files, so we need to turn this error into a warning
(let's keep the warning, hopefully it will disappear in newer versions
of DPDK).
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
- Currently only a single port and a single queue are supported.
- All DPDK EAL configuration is hard-coded in the dpdk_net_device constructor instead
of coming from the app parameters.
- No offload features are enabled.
- Tx: will spin in the dpdk_net_device::send() till there is a place in the HW ring to
place a current packet.
- Tx: copy data from the `packet` frags into the rte_mbuf's data.
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
