The current code (this will change soon with my reactor patches)
constructs a default (Posix) network stack before reactore::configure()
reassigns it to the requested network stack.
It turns out there is one place we use the network stack before calling
reactore::configure(), which ends up using the Posix stack even though
we want the native stack - this is both silly and plainly doesn't work on
the OSv setup.
The problem is that app_template.hh tries to configure scollectd before
the engine is started. This calls scollectd::impl::start() which calls
engine.net().make_udp_channel(). When this happens this early, it creates
a Posix socket...
This patch moves the scollectd configuration to after the engine is
started. It makes sense to me: As far as I understand, scollectd is all
about sending packets (diagnostic packets), and it's kind of silly to
start sending packets before starting the machinary which allows us to
send packets.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
[avi: use customary indentation, remove unneeded make_ready_future()]
Use when you don't want to care about the result and just want to
return a future<>.
The current implementation may not be the most optimal way to do it
but it can be improved later if there's need.
It's more convenient for users that way. If someone wants to pass a
reference, we use a reference. If he passes an r-value, we accept it
and use parameter l-value instead.
This patch adds "smp queue polling before going idle" to the reactor.
It allows to avoid signalfd overhead in case receiver thread is not idle
when message is sent. With this patch on top of two other patches from
me that are still waiting to be committed I see 450120 Requests/sec with
wrk and "httpd -c 2 --network-stack native" with native stack. With one
cpu the result is 316002, so we have around 40% scaling. The bottleneck
in this test is cpu 0 which takes 100% cpu time.
If we don't, we start the system before we have an IP address, and when
we actually do get the IP address, we fail an assert on the _config promise,
which was already fulfilled.
This is useful for features that are provided incrementally, so may not
be present on all hypervisors. If the value is not present, return a
user-provided default, which also has a system-provided default (0).
We current have one port per event channel. We need to have a list of
semaphores that will all be made ready when an interrupt kicks in. This is
useful in the case where both tx and rx are bound to the same event channel.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
If we do that, plus make it an instance method, we should be able to use
make_ready_port. This is consistent with the userspace implementation and
from that point any changes there will be propagated to both.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
The representation of an event channel as an integer poses a problem, in which
waiting on an integer port doesn't work well when the same event channel is
assigned for both tx and rx. The future will be ready for one of the sides, but
we won't process the other.
One alternative is to have conditions in the future processing, and in case the
event channels are bound to the same port, process both events. But a better
solution is to use a class to represent the bound ports, and instances of those
classes will have their own pending methods.
Infrastructure will be written in a following patch to make sure that all
listeners to the same port will be made ready when an interrupt kicks in
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
The backend may be completely silent about the existence of the split channels feature.
In that case, trying to read through the template directly would cause an exception,
since we can't convert the empty string.
The backend-id, OTOH, is guaranteed to exist and wasn't using the template signature.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
We copy our grant reference into a temporary, so free_ref() does not
clear the real entry, causing an assert() to trigger later on.
Fix by capturing the grant reference entry by reference.
With this, the xen network driver survives multiple trips around the ring.
Keep recycling free ring entries back into the receive ring so we can
receive more than 256 packets.
The code is a little lame at the moment since it writes the index and
notifies the host for every frame, but that can be adjusted later.
There is no reason to wait when pushing back a free id - there is nothing
that could possibly block there.
Switch from a queue<> to an std::queue<> and use a semaphore to guard
popping from the queue.
Running tcp stream test with --smp > 1, sometimes the server sends TSO
frame, sometimes it does not. If we set --smp = 1, the server always
sends TSO frame. This is because the proxy device does not parse all the
features in the opts. We should copy the _hw_features from the real
device but it is not easy. For now, we simply duplicate the parse code.
Fix UDP for memcache with native stack
memcached: apps/memcached/memcached.cc:807:
void memcache::assert_resolved(future<>): Assertion `f.available()'
failed.
Tomek writes:
UDP path relied on the fact that handle() could not block, because
the output stream does not block, and passed references to variables
which live on stack. Since you now can block in handle_get(), this
no longer holds. We should chnage that, ie allocate conversation
state like we do in TCP.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Fix tcp_server tx test. We still have more to do.
Native stack:
$ go run client-txtx.go
Bytes Received(MiB): 1000
Total Time(Secs): 1.567927562
Bandwidth(MiB/Sec): 637.7845662234746
Posix stack:
$ go run client-txtx.go
Bytes Received(MiB): 1000
Total Time(Secs): 1.014354958
Bandwidth(MiB/Sec): 985.8481906291427
Note: client-txtx uses 100 concurrent connections.
With TSO enabled, we can see a Ethernet frame larger than 64K on tap
device. This makes wireshark unable to handle. It complains:
The capture file appears to be damaged or corrupt.
(pcapng_read_packet_block: cap_len 65549 is larger than
WTAP_MAX_PACKET_SIZE 65535.)
handle buffer recycles. Right now it is very simple: allocate a new receive
buffer after a succesful receival, and mark the tx spot free when we get the tx
event notification.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Instead of returning a reference to a grant that is already present in an
array, defer the initialization. This is how the OSv driver handles it, and I
honestly am not sure if this is really needed: it seems to me we should be able
to just reuse the old grants. I need to check in the backend code if we can be
any smarter than this.
However, right now we need to do something to recycle the buffers, and just
re-doing the refs would lead to inconsistencies. So the best by now is to close
and reopen the grants, and then later on rework this in a way that works for
both the initial setup and the recycle.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Right now, we allocate the whole index, and notify the backend that we have
produced nr_ents indexes. If we do that, we cannot increment the producer index
when we receive a new package. This would make the index overflow, and
basically, it is the responsible for the biggest part of the slowdown we are
seeing.
Before this patch, we're seeing 2s RTT for pings. After the patch:
64 bytes from 192.168.100.79: icmp_seq=1 ttl=64 time=0.437 ms
64 bytes from 192.168.100.79: icmp_seq=2 ttl=64 time=0.431 ms
64 bytes from 192.168.100.79: icmp_seq=3 ttl=64 time=0.475 ms
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Aside from managing the grant references, we also need to manage the positional
indexes in the array. We need to keep track of which indexes are free, and
which are used. Because we need the actual position number to fill xen's data
structures, I figured we could use a queue and then fill it up with all the
integers in our range. The queue is already futurized, so that's easy.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
