At the moment, when local node send a mutation to remote node, it will
wait for remote node to apply the mutation and send back a response,
then it will send the next mutation. This means the sender are sending
mutations one by one. To optimize, we can make the sender send more
mutations in parallel without waiting for the response. In order to
apply back pressure from remote node, a per shard mutation send limiter
is introduced so that the sender will not overwhelm the receiver.
Otherwise code in stream_session.cc
_transfers.emplace(cf_id, stream_transfer_task(shared_from_this(), cf_id)).first;
will copy the stream_transfer_task and in turn copy the
outgoing_file_message which is not copyable due to the semaphore member.
Thanks avi for figuring this out.
This patch rearrange scylla.yaml, without adding or omitting parameters, to two segments:
1. Supported parameters
2. Not supported parameters: saved for future use or backward compatibility
Signed-off-by: Tzach Livyatan <tzach@cloudius-systems.com>
"This adds the get_snitch_name and update_snitch functionality to the API. After
this series it would be possible to return the snitch name and to update the
snitch."
While #152 is still open, we need to allow for moderately sized allocations
to succeed. Extend the segment size to 256k, which allows for threads to
be allocated.
Fixes#151.
"This series cleans the streaming_histogram and the estimated histogram that
were importad from origin, it then uses it to get the estimated min and max row
estimation in the API."
"Histograms are used to collect latency information, in Origin, many of the
operations are timed, this is a potential performance issue. This series adds
an option to sample the operations, where small amount will be timed and the
most will only be counted.
This will give an estimation for the statistics, while keeping an accurate
count of the total events and have neglectible performance impact.
The first to use the modified histogram are the column family for their read
and write."
Conflicts:
database.hh
We can catch most errors when we try to load an sstable. But if the TOC file is
the one missing, we won't try to load the sstable at all. This case is still an
invalid case, but it is way easier for us to treat it by waiting for all files
to be loaded, and then checking if we saw a file during scan_dir, without its
corresponding TOC.
Fixes#114
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
[in v2: 1. Fixed a few small bugs.
2. Added rudementary support parallel/sequential repair.
3. Verified that code works correctly with Asias's fix to streaming]
This patch adds the capability to track repair operations which we have
started, and check whether they are still running or completed (successfully
or unsuccessfully).
As before one starts a repair with the REST api:
curl -X GET --header "Content-Type: application/json" --header "Accept: application/json" "http://127.0.0.1:10000/storage_service/repair_async/try1"
where "try1" is the name of the keyspace. This returns a repair id -
a small integer starting with 0. This patch adds support for similar
request to *query* the status of a previously started repair, by adding
the "id=..." option to the query, which enquires about the status of the
repair with this id: For example.,
curl -i -X GET --header "Content-Type: application/json" --header "Accept: application/json" "http://127.0.0.1:10000/storage_service/repair_async/try1?id=0"
gets the current status of this repair 0. This status can be RUNNING,
SUCCESSFUL or FAILED, or a HTTP 400 "unknown repair id ..." in case an
invalid id is passed (not the id of any real repair that was previously
started).
This patch also adds two alternative code-paths in the main repair flow
do_repair_start(): One where each range is repaired one after another,
and one where all the ranges are repaired in parallel. At the moment, the
enabled code is the parallel version, just as before this patch. But the
will also be useful for implementing the "parallel" vs "sequential" repair
options of Cassandra.
Note that if you try to use repair, you are likely to run into a bug in
the streaming code which results in Scylla either crashing or a repair
hanging (never realising it finished). Asias already has a fix this this bug,
and will hopefully publish it soon, but it is unrelated to the repair code
so I think this patch can independently be committed.
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
The solution was proposed by Nadav. When writing a new sstable,
write all usual files, write the TOC to a temporary file, and
then rename it, which is atomic.
Files not belonging to any TOC are invalid, so we ensure that
partially written sstables aren't reused.
Avi also proposed using fsync on the sstable directory to guarantee
that the files reached the disk before sealing the sstable.
Subsequently, we should add code to avoid loading sstable which
TOC is either temporary or doesn't exist. Temporary TOC files
should also be deleted.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
There are two counters for the "current number of open connections" in the cql_server:
- _connects: incremented every time a new connection is opened. Should be used for
a derived statistics of connections/sec
- _connections: incremented and decremented every time a new connection is opened/closed
correspondingly.
_connects has been registered as a source for both derived and gauge collectd statistics by
mistake while it had to be registered for a derived counter only and _connections had to be
registered as a source for a gauge counter.
Fixes issue #143
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
mutation_result_merger can outlive query::read_command, so it have to
hold shared pointer to it instead of reference. The bug was introduced by
89e36541c3
This allows token::_data to be in a different representation
than the one expected by the token type.
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
_v.begin() points to the next element. If the size of last element
in a compound is zero then iterators pointing to second to last and
last element would seem equal. To fix this we also have to compare
_types_left.
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
* seastar b56a6eb...432e973 (3):
> dpdk: merge local patch to fix ixgbe
> dpdk: rebase to latest upstream
> net::dpdk: actually check the resulting cluster and not the original packet
Currently limit is enforced only on partition boundary, so real result
can contain 2*row_limit - 1 rows in the worst case. Fix it by trimming
rows from a mutation if only part of its rows fit the requested limit.
2.2 allows IN on any column and it seems that we support that fine, but
DTESTs except us to reject such queries.
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
Because of the reverse flag in partition slice rows inside bounds will
be returned in reversed order, however, we still have to make sure
that the bounds are in the expected order.
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
Values inside IN clause should be sorted and duplicates removed if the
restricted columns are part of the clustering key, which is always true
for multi column restrictions.
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
Values inside IN () restrictions may be either in a vector _in_values or
a marker (_in_marker or _value). To determine which one is appropriate
we check whether _in_values is empty, which is wrong because IN clause
can be empty (and there is no marker in such case). This is fixed by
using the presence of a marker to determine whether a vector of values
or a marker should be used.
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
