While #152 is still open, we need to allow for moderately sized allocations
to succeed. Extend the segment size to 256k, which allows for threads to
be allocated.
Fixes#151.
"Histograms are used to collect latency information, in Origin, many of the
operations are timed, this is a potential performance issue. This series adds
an option to sample the operations, where small amount will be timed and the
most will only be counted.
This will give an estimation for the statistics, while keeping an accurate
count of the total events and have neglectible performance impact.
The first to use the modified histogram are the column family for their read
and write."
Conflicts:
database.hh
To free memory, we need to allocate memory. In lsa compaction, we convert
N segments with average occupancy of (N-1)/N into N-1 new segments. However,
to do that, we need to allocate segments, which we may not be able to do
due to the low memory condition which caused us to compact anyway.
Fix by introducing a segment reserve, which we normally try to ensure is
full. During low memory conditions, we temporarily allow allocating from
the emergency reserve.
Currently, each column family creates a fiber to handle compaction requests
in parallel to the system. If there are N column families, N compactions
could be running in parallel, which is definitely horrible.
To solve that problem, a per-database compaction manager is introduced here.
Compaction manager is a feature used to service compaction requests from N
column families. Parallelism is made available by creating more than one
fiber to service the requests. That being said, N compaction requests will
be served by M fibers.
A compaction request being submitted will go to a job queue shared between
all fibers, and the fiber with the lowest amount of pending jobs will be
signalled.
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
The histogrm object is used both as a general counter for the number of
events and for statistics and sampling.
This chanage the histogram implementation, so it would support spares
sampling while keeping the total number of event accurate.
The implementation includes the following:
Remove the template nature of the histogram, as it is used only for
timer and use the name ihistogram instead.
If in the future we'll need a histogram for other types, we can use the
histogrma name for it.
a total counter was added that count the number of events that are part
of the statistic calculation.
A helper methods where added to the ihistogram to handle the latency
counter object.
According to the sample mask it would mark the latency object as start
if the counter and the mask are non zero and it would accept the latency
object in its mark method, in which if the latency was not start, it
will not be added and only the 'count' counter that counts the total
number of events will be incremented.
This should reduce the impact of latency calculation to a neglectable
effect.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
When doing a spares latency check, it is required to know if a latency
object was started.
This returns true if the start timer was set.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
When LSA reclaimer cannot reclaim more space by compaction, it
will reclaim data by evicting from evictable regions.
Currently the only evictable region is the one owned by the row cache.
Requiring alignment means that there must be 64K of contiguous space
to allocate each 32K segment. When memory is fragmented, we may fail
to allocate such segment, even though there's plenty of free space.
This especially hurts forward progress of compaction, which frees
segments randomly and relies on the fact that freeing a segment will
make it available to the next segment request.
Some code may attempt to use it during finalization after "instance"
was destroyed.
Reported by Pekka:
/usr/include/c++/4.9.2/bits/unique_ptr.h:291:14: runtime error:
reference binding to null pointer of type 'struct
standard_allocation_strategy'
./utils/allocation_strategy.hh:105:13: runtime error: reference
binding to null pointer of type 'struct standard_allocation_strategy'
./utils/allocation_strategy.hh:118:35: runtime error: reference
binding to null pointer of type 'struct allocation_strategy'
./utils/managed_bytes.hh:59:45: runtime error: member call on null
pointer of type 'struct allocation_strategy'
./utils/allocation_strategy.hh:82:9: runtime error: member access
within null pointer of type 'struct allocation_strategy'
The latency object is used to simplify calculating latencies. It uses a
start and stop time_point so the latency can be queried multiple time.
The start need to be done explicitely and not in the constructor to
allow reuse of the object.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
The histogram object is equivalent to the Histogram used in Origin. It
collect multiple values about the data:
Count, Min, Max, Sum, variance and the sum of square that are used for
std calculation.
It also contain a sample of the last n elements, that are stored in a
circular buffer.
The histogram is used by the API to report histogram statistics.
As the API does not support unsigned integer, the count is signed.
Typically the base type of the histogram is int64_t, so ihistogram was
defined as such.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Return a std::chrono::steady_clock::duration and switch the caller in
migration manager to also use proper C++ durations.
Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
The functionality is similar to RuntimeMBean.getUptime() that's needed
in schema pulling logic.
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
This is meant to allow std::moving the returned object when needed.
Otherwise std::move(s.get_vector()) will be degraded to copying.
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
The logger class constructor registers itself with the logger registry,
in order to enable dynamically setting log levels. However, since
thread_local variables may be (and are) initialized at the time of first
use, when the program starts up no loggers are registered.
Fix by making loggers global, not thread_local. This requires that the
registry use locking to prevent registration happening on different threads
from corrupting the registry.
Note that technically global variables can also be initialized at the
point of first use, and there is no portable way for classes to self-register.
However this is the best we can do.
This class implements an add-only vector that ensures that the elements are
unique.
As long as items are only added this class does essentially the same what Java's
LinkedHashSet does.
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
New in v2:
- Moved sequenced_set to its own .hh file.
There was a possibility for initialization disorder of static member _classes
and its usage in another static class.
Defining the _classes inside the static method that is called when it's accessed ensures
the proper initialization (aka "standard trick", quoting Avi ;)).
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
New in v2:
- storage_service: add a non-const version of get_token_metadata().
- get_broadcast_address(): check if net::get_messaging_service().local_is_initialized()
before calling net::get_local_messaging_service().listen_address().
- get_broadcast_address(): return an inet_address by value.
- system_keyspace: introduce db::system_keyspace::endpoint_dc_rack
- fb_utilities: use listen_address as broadcast_address for now
Without the check added in this patch if the class doesn't exist
a std::bad_function_call is thrown which is not very informative.
Signed-off-by: Paweł Dziepak <pdziepak@cloudius-systems.com>
max_size, as currently used, will return -1 and while it will fix the previous
bug, it will uncover another.
We can go back to using size, as long as we make sure that all sites correctly
pick a size when creating the bitset. Aside from that, for compatibility with
the java code, the total number of bits has to be a power of two.
The best way to achieve those goals, is to just set the size ourselves through
resize() in the filter constructor. num_blocks() * bits_per_block is guaranteed
to yield a power of two as we need, and in case one caller did not explicitly
set a size, it will be set from this moment on.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
We are using signed quantities to be compatible with the code java uses.
However, the current code will eventually overflow.
To avoid that, let's cast the quantities to unsigned, and then back to signed
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
There is a tricky bug in our current filter implementation: is_present will
return a different value depending on the order keys are inserted.
The problem here, is that _bitmap.size() will return the maximum *currently*
used bit in the set. Therefore, when we hash a given key, the maximum bit it
sets in the bitmap is used as "max" in the expression
results[i] = (base % max);
If the next keys do not set any bit higher than this one, everything works as
expected, because the keys will always hash the same way.
However, if one of the following keys happens to set a bit higher than the
highest bit which was set at the time a certain key was set, it will hash using
two different values of "max" in the aforementioned expression; one at insertion,
and another one at the test.
We should be using max_size() to be sure that we will always have the same
hash results.
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
