Now that the histogram has its own unit expressed in its template
parameter, there is no reason to convert it to nano just so we may need
to convert it back if the histogram needs another unit.
This patch will keep everything as a duration until last moment, and
then we'll convert when needed.
This was suggested by Amnon.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <218efa83e1c4ddc6806c51913d4e5f82dc6d231e.1479139020.git.glauber@scylladb.com>
We are tracking latencies in microseconds, but almost everywhere else
they are reported in microseconds. Instead of just converting, this
patch tries to be a bit more future proof and embed the unit into the
type - and we then default to microseconds.
I have verified that the JMX measures now report sane values for both
the storage proxy and the column family. nodetool cfhistograms still
works fine. That one is reported in nanoseconds, but through the
estimated_histogram, not ihistogram.
Fixes#1836
Signed-off-by: Glauber Costa <glauber@scylladb.com>
We have recently fixed a bug due to which the constructor parameters for
moving average were inverted, leading to the numbers being just plain
wrong. However, the calculation of alpha was already inverted, meaning
it was right by accident and now that's wrong.
With the wrong alpha, the values we see are still correct, but they move
very quickly. The intention of this code is obviously to smooth things
out.
This was found out by Nadav. I have tested and confirmed that the smoothing
factor now works as expected.
Fixes #1837
Signed-off-by: Glauber Costa <glauber@scylladb.com>
moving_averages constructor is defined like this:
moving_average(latency_counter::duration interval, latency_counter::duration tick_interval)
But when it is time to initialize them, we do this:
... {tick_interval(), std::chrono::minutes(1)} ...
As it can be seen, the interval and tick interval are inverted. This
leads to the metrics being assigned bogus values.
Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <d83f09eed20ea2ea007d120544a003b2e0099732.1478798595.git.glauber@scylladb.com>
The histogram implementation uses sampling to estimate the mean and sum.
This patch adds a method that returns an estimated sum based on the mean
and the total number of events measured.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1467547341-30438-2-git-send-email-amnon@scylladb.com>
The rate_moving_average is used by timed_rate_moving_average to return
its internal values.
If there are no timed event, the mean_rate is not propertly initilized.
To solve that the mean_rate is now initilized to 0 in the structure
definition.
Refs #1306
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1465231006-7081-1-git-send-email-amnon@scylladb.com>
This patch adds a few data structure for derived and accumulative
statistics that are similiar to the yammer implementation used by the
JMX.
It also adds a plus operator to histogram which cleans the histogram
usage.
moving_average - An exponentially-weighted moving average. calculate an event rate
on a given interval.
rate_moving_average and timed_rate_moving_average - Calculate 1m, 5m and
15m ewma an all time avrage and a counter.
rate_moving_average_and_histogram and
timed_rate_moving_average_and_histogram - Combines a histogram with a
rate_moving_average. It also expose a histogram API so it will be an
easy task to replace a histogram with a
timed_rate_moving_average_and_histogram.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
This patch adds a started counter, that is used to mark the number of
operation that were started.
This counter serves two purposes, it is a better indication for when to
sample the data and it is used to indicate how many pending operations
are.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
This patch contains two changes to the histogram implementation. It uses
a simpler method to calculate the estimated mean (simply divide the
estimated sum with the number of samples) and to make sure that there
will always be values in the histogram, it start with taking a sample
(when there are no samples) and then use the mask to decide if to sample
or not.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
The histogrm object is used both as a general counter for the number of
events and for statistics and sampling.
This chanage the histogram implementation, so it would support spares
sampling while keeping the total number of event accurate.
The implementation includes the following:
Remove the template nature of the histogram, as it is used only for
timer and use the name ihistogram instead.
If in the future we'll need a histogram for other types, we can use the
histogrma name for it.
a total counter was added that count the number of events that are part
of the statistic calculation.
A helper methods where added to the ihistogram to handle the latency
counter object.
According to the sample mask it would mark the latency object as start
if the counter and the mask are non zero and it would accept the latency
object in its mark method, in which if the latency was not start, it
will not be added and only the 'count' counter that counts the total
number of events will be incremented.
This should reduce the impact of latency calculation to a neglectable
effect.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
The histogram object is equivalent to the Histogram used in Origin. It
collect multiple values about the data:
Count, Min, Max, Sum, variance and the sum of square that are used for
std calculation.
It also contain a sample of the last n elements, that are stored in a
circular buffer.
The histogram is used by the API to report histogram statistics.
As the API does not support unsigned integer, the count is signed.
Typically the base type of the histogram is int64_t, so ihistogram was
defined as such.
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
