Recently we started to use more the concept of metric labels - several
metrics which share the same name, but differ in the value of some label
such a "group" (for different scheduling groups).
This patch documents this feature in docs/metrics.md, gives the example of
scheduling groups, and explains a couple more relevant Promethueus syntax
tricks.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20190909113803.15383-1-nyh@scylladb.com>
Prometheus needs to remember which "instance" (node) each measurement
came from. But it doesn't actually need Scylla to tell it the instance
name - it knows which node it got each measurement from.
After Seastar commit 79281ef287
which fixed Seastar issue https://github.com/scylladb/seastar/issues/477,
the "instance" label on measurements no longer comes from Scylla but rather
is added by Prometheus. This patch corrects the documentation to explain the
current situation, instead of incorrectly saying that Scylla adds the
"instance" label itself.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20190602074629.14336-1-nyh@scylladb.com>
docs/metrics.md so far explained just the REST API for retrieving current
metrics from a single Scylla node. In this patch, I add basic explanations
on how to use the Prometheus and Grafana tools included in the
"scylla-grafana-monitoring" project.
It is true that technically, what is being explained here doesn't come
with the Scylla project and requires the separate scylla-grafana-monitoring
to be installed as well. Nevertheless, most Scylla developers will need this
knowledge eventually and suprisingly it appears it was never documented
anywhere accessible to newbie developers, and I think metrics.md is the
right place to introduce it.
In fact, I myself wasn't aware until today that Prometheus actually had
its own Web UI on port 9090, and that it is probably more useful for
developers than Grafana is.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Reviewed-by: Botond Denes <bdenes@scylladb.com>
Message-Id: <20190129114214.17786-1-nyh@scylladb.com>
Today I realised that although we have per-table metrics, they are not
*really* available by default. I was suprised to find that we don't have
(as far as I can tell) a document explaining why it is so, or how to enable
them anyway. Moreover, the more I investigated this issue, the more I
realised how little I know on Scylla's metrics - how they are calculated,
how they are collected, their different types, and so on.
So I sat down to figure out everything I wanted to learn about Scylla metrics,
and then wrote it all down in a new document, docs/metrics.md.
There are some missing pieces in this document marked by TODO, and probably
additional missing pieces that I'm not aware of, but I think this is already
a good start and can be (and should be) improved-on later.
We really need to have more of these documents describing various Scylla
subsystems to new developers - what each subsystem does, why it does what
it does, where is the code, and so on. I am facing these problems every
day as a seasoned developer - I can't even imagine what our new developers
face when trying to understand a subsystem they are not yet familiar with.
Signed-off-by: Nadav Har'El <nyh@scylladb.com>
Message-Id: <20180920131103.20590-1-nyh@scylladb.com>
