mirror of
https://github.com/scylladb/scylladb.git
synced 2026-05-30 19:46:48 +00:00
169 lines
4.6 KiB
ReStructuredText
169 lines
4.6 KiB
ReStructuredText
Scylla Monitoring Stack
|
|
=======================
|
|
|
|
This document describes the setup of Scylla monitoring Stack, base on :doc:`Scylla Prometheus API <monitoring-apis>`.
|
|
|
|
Scylla monitoring stack consists of three components, wrapped in Docker containers:
|
|
|
|
* `prometheus` - collects and stores metrics
|
|
* `alertmanager` - handles alerts
|
|
* `grafana` - dashboard server
|
|
|
|
.. image:: monitor.png
|
|
|
|
The monitoring stack needs to be installed on a dedicated server, external to the Scylla cluster. Make sure the monitoring server have access to the Scylla nodes so that it can pull the metrics over the Prometheus API.
|
|
|
|
For evaluation, you can run Scylla monitoring stack on any server (or laptop) that can handle two Docker instances at the same time. For production, see recommendation below.
|
|
|
|
.. include:: min-prod-hw.rst
|
|
|
|
|
|
Prerequisites
|
|
.............
|
|
|
|
* `docker`_
|
|
|
|
.. _`docker`: https://www.docker.com/
|
|
|
|
|
|
Procedure
|
|
.........
|
|
|
|
1. Download and extract the latest `Scylla Monitoring Stack binary`_; for example, for release 2.2.0
|
|
|
|
.. _`Scylla Monitoring Stack binary`: https://github.com/scylladb/scylla-monitoring/releases
|
|
|
|
.. code-block:: sh
|
|
|
|
wget https://github.com/scylladb/scylla-monitoring/archive/scylla-monitoring-2.2.tar.gz
|
|
tar -xvf scylla-monitoring-2.2.tar.gz
|
|
cd scylla-monitoring-scylla-monitoring-2.2
|
|
|
|
As an alternative, you can clone and use the git repository directly.
|
|
|
|
.. code-block:: sh
|
|
|
|
git clone https://github.com/scylladb/scylla-monitoring.git
|
|
cd scylla-monitoring
|
|
git checkout branch-2.2
|
|
|
|
2. Start docker service if needed
|
|
|
|
.. code-block:: sh
|
|
|
|
centos $ sudo service docker start
|
|
ubuntu $ sudo systemctl restart docker
|
|
|
|
|
|
|
|
3. Update ``prometheus/scylla_servers.yml`` with the targets' IPs (the servers you wish to monitor).
|
|
|
|
.. note::
|
|
It is important that the dc in the target files will match the datacenters names used by Scylla.
|
|
Use the ``nodetool status`` command to validate the datacenter names used by Scylla.
|
|
|
|
For example:
|
|
|
|
.. code-block:: yaml
|
|
|
|
targets:
|
|
- 172.17.0.2
|
|
- 172.17.0.3
|
|
labels:
|
|
cluster: cluster1
|
|
dc: dc1
|
|
|
|
|
|
For general node information (disk, network, etc.) Scylla Monitoring Stack uses the node_exporter agent that runs on the same machine as Scylla does.
|
|
By default, Prometheus will assume you have a node_exporter running on each machine. If this is not the case, you can override the node_exporter
|
|
targets configuration file by creating an additional
|
|
file and pass it with the ``-n`` flag.
|
|
|
|
.. note::
|
|
By default, there is no need to configure ``node_exporter_server.yml``. Prometheus will use the same targets it uses for
|
|
Sylla and will assume you have a node_exporter
|
|
running on each Scylla server.
|
|
|
|
|
|
It is possible to configure your own target file instead of updating ``scylla_servers.yml``, using the ``-s`` for scylla target file.
|
|
|
|
For example:
|
|
|
|
.. code-block:: yaml
|
|
|
|
./start-all.sh -s my_scylla_server.yml -d data_dir
|
|
|
|
|
|
Use Labels to mark different Data Centers
|
|
|
|
As can be seen in the examples, each target has its own set of labels to mark the cluster name and the data center (dc).
|
|
You can add multiple targets in the same file for multiple clusters or multiple data centers.
|
|
|
|
4. Connect to `Scylla Manager`_ by updating ``prometheus/scylla_manager_servers.yml``
|
|
If you are using Scylla Manager, you should set its ip.
|
|
|
|
.. _`Scylla Manager`: /operating-scylla/manager/
|
|
|
|
For example
|
|
|
|
.. code-block:: yaml
|
|
|
|
# List Scylla Manager end points
|
|
|
|
- targets:
|
|
- 127.0.0.1:56090
|
|
|
|
|
|
Note that you do not need to add labels to the Scylla Manager targets.
|
|
|
|
|
|
Start and Stop
|
|
..............
|
|
|
|
Start
|
|
|
|
.. code-block:: yaml
|
|
|
|
./start-all.sh -d data_dir
|
|
|
|
|
|
Stop
|
|
|
|
.. code-block:: yaml
|
|
|
|
./kill-all.sh
|
|
|
|
Setting Specific Version
|
|
........................
|
|
|
|
By default, start-all.sh will start with dashboards for the latest two Scylla versions and the latest Scylla Manager version.
|
|
|
|
You can specify specific scylla version with the ``-v`` flag and Scylla Manager version with ``-M`` flag
|
|
|
|
For example:
|
|
|
|
.. code-block:: sh
|
|
|
|
./start-all.sh -v 3.0,master -M 1.3
|
|
|
|
will load the dashboards for Scylla versions ``3.0`` and ``master`` and the dashboard for Scylla Manager ``1.3``
|
|
|
|
View Grafana Monitor
|
|
....................
|
|
|
|
Point your browser to ``your-server-ip:3000``
|
|
By default, Grafana authentication is disabled. To enable it and set a password for user admin use the ``-a`` option
|
|
|
|
For example:
|
|
|
|
.. code-block:: yaml
|
|
|
|
10.10.0.1:3000
|
|
|
|
* Choose Disk and network interface
|
|
|
|
The dashboard holds a drop down menu at its upper left corner for disk and network interface.
|
|
You should choose relevant disk and interface for the dashboard to show the graphs.
|
|
|
|
|