Add a new "ContainerHealth" alert type that fires when a Docker container's
health check reports unhealthy, and resolves when it recovers. This mirrors
the existing Status (up/down) alert pattern: an alert can be armed per system
and honors the "min minutes" delay before firing.
When the alert fires, the notification (email and any configured webhook,
including Discord via shoutrrr) includes a log excerpt fetched live from the
agent for up to 2 of the unhealthy containers, prioritizing lines containing
"error" or "fatal" (falling back to the log tail if none match), capped to
keep the message well under Discord's size limit.
---------
Co-authored-by: hank <hank@henrygd.me>
Adds a user-configurable "Failed Services" alert that notifies when any
tracked systemd service enters the failed state, and again when all services
recover.
---------
Signed-off-by: Martin Stenröse <martin@stenrose.se>
Co-authored-by: henrygd <hank@henrygd.me>
- track pool capacity, health, I/O, scrub status, and vdev errors
- report dataset usage and correct ZFS filesystem metrics
- add pool charts, detail views, refresh controls, and health alerts
- persist pool details and include ZFS usage in disk alerts
- support configurable detail intervals and legacy agent compatibility
---------
Co-authored-by: hank <hank@henrygd.me>
createSessionWithTimeout checked sys.client for nil and then dereferenced
it again inside the goroutine that calls NewSession. update() runs the
SMART fetch in its own goroutine, so closeSSHConnection can clear the
field between those two reads and the goroutine dereferences a nil
client, panicking the whole hub process.
Make client an atomic.Pointer, load it once before starting the
goroutine, and clear it with Swap so a concurrent close cannot be
observed mid-session-creation. NewSession on an already-closed client
returns an error, which the existing retry path already handles.
Closes#2157
- Report battery data for individual devices
- Select a representative battery for legacy fields and alerts
- Average named battery data independently
- Display multiple batteries in system charts
- Add cross-platform coverage and transport tests
intel_gpu_top does not support the xe driver, so skip it for xe devices
and let the existing nvtop last-resort collector handle them. nvtop leaves
device_name unset on xe, so name the GPU from its PCI device id ("Intel GPU
(<id>)"). Adds nvtop to the Intel agent image (gputop already ships with
igt-gpu-tools).
The per-system updater reused an SSH client across ticks and ran the data
exchange with no deadline. If a connection went half-open (dead peer that
never sends RST/FIN) or an agent accepted the session but never wrote a
response, the read in fetchDataViaSSH blocked forever. Because
StartUpdater calls update() synchronously on its ticker, a blocked read
froze the whole per-system goroutine: the ticker's subsequent ticks were
dropped, no error was returned so the system stayed "up", and the agent
was never re-dialed until the hub process restarted.
Bound each SSH data exchange with sshOperationTimeout via runWithTimeout:
on timeout the connection is torn down (unwinding the blocked read) and a
retryable error is returned, so the next tick re-dials. Also enable TCP
keep-alive on dialed connections as a backstop for genuine network death.
Fixes#2041
CalculateCpuPercentLinux subtracted the stored previous counters from the
current ones without checking direction. When a stats response is processed
after a newer one for the same container, or an accounting counter resets,
the current total reads lower and the unsigned subtraction wraps to ~2^64
instead of going negative.
On the container counter that surfaces as the reported error, and the sample
is discarded along with the container's network stats:
cpu pct greater than 100: 1.15292150348562e+13
On the system counter it is quieter and worse: the wrapped value inflates the
divisor, so the percentage collapses toward zero and is stored as a healthy
sample rather than rejected. A synthetic rollback measures 2.7e-12 percent.
Both directions are now treated as a new baseline (0% for one sample), which
matches how the function already handles the first-run case.
CalculateCpuPercentWindows had the same unguarded subtraction and is fixed
the same way.
Fixes#2149
Co-authored-by: Ryan Chou <ryanchou1994@users.noreply.github.com>
henrygd
Santhi PrakashandGitHub
spatiumstasandGitHub
Steven HonsonandGitHub
Sven van GinkelandGitHub
Aditya Raj SinghandGitHub
Erkinjon YusupovandGitHub
Pavel ArtsishevskyandGitHub
Donggyu KwonandGitHub
Sai Asish YandGitHub
Miłosz KolberandGitHub
Lars LehtonenandGitHub
TowyTowyandGitHub
