Add a new "ContainerHealth" alert type that fires when a Docker container's
health check reports unhealthy, and resolves when it recovers. This mirrors
the existing Status (up/down) alert pattern: an alert can be armed per system
and honors the "min minutes" delay before firing.
When the alert fires, the notification (email and any configured webhook,
including Discord via shoutrrr) includes a log excerpt fetched live from the
agent for up to 2 of the unhealthy containers, prioritizing lines containing
"error" or "fatal" (falling back to the log tail if none match), capped to
keep the message well under Discord's size limit.
---------
Co-authored-by: hank <hank@henrygd.me>
Adds a user-configurable "Failed Services" alert that notifies when any
tracked systemd service enters the failed state, and again when all services
recover.
---------
Signed-off-by: Martin Stenröse <martin@stenrose.se>
Co-authored-by: henrygd <hank@henrygd.me>
- track pool capacity, health, I/O, scrub status, and vdev errors
- report dataset usage and correct ZFS filesystem metrics
- add pool charts, detail views, refresh controls, and health alerts
- persist pool details and include ZFS usage in disk alerts
- support configurable detail intervals and legacy agent compatibility
---------
Co-authored-by: hank <hank@henrygd.me>
createSessionWithTimeout checked sys.client for nil and then dereferenced
it again inside the goroutine that calls NewSession. update() runs the
SMART fetch in its own goroutine, so closeSSHConnection can clear the
field between those two reads and the goroutine dereferences a nil
client, panicking the whole hub process.
Make client an atomic.Pointer, load it once before starting the
goroutine, and clear it with Swap so a concurrent close cannot be
observed mid-session-creation. NewSession on an already-closed client
returns an error, which the existing retry path already handles.
Closes#2157
* fix(install): generate /etc/machine-id on systems without /proc
The install script read the fingerprint UUID from
/proc/sys/kernel/random/uuid, which FreeBSD does not have. On pfSense
the redirect still created /etc/machine-id, cat failed, and the agent
was left with an empty machine-id file. The [ ! -f ] guard then skipped
regeneration on every later run.
Fall back to uuidgen when /proc is unavailable, mirroring how the
script already picks between sha256sum and FreeBSD's sha256, and leave
no file behind when neither source exists.
* fix(install): regenerate empty machine id
refreshSystemDetails() takes NCPU and MemTotal from the Docker daemon's
/info response. That only describes this machine when the daemon shares its
kernel. On macOS and Windows Docker runs inside a Linux VM, so the system
details header shows the VM's memory as the host total, and the VM's CPU
count clamps both cores and threads through the lxc branch below it.
Only consult Docker's host info on platforms where the daemon runs natively.
Everything else already falls back to gopsutil, which reads this host.
SwapCached pages have been read back into memory but still occupy allocated swap slots. Subtracting them from SwapTotal - SwapFree underreported swap usage compared with free, Glances, and gopsutil's canonical SwapMemory metric.
Intel GPUs (intel_gpu_top) never report temperature or memory, so the
"suspended card" heuristic in calculateGPUAverage (temp == 0 &&
memoryUsed == 0) fired on every collection that landed between samples.
intel_gpu_top samples every 3.3s (intelGpuStatsInterval) while the hub's
realtime worker collects every 1s, so most realtime collections had no
new sample (delta count 0) and returned an empty GPUData with power
omitted (json "p"/"pp" are omitempty). The frontend derives the GPU
Power Draw series and legend from the latest sample, so the chart and
legend blanked on roughly two of every three or four one-second cycles.
NVIDIA/AMD were unaffected because they report temperature even when
idle, so the heuristic never fired and the last average was already
carried forward.
Gate the zero-return on non-engine (discrete) GPUs so Intel GPUs carry
the last average forward during between-sample gaps, matching the
existing NVIDIA/AMD behavior. Add a regression test.
Every other metric in getSystemStats is stored through utils.TwoDecimals.
The load averages were assigned straight from gopsutil, so whatever the
platform reported was recorded verbatim.
On Linux that goes unnoticed because /proc/loadavg is already two decimal
places. Everywhere else it is not. macOS and BSD divide a fixed point
value by fscale and produce numbers like 2.55322265625, and the Windows
implementation synthesises the average as a decaying EWMA over the
processor queue length counter, so an idle machine reports values like
1.3667392689044936e-73 instead of 0.
The hub already treats two decimals as the canonical precision for this
field, since records.go rounds the load average when it averages records.
That left the raw agent records as the only place carrying full precision.
The final message always echoed $PORT, which falls back to the default when -p
is not passed. Existing service files are kept as they are, so a plain upgrade
on a host with a custom port reported that the agent runs on 45876 regardless
of the actual configuration.
Read the address from the active service file instead. LISTEN is checked before
PORT to match the agent's own precedence in GetAddress, and the value is read as
text so host:port and unix socket paths are reported as configured.
Battery names come from firmware (sysfs model_name on Linux), which does not
guarantee valid UTF-8. The hub decodes agent payloads using the default
fxamacker/cbor decode mode, which rejects invalid UTF-8, so a single bad byte
in a battery name makes the hub drop the entire payload and mark the system
down until the agent is downgraded.
gopsutil's host.Uptime() calls the sysinfo(2) syscall. Inside an LXC
container lxcfs virtualizes /proc/uptime but cannot intercept a
syscall, so every container reported the host's uptime.
Reads /proc/uptime on linux and falls back to host.Uptime() if the file
is missing or unparseable, so other platforms are unchanged.
gopsutil's cpu.Info() does not parse the 'cpu model' field from
/proc/cpuinfo, which is the only source of CPU model names on MIPS.
Add a fallback that reads /proc/cpuinfo directly and combines
'cpu model' (e.g. 'MIPS 1004Kc V2.15') with 'system type'
(e.g. 'MediaTek MT7621 ver:1 eco:3') for a complete identifier.
The fallback only triggers when gopsutil returns an empty ModelName,
so x86/ARM/other architectures are unaffected.
henrygd
Santhi PrakashandGitHub
spatiumstasandGitHub
Steven HonsonandGitHub
Sven van GinkelandGitHub
Aditya Raj SinghandGitHub
Ryan ChouandGitHub
Erkinjon YusupovandGitHub
Luke WassandGitHub
dependabot[bot]andGitHub
Aditya Raj SinghandGitHub
MartinandGitHub
Toomore ChiangandGitHub
Alec RubinandGitHub
Ilya MuratovandGitHub
Jan DziąsłoandGitHub
Pavel ArtsishevskyandGitHub
