The hub collects monitor results only with stats requests, which stop
while
a system is paused or the agent is disconnected. Paused WebSocket agents
stay
connected, so they kept probing. Skip scheduled probes after three
default
update intervals without a stats request and resume on the next one.
Co-authored-by: Sven van Ginkel <svenvanginkel@icloud.com>
updateTemperatures detected duplicate SensorKeys by looking at the output
map, which only holds sensors that passed every filter. A sensor dropped by
SENSORS or SKIP_GPU therefore never claimed its name, so excluding the first
of two same-named sensors renamed the second one from nvme_sensor_2_4 to
nvme_sensor_2 and dropped it too, making it impossible to exclude just one.
Track handed-out names in a separate set that is populated before any filter
runs, so a sensor's key depends only on the hardware list.
Fan sensors were read with a plain os.ReadFile and no timeout. An i915
fan1_input read that blocks in the kernel held the agent lock forever,
stalling every hub request and the WebSocket read loop, so the system
went down and the agent never reconnected.
Fan reads now go through the same timeout as temperatures
(SENSORS_TIMEOUT, default 2s). A shared readWithTimeout helper tracks a
read still blocked after timing out and skips later polls until it
returns, instead of leaking another blocked goroutine each time. The
temperature path uses the helper too, fixing the same pile-up there.
Disk I/O has been sent in bytes since 0.13.0 (DiskIO for the system,
DiskReadBytes/DiskWriteBytes per filesystem). The agent no longer
populates the old MB-based DiskReadPs/DiskWritePs fields.
The fields are kept on Stats and FsStats only so agents < 0.13.0 still
decode. The hub migrates them on ingest: system-level to DiskIO and
per-filesystem to DiskReadBytes/DiskWriteBytes, so stored records and
the UI only use the byte fields.
Remove the MaxDiskReadPs/MaxDiskWritePs fields, their records averaging,
the UI MB fallbacks and the related TS types.
---------
Co-authored-by: hank <hank@henrygd.me>
NetworkSent/NetworkRecv were deprecated in 0.18.3 in favor of
Bandwidth. The hub already falls back to the old fields for older
agents, so the agent no longer needs to populate them.
Rebuild the SMART device list on every refresh instead of keeping
per-device type and parser state across rescans. Each device gets a
short ordered list of smartctl queries, and the three parsers are merged
into one that rejects output without health data so the next query runs.
- scsi devices try auto-detect, then -d sat (#2295)
- NVMe controllers fall back to their n1 namespace
- disks in standby reuse their last reading; a disk not yet collected
is woken once per agent run
- the last good reading survives temporary read failures
- failed or empty rescans keep the previous device list
- a missing SMART status is UNKNOWN instead of FAILED
- the reported disk type is the type that worked (e.g. sat)
- disks seen through two paths are reported once
---------
Co-authored-by: hank <hank@henrygd.me>
Agent.Start and ConnectionManager.Start now take a context.Context. The
manager runs until that context is cancelled or the process receives
SIGINT/SIGTERM. On shutdown it waits for in-flight reconnect attempts
before closing connections.
The reconnect delay, WebSocket dial and pending handshake now stop when
the context is cancelled. A context-aware dialer keeps proxy selection
from the environment. WebSocket callbacks no longer block on eventChan
after the event loop exits. A late reconnect can no longer restart the
SSH server once shutdown begins.
BREAKING CHANGE: agent.Agent.Start now requires a context.Context.
---------
Co-authored-by: hank <hank@henrygd.me>
Kernels before sysfs devinfo (pre-5.13, e.g. Synology DSM 4.4) report
every Btrfs pool health as UNKNOWN and expose no member devices, even
though the same facts are available through BTRFS_IOC_FS_INFO and
BTRFS_IOC_DEV_INFO on the mountpoint.
When the devinfo directory does not exist, enumerate devids via DEV_INFO
after confirming the filesystem through FS_INFO: members that answer are
listed with their recorded capacity, a member reporting no device path is
missing (DEGRADED), freed devids (ENODEV gaps from remove/replace) are
skipped, and the kernel device count is cross-checked against the sysfs
devices links so an unattributed missing member still reads DEGRADED.
Any ioctl failure or count mismatch leaves health UNKNOWN rather than
reporting unverified ONLINE.
The agent runs as the beszel user under ProtectSystem=strict, so package
managers that write temp, cache or log files failed with only
"exit status 1" in the log.
- Include the command's stderr in package update check errors.
- dnf5: point XDG_STATE_HOME and XDG_CACHE_HOME into the agent's data
dir. They default to $HOME, which is /nonexistent for the beszel user.
With -C, dnf5 still reads the system metadata cache.
- zypper: keep the solv cache in the data dir with --solv-cache-dir,
since the agent user cannot rebuild it in /var/cache/zypp.
- install-agent.sh, Debian unit: add PrivateTmp=yes. dnf4 (Rocky, Alma)
and zypper keep temporary files in /var/tmp, which ProtectSystem=strict
makes read-only. The install script also adds it to existing units
when re-run.
A refquota on a pool's root dataset caps its avail by the root's own
referenced data, so used+avail can understate pool capacity when child
datasets hold most of the data and trigger false disk usage alerts.
Fetch refquota with the root datasets and fall back to the raw zpool
figures for those pools.
The pool overview used `zpool list`, whose size and alloc count raw vdev
space. On raidz that includes parity, so a TrueNAS pool with 65 TB usable
showed 98 TB capacity, while the details page (root dataset) matched TrueNAS.
Take Size, Alloc and Free from the pool's root dataset (used+avail), the same
way Btrfs reports effective capacity. If the root dataset cannot be read, keep
the raw figures and set Raw so the UI labels them as raw usage.
Fixes#2539
unsafe_shutdowns was uint16, so drives reporting more than 65535 unsafe
shutdowns made json.Unmarshal fail and the entire NVMe SMART record was
discarded with only "no valid SMART data found" in the logs. Widen it
to uint, matching power_cycles (#2484), and log the unmarshal error at
debug level so similar failures are diagnosable.
Fixes#2582
Drop the cpu prefix from the cgroup path vars now that memory
accounting uses them too, and extract cgroupV2Dir() for the v2 mount
lookup previously repeated in CPU, memory and the root cgroup check.
Check /sys/class/ieee80211 before opening netlink sockets. Looking up
the
nl80211 family while cfg80211 is not loaded makes the kernel run
modprobe,
which loads the module on hosts without wireless hardware, or fails
again
on every poll when the module is unavailable.
Inside an LXC guest, lxcfs serves /proc/stat with the counters of the
host cores in the guest's cpuset, so an idle guest sharing a core with a
busy neighbor reported near-100% CPU.
When the agent detects it is running in an LXC guest (lxcfs mounted on
/proc/stat, container=lxc, or /run/systemd/container), derive CPU usage
from the guest's own cgroup instead: cpu.stat on cgroup v2, cpuacct on
v1, read at the cgroup mount root so it covers every process in the
guest. Usage is normalized by the usable cores (the smallest of the
affinity mask, cpuset and CPU quota).
Per-core usage is omitted in this mode, since the per-core /proc/stat
counters describe shared host cores and would contradict the total.
Iowait and steal are not available from cgroup accounting and report as
zero.
Hosts and Docker/Podman agents are unaffected and keep reading
/proc/stat, as does an LXC guest whose cgroup accounting is unreadable.
---------
Co-authored-by: hank <hank@henrygd.me>
Make WebSocket reconnect and SSH fallback transitions reliable across asynchronous disconnects, stale callbacks, and overlapping connections. Keep the SSH listener available while disconnected and allow a verified WebSocket connection to take precedence when it recovers.
Co-authored-by: henrygd <hank@henrygd.me>
Right after agent start the hub requests stats immediately, and the first
sample of the interval was measured from the baseline taken at startup, only
a second or so earlier. A burst of startup I/O was then stored as the rate
for the whole minute, causing large spikes in disk I/O charts.
The first sample of an interval now must span at least half the interval.
Otherwise it only stores the snapshot, and the next sample is measured from it.
Vincent FranstyoandGitHub
ErnestoMunizandGitHub
henrygdandSven van Ginkel
yi111andGitHub
Santhi PrakashandGitHub
kenny-ishandGitHub
Rajeev GadgilandGitHub
spatiumstasandGitHub
Bharat KauravandGitHub
Fuwuwuwuandhank
MarrrrrrrrryandGitHub
Jeroen De WildeandGitHub