The hub collects monitor results only with stats requests, which stop
while
a system is paused or the agent is disconnected. Paused WebSocket agents
stay
connected, so they kept probing. Skip scheduled probes after three
default
update intervals without a stats request and resume on the next one.
Co-authored-by: Sven van Ginkel <svenvanginkel@icloud.com>
updateTemperatures detected duplicate SensorKeys by looking at the output
map, which only holds sensors that passed every filter. A sensor dropped by
SENSORS or SKIP_GPU therefore never claimed its name, so excluding the first
of two same-named sensors renamed the second one from nvme_sensor_2_4 to
nvme_sensor_2 and dropped it too, making it impossible to exclude just one.
Track handed-out names in a separate set that is populated before any filter
runs, so a sensor's key depends only on the hardware list.
Fan sensors were read with a plain os.ReadFile and no timeout. An i915
fan1_input read that blocks in the kernel held the agent lock forever,
stalling every hub request and the WebSocket read loop, so the system
went down and the agent never reconnected.
Fan reads now go through the same timeout as temperatures
(SENSORS_TIMEOUT, default 2s). A shared readWithTimeout helper tracks a
read still blocked after timing out and skips later polls until it
returns, instead of leaking another blocked goroutine each time. The
temperature path uses the helper too, fixing the same pile-up there.
Authenticate the request and expect 404 so the test checks that the
route is not registered without CHECK_UPDATES. Tests run with the
no_ui tag, so the status is stable.
Disk I/O has been sent in bytes since 0.13.0 (DiskIO for the system,
DiskReadBytes/DiskWriteBytes per filesystem). The agent no longer
populates the old MB-based DiskReadPs/DiskWritePs fields.
The fields are kept on Stats and FsStats only so agents < 0.13.0 still
decode. The hub migrates them on ingest: system-level to DiskIO and
per-filesystem to DiskReadBytes/DiskWriteBytes, so stored records and
the UI only use the byte fields.
Remove the MaxDiskReadPs/MaxDiskWritePs fields, their records averaging,
the UI MB fallbacks and the related TS types.
---------
Co-authored-by: hank <hank@henrygd.me>
NetworkSent/NetworkRecv were deprecated in 0.18.3 in favor of
Bandwidth. The hub already falls back to the old fields for older
agents, so the agent no longer needs to populate them.
Rebuild the SMART device list on every refresh instead of keeping
per-device type and parser state across rescans. Each device gets a
short ordered list of smartctl queries, and the three parsers are merged
into one that rejects output without health data so the next query runs.
- scsi devices try auto-detect, then -d sat (#2295)
- NVMe controllers fall back to their n1 namespace
- disks in standby reuse their last reading; a disk not yet collected
is woken once per agent run
- the last good reading survives temporary read failures
- failed or empty rescans keep the previous device list
- a missing SMART status is UNKNOWN instead of FAILED
- the reported disk type is the type that worked (e.g. sat)
- disks seen through two paths are reported once
---------
Co-authored-by: hank <hank@henrygd.me>
Adds a NetworkMonitorCert alert with a threshold in days, evaluated on the hub from the certInfo stored by #2401. Follows the NetworkMonitorLoss alert: per-monitor incidents, alert history, recovery notifications.
Refs #391
Agent.Start and ConnectionManager.Start now take a context.Context. The
manager runs until that context is cancelled or the process receives
SIGINT/SIGTERM. On shutdown it waits for in-flight reconnect attempts
before closing connections.
The reconnect delay, WebSocket dial and pending handshake now stop when
the context is cancelled. A context-aware dialer keeps proxy selection
from the environment. WebSocket callbacks no longer block on eventChan
after the event loop exits. A late reconnect can no longer restart the
SSH server once shutdown begins.
BREAKING CHANGE: agent.Agent.Start now requires a context.Context.
---------
Co-authored-by: hank <hank@henrygd.me>
Kernels before sysfs devinfo (pre-5.13, e.g. Synology DSM 4.4) report
every Btrfs pool health as UNKNOWN and expose no member devices, even
though the same facts are available through BTRFS_IOC_FS_INFO and
BTRFS_IOC_DEV_INFO on the mountpoint.
When the devinfo directory does not exist, enumerate devids via DEV_INFO
after confirming the filesystem through FS_INFO: members that answer are
listed with their recorded capacity, a member reporting no device path is
missing (DEGRADED), freed devids (ENODEV gaps from remove/replace) are
skipped, and the kernel device count is cross-checked against the sysfs
devices links so an unattributed missing member still reads DEGRADED.
Any ioctl failure or count mismatch leaves health UNKNOWN rather than
reporting unverified ONLINE.
The agent runs as the beszel user under ProtectSystem=strict, so package
managers that write temp, cache or log files failed with only
"exit status 1" in the log.
- Include the command's stderr in package update check errors.
- dnf5: point XDG_STATE_HOME and XDG_CACHE_HOME into the agent's data
dir. They default to $HOME, which is /nonexistent for the beszel user.
With -C, dnf5 still reads the system metadata cache.
- zypper: keep the solv cache in the data dir with --solv-cache-dir,
since the agent user cannot rebuild it in /var/cache/zypp.
- install-agent.sh, Debian unit: add PrivateTmp=yes. dnf4 (Rocky, Alma)
and zypper keep temporary files in /var/tmp, which ProtectSystem=strict
makes read-only. The install script also adds it to existing units
when re-run.
A refquota on a pool's root dataset caps its avail by the root's own
referenced data, so used+avail can understate pool capacity when child
datasets hold most of the data and trigger false disk usage alerts.
Fetch refquota with the root datasets and fall back to the raw zpool
figures for those pools.
The pool overview used `zpool list`, whose size and alloc count raw vdev
space. On raidz that includes parity, so a TrueNAS pool with 65 TB usable
showed 98 TB capacity, while the details page (root dataset) matched TrueNAS.
Take Size, Alloc and Free from the pool's root dataset (used+avail), the same
way Btrfs reports effective capacity. If the root dataset cannot be read, keep
the raw figures and set Raw so the UI labels them as raw usage.
Fixes#2539
pb.realtime.subscribe() resolves only once the server has confirmed the
subscription. When the chart time changed away from 1m before that, the
effect cleanup ran with the initial no-op, the subscription stayed active,
and its 1m points kept overwriting the 1h history. Track cancellation,
unsubscribe as soon as the late subscription resolves, and ignore messages
that arrive after cleanup. Fixes#2598.
getRecord removed the system from the manager on any error, so a query that
timed out while the hub was stalled stopped the updater for good. The system
stayed 'up' in the database and was never polled again until a restart.
Only a record that is really gone removes the system now. (#2544)
PocketBase now marshals record JSON fields with encoding/json/v2, where
omitempty no longer omits zero numbers or false. This roughly doubled the
size of stored system_stats rows with zero-value keys. Switch scalar
fields to omitzero to restore the v1 output shape, and add a test that
compares v1 and v2 output for zero-heavy values.
unsafe_shutdowns was uint16, so drives reporting more than 65535 unsafe
shutdowns made json.Unmarshal fail and the entire NVMe SMART record was
discarded with only "no valid SMART data found" in the logs. Widen it
to uint, matching power_cycles (#2484), and log the unmarshal error at
debug level so similar failures are diagnosable.
Fixes#2582
henrygdandSven van Ginkel
yi111andGitHub
daniyyellandhank
Mike BogdanandGitHub
NielDBandGitHub
Santhi PrakashandGitHub
spatiumstasandGitHub
kenny-ishandGitHub
Rajeev GadgilandGitHub
MarrrrrrrrryandGitHub
Mario MoharandGitHub
Bharat KauravandGitHub