* telemetry: keep the reported version in the daily history
The version only ever lived on the instance record, which holds a cluster's
latest report, so there was no way to ask what anything ran last Tuesday.
Record it per sample, and let the daily axis carry strings as well as counts.
State written before this has no version on its samples. The newest sample is
the report the instance record itself came from, so fill that one in on load
rather than starting a version series a day late.
* telemetry: serve the fleet's version make-up over time
/api/versions gives how many clusters ran each release per day, on the same
axis and hold-forward rule as the cluster sizes. Releases are ordered by
number rather than by size: the caller stacks them, and a stack whose order
changes with the counts is unreadable over time. The tail past the limit is
summed into "other" so the stack still adds up.
Days with no version are dropped before the axis is built, so the series
spans the days it knows a version for instead of climbing out of blanks.
* telemetry: draw version distribution as a stacked growth chart
The pie only ever showed today. Stacked over 30 days the height is the
confirmed fleet and each band is a release, so one chart carries the growth
and the rollouts at once. Newest release on the floor, so the band being read
is anchored to the axis instead of riding on everything below it.
Eight fixed hues instead of the evenly spaced ones the cluster stacks use:
evenly spaced put a green and a cyan close enough to be hard to tell apart,
which matters for a set you read rather than a wall of anonymous ids.
Versions past the eighth fold into "other", and each band carries its own
number so the chart reads without matching colours against the legend.
The dashboard charted one summed disk-usage line, so a step in the total
gave no hint which cluster moved. A new panel stacks each cluster's daily
size as its own band: the top of the stack is the fleet total, each band
is one cluster, and the clusters past the twentieth are summed into an
"other" band so the stack still adds up to the total.
The series is built from the per-cluster daily histories and served by
/api/cluster-sizes. Clusters report roughly once a day at no fixed hour,
so a day with no report carries the previous value forward — dropping it
to zero would sag the total every day as the clusters that have not
reported yet fall out from under it. A cluster that stops reporting past
the active window ends at its last sample instead of holding capacity
forever. Ranking is by the most recent day, tie-broken on cluster id so
the colors do not shuffle between refreshes.
Hover and click resolve to the band under the pointer: Chart.js's builtin
interaction modes match the nearest line, which on a stack of thin bands
is rarely the band being pointed at. Clicking one fills the per-cluster
history lookup below it.
telemetry: per-cluster usage history
Keep one compact sample per cluster per UTC day (disk bytes, volume
count, volume servers), retained for -max-age and persisted in the
state file. Serve it at /api/history?cluster_id=...&days=90 and add a
per-cluster lookup with disk/volume charts to the built-in dashboard.
* telemetry: persist server state across restarts
The telemetry server kept the instance map and Prometheus gauges only
in process memory, so every deploy or restart reset all collected
metrics until clusters re-reported over the next 24h.
Snapshot the instance map to a JSON state file (atomic tmp+rename) on
a debounced interval and on SIGTERM, and restore it on startup,
preserving received_at so the cleanup and active-cluster windows stay
correct. Defaults to data/telemetry-state.json, which the deployed
systemd unit's WorkingDirectory already provides; -state-file=''
disables.
* telemetry: keep instances for 90 days by default
With state now persisted across restarts, a longer retention default is
meaningful; raise -max-age from 30 to 90 days so per-cluster data
survives long enough for quarterly views.
Chris LuandGitHub