Short-lived clusters report once under a fresh raft topology id and never
again, so CI runs and demo stacks each become their own cluster. Held
forward for the whole active window they pile up, and the volume server
line climbs every day while capacity stays flat.
Sum the fleet series over confirmed clusters only, the same set the
version and OS charts already use.
* telemetry: total disk usage over time counted each cluster on one day
GetMetrics aggregated s.instances, which holds only each cluster's most
recent report. Every cluster therefore landed in a single date bucket --
the day it last reported on -- so the chart plotted the disk usage of
clusters that went silent that day, and piled the whole live fleet onto
today. Aggregate the daily histories instead, reusing the day alignment
that the per-cluster size series already does.
* telemetry: don't pad the charts with days the server has no history for
The dashboard asks for 30 days, but daily history only starts when a
server first collects it, so the charts opened on a run of zeros and then
jumped -- reading as a fleet that appeared overnight. Start the window at
the oldest sample on hand when it is younger than the requested range.
The dashboard charted one summed disk-usage line, so a step in the total
gave no hint which cluster moved. A new panel stacks each cluster's daily
size as its own band: the top of the stack is the fleet total, each band
is one cluster, and the clusters past the twentieth are summed into an
"other" band so the stack still adds up to the total.
The series is built from the per-cluster daily histories and served by
/api/cluster-sizes. Clusters report roughly once a day at no fixed hour,
so a day with no report carries the previous value forward — dropping it
to zero would sag the total every day as the clusters that have not
reported yet fall out from under it. A cluster that stops reporting past
the active window ends at its last sample instead of holding capacity
forever. Ranking is by the most recent day, tie-broken on cluster id so
the colors do not shuffle between refreshes.
Hover and click resolve to the band under the pointer: Chart.js's builtin
interaction modes match the nearest line, which on a stack of thin bands
is rarely the band being pointed at. Clicking one fills the per-cluster
history lookup below it.
* telemetry: validate reports on the collect endpoint
/api/collect is anonymous, so reports can't be authenticated, but a
real master can't produce a non-UUID topology_id, a version outside
N.NN(-enterprise), an unknown GOOS/GOARCH, or absurd counts — reject
those to keep casual junk out of the collected data, and cap the
request body at 4 KB.
* telemetry: integration test fixtures pass collect validation
The test's topology id and version were exactly the junk shapes the
new validation rejects; use a UUID and a plain version number.
telemetry: confirmed-cluster stats
Count a cluster as confirmed once it has reported on >=2 distinct UTC
days (per-cluster history makes this a length check). Version/OS
distributions in /api/stats are computed over confirmed clusters, so a
one-shot injected report can't appear in them; falls back to all active
clusters while no confirmed ones exist (fresh server). Adds the
seaweedfs_telemetry_confirmed_clusters gauge and a dashboard card.
telemetry: per-cluster usage history
Keep one compact sample per cluster per UTC day (disk bytes, volume
count, volume servers), retained for -max-age and persisted in the
state file. Serve it at /api/history?cluster_id=...&days=90 and add a
per-cluster lookup with disk/volume charts to the built-in dashboard.
* telemetry: persist server state across restarts
The telemetry server kept the instance map and Prometheus gauges only
in process memory, so every deploy or restart reset all collected
metrics until clusters re-reported over the next 24h.
Snapshot the instance map to a JSON state file (atomic tmp+rename) on
a debounced interval and on SIGTERM, and restore it on startup,
preserving received_at so the cleanup and active-cluster windows stay
correct. Defaults to data/telemetry-state.json, which the deployed
systemd unit's WorkingDirectory already provides; -state-file=''
disables.
* telemetry: keep instances for 90 days by default
With state now persisted across restarts, a longer retention default is
meaningful; raise -max-age from 30 to 90 days so per-cluster data
survives long enough for quarterly views.
telemetry: key value gauges by cluster_id only
The value gauges were labeled {cluster_id, version, os}, so a cluster
reporting back after an upgrade started a new series while the old one
kept its last value forever: sum() double-counted every upgraded
cluster, and per-cluster history broke at every version change.
Key the five value gauges by cluster_id alone so each cluster keeps one
continuous series across upgrades; version/os metadata stays on
cluster_info (deleted and re-set on change), available to value queries
via 'on(cluster_id) group_left' joins. Update README accordingly.
telemetry: aggregate /api/metrics per day so dashboard charts render
The dashboard expects {dates, server_counts, disk_usage} as parallel
arrays, but GetMetrics returned per-instance {date,value} lists under
different keys, so the two over-time charts never rendered.
prometheus/common v0.67.5, golang.org/x/sys v0.45.0, klauspost/compress
v1.18.6 to match the root module; readonly build was rejecting the stale
versions.
* fix(telemetry): use correct TopologyId field in integration test
The proto field was renamed from cluster_id to topology_id but the
integration test was not updated, causing a compilation error.
* ci: add telemetry integration test workflow
Runs the telemetry integration test (server startup, protobuf
marshaling, client send, metrics/stats/instances API checks) on
changes to telemetry/ or weed/telemetry/.
* fix(telemetry): improve error message specificity in integration test
* fix(ci): pre-build telemetry server binary for integration test
go run compiles the server on the fly, which exceeds the 15s startup
timeout in CI. Build the binary first so the test starts instantly.
* fix(telemetry): fix ClusterId references in server and CI build path
- Replace ClusterId with TopologyId in server storage and API handler
(same rename as the integration test fix)
- Fix CI build: telemetry server has its own go.mod, so build from
within its directory
* ci(telemetry): add least-privilege permissions to workflow
Scope the workflow token to read-only repository contents, matching
the convention used in go.yml.
* fix(telemetry): set TopologyId in client integration test
The client only populates TopologyId when SetTopologyId has been
called. The test was missing this call, causing the server to reject
the request with 400 (missing required field).
* fix(telemetry): delete clusterInfo metric on instance cleanup
The cleanup loop removed all per-instance metrics except clusterInfo,
leaking that label set after eviction.
* fix(telemetry): make server build reproducible with proper context and deps
* Update telemetry/server/go.mod: go version
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* telemetry/server: optimize Dockerfile (organize cache deps, copy proto); run as non-root
* telemetry: update deployment docs for new Docker build context
* telemetry: clarify Docker build/run docs and improve Dockerfile caching
- DEPLOYMENT.md: specify docker build must run from repo root; provide full docker run example with flags/port mapping
- README.md: remove fragile 'cd ..'; keep instruction to run build from repo root
- Dockerfile: remove unnecessary pre-copy before 'go mod download' to improve cache utilization
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix(telemetry): make server build reproducible with proper context and deps
* Update telemetry/server/go.mod: go version
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* telemetry/server: optimize Dockerfile (organize cache deps, copy proto); run as non-root
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Chris LuandGitHub
Chris Lu
