Documentation cleanup pass across the tree, driven by the upcoming merge
of the dmrd-translation branch to main.
Deletions (obsolete / superseded):
- docs/user_csv_automation_proposal.md — implemented as dashboard/user_db.py
- docs/QUICK_TEST_REFERENCE.md — stale test counts, referenced a
TEST_COVERAGE.md that no longer exists
- user.old.csv — development backup artifact (also added user.old* and
user.bak* to .gitignore)
Release notes renumbered to the 4.x series (file kept 1.x-labelled from
a prior numbering scheme that predated the 4.1.0 release):
- docs/RELEASE_NOTES_v1.5.md -> docs/RELEASE_NOTES_v4.5.md
- docs/RELEASE_NOTES_v1.6.1.md -> docs/RELEASE_NOTES_v4.6.1.md
- Also fixed a stray "v1.5+" -> "v4.5+" in stream_tracking_diagrams.md
Content fixes where docs contradicted code or were stale:
- docs/dmrd_translation.md: replaced the "Payload blanking" section with
"Link Control rewriting" describing VHEAD/VTERM full-LC and voice-burst
EMB_LC re-encoding via dmr_utils3.bptc
- docs/connecting_to_hblink4.md: cross-reference updated to match
- docs/protocol.md: corrected call-type bit direction (0=Group,
1=Private/Unit — previously reversed), replaced "Bits 0-3: Reserved"
with actual dtype_vseq semantics, updated translation-doc link
- readme.md: "Built on Twisted" -> asyncio (Twisted was removed in the
outbound-connection release); expanded docs index to include routing,
DMRD translation, diagrams, TODO, and OpenBridge analysis; added DMRD
translation to the features list
- docs/stream_tracking.md: StreamState snippet now shows lc_base and
lc_cache fields added for LC rewriting
- scripts/README.md: noted that dashboard/user_db.py is the primary
refresh path; script remains as manual fallback
- dashboard/user_db.py: dropped the doctring reference to the deleted
proposal file
Note: dmrd-translation branch remains in active development but is
fully functional as of this commit. Tests: 75/75 pass (pre-existing
test_access_control.py failures are unrelated and predate this branch).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dashboard now pulls user.csv from radioid.net on a daily schedule,
filters it to the operator's configured country/callsign/ID scope, and
hot-swaps the in-memory radio_id -> callsign map without a restart. No more
stale callsigns when someone new gets their DMR ID.
Pipeline (dashboard/user_db.py):
- Stage 1: conditional HTTPS GET (If-Modified-Since) against
database.radioid.net/static/user.csv with identifying User-Agent. Uses
stdlib urllib.request, gzip-aware, 100 MB safety cap, 120s timeout. No
new dependencies.
- Stage 2: stream through csv.DictReader applying country / callsign_regex
/ radio_id_range filters. Default countries are ["United States",
"Canada"], reproducing the current scripts/filter_user_csv.py behavior
byte-for-byte.
- Stage 3: atomic write to data/user_db.json (stringified dict, ~2.4 MB vs
the current 7.7 MB CSV), sidecar metadata, then wholesale replacement of
the in-memory dict reference — single bytecode op under the GIL, no lock
needed. Readers see either the old or new dict, never a partial state.
Failure handling: any stage fault leaves the previous snapshot in place
and logs a warning. A filter that produces fewer than
fallback.min_rows_required rows is treated as a malformed download and
rejected. 304 Not Modified touches the timestamp so the "too stale" check
doesn't re-fire on every restart. The guiding rule: callsign lookups must
never get worse than they were before.
Scheduling (user_db_refresh_task coroutine):
- Catch-up refresh at startup if the snapshot is missing or older than
on_startup_if_older_than_hours (default 36).
- Then schedules the next run at time_of_day + [0, jitter_minutes)
random offset. Defaults: 03:17 local, 15 min jitter — deliberately
off-hour and off-minute to avoid the top-of-hour stampede hitting
radioid.net from Pi-Star fleets every night.
- "hourly" in config is clamped to "daily" with a warning. Upstream only
regenerates daily; sub-daily polling is pure waste.
Config: new user_database block in dashboard/config.json with source URL,
User-Agent (with explicit contact placeholder), schedule, filter rules, and
fallback thresholds. Sample and defaults in load_config() match. Startup
warns if the placeholder contact 'operator@example.org' wasn't changed so
radioid.net can reach the operator if there's ever an issue.
Dashboard UI: small "Callsign DB: N entries, updated Xh ago" indicator to
the right of the "Last Heard" section header, with a verbose hover tooltip
showing source URL, last refresh status, upstream Last-Modified, and last
error if any. Gold when the last refresh failed; re-ticks every 60s
client-side. The single call site that consumes user_database (stream_start
handler) was updated from module global to state.user_db.
scripts/filter_user_csv.py gets a deprecation notice but stays functional
for air-gapped workflows. Its filter logic now lives in
dashboard.user_db.filter_rows_from_csv_bytes() as a pure function so the
legacy script could be rewritten to import it without duplicating rules.
Design rationale and sources documented in
docs/user_csv_automation_proposal.md — notably, radioid.net's terms
explicitly flag "bulk mirroring" of the JSON API as blockable behavior, so
the daily static-CSV pull is the intended channel (Pi-Star, HBMonv2, and
others all use this same URL). The dashboard now follows the same
convention.
Locally verified: filter round-trips the existing user.csv to 136,613
rows in ~750ms, snapshot is 2.41 MB on disk, and the import path cleanly
wires into DashboardState without regressing existing stats/last-heard
persistence.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New utility script:
- scripts/filter_user_csv.py - Filter DMR user database
- scripts/README.md - Documentation and usage examples
Features:
- Process full user database to extract US/Canada only
- Automatic statistics and file size reporting
- Can overwrite input or create new output file
- Preserves all CSV columns
Usage:
python3 scripts/filter_user_csv.py user_full.csv user.csv
Benefits:
- Reproducible filtering for future database updates
- Clear documentation for maintenance
- Easy to add more countries if needed
cortandClaude Opus 4.7