The dashboard has built its callsign table from radioid.net since v4.7.0,
but the old manual process was still documented as usable. Nothing reads
user.csv, so an operator who followed it got a correct-looking file and a
dash in Last Heard. Remove the file, scripts/filter_user_csv.py and its
instructions.
min_rows_required rejected any filter result under 1000 rows, which ruled
out 153 of the 190 countries in the current data. Refuse only an empty
result, which means a misspelled country. Truncated or malformed downloads
still fail earlier. A valid but unexpectedly short file is now accepted
until the next good download. That leaves nothing in the fallback section
that is read, as keep_stale_on_failure never was, so remove it; existing
configs that carry it are unaffected.
The filter still defaults to the United States and Canada. Document how to
change it and when a change takes effect: at the next daily download, or
immediately by deleting the snapshot and meta files and restarting. Both
files have to go, because the meta file's timestamp suppresses the startup
refresh.
Reported in #7.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HBlink4 draws operators who have never run a network server before, and
several things in the install path assumed otherwise.
Sample configuration split in two. config_sample.json was a feature
showcase, and copying it as documented produced a broken server: its
example repeater pattern claimed IDs 310000-312999 with a passphrase of
its own, so an operator whose repeater fell in that range was rejected
with no way to diagnose it (patterns match first-wins and never fall
through to "default"). It also shipped an outbound link and an OpenBridge
trunk enabled against hosts that do not exist, and bound both address
families to one port, which cannot work. It is now a minimal working
config needing one edit -- the passphrase. The full reference moves to
config_advanced_sample.json with its link examples disabled and the ID
overlap called out.
Ship IPv4-only. Dual-stack is fully supported and now documented as four
concrete layouts, each verified to bind; it is simply not the default,
because most repeaters are IPv4 and the same-port bind conflict is not
something a new operator should have to learn first. The built-in default
when the key is absent stays false, so existing installs are unaffected.
"IPv6 is globally disabled" drops from WARNING to INFO -- it is now the
recommended setting and should not read as a fault.
install_services.sh: refuse paths containing a space, $ or % (systemd
mis-parses them in ExecStart=, and a space previously produced a unit
that passed verification and then failed at start); escape & in the path
substitution; render and verify units in a temp dir so a failed
substitution cannot leave a broken unit in /etc/systemd/system. Check the
things that otherwise surface only in the journal: complete checkout,
Python 3.9+, server and dashboard dependencies, both config files, and
agreement between the two halves of the dashboard event link.
readme: seven numbered steps with no diversions, adding the prerequisites
(python3-venv, which Debian and Ubuntu users hit constantly) and the
firewall ports, neither of which were mentioned.
Also: the dashboard logged a hardcoded localhost:8080 regardless of the
configured port; run_all.sh now fails clearly on missing config instead of
a traceback; pyproject's comment claimed systemd should invoke the console
scripts, contradicting how the units actually work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Leaving <unit>.bak-<stamp> next to the live units puts clutter in a
system directory that systemd itself scans, and it accumulates where
nobody looks. Back up into the project root instead, chowned back to the
invoking user rather than left owned by root.
Renamed to <unit>.<timestamp>.bak so the existing *.bak ignore rule
actually matches -- the previous .bak-<timestamp> form does not end in
.bak and would have shown up as untracked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The installer script, the user/group/path warning and the service
management steps were described in readme.md, SYSTEMD.md and
scripts/README.md, which is three places to keep in sync.
SYSTEMD.md is now authoritative. readme.md's Running section points at it
instead of repeating the install command and its caveats, and its
Installation warning drops the systemd-specific detail it duplicated,
keeping only the write-access requirement that belongs there.
scripts/README.md keeps a two-line index entry that links onward.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shipped unit files carry the author's user and paths (User=cort,
/home/cort/hblink4), so copying them verbatim produces two services that
fail to start. The docs said so only in a Customization section below the
copy instructions, and told people to edit the repo copies -- which then
conflict on the next git pull.
The installer prompts for the service user and group (defaulting to the
invoking sudo user and their primary group), substitutes them and the
detected installation path into both units, installs to
/etc/systemd/system/, reloads systemd, and optionally enables and starts
the services. It backs up existing units, refuses unknown users/groups,
checks the venv exists, warns when the service user cannot write the
installation directory, and verifies the substitution landed before
leaving anything in place. DESTDIR previews the output without touching
the live system.
The manual instructions now say to copy first and edit the installed
copy, and note that daemon-reload is required after any unit change. Both
unit files carry a header saying the same.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Documentation cleanup pass across the tree, driven by the upcoming merge
of the dmrd-translation branch to main.
Deletions (obsolete / superseded):
- docs/user_csv_automation_proposal.md — implemented as dashboard/user_db.py
- docs/QUICK_TEST_REFERENCE.md — stale test counts, referenced a
TEST_COVERAGE.md that no longer exists
- user.old.csv — development backup artifact (also added user.old* and
user.bak* to .gitignore)
Release notes renumbered to the 4.x series (file kept 1.x-labelled from
a prior numbering scheme that predated the 4.1.0 release):
- docs/RELEASE_NOTES_v1.5.md -> docs/RELEASE_NOTES_v4.5.md
- docs/RELEASE_NOTES_v1.6.1.md -> docs/RELEASE_NOTES_v4.6.1.md
- Also fixed a stray "v1.5+" -> "v4.5+" in stream_tracking_diagrams.md
Content fixes where docs contradicted code or were stale:
- docs/dmrd_translation.md: replaced the "Payload blanking" section with
"Link Control rewriting" describing VHEAD/VTERM full-LC and voice-burst
EMB_LC re-encoding via dmr_utils3.bptc
- docs/connecting_to_hblink4.md: cross-reference updated to match
- docs/protocol.md: corrected call-type bit direction (0=Group,
1=Private/Unit — previously reversed), replaced "Bits 0-3: Reserved"
with actual dtype_vseq semantics, updated translation-doc link
- readme.md: "Built on Twisted" -> asyncio (Twisted was removed in the
outbound-connection release); expanded docs index to include routing,
DMRD translation, diagrams, TODO, and OpenBridge analysis; added DMRD
translation to the features list
- docs/stream_tracking.md: StreamState snippet now shows lc_base and
lc_cache fields added for LC rewriting
- scripts/README.md: noted that dashboard/user_db.py is the primary
refresh path; script remains as manual fallback
- dashboard/user_db.py: dropped the doctring reference to the deleted
proposal file
Note: dmrd-translation branch remains in active development but is
fully functional as of this commit. Tests: 75/75 pass (pre-existing
test_access_control.py failures are unrelated and predate this branch).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dashboard now pulls user.csv from radioid.net on a daily schedule,
filters it to the operator's configured country/callsign/ID scope, and
hot-swaps the in-memory radio_id -> callsign map without a restart. No more
stale callsigns when someone new gets their DMR ID.
Pipeline (dashboard/user_db.py):
- Stage 1: conditional HTTPS GET (If-Modified-Since) against
database.radioid.net/static/user.csv with identifying User-Agent. Uses
stdlib urllib.request, gzip-aware, 100 MB safety cap, 120s timeout. No
new dependencies.
- Stage 2: stream through csv.DictReader applying country / callsign_regex
/ radio_id_range filters. Default countries are ["United States",
"Canada"], reproducing the current scripts/filter_user_csv.py behavior
byte-for-byte.
- Stage 3: atomic write to data/user_db.json (stringified dict, ~2.4 MB vs
the current 7.7 MB CSV), sidecar metadata, then wholesale replacement of
the in-memory dict reference — single bytecode op under the GIL, no lock
needed. Readers see either the old or new dict, never a partial state.
Failure handling: any stage fault leaves the previous snapshot in place
and logs a warning. A filter that produces fewer than
fallback.min_rows_required rows is treated as a malformed download and
rejected. 304 Not Modified touches the timestamp so the "too stale" check
doesn't re-fire on every restart. The guiding rule: callsign lookups must
never get worse than they were before.
Scheduling (user_db_refresh_task coroutine):
- Catch-up refresh at startup if the snapshot is missing or older than
on_startup_if_older_than_hours (default 36).
- Then schedules the next run at time_of_day + [0, jitter_minutes)
random offset. Defaults: 03:17 local, 15 min jitter — deliberately
off-hour and off-minute to avoid the top-of-hour stampede hitting
radioid.net from Pi-Star fleets every night.
- "hourly" in config is clamped to "daily" with a warning. Upstream only
regenerates daily; sub-daily polling is pure waste.
Config: new user_database block in dashboard/config.json with source URL,
User-Agent (with explicit contact placeholder), schedule, filter rules, and
fallback thresholds. Sample and defaults in load_config() match. Startup
warns if the placeholder contact 'operator@example.org' wasn't changed so
radioid.net can reach the operator if there's ever an issue.
Dashboard UI: small "Callsign DB: N entries, updated Xh ago" indicator to
the right of the "Last Heard" section header, with a verbose hover tooltip
showing source URL, last refresh status, upstream Last-Modified, and last
error if any. Gold when the last refresh failed; re-ticks every 60s
client-side. The single call site that consumes user_database (stream_start
handler) was updated from module global to state.user_db.
scripts/filter_user_csv.py gets a deprecation notice but stays functional
for air-gapped workflows. Its filter logic now lives in
dashboard.user_db.filter_rows_from_csv_bytes() as a pure function so the
legacy script could be rewritten to import it without duplicating rules.
Design rationale and sources documented in
docs/user_csv_automation_proposal.md — notably, radioid.net's terms
explicitly flag "bulk mirroring" of the JSON API as blockable behavior, so
the daily static-CSV pull is the intended channel (Pi-Star, HBMonv2, and
others all use this same URL). The dashboard now follows the same
convention.
Locally verified: filter round-trips the existing user.csv to 136,613
rows in ~750ms, snapshot is 2.41 MB on disk, and the import path cleanly
wires into DashboardState without regressing existing stats/last-heard
persistence.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New utility script:
- scripts/filter_user_csv.py - Filter DMR user database
- scripts/README.md - Documentation and usage examples
Features:
- Process full user database to extract US/Canada only
- Automatic statistics and file size reporting
- Can overwrite input or create new output file
- Preserves all CSV columns
Usage:
python3 scripts/filter_user_csv.py user_full.csv user.csv
Benefits:
- Reproducible filtering for future database updates
- Clear documentation for maintenance
- Easy to add more countries if needed
Cort BuffingtonandClaude Opus 5.5
cortandClaude Opus 5