mirror of
https://github.com/n0mjs710/HBlink4.git
synced 2026-11-08 12:26:31 +00:00
cortandClaude Opus 4.7
16db4f0421
The dashboard now pulls user.csv from radioid.net on a daily schedule, filters it to the operator's configured country/callsign/ID scope, and hot-swaps the in-memory radio_id -> callsign map without a restart. No more stale callsigns when someone new gets their DMR ID. Pipeline (dashboard/user_db.py): - Stage 1: conditional HTTPS GET (If-Modified-Since) against database.radioid.net/static/user.csv with identifying User-Agent. Uses stdlib urllib.request, gzip-aware, 100 MB safety cap, 120s timeout. No new dependencies. - Stage 2: stream through csv.DictReader applying country / callsign_regex / radio_id_range filters. Default countries are ["United States", "Canada"], reproducing the current scripts/filter_user_csv.py behavior byte-for-byte. - Stage 3: atomic write to data/user_db.json (stringified dict, ~2.4 MB vs the current 7.7 MB CSV), sidecar metadata, then wholesale replacement of the in-memory dict reference — single bytecode op under the GIL, no lock needed. Readers see either the old or new dict, never a partial state. Failure handling: any stage fault leaves the previous snapshot in place and logs a warning. A filter that produces fewer than fallback.min_rows_required rows is treated as a malformed download and rejected. 304 Not Modified touches the timestamp so the "too stale" check doesn't re-fire on every restart. The guiding rule: callsign lookups must never get worse than they were before. Scheduling (user_db_refresh_task coroutine): - Catch-up refresh at startup if the snapshot is missing or older than on_startup_if_older_than_hours (default 36). - Then schedules the next run at time_of_day + [0, jitter_minutes) random offset. Defaults: 03:17 local, 15 min jitter — deliberately off-hour and off-minute to avoid the top-of-hour stampede hitting radioid.net from Pi-Star fleets every night. - "hourly" in config is clamped to "daily" with a warning. Upstream only regenerates daily; sub-daily polling is pure waste. Config: new user_database block in dashboard/config.json with source URL, User-Agent (with explicit contact placeholder), schedule, filter rules, and fallback thresholds. Sample and defaults in load_config() match. Startup warns if the placeholder contact 'operator@example.org' wasn't changed so radioid.net can reach the operator if there's ever an issue. Dashboard UI: small "Callsign DB: N entries, updated Xh ago" indicator to the right of the "Last Heard" section header, with a verbose hover tooltip showing source URL, last refresh status, upstream Last-Modified, and last error if any. Gold when the last refresh failed; re-ticks every 60s client-side. The single call site that consumes user_database (stream_start handler) was updated from module global to state.user_db. scripts/filter_user_csv.py gets a deprecation notice but stays functional for air-gapped workflows. Its filter logic now lives in dashboard.user_db.filter_rows_from_csv_bytes() as a pure function so the legacy script could be rewritten to import it without duplicating rules. Design rationale and sources documented in docs/user_csv_automation_proposal.md — notably, radioid.net's terms explicitly flag "bulk mirroring" of the JSON API as blockable behavior, so the daily static-CSV pull is the intended channel (Pi-Star, HBMonv2, and others all use this same URL). The dashboard now follows the same convention. Locally verified: filter round-trips the existing user.csv to 136,613 rows in ~750ms, snapshot is 2.41 MB on disk, and the import path cleanly wires into DashboardState without regressing existing stats/last-heard persistence. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HBlink4 Scripts
filter_user_csv.py
Filters the DMR user database to only include US and Canada entries, reducing file size and memory usage.
Usage
Download latest user database:
# Download from radioid.net or other DMR database source
wget -O user_full.csv https://example.com/user.csv
Filter for US/Canada only:
# Process and save to user.csv
python3 scripts/filter_user_csv.py user_full.csv user.csv
# Or overwrite the original file:
python3 scripts/filter_user_csv.py user.csv
Output:
Processing user_full.csv...
✅ Filtering complete!
Kept: 132,999 entries (45.6%)
- Canada: 7,047
- United States: 125,952
Skipped: 158,969 entries (54.4%)
📊 File sizes:
Input: 14.79 MB
Output: 7.21 MB
Saved: 7.57 MB (51.2%)
✨ Filtered CSV written to: user.csv
Why Filter?
- Memory efficiency: 54% fewer entries to load
- Disk space: 51% smaller file size
- Target audience: Primary users are US/Canada
- Performance: Faster startup time for dashboard
Adding More Countries
To include additional countries, edit filter_user_csv.py line 52:
if country in ('United States', 'Canada', 'United Kingdom'): # Add more here
Automation
To automatically update and filter the user database:
#!/bin/bash
# update_user_database.sh
echo "Downloading latest user database..."
wget -O user_full.csv https://example.com/user.csv
echo "Filtering for US/Canada..."
python3 scripts/filter_user_csv.py user_full.csv user.csv
echo "Cleaning up..."
rm user_full.csv
echo "Done! Restart dashboard to load new database."