mirror of
https://github.com/n0mjs710/HBlink4.git
synced 2026-11-08 04:19:26 +00:00
Remove the manual user CSV pipeline; load a filter of any size
The dashboard has built its callsign table from radioid.net since v4.7.0, but the old manual process was still documented as usable. Nothing reads user.csv, so an operator who followed it got a correct-looking file and a dash in Last Heard. Remove the file, scripts/filter_user_csv.py and its instructions. min_rows_required rejected any filter result under 1000 rows, which ruled out 153 of the 190 countries in the current data. Refuse only an empty result, which means a misspelled country. Truncated or malformed downloads still fail earlier. A valid but unexpectedly short file is now accepted until the next good download. That leaves nothing in the fallback section that is read, as keep_stale_on_failure never was, so remove it; existing configs that carry it are unaffected. The filter still defaults to the United States and Canada. Document how to change it and when a change takes effect: at the next daily download, or immediately by deleting the snapshot and meta files and restarting. Both files have to go, because the meta file's timestamp suppresses the startup refresh. Reported in #7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Cort BuffingtonandClaude Opus 5.5 committed
1 parent
2348c3bcc5
commit
72cf6423f5
11 files changed
+138
-136821
No files matched your search
@@ -79,6 +79,38 @@ Add an optional "Network Info" button to the dashboard header that links users t
|
||||
|
||||
The button appears as the first item in the header status area with a light blue color to distinguish it from system status indicators. Perfect for linking to talkgroup lists, network rules, connection information, or help pages.
|
||||
|
||||
### Callsign Lookups
|
||||
|
||||
The dashboard downloads the radio ID database from radioid.net once a day and uses it to show callsigns in Last Heard. **By default it only loads the United States and Canada**, so a radio from anywhere else shows `-` instead of a callsign. If that's you, change the `countries` list in the `user_database` section of `dashboard/config.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"user_database": {
|
||||
"filter": {
|
||||
"countries": ["New Zealand"],
|
||||
"callsign_regex": null,
|
||||
"radio_id_ranges": null
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Filter options:**
|
||||
- `countries`: A list of country names, spelled exactly as radioid.net spells them (e.g., `["United Kingdom", "Ireland"]`), or the string `"all"`, without brackets, for every country. `"all"` loads over 300,000 entries, roughly twice the default, so list only the countries you need if you can
|
||||
- `callsign_regex`: Only load callsigns matching this Python regular expression, or `null` for no restriction
|
||||
- `radio_id_ranges`: Only load IDs in these ranges, e.g. `[[5300000, 5309999]]`, or `null` for no restriction
|
||||
|
||||
**When a change takes effect:** restart the dashboard to read the new config. The new filter is then applied at the next daily download (03:17 by default, set by `refresh.time_of_day`) once radioid.net has published a new file, which it normally does daily. Until then, the dashboard keeps using the data it already has.
|
||||
|
||||
**To apply it now**, delete the saved database and restart. The dashboard downloads a fresh copy within a few seconds of starting:
|
||||
|
||||
```bash
|
||||
rm dashboard/data/user_db.json dashboard/data/user_db.meta.json
|
||||
sudo systemctl restart hblink4-dash
|
||||
```
|
||||
|
||||
Delete both files: the second one records when the last download happened, and keeping it can prevent the new download. If the filter matches nothing, such as a misspelled country, the dashboard logs a warning and keeps the data it had. Callsigns are looked up when a call starts, so a call already in Last Heard keeps showing `-` until that radio transmits again.
|
||||
|
||||
## Usage
|
||||
|
||||
The dashboard is started automatically with `./run_all.sh`, or separately with `python3 run_dashboard.py` (or `python3 dashboard/server.py`) from the repository root. It binds the address and port from the `web` section of `dashboard/config.json` (default `0.0.0.0:8080`); passing `[bind] [port]` on the command line overrides that for one-off runs. Access at http://localhost:8080 (or your server IP for remote access).
|
||||
@@ -175,6 +207,11 @@ Real-time updates via WebSocket
|
||||
- Verify HBlink4 is running and repeaters are connected
|
||||
- Check browser console for errors
|
||||
|
||||
**Last Heard shows `-` instead of a callsign**
|
||||
- The radio's country is probably not in your filter: see [Callsign Lookups](#callsign-lookups)
|
||||
- The ID may not be registered at radioid.net
|
||||
- Check the dashboard log for `user_db` warnings: `journalctl -u hblink4-dash | grep user_db`
|
||||
|
||||
**WebSocket connection fails**
|
||||
- Verify uvicorn is running with WebSocket support
|
||||
- Try a different browser
|
||||
|
||||
@@ -28,6 +28,7 @@
|
||||
"unix_socket": "/tmp/hblink4.sock"
|
||||
},
|
||||
"user_database": {
|
||||
"_comment": "Radio ID to callsign lookups, downloaded daily from radioid.net. Only the countries listed below are loaded, so a radio from anywhere else shows '-' in Last Heard. Add yours using the exact name radioid.net uses, e.g. [\"New Zealand\"], or use \"all\" (no list brackets) for every country. See dashboard/README.md for when a change takes effect.",
|
||||
"enabled": true,
|
||||
"source_url": "https://database.radioid.net/static/user.csv",
|
||||
"user_agent": "HBlink4-Dashboard (+https://github.com/n0mjs710/HBlink4; contact=operator@example.org)",
|
||||
@@ -41,10 +42,6 @@
|
||||
"countries": ["United States", "Canada"],
|
||||
"callsign_regex": null,
|
||||
"radio_id_ranges": null
|
||||
},
|
||||
"fallback": {
|
||||
"keep_stale_on_failure": true,
|
||||
"min_rows_required": 1000
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -6,7 +6,6 @@ Updates every 10 superframes (60 packets = 1 second) for smooth real-time feel
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import csv
|
||||
import socket
|
||||
import os
|
||||
import ipaddress
|
||||
@@ -76,10 +75,6 @@ def load_config() -> dict:
|
||||
"countries": ["United States", "Canada"],
|
||||
"callsign_regex": None,
|
||||
"radio_id_ranges": None
|
||||
},
|
||||
"fallback": {
|
||||
"keep_stale_on_failure": True,
|
||||
"min_rows_required": 1000
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -145,7 +145,7 @@ class UserDatabase:
|
||||
"ok", "not_modified", "disabled", "error".
|
||||
|
||||
Config shape (dashboard config["user_database"]):
|
||||
source_url, user_agent, filter {...}, fallback {...}
|
||||
source_url, user_agent, filter {...}
|
||||
|
||||
All blocking work runs via asyncio.to_thread so the event loop stays
|
||||
responsive even if radioid.net is slow.
|
||||
@@ -166,7 +166,6 @@ class UserDatabase:
|
||||
source_url = config.get("source_url", "https://database.radioid.net/static/user.csv")
|
||||
user_agent = config.get("user_agent", "HBlink4-Dashboard (contact=operator@example.org)")
|
||||
filter_cfg = config.get("filter", {}) or {}
|
||||
fallback_cfg = config.get("fallback", {}) or {}
|
||||
|
||||
self._meta.source_url = source_url
|
||||
|
||||
@@ -218,10 +217,9 @@ class UserDatabase:
|
||||
self._write_meta_best_effort()
|
||||
return "error"
|
||||
|
||||
min_rows = int(fallback_cfg.get("min_rows_required", 1000))
|
||||
if len(new_data) < min_rows:
|
||||
if not new_data:
|
||||
msg = (
|
||||
f"filter produced {len(new_data)} rows, below min_rows_required={min_rows} "
|
||||
f"filter matched no rows, check the spelling of filter.countries "
|
||||
f"(keeping old snapshot of {len(self._data)} rows)"
|
||||
)
|
||||
logger.warning(f"⚠️ user_db refresh: {msg}")
|
||||
|
||||
@@ -75,7 +75,7 @@ The dashboard previously relied on a manual `wget + filter_user_csv.py` pipeline
|
||||
- JSON snapshot + sidecar metadata on disk; hot-swap of the in-memory `radio_id → callsign` dict with no dashboard restart
|
||||
- Three-stage pipeline (download → filter → store); failure in any stage is non-fatal and the last known-good snapshot keeps serving
|
||||
- No new external dependencies (stdlib `urllib.request` only)
|
||||
- [scripts/filter_user_csv.py](../scripts/filter_user_csv.py) remains as a manual fallback
|
||||
- `scripts/filter_user_csv.py` remained as a manual fallback (removed in a later release)
|
||||
|
||||
### 7. Code organization refactor
|
||||
|
||||
|
||||
@@ -292,6 +292,8 @@ The `event_receiver` section is the **listening socket for the link from HBlink4
|
||||
|
||||
> **Renamed**: these two keys were previously `host_ipv4`/`host_ipv6`, which collided with the identically-named connect-target keys in HBlink4's own `dashboard` section. The old names are still accepted, with a warning logged at startup.
|
||||
|
||||
The `user_database` section controls which countries get callsign lookups (United States and Canada by default). See [Callsign Lookups](../dashboard/README.md#callsign-lookups) for how to change it.
|
||||
|
||||
See [Dashboard Documentation](../dashboard/README.md) for the rest of the dashboard-side configuration.
|
||||
|
||||
## Connection Type Detection
|
||||
|
||||
@@ -99,6 +99,10 @@ you connect must be set to the same passphrase.
|
||||
nano config/config.json
|
||||
```
|
||||
|
||||
Outside the United States and Canada? The dashboard only shows callsigns for
|
||||
those two countries until you
|
||||
[add yours](dashboard/README.md#callsign-lookups).
|
||||
|
||||
### 5. Try it out
|
||||
|
||||
```bash
|
||||
|
||||
@@ -10,79 +10,3 @@ sudo ./scripts/install_services.sh
|
||||
|
||||
Documented in **[SYSTEMD.md](../SYSTEMD.md)**, which is the reference for
|
||||
systemd deployment.
|
||||
|
||||
## filter_user_csv.py
|
||||
|
||||
Filters the DMR user database to only include US and Canada entries, reducing file size and memory usage.
|
||||
|
||||
> **Note**: The dashboard now refreshes `user.csv` automatically on a daily schedule via [dashboard/user_db.py](../dashboard/user_db.py), applying configurable country/callsign/ID filters without a restart. This script remains available for manual / one-off filtering when the automated pipeline isn't in use.
|
||||
|
||||
### Usage
|
||||
|
||||
#### Download latest user database:
|
||||
```bash
|
||||
# Download from radioid.net or other DMR database source
|
||||
wget -O user_full.csv https://example.com/user.csv
|
||||
```
|
||||
|
||||
#### Filter for US/Canada only:
|
||||
```bash
|
||||
# Process and save to user.csv
|
||||
python3 scripts/filter_user_csv.py user_full.csv user.csv
|
||||
|
||||
# Or overwrite the original file:
|
||||
python3 scripts/filter_user_csv.py user.csv
|
||||
```
|
||||
|
||||
#### Output:
|
||||
```
|
||||
Processing user_full.csv...
|
||||
|
||||
✅ Filtering complete!
|
||||
Kept: 132,999 entries (45.6%)
|
||||
- Canada: 7,047
|
||||
- United States: 125,952
|
||||
Skipped: 158,969 entries (54.4%)
|
||||
|
||||
📊 File sizes:
|
||||
Input: 14.79 MB
|
||||
Output: 7.21 MB
|
||||
Saved: 7.57 MB (51.2%)
|
||||
|
||||
✨ Filtered CSV written to: user.csv
|
||||
```
|
||||
|
||||
### Why Filter?
|
||||
|
||||
- **Memory efficiency**: 54% fewer entries to load
|
||||
- **Disk space**: 51% smaller file size
|
||||
- **Target audience**: Primary users are US/Canada
|
||||
- **Performance**: Faster startup time for dashboard
|
||||
|
||||
### Adding More Countries
|
||||
|
||||
To include additional countries, edit `filter_user_csv.py` line 52:
|
||||
|
||||
```python
|
||||
if country in ('United States', 'Canada', 'United Kingdom'): # Add more here
|
||||
```
|
||||
|
||||
### Automation
|
||||
|
||||
To automatically update and filter the user database:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# update_user_database.sh
|
||||
|
||||
echo "Downloading latest user database..."
|
||||
wget -O user_full.csv https://example.com/user.csv
|
||||
|
||||
echo "Filtering for US/Canada..."
|
||||
python3 scripts/filter_user_csv.py user_full.csv user.csv
|
||||
|
||||
echo "Cleaning up..."
|
||||
rm user_full.csv
|
||||
|
||||
echo "Done! Restart dashboard to load new database."
|
||||
```
|
||||
@@ -1,116 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Filter user.csv to only include US and Canada entries.
|
||||
|
||||
DEPRECATED: as of April 2026, the dashboard fetches and filters user.csv
|
||||
automatically on a daily schedule (see dashboard/user_db.py and
|
||||
docs/user_csv_automation_proposal.md). This script is kept functional as a
|
||||
manual fallback for air-gapped deployments or operators who prefer to shuttle
|
||||
the file in by hand. It may be removed in a future release.
|
||||
|
||||
Usage:
|
||||
python3 scripts/filter_user_csv.py <input.csv> [output.csv]
|
||||
|
||||
Example:
|
||||
python3 scripts/filter_user_csv.py user_full.csv user.csv
|
||||
|
||||
If no output file specified, will overwrite the input file.
|
||||
"""
|
||||
|
||||
import csv
|
||||
import sys
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def filter_user_csv(input_path: str, output_path: str = None) -> None:
|
||||
"""
|
||||
Filter user.csv to only keep US and Canada entries.
|
||||
|
||||
Args:
|
||||
input_path: Path to input CSV file
|
||||
output_path: Path to output CSV file (defaults to overwriting input)
|
||||
"""
|
||||
if output_path is None:
|
||||
output_path = input_path
|
||||
temp_path = input_path + '.tmp'
|
||||
else:
|
||||
temp_path = output_path
|
||||
|
||||
input_file = Path(input_path)
|
||||
if not input_file.exists():
|
||||
print(f"Error: Input file not found: {input_path}")
|
||||
sys.exit(1)
|
||||
|
||||
# Track statistics
|
||||
kept = 0
|
||||
skipped = 0
|
||||
countries_kept = {}
|
||||
|
||||
print(f"Processing {input_path}...")
|
||||
|
||||
with open(input_path, 'r', encoding='utf-8') as infile, \
|
||||
open(temp_path, 'w', encoding='utf-8', newline='') as outfile:
|
||||
|
||||
reader = csv.DictReader(infile)
|
||||
|
||||
# Verify required columns exist
|
||||
if 'COUNTRY' not in reader.fieldnames:
|
||||
print("Error: COUNTRY column not found in CSV")
|
||||
sys.exit(1)
|
||||
|
||||
writer = csv.DictWriter(outfile, fieldnames=reader.fieldnames)
|
||||
writer.writeheader()
|
||||
|
||||
for row in reader:
|
||||
country = row.get('COUNTRY', '').strip()
|
||||
|
||||
# Keep only US and Canada
|
||||
if country in ('United States', 'Canada'):
|
||||
writer.writerow(row)
|
||||
kept += 1
|
||||
countries_kept[country] = countries_kept.get(country, 0) + 1
|
||||
else:
|
||||
skipped += 1
|
||||
|
||||
# If we wrote to a temp file, move it to the final location
|
||||
if output_path == input_path:
|
||||
os.replace(temp_path, output_path)
|
||||
|
||||
# Show statistics
|
||||
total = kept + skipped
|
||||
print(f"\n✅ Filtering complete!")
|
||||
print(f" Kept: {kept:,} entries ({kept/total*100:.1f}%)")
|
||||
for country, count in sorted(countries_kept.items()):
|
||||
print(f" - {country}: {count:,}")
|
||||
print(f" Skipped: {skipped:,} entries ({skipped/total*100:.1f}%)")
|
||||
|
||||
# Show file sizes
|
||||
output_file = Path(output_path)
|
||||
if input_path != output_path and Path(input_path).exists():
|
||||
input_size = Path(input_path).stat().st_size / 1024 / 1024
|
||||
output_size = output_file.stat().st_size / 1024 / 1024
|
||||
print(f"\n📊 File sizes:")
|
||||
print(f" Input: {input_size:.2f} MB")
|
||||
print(f" Output: {output_size:.2f} MB")
|
||||
print(f" Saved: {input_size - output_size:.2f} MB ({(input_size - output_size)/input_size*100:.1f}%)")
|
||||
else:
|
||||
output_size = output_file.stat().st_size / 1024 / 1024
|
||||
print(f"\n📊 Output file size: {output_size:.2f} MB")
|
||||
|
||||
print(f"\n✨ Filtered CSV written to: {output_path}")
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
print(__doc__)
|
||||
sys.exit(1)
|
||||
|
||||
input_path = sys.argv[1]
|
||||
output_path = sys.argv[2] if len(sys.argv) > 2 else input_path
|
||||
|
||||
filter_user_csv(input_path, output_path)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,90 @@
|
||||
"""
|
||||
User database filter acceptance and forced refresh.
|
||||
|
||||
A country is never too small to load: the only filter result refused is an
|
||||
empty one, which means a misspelled country rather than a small one. Deleting
|
||||
the two files on disk is the documented way to force a download, so a restart
|
||||
with neither file present must refresh and must not send If-Modified-Since.
|
||||
"""
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
from dashboard.user_db import UserDatabase, filter_rows_from_csv_bytes
|
||||
|
||||
|
||||
_CSV = (
|
||||
b"RADIO_ID,CALLSIGN,FIRST_NAME,LAST_NAME,CITY,STATE,COUNTRY\n"
|
||||
b"3100001,W1AAA,A,,City,State,United States\n"
|
||||
b"5300001,ZL1AAA,B,,City,State,New Zealand\n"
|
||||
)
|
||||
_LAST_MODIFIED = "Tue, 03 Nov 2026 05:01:55 GMT"
|
||||
|
||||
|
||||
def _config(countries):
|
||||
return {"filter": {"countries": countries}}
|
||||
|
||||
|
||||
class TestFilterAcceptance(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.data_dir = Path(tmp.name)
|
||||
self.db = UserDatabase(self.data_dir)
|
||||
|
||||
def _refresh(self, countries, status=200):
|
||||
with mock.patch(
|
||||
"dashboard.user_db._http_get_with_conditional",
|
||||
return_value=(status, _CSV, _LAST_MODIFIED),
|
||||
) as http_get:
|
||||
result = self.db._refresh_sync(_config(countries))
|
||||
return result, http_get
|
||||
|
||||
def test_single_row_country_is_loaded(self):
|
||||
result, _ = self._refresh(["New Zealand"])
|
||||
self.assertEqual(result, "ok")
|
||||
self.assertEqual(self.db.get(5300001), "ZL1AAA")
|
||||
self.assertEqual(self.db.get(3100001), "")
|
||||
|
||||
def test_no_match_keeps_old_snapshot(self):
|
||||
self._refresh(["New Zealand"])
|
||||
result, _ = self._refresh(["New Zeland"])
|
||||
self.assertEqual(result, "error")
|
||||
self.assertEqual(self.db.get(5300001), "ZL1AAA")
|
||||
self.assertIn("no rows", self.db.meta.last_error)
|
||||
|
||||
def test_all_loads_every_country(self):
|
||||
self.assertEqual(
|
||||
set(filter_rows_from_csv_bytes(_CSV, {"countries": "all"})),
|
||||
{3100001, 5300001},
|
||||
)
|
||||
|
||||
def test_deleting_both_files_forces_full_download(self):
|
||||
self._refresh(["United States"])
|
||||
|
||||
for name in ("user_db.json", "user_db.meta.json"):
|
||||
(self.data_dir / name).unlink()
|
||||
|
||||
restarted = UserDatabase(self.data_dir)
|
||||
restarted.load_from_disk()
|
||||
self.assertIsNone(restarted.snapshot_age_hours())
|
||||
|
||||
self.db = restarted
|
||||
result, http_get = self._refresh(["New Zealand"])
|
||||
self.assertEqual(result, "ok")
|
||||
self.assertIsNone(http_get.call_args.kwargs["if_modified_since"])
|
||||
self.assertEqual(restarted.get(5300001), "ZL1AAA")
|
||||
|
||||
def test_keeping_meta_file_blocks_startup_refresh(self):
|
||||
self._refresh(["United States"])
|
||||
(self.data_dir / "user_db.json").unlink()
|
||||
|
||||
restarted = UserDatabase(self.data_dir)
|
||||
restarted.load_from_disk()
|
||||
self.assertIsNotNone(restarted.snapshot_age_hours())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user