sa6anw/heardlog
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: sa6anw:9171a5134d5ef3267e5614202dd5ef7ef364ef84
Branches Tags
sa6anw:main
..
compare: sa6anw:main
Branches Tags
sa6anw:main

2 Commits
9171a5134d ... main

Author SHA1 Message Date
root
3a066ea589 Rttningar 2026-05-03 06:39:58 +00:00
root
b5d65c4a52 N/A 2026-05-02 18:37:10 +00:00
4 changed files with 168 additions and 99 deletions
Expand all files Collapse all files

219
README.md
View File

@@ -1,187 +1,236 @@
# heardlog # heardlog
Samlar APRS-data från två källor och lagrar i TimescaleDB + PostGIS som grund för en täckningskarta. APRS RF coverage logger and map for amateur radio digipeaters.
Collects every packet heard on RF by a Direwolf digipeater and builds an
interactive coverage map showing which Maidenhead subsquares have been reached,
which paths were used, and which stations have been heard.
Live: [aprs.sa6anw.se](https://aprs.sa6anw.se)
---
## Architecture
``` ```
[Shack-LAN] [DMZ] [Shack LAN] [DMZ]
Direwolf AGW:8000 Direwolf AGW :8000
│ |
agw_forwarder.py ──POST/Bearer──► FastAPI :8085 ──► TimescaleDB + PostGIS agw_forwarder.py --POST/Bearer--> FastAPI :8085 --> TimescaleDB + PostGIS
│ |
rotate.aprs2.net┘ (APRS-IS, outbound från DMZ) rotate.aprs2.net| (APRS-IS regional feed, outbound)
|
nginx :80/443
|
index.html (Leaflet map)
``` ```
| Källa | Tabell | Nyckeldata | | Source | Table | Key data |
|---|---|---| |---|---|---|
| Direwolf AGW (RF) | `rf_frames` | `heard_direct` – hördes utan mellanliggande digi | | Direwolf AGW (RF) | `rf_frames` | Every packet received over the air |
| APRS-IS regionalt | `is_frames` | Stationer inom 200 km som digipeatas till internet | | APRS-IS regional | `is_frames` | Regional feed within 200 km radius |
`heard_direct = TRUE` är grundstenen för täckningskartan – ett bevis på att en station nådde vår mottagare direkt.
--- ---
## Krav ## Repository layout
- Docker + Docker Compose (DMZ-server) ```
- Direwolf med `AGWPORT 8000` i `direwolf.conf` (shack-dator) heardlog/
- Python 3.10+ och `pip3 install requests` (shack-dator) ????????? .env.example
????????? README.md
????????? docker-compose.yml
????????? collector/ # DMZ - runs in Docker
??? ????????? Dockerfile
??? ????????? requirements.txt
??? ????????? config.py
??? ????????? db.py # TimescaleDB schema + insert helpers
??? ????????? maidenhead.py # Maidenhead subsquare calculations
??? ????????? aprs_is.py # APRS-IS client
??? ????????? main.py # FastAPI app + APRS-IS background task
????????? agw-forwarder/ # Shack - runs directly on the Direwolf machine
??? ????????? agw_forwarder.py
??? ????????? requirements.txt
??? ????????? agw-forwarder.service
????????? web/
????????? index.html # Leaflet map (served by nginx)
????????? nginx.conf
```
--- ---
## DMZ – snabbstart ## Getting started
### DMZ
```bash ```bash
git clone <repo> heardlog git clone <repo> heardlog
cd heardlog cd heardlog
cp .env.example .env cp .env.example .env
# Edit .env - set DB_PASSWORD and API_KEY
# Generera API-nyckel # Generate API key
python3 -c "import secrets; print(secrets.token_hex(32))" python3 -c "import secrets; print(secrets.token_hex(32))"
# Klistra in som API_KEY i .env
docker compose up -d docker compose up -d
docker compose logs -f collector docker compose logs -f collector
``` ```
Förväntad output: Expected output:
``` ```
INFO db Database schema ready INFO db Database schema ready
INFO main Startup complete – listening for RF frames INFO main Startup complete - listening for RF frames
INFO aprs_is APRS-IS login: # logresp SA6ANW-1 unverified, server T2... INFO aprs_is APRS-IS login: # logresp SA6ANW-1 unverified, server T2...
``` ```
`unverified` på APRS-IS är förväntat – passcode `-1` ger receive-only vilket är tillräckligt. ### Shack (agw-forwarder)
---
## Shack – agw-forwarder
```bash ```bash
pip3 install requests pip3 install requests
# Skapa konfigurationsfil
cat > heardlog/agw-forwarder/.env << 'EOF' cat > heardlog/agw-forwarder/.env << 'EOF'
AGW_HOST=localhost AGW_HOST=localhost
AGW_PORT=8000 AGW_PORT=8000
COLLECTOR_URL=http://<dmz-ip>:8085 COLLECTOR_URL=http://<dmz-ip>:8085
API_KEY=<samma nyckel som i DMZ .env> API_KEY=<same key as DMZ .env>
STATION_CALL=SA6ANW-1 STATION_CALL=SA6ANW-1
LOG_LEVEL=INFO LOG_LEVEL=INFO
EOF EOF
# Testa manuellt # Test manually first
cd heardlog/agw-forwarder cd heardlog/agw-forwarder
python3 agw_forwarder.py python3 agw_forwarder.py
``` ```
### Kör som systemd-tjänster ### Run as systemd services (shack)
```bash ```bash
# Kopiera service-filer
sudo cp heardlog/agw-forwarder/direwolf.service /etc/systemd/system/ sudo cp heardlog/agw-forwarder/direwolf.service /etc/systemd/system/
sudo cp heardlog/agw-forwarder/agw-forwarder.service /etc/systemd/system/ sudo cp heardlog/agw-forwarder/agw-forwarder.service /etc/systemd/system/
sudo systemctl daemon-reload sudo systemctl daemon-reload
sudo systemctl enable --now direwolf sudo systemctl enable --now direwolf
sudo systemctl enable --now agw-forwarder sudo systemctl enable --now agw-forwarder
# Verifiera
journalctl -u direwolf -u agw-forwarder -f journalctl -u direwolf -u agw-forwarder -f
``` ```
`agw-forwarder` har `Wants=direwolf.service` – systemd startar Direwolf automatiskt om den inte kör. `agw-forwarder` has `Wants=direwolf.service` - systemd starts Direwolf automatically.
--- ---
## Konfiguration ## Configuration
### DMZ (`.env`) ### DMZ (`.env`)
| Variabel | Standard | Beskrivning | | Variable | Default | Description |
|---|---|---| |---|---|---|
| `STATION_CALL` | `SA6ANW-1` | Din anropssignal | | `STATION_CALL` | `SA6ANW-1` | Your callsign |
| `DB_PASSWORD` | – | Postgres-lösenord | | `DB_PASSWORD` | - | Postgres password |
| `API_KEY` | – | Delad hemlighet med forwardern | | `API_KEY` | - | Shared secret with forwarder |
| `APRS_IS_PASSCODE` | `-1` | `-1` för receive-only | | `APRS_IS_PASSCODE` | `-1` | `-1` for receive-only |
| `APRS_IS_FILTER` | `r/58.35/14.05/200` | lat/lon/km runt din QTH | | `APRS_IS_FILTER` | `r/58.35/14.05/200` | lat/lon/km around your QTH |
| `LOG_LEVEL` | `INFO` | `DEBUG` för varje frame | | `LOG_LEVEL` | `INFO` | `DEBUG` for per-frame logging |
### Shack (`agw-forwarder/.env`) ### Shack (`agw-forwarder/.env`)
| Variabel | Standard | Beskrivning | | Variable | Default | Description |
|---|---|---| |---|---|---|
| `AGW_HOST` | `localhost` | Direwolf-host | | `AGW_HOST` | `localhost` | Direwolf host |
| `AGW_PORT` | `8000` | Direwolf AGW-port | | `AGW_PORT` | `8000` | Direwolf AGW port |
| `COLLECTOR_URL` | – | URL till DMZ-API:t | | `COLLECTOR_URL` | - | DMZ API URL |
| `API_KEY` | – | Samma som i DMZ `.env` | | `API_KEY` | - | Same as DMZ `.env` |
| `STATION_CALL` | `SA6ANW-1` | Din anropssignal | | `STATION_CALL` | `SA6ANW-1` | Your callsign |
--- ---
## Implementationsnoteringar ## Map
### AGW monitoring-format The map has three zoom levels:
Direwolf skickar `U`-frames i monitoring-mode med ett ledande mellanslag och full monitoring-header före `\r`:
| Zoom | Display |
|---|---|
| < 7 | One dot per Maidenhead subsquare |
| 7 - 13 | Transparent subsquare rectangles (~5 x 4.6 km each) |
| >= 14 | Individual observation points |
Clicking a square opens a popup with two tabs:
- **PATHS** - actual digipeater hops heard from that square, sorted by count. Digipeaters without a known position are shown in red.
- **STATIONS** - callsigns heard from that square. Clicking a station shows detailed statistics including distance, timestamps, and top paths.
Hovering over a path row draws the route on the map: square center to each digipeater to RX station. The RX station marker (pulsing yellow dot) shows all paths heard aggregated across all squares.
Clicking **SQUARES** in the header opens a feed of all squares sorted by most recently added, with distance from the RX station.
Popups can be dragged to move them out of the way.
---
## API
| Endpoint | Auth | Description |
|---|---|---|
| `POST /ingest/rf` | Bearer | RF frame from forwarder |
| `GET /health` | - | Liveness check |
| `GET /coverage/squares` | - | Subsquares with path and station breakdown |
| `GET /coverage/points` | - | Individual observation positions |
| `GET /coverage/digis` | - | Latest known position per callsign |
| `GET /coverage/rx-stats` | - | All paths aggregated across all squares |
| `GET /coverage/station/{callsign}` | - | Detailed statistics for one station |
Swagger UI: `http://<host>:8085/docs`
---
## Implementation notes
### AGW monitoring format
Direwolf sends `U`-frames in monitoring mode with a leading space and full
monitoring header before `\r`:
``` ```
1:Fm SM6MJW-9 To UXRT8L Via WIDE1-1,WIDE2-1 <UI pid=F0 Len=17 F=0 >[16:46:13]\r<info> 1:Fm SM6MJW-9 To UXRT8L Via WIDE1-1,WIDE2-1 <UI pid=F0 Len=17 F=0 >[16:46:13]\r<info>
``` ```
Via-pathen extraheras med regex mellan `Via ` och ` <UI`. `heard_direct` avgörs av om någon hop i pathen har `*` (har-repeaterats-flaggan). The via path is extracted with a regex between `Via ` and ` <UI`.
### Null-bytes ### Path reduction
APRS-frames kan innehålla `0x00`-bytes i info-fältet. Dessa stoppas ut innan DB-insert eftersom Postgres UTF8 inte accepterar null-bytes. Raw APRS paths like `SK6WW,WIDE2-1` are reduced to actual physical digipeaters
by stripping alias hops (`WIDE*`, `RELAY`, `TRACE`, `ECHO`, `GATE`). This means
`SK6WW,WIDE1` and `SK6WW,WIDE2-1` both reduce to `SK6WW` and are counted together.
### Null bytes
APRS frames can contain `0x00` bytes in the info field. These are stripped
before database insert since Postgres UTF-8 rejects null bytes.
--- ---
## Nyttiga queries ## Useful queries
```sql ```sql
-- Senaste RF-frames -- Latest RF frames
SELECT ts, src_call, lat, lon, heard_direct, path SELECT ts, src_call, lat, lon, path
FROM rf_frames FROM rf_frames
ORDER BY ts DESC ORDER BY ts DESC
LIMIT 20; LIMIT 20;
-- Direkt-hörda stationer senaste 7 dagarna -- Stations heard in the last 30 minutes
SELECT ts, src_call, lat, lon, path
FROM rf_frames
WHERE heard_direct = TRUE
AND ts > NOW() - INTERVAL '7 days'
ORDER BY ts DESC;
-- Digis/stationer hörda senaste 30 min
SELECT src_call, MAX(ts) AS last_seen, COUNT(*) AS frames SELECT src_call, MAX(ts) AS last_seen, COUNT(*) AS frames
FROM rf_frames FROM rf_frames
WHERE ts > NOW() - INTERVAL '30 minutes' WHERE ts > NOW() - INTERVAL '30 minutes'
GROUP BY src_call GROUP BY src_call
ORDER BY last_seen DESC; ORDER BY last_seen DESC;
-- Täckningspunkter per rutnätscell (0.01°) -- Coverage grid
SELECT SELECT
ROUND(lat::numeric, 2) AS grid_lat, ROUND(lat::numeric, 2) AS grid_lat,
ROUND(lon::numeric, 2) AS grid_lon, ROUND(lon::numeric, 2) AS grid_lon,
COUNT(*) AS hits COUNT(*) AS hits
FROM rf_frames FROM rf_frames
WHERE heard_direct = TRUE WHERE lat IS NOT NULL
AND lat IS NOT NULL
GROUP BY grid_lat, grid_lon; GROUP BY grid_lat, grid_lon;
-- APRS-IS senaste timmen
SELECT ts, src_call, lat, lon
FROM is_frames
WHERE ts > NOW() - INTERVAL '1 hour'
ORDER BY ts DESC;
``` ```
--- Access the database:
```bash
## API docker exec -it heardlog-db-1 psql -U aprs -d aprs
```
| Endpoint | Metod | Auth | Beskrivning |
|---|---|---|---|
| `/ingest/rf` | POST | Bearer | RF-frame från forwarder |
| `/health` | GET | – | Liveness check |
Swagger UI: `http://<dmz-ip>:8085/docs`

19
collector/main.py
View File

@@ -28,6 +28,21 @@ from maidenhead import latlon_to_subsquare, subsquare_bounds
import re as _re import re as _re
def _valid_position(lat, lon) -> bool:
"""Filter out corrupt or irrelevant positions."""
if lat is None or lon is None:
return False
if lat != lat or lon != lon: # NaN check
return False
if abs(lat) < 0.01 and abs(lon) < 0.01: # 0,0 = no GPS fix
return False
if lat < 54 or lat > 72: # outside Scandinavia
return False
if lon < 4 or lon > 32: # outside Scandinavia
return False
return True
def _extract_digis(path: str) -> str: def _extract_digis(path: str) -> str:
""" """
Reduce a raw APRS path to the actual physical digipeaters that handled it. Reduce a raw APRS path to the actual physical digipeaters that handled it.
@@ -187,6 +202,10 @@ async def ingest_rf(frame: RFFrameIn) -> None:
lat, lon, _ = _parse_position(tnc2) lat, lon, _ = _parse_position(tnc2)
# Nullify invalid positions so they are stored as NULL rather than corrupt data
if not _valid_position(lat, lon):
lat, lon = None, None
logger.info( logger.info(
"RF %-6s %-9s [%s] lat=%s lon=%s", "RF %-6s %-9s [%s] lat=%s lon=%s",
"DIRECT" if frame.heard_direct else "VIA", "DIRECT" if frame.heard_direct else "VIA",

26
web/index.html
View File

@@ -3,7 +3,7 @@
<head> <head>
<meta charset="UTF-8"> <meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>heardlog ??? APRS RF Coverage</title> <title>heardlog – APRS RF Coverage</title>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/leaflet/1.9.4/leaflet.min.css"> <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/leaflet/1.9.4/leaflet.min.css">
<script src="https://cdnjs.cloudflare.com/ajax/libs/leaflet/1.9.4/leaflet.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/leaflet/1.9.4/leaflet.min.js"></script>
<style> <style>
@@ -115,10 +115,10 @@
<header> <header>
<div class="logo">heard<span>log</span> &mdash; <em>APRS RF COVERAGE</em></div> <div class="logo">heard<span>log</span> &mdash; <em>APRS RF COVERAGE</em></div>
<div class="controls"> <div class="controls">
<div class="stat">SQUARES <strong id="sq-count" onclick="toggleSquaresPanel()">???</strong></div> <div class="stat">SQUARES <strong id="sq-count" onclick="toggleSquaresPanel()">–</strong></div>
<div class="stat">POSITIONS <strong id="pt-count">???</strong></div> <div class="stat">POSITIONS <strong id="pt-count">–</strong></div>
<div class="stat">RX <strong>SA6ANW-1</strong></div> <div class="stat">RX <strong>SA6ANW-1</strong></div>
<button class="btn" onclick="refresh()">??? REFRESH</button> <button class="btn" onclick="refresh()">&#8635; REFRESH</button>
</div> </div>
</header> </header>
<div id="map"></div> <div id="map"></div>
@@ -167,7 +167,7 @@ async function fetchAll() {
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Popup HTML ??? square view + station view both in DOM, toggled with CSS // Popup HTML – square view + station view both in DOM, toggled with CSS
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
function makePopupHTML(sq) { function makePopupHTML(sq) {
const pathRows = sq.paths.map(p => { const pathRows = sq.paths.map(p => {
@@ -242,7 +242,7 @@ async function showStation(sqId, callsign) {
try { try {
const res = await fetch(`${API}/api/coverage/station/${encodeURIComponent(callsign)}`); const res = await fetch(`${API}/api/coverage/station/${encodeURIComponent(callsign)}`);
const d = await res.json(); const d = await res.json();
const fmt = iso => iso ? new Date(iso).toLocaleString('sv-SE',{dateStyle:'short',timeStyle:'short'}) : '???'; const fmt = iso => iso ? new Date(iso).toLocaleString('sv-SE',{dateStyle:'short',timeStyle:'short'}) : '–';
const topPaths = (d.top_paths||[]).map(p => const topPaths = (d.top_paths||[]).map(p =>
`<div class="detail-row"><span class="detail-label">${p.path}</span><span class="detail-value">${p.count}</span></div>` `<div class="detail-row"><span class="detail-label">${p.path}</span><span class="detail-value">${p.count}</span></div>`
).join(''); ).join('');
@@ -253,16 +253,16 @@ async function showStation(sqId, callsign) {
<div class="detail-row"><span class="detail-label">Unique positions</span><span class="detail-value">${d.unique_positions}</span></div> <div class="detail-row"><span class="detail-label">Unique positions</span><span class="detail-value">${d.unique_positions}</span></div>
<div class="detail-row"><span class="detail-label">Unique paths</span><span class="detail-value">${d.unique_paths}</span></div> <div class="detail-row"><span class="detail-label">Unique paths</span><span class="detail-value">${d.unique_paths}</span></div>
<div class="detail-section">DISTANCE (LINE OF SIGHT)</div> <div class="detail-section">DISTANCE (LINE OF SIGHT)</div>
<div class="detail-row"><span class="detail-label">Closest</span><span class="detail-value">${d.distance_min_km??'???'} km</span></div> <div class="detail-row"><span class="detail-label">Closest</span><span class="detail-value">${d.distance_min_km??'–'} km</span></div>
<div class="detail-row"><span class="detail-label">Farthest</span><span class="detail-value">${d.distance_max_km??'???'} km</span></div> <div class="detail-row"><span class="detail-label">Farthest</span><span class="detail-value">${d.distance_max_km??'–'} km</span></div>
<div class="detail-row"><span class="detail-label">Average</span><span class="detail-value">${d.distance_avg_km??'???'} km</span></div> <div class="detail-row"><span class="detail-label">Average</span><span class="detail-value">${d.distance_avg_km??'–'} km</span></div>
<div class="detail-section">TIMESTAMPS</div> <div class="detail-section">TIMESTAMPS</div>
<div class="detail-row"><span class="detail-label">First heard</span><span class="detail-value">${fmt(d.first_heard)}</span></div> <div class="detail-row"><span class="detail-label">First heard</span><span class="detail-value">${fmt(d.first_heard)}</span></div>
<div class="detail-row"><span class="detail-label">Last heard</span><span class="detail-value">${fmt(d.last_heard)}</span></div> <div class="detail-row"><span class="detail-label">Last heard</span><span class="detail-value">${fmt(d.last_heard)}</span></div>
<div class="detail-section">TOP PATHS</div> <div class="detail-section">TOP PATHS</div>
${topPaths} ${topPaths}
<div class="detail-section">SQUARES</div> <div class="detail-section">SQUARES</div>
<div class="squares-list">${(d.squares||[]).join(' &nbsp;') || '???'}</div>`; <div class="squares-list">${(d.squares||[]).join(' &nbsp;') || '–'}</div>`;
} catch(e) { } catch(e) {
body.innerHTML = '<div class="loading-msg">Failed to load data.</div>'; body.innerHTML = '<div class="loading-msg">Failed to load data.</div>';
} }
@@ -509,7 +509,7 @@ function clearPathLines() {
function drawPath(rawPath, stationLat, stationLon) { function drawPath(rawPath, stationLat, stationLon) {
clearPathLines(); clearPathLines();
// Build waypoints: station ??? each digi ??? RX // Build waypoints: station → each digi → RX
const waypoints = [{ lat: stationLat, lon: stationLon, label: 'Station' }]; const waypoints = [{ lat: stationLat, lon: stationLon, label: 'Station' }];
if (rawPath && rawPath !== 'DIRECT') { if (rawPath && rawPath !== 'DIRECT') {
@@ -613,8 +613,8 @@ map.on('popupopen', e => {
map.on('zoomend', renderLayers); map.on('zoomend', renderLayers);
async function refresh() { async function refresh() {
document.getElementById('sq-count').textContent = '???'; document.getElementById('sq-count').textContent = '…';
document.getElementById('pt-count').textContent = '???'; document.getElementById('pt-count').textContent = '…';
await fetchAll(); await fetchAll();
} }

1
web/nginx.conf
View File

@@ -1,6 +1,7 @@
server { server {
listen 80; listen 80;
charset utf-8; charset utf-8;
source_charset utf-8;
location / { location / {
root /usr/share/nginx/html; root /usr/share/nginx/html;