Rttningar

N/A
2026-05-03 06:39:58 +00:00 · 2026-05-02 18:37:10 +00:00
4 changed files with 168 additions and 99 deletions
--- a/README.md
+++ b/README.md
@@ -1,187 +1,236 @@
 # heardlog

-Samlar APRS-data från två källor och lagrar i TimescaleDB + PostGIS som grund för en täckningskarta.
+APRS RF coverage logger and map for amateur radio digipeaters.
+
+Collects every packet heard on RF by a Direwolf digipeater and builds an
+interactive coverage map showing which Maidenhead subsquares have been reached,
+which paths were used, and which stations have been heard.
+
+Live: [aprs.sa6anw.se](https://aprs.sa6anw.se)
+
+---
+
+## Architecture

 ```
-[Shack-LAN]                        [DMZ]
+[Shack LAN]                         [DMZ]
 Direwolf AGW :8000
-      │
-agw_forwarder.py  ──POST/Bearer──► FastAPI :8085  ──► TimescaleDB + PostGIS
-                                          │
-                          rotate.aprs2.net┘  (APRS-IS, outbound från DMZ)
+      |
+agw_forwarder.py  --POST/Bearer-->  FastAPI :8085  -->  TimescaleDB + PostGIS
+                                          |
+                          rotate.aprs2.net|  (APRS-IS regional feed, outbound)
+                                          |
+                                     nginx :80/443
+                                          |
+                                     index.html  (Leaflet map)
 ```

-| Källa | Tabell | Nyckeldata |
+| Source | Table | Key data |
 |---|---|---|
-| Direwolf AGW (RF) | `rf_frames` | `heard_direct` – hördes utan mellanliggande digi |
-| APRS-IS regionalt | `is_frames` | Stationer inom 200 km som digipeatas till internet |
-
-`heard_direct = TRUE` är grundstenen för täckningskartan – ett bevis på att en station nådde vår mottagare direkt.
+| Direwolf AGW (RF) | `rf_frames` | Every packet received over the air |
+| APRS-IS regional | `is_frames` | Regional feed within 200 km radius |

 ---

-## Krav
+## Repository layout

- Docker + Docker Compose (DMZ-server)
- Direwolf med `AGWPORT 8000` i `direwolf.conf` (shack-dator)
- Python 3.10+ och `pip3 install requests` (shack-dator)
+```
+heardlog/
+????????? .env.example
+????????? README.md
+????????? docker-compose.yml
+????????? collector/               # DMZ - runs in Docker
+???   ????????? Dockerfile
+???   ????????? requirements.txt
+???   ????????? config.py
+???   ????????? db.py                # TimescaleDB schema + insert helpers
+???   ????????? maidenhead.py        # Maidenhead subsquare calculations
+???   ????????? aprs_is.py           # APRS-IS client
+???   ????????? main.py              # FastAPI app + APRS-IS background task
+????????? agw-forwarder/           # Shack - runs directly on the Direwolf machine
+???   ????????? agw_forwarder.py
+???   ????????? requirements.txt
+???   ????????? agw-forwarder.service
+????????? web/
+    ????????? index.html           # Leaflet map (served by nginx)
+    ????????? nginx.conf
+```

 ---

-## DMZ – snabbstart
+## Getting started
+
+### DMZ

 ```bash
 git clone <repo> heardlog
 cd heardlog

 cp .env.example .env
+# Edit .env - set DB_PASSWORD and API_KEY

-# Generera API-nyckel
+# Generate API key
 python3 -c "import secrets; print(secrets.token_hex(32))"
-# Klistra in som API_KEY i .env

 docker compose up -d
 docker compose logs -f collector
 ```

-Förväntad output:
+Expected output:
 ```
 INFO  db    Database schema ready
-INFO  main  Startup complete – listening for RF frames
+INFO  main  Startup complete - listening for RF frames
 INFO  aprs_is  APRS-IS login: # logresp SA6ANW-1 unverified, server T2...
 ```

-`unverified` på APRS-IS är förväntat – passcode `-1` ger receive-only vilket är tillräckligt.
-
---
-
-## Shack – agw-forwarder
+### Shack (agw-forwarder)

 ```bash
 pip3 install requests

-# Skapa konfigurationsfil
 cat > heardlog/agw-forwarder/.env << 'EOF'
 AGW_HOST=localhost
 AGW_PORT=8000
 COLLECTOR_URL=http://<dmz-ip>:8085
-API_KEY=<samma nyckel som i DMZ .env>
+API_KEY=<same key as DMZ .env>
 STATION_CALL=SA6ANW-1
 LOG_LEVEL=INFO
 EOF

-# Testa manuellt
+# Test manually first
 cd heardlog/agw-forwarder
 python3 agw_forwarder.py
 ```

-### Kör som systemd-tjänster
+### Run as systemd services (shack)

 ```bash
-# Kopiera service-filer
 sudo cp heardlog/agw-forwarder/direwolf.service /etc/systemd/system/
 sudo cp heardlog/agw-forwarder/agw-forwarder.service /etc/systemd/system/
-
 sudo systemctl daemon-reload
 sudo systemctl enable --now direwolf
 sudo systemctl enable --now agw-forwarder
-
-# Verifiera
 journalctl -u direwolf -u agw-forwarder -f
 ```

-`agw-forwarder` har `Wants=direwolf.service` – systemd startar Direwolf automatiskt om den inte kör.
+`agw-forwarder` has `Wants=direwolf.service` - systemd starts Direwolf automatically.

 ---

-## Konfiguration
+## Configuration

 ### DMZ (`.env`)

-| Variabel | Standard | Beskrivning |
+| Variable | Default | Description |
 |---|---|---|
-| `STATION_CALL` | `SA6ANW-1` | Din anropssignal |
-| `DB_PASSWORD` | – | Postgres-lösenord |
-| `API_KEY` | – | Delad hemlighet med forwardern |
-| `APRS_IS_PASSCODE` | `-1` | `-1` för receive-only |
-| `APRS_IS_FILTER` | `r/58.35/14.05/200` | lat/lon/km runt din QTH |
-| `LOG_LEVEL` | `INFO` | `DEBUG` för varje frame |
+| `STATION_CALL` | `SA6ANW-1` | Your callsign |
+| `DB_PASSWORD` | - | Postgres password |
+| `API_KEY` | - | Shared secret with forwarder |
+| `APRS_IS_PASSCODE` | `-1` | `-1` for receive-only |
+| `APRS_IS_FILTER` | `r/58.35/14.05/200` | lat/lon/km around your QTH |
+| `LOG_LEVEL` | `INFO` | `DEBUG` for per-frame logging |

 ### Shack (`agw-forwarder/.env`)

-| Variabel | Standard | Beskrivning |
+| Variable | Default | Description |
 |---|---|---|
-| `AGW_HOST` | `localhost` | Direwolf-host |
-| `AGW_PORT` | `8000` | Direwolf AGW-port |
-| `COLLECTOR_URL` | – | URL till DMZ-API:t |
-| `API_KEY` | – | Samma som i DMZ `.env` |
-| `STATION_CALL` | `SA6ANW-1` | Din anropssignal |
+| `AGW_HOST` | `localhost` | Direwolf host |
+| `AGW_PORT` | `8000` | Direwolf AGW port |
+| `COLLECTOR_URL` | - | DMZ API URL |
+| `API_KEY` | - | Same as DMZ `.env` |
+| `STATION_CALL` | `SA6ANW-1` | Your callsign |

 ---

-## Implementationsnoteringar
+## Map

-### AGW monitoring-format
-Direwolf skickar `U`-frames i monitoring-mode med ett ledande mellanslag och full monitoring-header före `\r`:
+The map has three zoom levels:
+
+| Zoom | Display |
+|---|---|
+| < 7 | One dot per Maidenhead subsquare |
+| 7 - 13 | Transparent subsquare rectangles (~5 x 4.6 km each) |
+| >= 14 | Individual observation points |
+
+Clicking a square opens a popup with two tabs:
+
+- **PATHS** - actual digipeater hops heard from that square, sorted by count. Digipeaters without a known position are shown in red.
+- **STATIONS** - callsigns heard from that square. Clicking a station shows detailed statistics including distance, timestamps, and top paths.
+
+Hovering over a path row draws the route on the map: square center to each digipeater to RX station. The RX station marker (pulsing yellow dot) shows all paths heard aggregated across all squares.
+
+Clicking **SQUARES** in the header opens a feed of all squares sorted by most recently added, with distance from the RX station.
+
+Popups can be dragged to move them out of the way.
+
+---
+
+## API
+
+| Endpoint | Auth | Description |
+|---|---|---|
+| `POST /ingest/rf` | Bearer | RF frame from forwarder |
+| `GET /health` | - | Liveness check |
+| `GET /coverage/squares` | - | Subsquares with path and station breakdown |
+| `GET /coverage/points` | - | Individual observation positions |
+| `GET /coverage/digis` | - | Latest known position per callsign |
+| `GET /coverage/rx-stats` | - | All paths aggregated across all squares |
+| `GET /coverage/station/{callsign}` | - | Detailed statistics for one station |
+
+Swagger UI: `http://<host>:8085/docs`
+
+---
+
+## Implementation notes
+
+### AGW monitoring format
+Direwolf sends `U`-frames in monitoring mode with a leading space and full
+monitoring header before `\r`:

 ```
 1:Fm SM6MJW-9 To UXRT8L Via WIDE1-1,WIDE2-1 <UI pid=F0 Len=17 F=0 >[16:46:13]\r<info>
 ```

-Via-pathen extraheras med regex mellan `Via ` och ` <UI`. `heard_direct` avgörs av om någon hop i pathen har `*` (har-repeaterats-flaggan).
+The via path is extracted with a regex between `Via ` and ` <UI`.

-### Null-bytes
-APRS-frames kan innehålla `0x00`-bytes i info-fältet. Dessa stoppas ut innan DB-insert eftersom Postgres UTF8 inte accepterar null-bytes.
+### Path reduction
+Raw APRS paths like `SK6WW,WIDE2-1` are reduced to actual physical digipeaters
+by stripping alias hops (`WIDE*`, `RELAY`, `TRACE`, `ECHO`, `GATE`). This means
+`SK6WW,WIDE1` and `SK6WW,WIDE2-1` both reduce to `SK6WW` and are counted together.
+
+### Null bytes
+APRS frames can contain `0x00` bytes in the info field. These are stripped
+before database insert since Postgres UTF-8 rejects null bytes.

 ---

-## Nyttiga queries
+## Useful queries

 ```sql
-- Senaste RF-frames
-SELECT ts, src_call, lat, lon, heard_direct, path
+-- Latest RF frames
+SELECT ts, src_call, lat, lon, path
 FROM rf_frames
 ORDER BY ts DESC
 LIMIT 20;

-- Direkt-hörda stationer senaste 7 dagarna
-SELECT ts, src_call, lat, lon, path
-FROM rf_frames
-WHERE heard_direct = TRUE
-  AND ts > NOW() - INTERVAL '7 days'
-ORDER BY ts DESC;
-
-- Digis/stationer hörda senaste 30 min
+-- Stations heard in the last 30 minutes
 SELECT src_call, MAX(ts) AS last_seen, COUNT(*) AS frames
 FROM rf_frames
 WHERE ts > NOW() - INTERVAL '30 minutes'
 GROUP BY src_call
 ORDER BY last_seen DESC;

-- Täckningspunkter per rutnätscell (0.01°)
+-- Coverage grid
 SELECT
    ROUND(lat::numeric, 2) AS grid_lat,
    ROUND(lon::numeric, 2) AS grid_lon,
    COUNT(*) AS hits
 FROM rf_frames
-WHERE heard_direct = TRUE
-  AND lat IS NOT NULL
+WHERE lat IS NOT NULL
 GROUP BY grid_lat, grid_lon;
-
-- APRS-IS senaste timmen
-SELECT ts, src_call, lat, lon
-FROM is_frames
-WHERE ts > NOW() - INTERVAL '1 hour'
-ORDER BY ts DESC;
 ```

---
-
-## API
-
-| Endpoint | Metod | Auth | Beskrivning |
-|---|---|---|---|
-| `/ingest/rf` | POST | Bearer | RF-frame från forwarder |
-| `/health` | GET | – | Liveness check |
-
-Swagger UI: `http://<dmz-ip>:8085/docs`
+Access the database:
+```bash
+docker exec -it heardlog-db-1 psql -U aprs -d aprs
+```
--- a/collector/main.py
+++ b/collector/main.py
@@ -28,6 +28,21 @@ from maidenhead import latlon_to_subsquare, subsquare_bounds

 import re as _re

+def _valid_position(lat, lon) -> bool:
+    """Filter out corrupt or irrelevant positions."""
+    if lat is None or lon is None:
+        return False
+    if lat != lat or lon != lon:  # NaN check
+        return False
+    if abs(lat) < 0.01 and abs(lon) < 0.01:  # 0,0 = no GPS fix
+        return False
+    if lat < 54 or lat > 72:  # outside Scandinavia
+        return False
+    if lon < 4 or lon > 32:   # outside Scandinavia
+        return False
+    return True
+
+
 def _extract_digis(path: str) -> str:
    """
    Reduce a raw APRS path to the actual physical digipeaters that handled it.
@@ -187,6 +202,10 @@ async def ingest_rf(frame: RFFrameIn) -> None:

        lat, lon, _ = _parse_position(tnc2)

+        # Nullify invalid positions so they are stored as NULL rather than corrupt data
+        if not _valid_position(lat, lon):
+            lat, lon = None, None
+
        logger.info(
            "RF  %-6s  %-9s  [%s]  lat=%s lon=%s",
            "DIRECT" if frame.heard_direct else "VIA",
--- a/web/index.html
+++ b/web/index.html
@@ -3,7 +3,7 @@
 <head>
 <meta charset="UTF-8">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
-<title>heardlog ??? APRS RF Coverage</title>
+<title>heardlog – APRS RF Coverage</title>
 <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/leaflet/1.9.4/leaflet.min.css">
 <script src="https://cdnjs.cloudflare.com/ajax/libs/leaflet/1.9.4/leaflet.min.js"></script>
 <style>
@@ -115,10 +115,10 @@
 <header>
  <div class="logo">heard<span>log</span> &mdash; <em>APRS RF COVERAGE</em></div>
  <div class="controls">
-    <div class="stat">SQUARES <strong id="sq-count" onclick="toggleSquaresPanel()">???</strong></div>
-    <div class="stat">POSITIONS <strong id="pt-count">???</strong></div>
+    <div class="stat">SQUARES <strong id="sq-count" onclick="toggleSquaresPanel()">–</strong></div>
+    <div class="stat">POSITIONS <strong id="pt-count">–</strong></div>
    <div class="stat">RX <strong>SA6ANW-1</strong></div>
-    <button class="btn" onclick="refresh()">??? REFRESH</button>
+    <button class="btn" onclick="refresh()">&#8635; REFRESH</button>
  </div>
 </header>
 <div id="map"></div>
@@ -167,7 +167,7 @@ async function fetchAll() {
 }

 // ---------------------------------------------------------------------------
-// Popup HTML ??? square view + station view both in DOM, toggled with CSS
+// Popup HTML – square view + station view both in DOM, toggled with CSS
 // ---------------------------------------------------------------------------
 function makePopupHTML(sq) {
  const pathRows = sq.paths.map(p => {
@@ -242,7 +242,7 @@ async function showStation(sqId, callsign) {
  try {
    const res = await fetch(`${API}/api/coverage/station/${encodeURIComponent(callsign)}`);
    const d = await res.json();
-    const fmt = iso => iso ? new Date(iso).toLocaleString('sv-SE',{dateStyle:'short',timeStyle:'short'}) : '???';
+    const fmt = iso => iso ? new Date(iso).toLocaleString('sv-SE',{dateStyle:'short',timeStyle:'short'}) : '–';
    const topPaths = (d.top_paths||[]).map(p =>
      `<div class="detail-row"><span class="detail-label">${p.path}</span><span class="detail-value">${p.count}</span></div>`
    ).join('');
@@ -253,16 +253,16 @@ async function showStation(sqId, callsign) {
      <div class="detail-row"><span class="detail-label">Unique positions</span><span class="detail-value">${d.unique_positions}</span></div>
      <div class="detail-row"><span class="detail-label">Unique paths</span><span class="detail-value">${d.unique_paths}</span></div>
      <div class="detail-section">DISTANCE (LINE OF SIGHT)</div>
-      <div class="detail-row"><span class="detail-label">Closest</span><span class="detail-value">${d.distance_min_km??'???'} km</span></div>
-      <div class="detail-row"><span class="detail-label">Farthest</span><span class="detail-value">${d.distance_max_km??'???'} km</span></div>
-      <div class="detail-row"><span class="detail-label">Average</span><span class="detail-value">${d.distance_avg_km??'???'} km</span></div>
+      <div class="detail-row"><span class="detail-label">Closest</span><span class="detail-value">${d.distance_min_km??'–'} km</span></div>
+      <div class="detail-row"><span class="detail-label">Farthest</span><span class="detail-value">${d.distance_max_km??'–'} km</span></div>
+      <div class="detail-row"><span class="detail-label">Average</span><span class="detail-value">${d.distance_avg_km??'–'} km</span></div>
      <div class="detail-section">TIMESTAMPS</div>
      <div class="detail-row"><span class="detail-label">First heard</span><span class="detail-value">${fmt(d.first_heard)}</span></div>
      <div class="detail-row"><span class="detail-label">Last heard</span><span class="detail-value">${fmt(d.last_heard)}</span></div>
      <div class="detail-section">TOP PATHS</div>
      ${topPaths}
      <div class="detail-section">SQUARES</div>
-      <div class="squares-list">${(d.squares||[]).join(' &nbsp;') || '???'}</div>`;
+      <div class="squares-list">${(d.squares||[]).join(' &nbsp;') || '–'}</div>`;
  } catch(e) {
    body.innerHTML = '<div class="loading-msg">Failed to load data.</div>';
  }
@@ -509,7 +509,7 @@ function clearPathLines() {
 function drawPath(rawPath, stationLat, stationLon) {
  clearPathLines();

-  // Build waypoints: station ??? each digi ??? RX
+  // Build waypoints: station → each digi → RX
  const waypoints = [{ lat: stationLat, lon: stationLon, label: 'Station' }];

  if (rawPath && rawPath !== 'DIRECT') {
@@ -613,8 +613,8 @@ map.on('popupopen', e => {
 map.on('zoomend', renderLayers);

 async function refresh() {
-  document.getElementById('sq-count').textContent = '???';
-  document.getElementById('pt-count').textContent = '???';
+  document.getElementById('sq-count').textContent = '…';
+  document.getElementById('pt-count').textContent = '…';
  await fetchAll();
 }

--- a/web/nginx.conf
+++ b/web/nginx.conf
@@ -1,6 +1,7 @@
 server {
    listen 80;
    charset utf-8;
+    source_charset utf-8;

    location / {
        root /usr/share/nginx/html;
Author	SHA1	Message	Date
root	3a066ea589	Rttningar	2026-05-03 06:39:58 +00:00
root	b5d65c4a52	N/A	2026-05-02 18:37:10 +00:00