From 261ef3f4db0cfcfb77157fc342a3703086b866eb Mon Sep 17 00:00:00 2001 From: Mathieu Moran Date: Sat, 8 Aug 2026 18:01:35 -0400 Subject: [PATCH] Revive yaris-obd: USB OBD adapter, real GPS/GNSS, offline map, hotspot, SQLite logging, dashcam retrieval Squashed rewrite of the project revival work (replaces prior history on this branch to remove personal data that had been accidentally committed - real LAN IP, GPS coordinates, home WiFi SSID, and phone model). See PLAN.md for full phase-by-phase details. Summary: - Phase 2: adapt app_obd.py for a USB OBD-II adapter (was Bluetooth-only), auto-detect /dev/ttyUSB*, fix several pre-existing crash bugs. - Phase 3/4: make the web UI work fully offline (vendored Socket.IO/Leaflet/ Chart.js/FontAwesome, offline map tile downloader script, live GPS marker tracking fix). - Phase 5: systemd deployment/autostart on the Pi. - Phase 6: WiFi hotspot mode via NetworkManager (auto-switch between home WiFi and a local AP for in-car web UI access), mDNS hostname + port 80 redirect so the UI is reachable at a plain hostname with no port number. - Phase 7: fix app_db.py/app_sql.py bugs, wire up local SQLite trip logging. - Phase 8: brand-agnostic USB dashcam file retrieval (lsblk transport-based detection, works with any dashcam brand/folder layout). - Rewritten readme.md with a proper project overview. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .gitignore | 27 + PLAN.md | 330 +++++++++ app.py | 17 +- deploy/install_netwatch.sh | 21 + deploy/install_port_redirect.sh | 23 + deploy/install_service.sh | 24 + deploy/nftables-yaris.conf | 10 + deploy/yaris-netwatch.service | 8 + deploy/yaris-netwatch.sh | 47 ++ deploy/yaris-netwatch.timer | 11 + deploy/yaris-obd.service | 22 + .../__pycache__/app_account.cpython-310.pyc | Bin 452 -> 0 bytes .../__pycache__/app_account.cpython-311.pyc | Bin 592 -> 0 bytes modules/__pycache__/app_car.cpython-310.pyc | Bin 427 -> 0 bytes modules/__pycache__/app_car.cpython-311.pyc | Bin 554 -> 0 bytes .../__pycache__/app_events.cpython-310.pyc | Bin 1194 -> 0 bytes .../__pycache__/app_events.cpython-311.pyc | Bin 2227 -> 0 bytes modules/__pycache__/app_gps.cpython-310.pyc | Bin 814 -> 0 bytes modules/__pycache__/app_gps.cpython-311.pyc | Bin 1194 -> 0 bytes modules/__pycache__/app_obd.cpython-310.pyc | Bin 2876 -> 0 bytes modules/__pycache__/app_obd.cpython-311.pyc | Bin 6528 -> 0 bytes modules/__pycache__/app_sql.cpython-310.pyc | Bin 2461 -> 0 bytes modules/__pycache__/app_sql.cpython-311.pyc | Bin 3676 -> 0 bytes modules/app_dash.py | 255 ++++++- modules/app_db.py | 38 +- modules/app_events.py | 13 +- modules/app_gps.py | 141 +++- modules/app_obd.py | 158 +++-- modules/app_sql.py | 214 +++--- readme.md | 78 ++- requirements.txt | 4 +- scripts/download_map_tiles.py | 132 ++++ static/js/chart.min.js | 21 +- static/js/dash.js | 65 ++ static/js/yaris.js | 31 +- static/vendor/leaflet/images/layers-2x.png | Bin 0 -> 1259 bytes static/vendor/leaflet/images/layers.png | Bin 0 -> 696 bytes .../vendor/leaflet/images/marker-icon-2x.png | Bin 0 -> 2464 bytes static/vendor/leaflet/images/marker-icon.png | Bin 0 -> 1466 bytes .../vendor/leaflet/images/marker-shadow.png | Bin 0 -> 618 bytes static/vendor/leaflet/leaflet.css | 661 ++++++++++++++++++ static/vendor/leaflet/leaflet.js | 6 + static/vendor/socketio/socket.io.min.js | 7 + templates/dash.html | 25 +- templates/dtc.html | 5 +- templates/index.html | 11 +- 46 files changed, 2150 insertions(+), 255 deletions(-) create mode 100644 .gitignore create mode 100644 PLAN.md create mode 100755 deploy/install_netwatch.sh create mode 100755 deploy/install_port_redirect.sh create mode 100755 deploy/install_service.sh create mode 100644 deploy/nftables-yaris.conf create mode 100644 deploy/yaris-netwatch.service create mode 100755 deploy/yaris-netwatch.sh create mode 100644 deploy/yaris-netwatch.timer create mode 100644 deploy/yaris-obd.service delete mode 100644 modules/__pycache__/app_account.cpython-310.pyc delete mode 100644 modules/__pycache__/app_account.cpython-311.pyc delete mode 100644 modules/__pycache__/app_car.cpython-310.pyc delete mode 100644 modules/__pycache__/app_car.cpython-311.pyc delete mode 100644 modules/__pycache__/app_events.cpython-310.pyc delete mode 100644 modules/__pycache__/app_events.cpython-311.pyc delete mode 100644 modules/__pycache__/app_gps.cpython-310.pyc delete mode 100644 modules/__pycache__/app_gps.cpython-311.pyc delete mode 100644 modules/__pycache__/app_obd.cpython-310.pyc delete mode 100644 modules/__pycache__/app_obd.cpython-311.pyc delete mode 100644 modules/__pycache__/app_sql.cpython-310.pyc delete mode 100644 modules/__pycache__/app_sql.cpython-311.pyc create mode 100644 scripts/download_map_tiles.py create mode 100644 static/js/dash.js create mode 100644 static/vendor/leaflet/images/layers-2x.png create mode 100644 static/vendor/leaflet/images/layers.png create mode 100644 static/vendor/leaflet/images/marker-icon-2x.png create mode 100644 static/vendor/leaflet/images/marker-icon.png create mode 100644 static/vendor/leaflet/images/marker-shadow.png create mode 100644 static/vendor/leaflet/leaflet.css create mode 100644 static/vendor/leaflet/leaflet.js create mode 100644 static/vendor/socketio/socket.io.min.js diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..16c75fa --- /dev/null +++ b/.gitignore @@ -0,0 +1,27 @@ +# Python +__pycache__/ +*.py[cod] +*.egg-info/ +.venv/ +venv/ + +# Editor +.vscode/ + +# Downloaded offline map tiles (large, regenerate via scripts/download_map_tiles.py +# and deploy directly to the Pi - not tracked in git) +static/vendor/tiles/ + +# Runtime logs +*.log + +# Local secrets (MAPTILER_KEY etc.) - never commit real credentials +.env +*.env +!db/creds.env + +# Local SQLite trip-tracking databases (per-device data, not source) +db/sql/ + +# Imported dashcam footage (large, per-device data, not source) +dash/videos/ diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..46e7d8a --- /dev/null +++ b/PLAN.md @@ -0,0 +1,330 @@ +# yaris-obd Revival Plan + +This document tracks the plan to revive and finish this project: a Raspberry Pi 3-based +OBD-II + GPS car tracker for a 2013 Toyota Yaris, hosting a local web UI over a self-broadcast +WiFi hotspot, with dashcam file retrieval and remote upload of itinerary/dashcam data. + +Keep this file up to date as phases complete or plans change, so any future session/agent can +pick up the project without needing prior chat history. + +## Hardware + +- **Board**: Raspberry Pi 3 Model B Rev 1.2, 906MB RAM. Confirmed too weak to run an AI coding + agent — all agent-assisted dev work happens on a separate local machine; the Pi is only used + for hardware bring-up/testing/deployment via SSH (`yaris@`). +- **GPS/GSM**: Waveshare **SIM868 GSM/GPRS/GNSS/Bluetooth HAT**. Also has unused cellular + (GSM/GPRS) data capability, earmarked as a fallback internet uplink (see Remote Server phase). +- **OBD-II adapter**: a cheap unbranded USB adapter using a **PL2303** USB-to-serial chip, + presumed ELM327-compatible (works with the `python-obd` library). Originally the project used + a Bluetooth OBD adapter; this has been switched to USB. +- **Car**: 2013 Toyota Yaris. + +### SIM868 HAT bring-up notes (important gotchas) +- UART routing is a 3-position jumper: **A** = USB↔SIM868 (via onboard CP2102), **B** = Pi + GPIO↔SIM868, **C** = USB↔Pi console. Must be on **B** for the Pi to talk to the module over + its own GPIO UART. +- Pi-side UART setup required: `enable_uart=1` and `dtoverlay=disable-bt` in + `/boot/firmware/config.txt` (frees the full PL011 UART — `/dev/ttyAMA0`/`/dev/serial0` — from + the onboard Bluetooth controller), `raspi-config nonint do_serial_cons 1` (stops a login shell + from claiming the serial port), and disable the `hciuart` systemd service. +- **The SIM868 module chip itself needs its PWRKEY button pressed and held (~1-2s) to power on** + — this is separate from the board's 5V/PWR LED, which can be lit while the module chip is + fully off. AT commands get zero response if the module chip isn't powered, even with perfect + UART wiring/config. This is a manual step needed on every power cycle unless a P4 + GPIO-power-on hardware mod (per Waveshare FAQ) is done — not pursued so far. +- Confirmed AT command baud rate: **115200**. +- GNSS workflow: `AT+CGNSPWR?` to check GNSS power state, `AT+CGNSPWR=1` to power on, + `AT+CGNSINF` to poll for a fix. Response format: + `+CGNSINF: run,fix,utc_time,lat,lon,alt,speed,course,fixmode,reserved1,hdop,pdop,vdop,reserved2,gps_sat_view,gnss_sat_used,gnss_sat_view,cn0,hpa,vpa` + — field 1 (`fix`) must be `'1'` for a valid fix. +- **The GPS/GNSS antenna is separate from the GSM antenna** — both need to be physically + connected to their respective u.FL connectors on the HAT. A real fix was obtained only after + discovering the GPS antenna wasn't plugged in (only GSM was). +- Confirmed real fix obtained near a window: ~80s cold-start TTFF, 5-7 satellites, HDOP ~1.2-1.6. + +## Software Architecture Decisions + +1. **Offline maps**: Pre-cache/download offline map tiles for the car's usual driving area so + the live map works fully in the car with no internet (not just numeric GPS readout). + Tooling: MapTiler Cloud free tier (raster XYZ tiles, `streets-v2` style) — one-time download + on the dev machine via `scripts/download_map_tiles.py`, output shipped to the Pi as static + PNGs. Bulk-downloading directly from `tile.openstreetmap.org` was ruled out — it violates the + OSM tile usage policy. Bounding box is a required CLI arg (the user's own driving area, not + committed here), zoom 8-15 by default (~14k tiles for a city-plus-suburbs-sized area, well + under MapTiler's 100k/month free quota). +2. **WiFi hotspot mode**: Auto-switch — the Pi joins home WiFi when in range (for + syncing/updates), and falls back to broadcasting its own AP otherwise. **Important scope + clarification**: the hotspot is *only* for local web-UI viewing on a phone/laptop while + driving — it is not part of the upload path. Remote-server uploads happen independently via + home WiFi (when in range) or the SIM868 cellular fallback (decision #5) — the phone connecting + to the hotspot never relays/carries upload traffic. +3. **Local/remote persistence**: Keep SQLite locally (reliable local write path per trip) + + MongoDB Atlas for remote sync when online. Fix existing bugs rather than replace the design. +4. **Remote server**: Self-hosted — a custom server/API (e.g. on a home server or VPS) receives + itinerary and dashcam uploads, rather than 3rd-party managed cloud services. +5. **Remote upload uplink**: Use the SIM868's GSM/GPRS cellular connection as a fallback uplink + for uploads when no WiFi is available (e.g. while driving), not just "sync when on home WiFi." + +## Phases + +### Phase 1: Real GPS integration — ✅ DONE +`modules/app_gps.py` rewritten to read real SIM868 GNSS data via serial AT commands +(`/dev/serial0` @ 115200 baud) instead of hardcoded fake coordinates. Background daemon thread +polls `AT+CGNSINF` every 2s, parses fix data, emits `gps_location` via socketio only on an actual +fix. Verified against real hardware (see bring-up notes above). + +### Phase 2: OBD fixes for USB adapter — ✅ DONE +`modules/app_obd.py` fixed (commit "Fix app_obd.py bugs and adapt for USB OBD-II adapter", +pushed to `origin/shameless-aislop`): +- `connection` is never left undefined on failed connect (was a `NameError` crash). +- Added `_find_obd_port()`/`connect_obd()`: auto-detects `/dev/ttyUSB*` (the PL2303 adapter), + honors an `OBD_PORT` env override, falls back to python-obd's own port auto-scan (still covers + the old `/dev/rfcomm*` Bluetooth path). +- Fixed stale-response bug: `obd_speed()`/`obd_rpm()` now re-query every loop iteration instead + of reusing one response forever. +- Fixed `obd_rpm()` using the wrong lock, never updating the global `rpm`, and emitting to the + wrong socketio event (`obd_speed` instead of `obd_rpm`). +- Fixed `kill_obd()` being a no-op — now uses `threading.Event` flags actually checked in the + polling loops. +- Fixed inverted thread-start logic (threads were starting on failed connection, not success). +- Fixed `obm_error` → `obd_error` typo (frontend listens for `obd_error`). +- Added `obd` (python-obd) to `requirements.txt`. +- **Still untested**: a real connection to the actual car (adapter has only been tested + standalone on the Pi so far — correctly reports "no connection to car" with nothing attached). + +### Phase 3: Web UI must work fully offline — ✅ DONE +The whole point of the project is a Pi hotspot with **no internet in the car**, but the current +UI silently depended on the internet for core functionality: +- `templates/index.html`/`dash.html`/`dtc.html` loaded Google Fonts, FontAwesome, Leaflet JS/CSS, + and the Socket.IO client from public CDNs — none of this loads offline, breaking layout, the + websocket connection itself, and the map entirely. +- `static/js/yaris.js` loaded live map tiles from `tile.openstreetmap.org` (also needs internet). +- A local `static/js/chart.min.js` existed but was unused — `index.html` loaded Chart.js from + `cdn.jsdelivr.net` instead (turned out the local copy was a stale Chart.js **v2.8.0** while + `yaris.js`'s config uses v3/v4-style `scales: {x:{...}, y:{...}}` syntax — that mismatch is + exactly why the CDN was used instead. Replaced the local file with a current v4.5.0 UMD build). + +**What was done**: +- Vendored Socket.IO client (4.8.1, matches Flask-SocketIO 5.6.1's Engine.IO v4 protocol) under + `static/vendor/socketio/`, and Leaflet (1.9.4 JS/CSS + marker/layers images) under + `static/vendor/leaflet/`. +- FontAwesome 5.12.0 was already vendored locally at `static/fonts/` (self-consistent, just never + referenced) — repointed all three templates to it instead of the CDN. +- Dropped the Google Fonts (Nunito) `` entirely rather than self-hosting 12 font-weight + files — Bootstrap's `--bs-font-sans-serif` variable already falls back cleanly to system fonts + (`-apple-system, Segoe UI, Roboto, ...`) after Nunito, so this is a clean no-cost simplification. +- Replaced `static/js/chart.min.js` with a current Chart.js v4.5.0 UMD build (auto-registering, + same drop-in behavior as the old CDN ` + diff --git a/templates/dtc.html b/templates/dtc.html index 905ee7a..909caa0 100644 --- a/templates/dtc.html +++ b/templates/dtc.html @@ -7,9 +7,8 @@ Dashboard - yaris vroom vroom de course - - - + + diff --git a/templates/index.html b/templates/index.html index 11abfae..1b042dd 100644 --- a/templates/index.html +++ b/templates/index.html @@ -7,9 +7,8 @@ Dashboard - yaris vroom vroom de course - - - + + @@ -180,9 +179,9 @@ - - - + + +