yaris-obd/PLAN.md

yaris-obd Revival Plan

This document tracks the plan to revive and finish this project: a Raspberry Pi 3-based OBD-II + GPS car tracker for a 2013 Toyota Yaris, hosting a local web UI over a self-broadcast WiFi hotspot, with dashcam file retrieval and remote upload of itinerary/dashcam data.

Keep this file up to date as phases complete or plans change, so any future session/agent can pick up the project without needing prior chat history.

Hardware

• Board: Raspberry Pi 3 Model B Rev 1.2, 906MB RAM. Confirmed too weak to run an AI coding agent — all agent-assisted dev work happens on a separate local machine; the Pi is only used for hardware bring-up/testing/deployment via SSH (yaris@<pi-ip>).
• GPS/GSM: Waveshare SIM868 GSM/GPRS/GNSS/Bluetooth HAT. Also has unused cellular (GSM/GPRS) data capability, earmarked as a fallback internet uplink (see Remote Server phase).
• OBD-II adapter: a cheap unbranded USB adapter using a PL2303 USB-to-serial chip, presumed ELM327-compatible (works with the python-obd library). Originally the project used a Bluetooth OBD adapter; this has been switched to USB.
• Car: 2013 Toyota Yaris.

SIM868 HAT bring-up notes (important gotchas)

• UART routing is a 3-position jumper: A = USB↔SIM868 (via onboard CP2102), B = Pi GPIO↔SIM868, C = USB↔Pi console. Must be on B for the Pi to talk to the module over its own GPIO UART.
• Pi-side UART setup required: enable_uart=1 and dtoverlay=disable-bt in /boot/firmware/config.txt (frees the full PL011 UART — /dev/ttyAMA0//dev/serial0 — from the onboard Bluetooth controller), raspi-config nonint do_serial_cons 1 (stops a login shell from claiming the serial port), and disable the hciuart systemd service.
• The SIM868 module chip itself needs its PWRKEY button pressed and held (~1-2s) to power on — this is separate from the board's 5V/PWR LED, which can be lit while the module chip is fully off. AT commands get zero response if the module chip isn't powered, even with perfect UART wiring/config. Update 2026-08-09: this HAT revision DOES have the "P4" GPIO-power-on mod wire installed (per the Waveshare FAQ/GSM_PWRKEY.rar script) — confirmed by pulsing physical header pin 7 (BCM GPIO4) LOW for 4s then HIGH while the module was unresponsive, after which it immediately answered AT. This is now automated in modules/app_gps.py (_ensure_module_powered()), which checks AT first and only pulses PWRKEY if the module isn't already responding (PWRKEY is a toggle — pulsing an already-on module would turn it back off, so this check-first behavior is required). No more manual button-pressing needed after a Pi power cycle.
• Confirmed AT command baud rate: 115200.
• GNSS workflow: AT+CGNSPWR? to check GNSS power state, AT+CGNSPWR=1 to power on, AT+CGNSINF to poll for a fix. Response format: +CGNSINF: run,fix,utc_time,lat,lon,alt,speed,course,fixmode,reserved1,hdop,pdop,vdop,reserved2,gps_sat_view,gnss_sat_used,gnss_sat_view,cn0,hpa,vpa — field 1 (fix) must be '1' for a valid fix.
• The GPS/GNSS antenna is separate from the GSM antenna — both need to be physically connected to their respective u.FL connectors on the HAT. A real fix was obtained only after discovering the GPS antenna wasn't plugged in (only GSM was).
• Confirmed real fix obtained near a window: ~80s cold-start TTFF, 5-7 satellites, HDOP ~1.2-1.6.

Software Architecture Decisions

1. Offline maps: Pre-cache/download offline map tiles for the car's usual driving area so the live map works fully in the car with no internet (not just numeric GPS readout). Tooling: MapTiler Cloud free tier (raster XYZ tiles, streets-v2 style) — one-time download on the dev machine via scripts/download_map_tiles.py, output shipped to the Pi as static PNGs. Bulk-downloading directly from tile.openstreetmap.org was ruled out — it violates the OSM tile usage policy. Bounding box is a required CLI arg (the user's own driving area, not committed here), zoom 8-15 by default (~14k tiles for a city-plus-suburbs-sized area, well under MapTiler's 100k/month free quota).
2. WiFi hotspot mode: Auto-switch — the Pi joins home WiFi when in range (for syncing/updates), and falls back to broadcasting its own AP otherwise. Important scope clarification: the hotspot is only for local web-UI viewing on a phone/laptop while driving — it is not part of the upload path. Remote-server uploads happen independently via home WiFi (when in range) or the SIM868 cellular fallback (decision #5) — the phone connecting to the hotspot never relays/carries upload traffic.
3. Local/remote persistence: Keep SQLite locally (reliable local write path per trip). The original plan paired this with MongoDB Atlas for remote sync (modules/app_db.py, fixed in Phase 7) — superseded by decision #4/Phase 9: the actual remote store is the self-hosted Postgres server, uploaded via modules/app_upload.py. app_db.py's Mongo path is not wired into app.py and is effectively dead code at this point, kept only for reference.
4. Remote server: Self-hosted — a custom server/API (e.g. on a home server or VPS) receives itinerary and dashcam uploads, rather than 3rd-party managed cloud services.
5. Remote upload uplink: Use the SIM868's GSM/GPRS cellular connection as a fallback uplink for uploads when no WiFi is available (e.g. while driving), not just "sync when on home WiFi."

Phases

Phase 1: Real GPS integration — ✅ DONE

modules/app_gps.py rewritten to read real SIM868 GNSS data via serial AT commands (/dev/serial0 @ 115200 baud) instead of hardcoded fake coordinates. Background daemon thread polls AT+CGNSINF every 2s, parses fix data, emits gps_location via socketio only on an actual fix. Verified against real hardware (see bring-up notes above).

Phase 2: OBD fixes for USB adapter — ✅ DONE

modules/app_obd.py fixed (commit "Fix app_obd.py bugs and adapt for USB OBD-II adapter", pushed to origin/shameless-aislop):

• connection is never left undefined on failed connect (was a NameError crash).
• Added _find_obd_port()/connect_obd(): auto-detects /dev/ttyUSB* (the PL2303 adapter), honors an OBD_PORT env override, falls back to python-obd's own port auto-scan (still covers the old /dev/rfcomm* Bluetooth path).
• Fixed stale-response bug: obd_speed()/obd_rpm() now re-query every loop iteration instead of reusing one response forever.
• Fixed obd_rpm() using the wrong lock, never updating the global rpm, and emitting to the wrong socketio event (obd_speed instead of obd_rpm).
• Fixed kill_obd() being a no-op — now uses threading.Event flags actually checked in the polling loops.
• Fixed inverted thread-start logic (threads were starting on failed connection, not success).
• Fixed obm_error → obd_error typo (frontend listens for obd_error).
• Added obd (python-obd) to requirements.txt.
• Still untested: a real connection to the actual car (adapter has only been tested standalone on the Pi so far — correctly reports "no connection to car" with nothing attached).

Phase 3: Web UI must work fully offline — ✅ DONE

The whole point of the project is a Pi hotspot with no internet in the car, but the current UI silently depended on the internet for core functionality:

• templates/index.html/dash.html/dtc.html loaded Google Fonts, FontAwesome, Leaflet JS/CSS, and the Socket.IO client from public CDNs — none of this loads offline, breaking layout, the websocket connection itself, and the map entirely.
• static/js/yaris.js loaded live map tiles from tile.openstreetmap.org (also needs internet).
• A local static/js/chart.min.js existed but was unused — index.html loaded Chart.js from cdn.jsdelivr.net instead (turned out the local copy was a stale Chart.js v2.8.0 while yaris.js's config uses v3/v4-style scales: {x:{...}, y:{...}} syntax — that mismatch is exactly why the CDN was used instead. Replaced the local file with a current v4.5.0 UMD build).

What was done:

• Vendored Socket.IO client (4.8.1, matches Flask-SocketIO 5.6.1's Engine.IO v4 protocol) under static/vendor/socketio/, and Leaflet (1.9.4 JS/CSS + marker/layers images) under static/vendor/leaflet/.
• FontAwesome 5.12.0 was already vendored locally at static/fonts/ (self-consistent, just never referenced) — repointed all three templates to it instead of the CDN.
• Dropped the Google Fonts (Nunito) <link> entirely rather than self-hosting 12 font-weight files — Bootstrap's --bs-font-sans-serif variable already falls back cleanly to system fonts (-apple-system, Segoe UI, Roboto, ...) after Nunito, so this is a clean no-cost simplification.
• Replaced static/js/chart.min.js with a current Chart.js v4.5.0 UMD build (auto-registering, same drop-in behavior as the old CDN <script src="https://cdn.jsdelivr.net/npm/chart.js">).
• Offline map tiles: added scripts/download_map_tiles.py, a dev-machine-only one-time tile downloader using the MapTiler Cloud free tier (raster XYZ, streets-v2 style, 256px tiles). Reads MAPTILER_KEY from an env var/CLI arg only — the key itself is never committed or deployed, only the resulting static PNG tiles are. Bounding box is a required CLI arg (your own driving area's coordinates - not committed/hardcoded) at zoom 8-15 (~14k tiles for a city-plus-suburbs-sized area, well under MapTiler's 100k tiles/month free quota). Output goes to static/vendor/tiles/{z}/{x}/{y}.png (gitignored — regenerate locally, rsync to the Pi as part of deployment, don't bloat the git repo with binary tile data).
  • Tiles populated: user signed up for a free MapTiler key and the full zoom 8-15 download ran to completion — 14,241 downloaded, 24 skipped (out-of-bounds), 0 failed (14,265 tiles total, ~285MB). Rsynced to the Pi's static/vendor/tiles/ (tile counts verified identical on both sides). Live Pi instance re-confirmed serving HTTP 200 with the complete offline tile set.
• Updated static/js/yaris.js's Leaflet init to point at /static/vendor/tiles/{z}/{x}/{y}.png (minZoom/maxZoom 8/15 to match). The initial map center is just a generic placeholder (not a real location) - it re-centers automatically to the real live GPS fix once one comes in, and the marker is exposed as carMarker so Phase 4 can move it on live fixes.
• Added a repo-root .gitignore (Python caches, venvs, .vscode/, static/vendor/tiles/, *.env except the already-tracked db/creds.env placeholder) — there wasn't one before, which is why stray __pycache__/.pyc files kept showing up in source control.
• Verified: ran the Flask app in a local venv, confirmed all three pages (/, /dtc, /dash) render with zero external URLs in the served HTML, every local static asset (JS, CSS, fonts, Leaflet images) returns HTTP 200, and all vendored JS files pass a Node.js syntax check.

Phase 4: Map/GPS display bug — ✅ DONE

yaris.js's Leaflet marker (carMarker) was never moved on incoming gps_location socket events — only two text spans (gpsLocationX/Y) got updated. Fixed: carMarker.setLatLng(...) is now called on every gps_location event, and the map re-centers (zoom 15) on the first fix only (mapCentered flag), so the user isn't fighting the map re-centering under them while panning/zooming during a trip.

Verified: also used this as an opportunity to test Phase 3 end-to-end — ran scripts/download_map_tiles.py for real with the user's actual MapTiler API key (smoke-tested zoom 8-10 first, confirmed idempotent re-run skips existing tiles, confirmed a downloaded tile is a real map image of the target driving area), then started the full zoom 8-15 download. Took a headless Chromium screenshot of the running app's / page and confirmed: sidebar/icons/fonts render correctly (FontAwesome working), the Speed Graph (Chart.js v4) renders, and the Leaflet map renders real offline map tiles with the marker in place — full offline end-to-end proof, not just "no CDN URLs in the HTML".

• Fixed (was noted here as out of scope, now addressed in Phase 8): /dash and /dtc used to render near-identical body content (both showed "DTC Codes") — dash.html now has real dashcam-specific content.
• Full tile download completed (see Phase 3 update above) and rsynced to the Pi — Phase 3/4's offline map is now 100% functional end-to-end on the actual deployed device, not just the dev machine.

Phase 5: Pi deployment / autostart — ✅ DONE

Set up a proper venv (/home/yaris/app/yaris-obd/venv), installed all requirements (all found as prebuilt ARM wheels via piwheels — no slow source compiles needed, even for gevent). Added deploy/yaris-obd.service (systemd unit: runs as the yaris user, Restart=on-failure, 5s ExecStartPre sleep to let USB/serial devices enumerate after boot, logs to yaris-obd.log) and deploy/install_service.sh (idempotent one-time install helper — also creates the venv if missing). Installed and enabled on the Pi (systemctl enable — survives reboot).

Also had to fix Pi → GitHub auth for git pull: the repo is private, and the Pi's existing ~/.ssh/id_ed25519.pub turned out to be an inbound SSH key (its private half lives on the dev machine, not the Pi) — useless for the Pi to authenticate outbound. Generated a dedicated ed25519 keypair on the Pi itself (~/.ssh/yaris_deploy_key), registered it as a read-only deploy key on the GitHub repo (least-privilege — the Pi only ever needs to pull), and added an ~/.ssh/config entry so git transparently uses it for github.com. Switched the Pi's origin remote to SSH. git pull now works standalone on the Pi with no PAT/interactive auth needed.

Verified end-to-end: pulled latest code onto the Pi (was 5 commits stale), confirmed all Python deps import cleanly, ran the app manually first (HTTP 200, all local static assets — Leaflet/Socket.IO/Chart.js/FontAwesome — served correctly from the Pi itself), then installed + started the systemd service and confirmed active (running) + enabled. Tested crash recovery: sent SIGKILL to the running process, confirmed systemd auto-restarted it (through the 5s ExecStartPre delay) and the app was serving HTTP 200 again within ~10s, with zero manual intervention.

Phase 6: WiFi hotspot mode — ✅ DONE

Used NetworkManager (already the network manager in use, not hostapd/dnsmasq directly — those were leftover/unused) since it natively supports AP mode via nmcli. Created two connection profiles on the Pi: home-wifi (your real home WiFi network, autoconnect-priority 10) and Hotspot (AP mode, ipv4.method shared, its own dedicated SSID/password set locally on the Pi - not committed here, WPA2-PSK, autoconnect-priority -10). NM's own autoconnect logic handles "leaving home WiFi range -> fall back to hotspot" automatically since home-wifi has higher priority and NM always tries it first.

For the reverse direction (home WiFi back in range while sitting in AP mode) added deploy/yaris-netwatch.sh + .service/.timer: since the Pi's onboard BCM43438 radio can't be an AP and background-scan for client networks simultaneously, a systemd timer runs every 3 minutes, briefly attempts reconnecting to home-wifi, and restores the hotspot if home WiFi still isn't in range. Installed via deploy/install_netwatch.sh.

Root-caused and fixed a real bug during testing: the hotspot's SSID broadcast worked immediately, but its DHCP server initially failed (dnsmasq: failed to create listening socket for 10.42.0.1: Address already in use) — a pre-existing orphaned system-wide dnsmasq.service (noted as "likely unused" in earlier investigation) was already bound to the address space NetworkManager's own internal dnsmasq needs for the hotspot. Confirmed the system dnsmasq's config was genuinely empty/default (nothing in this project depends on it) and disabled it (systemctl disable --now dnsmasq).

Also added easy access without needing to know the Pi's IP:

• Renamed the Pi's hostname from yarisobd to yaris — avahi-daemon (already installed and running) now publishes it as yaris.local via mDNS.
• Port 80 -> 5000 redirect via nftables (deploy/nftables-yaris.conf + install_port_redirect.sh, loaded through an include in /etc/nftables.conf), so the app is reachable at plain http://yaris.local/ (no port number) instead of :5000.

Verified on real hardware, very carefully since the SSH session used to do all this testing itself runs over wlan0: used a "dead-man's-switch" pattern for every risky test — a detached background command (setsid nohup bash -c 'sleep N && nmcli con up home-wifi' & disown) armed before each hotspot activation, so the Pi automatically reverts to home WiFi (and SSH access comes back) even if something goes wrong, no physical access needed. Confirmed: hotspot broadcasts and a real phone gets a genuine DHCP lease from NetworkManager's own DHCP server and can load the web UI; yaris-netwatch correctly detects AP mode and reconnects to home WiFi when in range (and is a no-op when already on home WiFi); http://yaris.local/ and http://<pi-lan-ip>/ both return HTTP 200 with no port number from another machine on the network. (Real hotspot SSID/password and the Pi's LAN IP are intentionally not recorded in this file — see local, non-committed notes.)

Update — real-hardware bug found + fixed (hotspot->home reconnect was flaky): while testing the phone-relay feature live (see Phase 11 update below), yaris-netwatch repeatedly logged "home WiFi not found" 4 times in a row over ~13 minutes while sitting right next to the router — a real regression from the original design, not a one-off fluke. Root cause: a plain nmcli con up home-wifi straight out of AP mode relies on NM's scan cache for wlan0, which is stale/empty coming out of AP mode (the radio can't scan while acting as an AP - see above). A single --wait 20 connect attempt gave up before a fresh scan could land. Fixed yaris-netwatch.sh to explicitly drop the AP connection first (releases the radio), force nmcli device wifi rescan, wait for it to settle, then retry the connect (longer 30s wait) up to 2 times before conceding and restoring the hotspot. Bumped yaris-netwatch.service's TimeoutStartSec to 180s (retry loop can take up to ~80s, default systemd 90s start timeout was too tight). Verified fixed on real hardware: both hotspot->home transitions after the fix recovered in ~7 seconds (previously failing outright). Deployed: sudo cp deploy/yaris-netwatch.sh /usr/local/bin/... && cp .service ... && daemon-reload.

Phase 7: app_db.py / app_sql.py fixes — ✅ DONE

Neither file was called from app.py or anywhere else - orphaned from a prior iteration. Fixed:

• app_db.py: applied a prior-session stash fix - resolves .env path via __file__ (was cwd-relative), fixes os.getenv() calls to pass the actual key names (was called with zero args, an immediate TypeError if ever invoked), added a clear RuntimeError guard if credentials are still empty placeholders, returns client[mDB].
• app_sql.py: rewritten - resolves db/sql/ via __file__ + os.makedirs (was cwd-relative), fixed create_conn() closing the connection immediately and returning None (so table creation never ran and every caller got a dead connection), added get_conn() (lazy singleton
  • table creation for both tracking and dtc tables), insert_tracking(), and a background start_logging()/_log_loop() thread (5s interval) that snapshots GPS (app_gps.LocationX/Y)
  • OBD speed (app_obd.speed) into the local tracking table. Used a deferred (function-local) import of app_obd inside _log_loop() to keep the pre-existing app_obd<->app_sql top-level circular import safe/explicit.
• Also fixed a Python 3.12+/3.14 compat bug found during testing: sqlite3.version (the sqlite3 module's version) was removed/deprecated - switched to sqlite3.sqlite_version (the actual SQLite library version), which is what was intended.
• Wired app_sql.start_logging() into app.py alongside the other startup calls (app_gps.start_gps(), app_obd.OBD()).
• db/sql/ added to .gitignore (per-device local trip data, not source).
• Verified locally: get_conn() creates db/sql/<account>_<car>.db with both tables; insert_tracking() writes a real row; start_logging() thread starts and logs "SQL tracking-logger thread started, alive: True"; full app.py run confirms the thread starts cleanly and (with no real GPS/OBD hardware on the dev machine) correctly logs read errors instead of crashing; app_db.get_database() raises the new clear RuntimeError instead of crashing, since db/creds.env still holds empty placeholder credentials (real Mongo credentials are a user-provided follow-up, out of scope here).

Phase 8: Dashcam retrieval (app_dash.py) — ✅ DONE, verified on real hardware

Rewrote app_dash.py to be brand/device-agnostic: detection via lsblk -J, matching partitions whose parent disk reports TRAN == "usb" (this is what safely distinguishes a plugged-in dashcam/USB stick from the Pi's own onboard storage, which is always mmc transport - no more hardcoded /dev/sda). Walks the whole mounted filesystem for video file extensions (.mp4/.mov/.avi/.ts/.mkv) instead of assuming a specific brand's folder layout (DCIM, MOVIE, NORM/EVENT/PARKING, etc. all just work), preserves each file's relative sub-folder path when copying (avoids filename collisions across dashcam event/date folders), and skips files already copied (same relative path + size) so re-running an import is cheap. Mounts read-only to avoid any risk of corrupting the dashcam's SD card. Dropped the unused/uninstalled tqdm CLI-progress dependency (never actually in requirements.txt) since this now runs headless inside the always- on web server, not interactively.

Wired into app.py: GET /api/dash/devices (list detected USB storage) and POST /api/dash/import (run the mount->copy->unmount pipeline), both JSON. Fixed the pre-existing /dash vs /dtc duplicate-template bug — dash.html now has real dashcam content (scan/import buttons + live status) instead of a copy-pasted "DTC Codes" body. Added a dedicated dash.js (not the shared yaris.js, which unconditionally initializes the /index-only map/chart and would throw/crash on /dash and /dtc since those pages don't have those DOM elements - a pre-existing bug, worked around by not depending on that shared file for this page's new JS). Added dash/videos/ to .gitignore (imported footage is per-device data, not source).

Tested thoroughly without the physical dashcam (user didn't have it on hand this session):

1. Unit-tested the lsblk-JSON detection/filtering logic against a realistic synthetic sample (Pi's own mmcblk0 boot+root partitions plus a simulated USB dashcam partition) - confirmed it correctly finds only the USB partition and ignores the Pi's own storage.
2. Integration-tested the real mount -> find-videos -> copy (preserving structure) -> dedupe -> unmount pipeline on the actual Pi using a loop-device-backed FAT32 image (mkfs.vfat + losetup) populated with fake nested dashcam video files (DCIM/100EVENT/..., DCIM/100NORMAL/...) - confirmed real mount/copy/unmount all work, confirmed a second import run correctly skips already-copied files, confirmed non-video files (e.g. thumbs.db) are ignored. All simulation artifacts (loop device, test image, temp files) cleaned up afterward.
3. Verified the full /dash page + both new API routes locally (Flask dev run + headless browser check): auto-scan on page load correctly reports "No USB dashcam/storage device detected" (accurate, since none is plugged into the dev machine), Import button correctly stays disabled until a device is found, no JS console errors, no regression on / or /dtc.

Verified against the real physical dashcam (Jieli-chipset USB ID 1224:2725, 29.7GB SD card, DCIM/1/VID_###.MOV layout, 87 files, ~28GB total). This first live hardware test found and fixed three real bugs the loop-device simulation couldn't have caught:

1. Superfloppy SD card - the card has no partition table at all (lsblk reports the whole /dev/sda disk directly as fstype: vfat, no child partition) - the original detection assumed a partition would always exist. Fixed to also match a TRAN == "usb" disk with its own fstype set directly, not just partitions under it.
2. Mount permission error - mount_device() called os.makedirs() directly, but the service runs as unprivileged yaris and /mnt is root-owned. Fixed to sudo mkdir -p <mount_point> before the existing sudo mount, mirroring that pattern.
3. OOM crash during upload (see Phase 10 below) - not a copy-path bug, but discovered immediately after the first successful real import.

End-to-end result: full 28GB/87-file import from the SD card completed successfully (resumed cleanly after being interrupted once by the OOM crash below - copy_files()'s skip-if-already- copied logic meant re-running /api/dash/import just picked up the remaining files, no data loss or re-copy of the first 19 files). SD card unmounted cleanly afterward.

Phase 9: Remote server for itinerary/dashcam upload — ✅ DONE

Per architecture decisions #4 and #5: self-hosted server/API to receive itinerary (from the SQLite pipeline) and dashcam file uploads.

Decided/built: Flask API + dedicated Postgres, deployed as a docker-compose stack on the user's unraid server (separate from any existing containers/DBs there), living in server/ in this same repo. Auth: simple X-Device-Name/X-Api-Key header pair per device, key hash stored server-side, real key never committed (raw key printed once by server/scripts/create_device.py, saved locally on the Pi only, in gitignored db/upload_creds.env). External access: exposed at https://yaris.ceaf.dev via the user's existing Cloudflare Tunnel (public-hostname rule added in the Cloudflare dashboard, pointing at 192.168.10.10:8098) — router port-forward + the user's Unifi firewall is the documented fallback if ever needed instead. Large dashcam uploads can still prefer a local-network path (e.g. an AdGuard DNS override to bypass the tunnel while on home WiFi) since tunnels typically cap per-request upload size — noted in server/README.md, not yet needed/built.

Built (server/ folder):

• server/sql/init.sql — Postgres schema: devices (name + sha256 api-key-hash), tracking_points (device_id FK, GPS/speed/odometer + recorded_at, unique per device+timestamp so retried batches can't duplicate rows), dashcam_files (device_id FK, relative_path + sha256, unique per device+path+hash so re-uploads are no-ops).
• server/modules/db.py — psycopg3 connection pool + query helpers (bulk tracking insert with ON CONFLICT DO NOTHING, dashcam dedupe-check-before-write).
• server/modules/auth.py — require_device_auth decorator checking the two headers against the stored key hash.
• server/app.py — Flask app: GET /health (unauthenticated), POST /api/v1/tracking/batch (bulk JSON insert, capped at 5000 points/request as a sanity guard), POST /api/v1/dashcam/upload (multipart, streamed to disk with sha256 verification + path-traversal guard on relative_path).
• server/Dockerfile + server/docker-compose.yml — api (gunicorn, published on host port 8098 by default) + db (Postgres 17, internal-only, not published to the LAN) services, data volume defaults to server/data/ but pointed at /mnt/user/appdata/yaris-obd-server on unraid.
• server/.env.example — template for POSTGRES_PASSWORD/API_PORT/DATA_DIR/upload size cap.
• server/scripts/create_device.py — run once per device inside the api container to register it and print its one-time API key. Ran once for device yaris-2013.
• server/README.md — full unraid deployment + Cloudflare Tunnel exposure + device-registration instructions.

Pi-side upload client (modules/app_upload.py):

• Background thread, uploads unsent tracking rows (batches of 500) and not-yet-uploaded dashcam files (sha256-verified) via the API above, with exponential retry/backoff on failure.
• Independent dashcam-upload-only pause toggle (persisted settings table + a switch on the /dash web UI) — trip/GPS logs always keep uploading regardless of the toggle, per explicit user request (useful on a slow/metered connection).
• Backlog-drain tuning: since tracking rows log every 5s locally, an offline/stale device can queue up a large backlog. The loop drains back-to-back (short 1s pause between batches) while a batch comes back full, only settling into the slow 30s idle cadence once actually caught up — a simulated 1250-row backlog now drains in 3 batches (~0.35s) instead of 13 batches over ~6.5 minutes with the original fixed 100-row/30s-sleep design.
• modules/app_sql.py extended: UPLOADED column + migration on tracking, settings table, dashcam_uploads dedupe table, and helper functions backing all of the above.

Tested end-to-end against the real deployed server (tracking batch + dedupe, dashcam upload + dedupe + path-traversal rejection, bad-API-key rejection, toggle-off correctly pausing only dashcam uploads) and on the real Pi (its actual ~1800-row pre-feature backlog drained and confirmed matching row counts on the server via direct psql query, live upload status visible on /dash). Committed+pushed (4ff99f0, c4db1ab, 8dc304e), deployed on both unraid and the Pi. Exposed externally at https://yaris.ceaf.dev via the user's Cloudflare Tunnel (public hostname added in the Cloudflare dashboard, pointing at 192.168.10.10:8098).

Phase 10: Dashcam upload memory/size fixes — ✅ DONE, verified on real hardware

Two real-hardware-only bugs surfaced once actual ~300MB+ video files (not small test fixtures) hit the upload path for the first time:

1. OOM crash from in-memory multipart uploads. modules/app_upload.py originally uploaded dashcam files via requests.post(..., files={"file": f}). The requests library builds the entire multipart/form-data body in memory before sending when using files= - a well-known gotcha, not specific to this codebase. On the Pi 3's ~900MB RAM, uploading one ~320MB video this way spiked the process past 600MB+ RSS, triggering the kernel OOM killer within minutes (confirmed via dmesg: repeated Out of memory: Killed process ... anon-rss:823928kB, and via pmap -x showing a single ~362MB anonymous mapping matching the file size almost exactly). Fixed: added requests-toolbelt and rewrote the upload body construction to use MultipartEncoder, which streams the file in chunks as the request is sent instead of materializing it up front. Confirmed via an isolated 300MB test (628MB RSS old way -> 29MB new way, byte-identical result) and live on the Pi (RSS stayed flat ~55-59MB during real uploads, vs. previously crashing every few minutes).
2. Cloudflare Tunnel's hard ~100MB request-body cap. Even after the memory fix, real uploads still failed once routed through https://yaris.ceaf.dev - a bare 413 Payload Too Large from Cloudflare itself (confirmed: the identical file succeeds direct-LAN, 413s only through the tunnel). This is a free-tier platform limit, unrelated to MAX_DASHCAM_UPLOAD_MB on the Flask side. Fixed via a chunked/resumable upload redesign (well under the 100MB cap):
   • Client (modules/app_upload.py): DASHCAM_CHUNK_SIZE = 40MB; new _upload_video_chunked() checks /status first (enables resume across restarts/failures), then POSTs missing chunks one at a time via MultipartEncoder.
   • Server (server/app.py): new GET /api/v1/dashcam/upload/status (reports already_uploaded or received_chunk_indices) and POST /api/v1/dashcam/upload/chunk (stages chunks under STORAGE_DIR/.upload_tmp/<device>/<sha256>/, reassembles + verifies SHA256 once all chunks arrive, records via the existing db.insert_dashcam_file, cleans up the staging dir). A .total_chunks marker guards against stale/mismatched partial sets if the chunk size ever changes between client versions. The original single-shot /api/v1/dashcam/upload endpoint is untouched, kept for reference/compatibility.

Verified end-to-end against the real deployed server through the real Cloudflare tunnel, first with a synthetic interrupted-then-resumed 250MB test (status transitions not_started -> partial upload -> in_progress w/ correct received_chunk_indices -> resume -> complete -> already_uploaded, SHA256 match confirmed on disk), then with real dashcam footage: the Pi copied all 87 real files locally and the background upload thread drained the real backlog one file at a time, each one verified byte-for-byte (SHA256 match Pi vs. server) and correctly recorded in Postgres, while the Pi's service ran 50+ minutes straight with RSS staying in a bounded ~55-141MB range (no OOM kills) - a stark contrast to the previous crash-every-few-minutes behavior. Given home-upload-bandwidth constraints, the full 28GB backlog drains at roughly one ~350MB file per 4-5 minutes and is expected to take several hours total - this runs unattended in the background with no user action needed once started.

Not yet built: parallel/concurrent chunk uploads (currently strictly sequential, one chunk then one file at a time) - would speed up the multi-hour full-backlog drain but adds complexity; not requested by the user yet, flagged here as a possible future optimization if the sequential pace becomes annoying in practice.

Known manual step (not yet automated): this dashcam model boots into a "select mode" menu (dashcam / mass-storage / webcam) when it senses USB data lines are live, and needs USB data communication physically absent to boot straight into recording mode, but present when plugging into the Pi for import. No code in this repo handles that switch - it's a manual physical step by the user (e.g. a data-blocking USB cable/adapter, or plugging/unplugging appropriately) each time before recording vs. before importing. Worth revisiting if it becomes tedious in daily use (e.g. a Pi-controlled USB data-line switch), but out of scope for now.

SIM868 cellular fallback for tracking uploads — ⚠️ built, UNTESTED (no SIM card available)

modules/app_cellular.py gives the tracking-upload path a fallback over the SIM868's cellular data connection when WiFi/Ethernet is unreachable. Design notes:

• Uses the SIM868's built-in AT-command HTTP(S) client (AT+HTTPINIT/HTTPPARA/HTTPACTION/ HTTPREAD) rather than a real ppp0 network interface. Reason: the SIM868's GNSS (polled continuously by modules/app_gps.py) and its GSM/GPRS radio share the same physical UART (/dev/serial0) — a classic pppd/chat dial-up would put that port into raw PPP framing for as long as the link is up, starving the GNSS polling thread. Staying in AT-command mode lets cellular HTTP calls interleave safely through the same serial lock (modules.app_gps._serial_lock) — each exchange just briefly delays the next GPS poll.
• Only carries small JSON tracking batches, never dashcam video — cellular data is far too slow/limited for multi-MB video files, and the modem's HTTP AT interface has a small internal buffer unsuited to them anyway. Dashcam uploads remain WiFi/Ethernet-only, unchanged.
• modules/app_upload.py's tracking-batch upload always tries WiFi/Ethernet (requests) first; only on a connection-level failure (no route/DNS/timeout — as opposed to an HTTP error status, which means the server WAS reached) does it retry the same batch over cellular.
• Feature-flagged off by default: YARIS_CELLULAR_ENABLED/YARIS_CELLULAR_APN/ YARIS_CELLULAR_APN_USER/YARIS_CELLULAR_APN_PASS added to db/upload_creds.env, all blank by default so this has zero effect on the already-verified WiFi upload path until the user fills in an APN once they get a SIM card. Regression-tested (against the real server) that normal WiFi uploads are completely unaffected while cellular is unconfigured.
• /dash upload-status card now shows which path (wifi/cellular) the last successful batch used, and whether cellular fallback is configured.
• Needs validation once a SIM card is available — exact AT command dialect/timing (SIMCOM firmware revisions vary slightly), and whether AT+HTTPSSL works for the HTTPS yaris.ceaf.dev endpoint on this specific module's firmware (if not, a documented fallback is a plain-HTTP router port-forward path, avoiding the need for the modem's TLS stack to validate a cert chain).

Phase 11: Phone-relay dashcam upload — ⛔ REMOVED (see update below - hit a hard platform limitation)

Root cause investigation into "why is the transfer from the Pi to the server so slow?" (real dashcam backlog draining at only ~350MB/4-5min) found the bottleneck is the Pi's own onboard BCM43438 WiFi chip (2.4GHz-only, congested/interfered channel) — confirmed via iperf3 raw-TCP tests showing only ~14.7Mbit/s with active retransmits even over a same-subnet direct-LAN hop that never touches the internet/Cloudflare, ruling out ISP/Cloudflare limits. Since the Pi 3 B has no 5GHz radio and can't be fixed in software, added a way to bypass its weak WiFi for the internet-bound leg entirely: a phone opens the same /dash page (over the Pi's hotspot/home WiFi) and manually relays pending dashcam files straight to the remote server using the phone's own network (WiFi or cellular data), instead of routing everything through the Pi.

Zero new remote-server upload logic — the phone's browser JS calls the exact same /api/v1/dashcam/upload/status + /api/v1/dashcam/upload/chunk endpoints the Pi's own Python client already uses, in the same 40MB-chunk protocol, just orchestrated by fetch()/FormData instead of requests/MultipartEncoder. The server dedupes purely by SHA256 regardless of which device sends the bytes, so no server-side changes were needed for the upload path itself — only a CORS after_request hook (server/app.py) was added, since the phone's browser calls yaris.ceaf.dev from a page served by a different origin (the Pi), which browsers block without Access-Control-Allow-* headers on both the real request and its preflight OPTIONS. Wildcard origin is fine here — auth is header-based (X-Device-Name/X-Api-Key), not cookie/session-based.

New Pi-side endpoints (app.py):

• GET /api/dash/pending — lists not-yet-uploaded dashcam files (relative_path, sha256, size) via a new app_upload.list_pending_dashcam_files().
• GET /api/dash/file/<path:relpath> — serves a dashcam file's bytes for the phone to download, via send_file(..., conditional=True) for Range-request support (206 Partial Content) so the phone downloads (and immediately re-uploads) one 40MB slice at a time instead of ever holding a whole 300+MB file in the browser's memory — the exact same memory-safety lesson learned the hard way in Phase 10, applied symmetrically to the download side. Path-traversal safe via Werkzeug's safe_join against app_dash.DEFAULT_DESTINATION.
• GET /api/dash/relay/config — hands the phone the same SERVER_URL/DEVICE_NAME/API_KEY the Pi's own client uses, so its JS can auth directly with the remote server. Same trust boundary as the rest of this project's unauthenticated local UI (anyone on the Pi's own hotspot/LAN is already implicitly trusted).
• POST /api/dash/relay/heartbeat / POST /api/dash/relay/stop — a lease-based pause (RELAY_LEASE_SECONDS = 45) for the Pi's own background dashcam uploader, chosen over a plain on/off toggle specifically to self-heal if the phone's browser tab closes/loses connection uncleanly (a simple toggle could get stuck "paused" forever). The phone page heartbeats every 15s while relaying; _upload_dashcam_files() skips entirely while a lease is active, re-checked between files so it stops promptly if a relay starts mid-drain. Also released via navigator.sendBeacon() on beforeunload as a best-effort immediate release.
• POST /api/dash/relay/mark_uploaded — lets the phone tell the Pi "I just uploaded this file" right after a successful relay, so the local dedupe table (and the pending list / Pi's own background loop) reflect it immediately instead of waiting for the next full backlog scan.

Performance side-fix: modules/app_upload.py's dashcam loop previously recomputed SHA256 for every candidate file on every loop iteration, including files long since uploaded — wasteful CPU/IO against a growing, possibly 20GB+, local backlog. Added a dashcam_hash_cache SQLite table (modules/app_sql.py, keyed by relative_path+size+mtime) so a file's hash is only computed once; both the existing background uploader and the new pending-files endpoint benefit.

UI (templates/dash.html / static/js/dash.js): a new "Relay via phone" card on /dash lists pending files with a per-file "Relay" button and a "Relay all pending" button (manual trigger only — chosen deliberately over an automatic one, per explicit user preference), plus a "Stop relay" button. Chunk downloads/uploads retry up to 3 times with backoff before surfacing an error, and a resumed relay re-checks /status first so a previously-interrupted file picks up from its first missing chunk instead of restarting.

Tested: a full local end-to-end simulation (no physical phone/network needed) drove the real Pi app.py and real server/app.py Flask apps side by side (Postgres mocked at the db module boundary only — all HTTP routing, chunking, auth, and CORS logic is real), reproducing the exact sequence the phone's browser JS performs: list pending -> check remote /status -> download each chunk from the Pi via Range requests (verified real 206 Partial Content responses) -> upload each chunk to the mock-backed server -> verify the reassembled file matches the original byte-for-byte via SHA256 -> mark uploaded on the Pi -> confirm it drops out of the pending list. Also verified: path-traversal attempts on /api/dash/file/ correctly 404, the relay heartbeat/stop lease correctly pauses/resumes _upload_dashcam_files(), and the sha256 cache table is populated and reused across repeated /api/dash/pending calls.

Deploy gotcha found: the CORS fix in server/app.py was initially deployed to the Pi but not rebuilt on the actual remote server (the server's Docker image bakes app.py/modules/ in at build time via COPY, so a plain git pull + container restart isn't enough — needs docker compose build api && docker compose up -d api). Without it, the phone's cross-origin fetch() calls to yaris.ceaf.dev were silently blocked by the browser's CORS policy, which surfaces in JS as a generic, unhelpful TypeError: Failed to fetch — worth remembering as the first thing to check if this error reappears after a future server-side change.

Tested for real against a real phone and the real deployed server (first end-to-end proof beyond the local simulation above): the initial /api/dash/pending call took ~25 minutes for a one-time full-backlog SHA256 hash of all 87 files (~28GB, no cache existed yet) — after that, repeat calls returned in ~0.13s. The user then successfully relayed a real file from their phone (home WiFi) partway through (10/~12 chunks) before stopping to check something important: is the relay actually faster, given the phone downloads from the Pi over the same congested radio diagnosed as the bottleneck? Measured directly from chunk-arrival timestamps on the server: ~1.11MB/s (~8.9Mbit/s) for the phone-relay path — statistically the same as the ~9-10Mbit/s measured for the Pi's own direct Cloudflare upload earlier this session. Conclusion: when the phone is on the same home WiFi as the Pi, the relay provides no speed benefit — the download-to-phone leg pays the exact same slow-radio cost as a direct Pi upload would, and the phone's reupload leg then goes out through the same home internet connection anyway, adding an extra hop for no gain. The relay only has genuine value when the phone's second leg differs meaningfully from the Pi's own path — e.g. cellular data with better upload than the home ISP, or uploading after physically leaving home with files already grabbed locally. Stopped the test relay (POST /api/dash/relay/stop) and confirmed the Pi's own background uploader resumed the same file's chunked upload exactly where the phone left off (shared /status/.upload_tmp state on the server made this seamless, no wasted re-transfer).

Next idea (not yet built): the user wants a way to browse/preview individual dashcam clips on the phone and select specific ones to view or upload (as opposed to bulk-relaying the entire backlog) — useful for "just grab this one important clip" rather than a wholesale speed optimization. Not scoped/designed yet.

Update — tested the "away from home" hotspot+cellular scenario for real, found a real phone-OS limitation (not a code bug): the user asked to test the specific scenario the relay feature was actually designed for — phone connects to the Pi's own hotspot (not home WiFi) to download pending files, then re-uploads using the phone's cellular data, bypassing the Pi's own (in hotspot mode, nonexistent) internet connection entirely. Confirmed via code review this needs zero changes: /api/dash/relay/config already hands the phone the public https://yaris.ceaf.dev SERVER_URL (not a LAN-only address), so the re-upload leg was always going to be internet-bound regardless of which physical network carries it.

Live-tested with a real dummy file (dashcam upload toggle paused first so the Pi's own uploader couldn't grab it before the phone did): got TypeError: Failed to fetch trying to relay. Confirmed via a direct test (loading https://yaris.ceaf.dev/health as a plain page navigation, not a fetch(), ruling out CORS) that the phone's own OS was declining to route internet-bound traffic over cellular while still associated with the Pi's no-internet-gateway hotspot WiFi — it only succeeded once the phone auto-fell-back to the home WiFi network (which does have internet) instead. This is a phone/OS-level "smart network switching" behavior (common on both iOS and Android to avoid using a WiFi network with no internet gateway) that cannot be fixed in this project's code — the architecture is sound and was already proven correct in the local end-to-end simulation and the byte-for-byte reassembly tests earlier in this phase.

Practical implication: testing this exact scenario at home is inherently confounded — the phone always has a known-good home WiFi it can silently fall back to, defeating the test. A true validation needs either (a) actually being away from home with no known WiFi in range, forcing the phone to use cellular for internet traffic, or (b) manually disabling the phone's WiFi (or its "switch to mobile data automatically" setting) while connected to the Pi's hotspot, to force cellular use for this one operation.

Update — retested more rigorously, confirmed this is a fundamental architecture-level dead end, not just an untested corner case. Re-ran the same live test multiple times with longer/managed "dead-man's-switch" windows (yaris-netwatch.timer explicitly stopped during each test run so it wouldn't yank the Pi back to home WiFi mid-test — an earlier run's failure was actually caused by netwatch reconnecting home WiFi under the phone, see Phase 6 update, not a phone-network issue). With netwatch correctly disabled and the Pi confirmed to stay in AP mode for the whole test window, still got Failed to fetch. Directly tested a plain page navigation (not fetch()) to https://yaris.ceaf.dev/health with WiFi fully turned off (pure cellular) — that also failed, while every other website loaded fine over the same cellular connection with WiFi off. This narrows the cause precisely: the browser/phone genuinely cannot reach the public internet at all while wlan0/the phone's WiFi radio is associated with the Pi's hotspot, full stop — this is not a settings toggle problem (checked Pixel-specific "switch to mobile data automatically" options, not present on this device/Android version) but a fundamental OS-level "default network" routing decision: a device's default route for all IP traffic (including a web page's fetch() calls) is a single, whole-device choice made by the OS's connectivity manager, not something a browser page's JavaScript can override per-request. Real per-request network binding (e.g. Android's ConnectivityManager.bindProcessToNetwork()) is an API only available to native apps, not exposed to web pages at all, in any browser, on any phone. Given this, the phone-relay feature as designed (a web page doing simultaneous WiFi-to-Pi + cellular-to-server transfers) cannot ever work while the phone is connected to the Pi's no-internet hotspot — this isn't a bug to fix, it's a hard platform ceiling.

Decision: removed the phone-relay feature entirely (was Phase 11, now reverted - see below). Two ways to actually deliver on the original "upload from the road without the Pi's slow WiFi" goal were discussed: (1) restructure as two distinct, non-simultaneous phases (fully download to the phone's local storage first while on the Pi's hotspot, then have the user manually switch to cellular, then upload from local storage) - no native app needed but a more awkward two-step user flow, or (2) a real native Android app, which can hold both networks open at once via bindProcessToNetwork() for a true simultaneous relay. Neither has been built - shelved for now in favor of simplifying the dashcam retrieval UI (see the new phases below). The Pi's own background uploader (plain WiFi/Ethernet, optionally SIM868 cellular fallback for tracking data only) remains the only upload path; it's just slower when the Pi's own WiFi is weak, with no workaround currently in place for that specific case.

Side-fix from this test session: found and fixed a real, unrelated bug in yaris-netwatch.sh during this same test — see the Phase 6 update above.

Phase 12: Remove phone-relay feature — ✅ DONE

Undo Phase 11 cleanly given the Phase 11 update above (fundamental browser platform limitation, not fixable). Removed:

• app.py: /api/dash/pending, /api/dash/file/<path>, /api/dash/relay/config, /api/dash/relay/heartbeat, /api/dash/relay/stop, /api/dash/relay/mark_uploaded routes (and the now-unused os/send_file/abort/safe_join imports they needed).
• modules/app_upload.py: list_pending_dashcam_files(), relay_heartbeat()/relay_stop()/ is_relay_active(), RELAY_LEASE_SECONDS/_relay_lease_until, and the is_relay_active() check inside _upload_dashcam_files()/get_status().
• templates/dash.html: the "Relay via phone" card.
• static/js/dash.js: all relay JS (relayOneFile, fetchChunkWithRetry, uploadChunkWithRetry, heartbeat timers, pending-file list rendering, etc).
• server/app.py: the CORS after_request hook, added solely for the phone's cross-origin browser calls (the Pi's own Python requests-based client was never subject to CORS - that's a browser-only enforcement mechanism).
• Kept untouched, as planned: the chunked upload protocol itself (/api/v1/dashcam/upload/chunk + /status), dashcam_hash_cache table + _hash_dashcam_file() (still used by the Pi's own background uploader to avoid rehashing its 20GB+ backlog every loop), DASHCAM_CHUNK_SIZE/Cloudflare 100MB workaround - none of that is relay-specific.

Verified: python3 -m py_compile on all touched Python files, node --check on dash.js, grepped all five touched files for any leftover relay/RELAY reference (none found). Smoke- tested by actually importing app.py in a scratch venv and hitting it with Flask's test client: app imports cleanly (GPS/OBD/SQL/upload startup all log their normal "no hardware" messages, no crash), app.url_map confirms every relay route is gone while /, /dash, /dtc, and the remaining dashcam/upload API routes are still registered, and both / and /dash render with HTTP 200 (/dash's HTML contains zero occurrences of "relay").

Phase 13: Power-loss resilience & watchdog — 🚧 PLANNED

New phase, not code-dependent on anything else — OS/hardware-level, can be done anytime, ideally before the Pi lives in the car full-time. A car cuts power abruptly at every key-off; nothing in the project currently protects against that, and repeated unclean power loss is a classic cause of SD card corruption on embedded Pi projects. Two complementary angles:

• Filesystem hardening: convert the Pi's root filesystem to read-only with a writable overlay (overlayroot/raspi-config's built-in overlay FS option), so an abrupt power cut can no longer corrupt the OS itself. App data that must persist (db/sql/, dash/videos/, logs) needs to live on an explicitly writable partition/bind-mount carved out of the overlay, since those still need real writes.
• Clean shutdown before power loss: either (a) a small delayed-shutdown power module/circuit (holds 5V for a few seconds after ignition-off so the Pi can run shutdown -h now cleanly), or (b) if the SIM868 HAT or another ADC/GPIO source can sense ignition/voltage state, a software voltage-drop watcher that triggers a clean shutdown proactively. Exact approach TBD - depends on what's easiest to wire into the existing power setup.
• Hardware watchdog: enable the Pi's built-in bcm2835_wdt watchdog (via systemd's RuntimeWatchdogSec/watchdog-device support) so a fully hung app or kernel gets a hard reboot automatically instead of sitting dead in the car until someone notices.

Phase 14: UI design-system foundation — 🚧 PLANNED

New phase, pulled forward from the old "do the whole redesign at the end" plan. Building the Phase 15/16 file browsers (and later the trip-history/Pi-health/settings pages) against the current old admin-template style just means rebuilding the same UI twice once the full redesign (Phase 22) happens. Instead, do a small, scoped slice of the redesign right now: a shared page layout/shell plus one reusable list/table component (sortable columns, per-row action buttons, delete-with-confirmation pattern, empty-state messaging) that every subsequent list-based feature in this plan (dashcam browser, video browser, trip history, settings) can be built on directly, instead of writing bespoke list/delete UI code from scratch each time. Not a full redesign of every existing page yet - just establishes the pattern once so nothing built after this point needs to be redone in Phase 22.

Phase 15: Dashcam USB file browser (list + delete) — 🚧 PLANNED

Replace the current all-or-nothing "Import videos" flow's blind spot (no visibility into what's actually on the camera) with a real file browser for the mounted USB dashcam device, built on the Phase 14 shared list component:

• New endpoint(s) in modules/app_dash.py to list every video file on the mounted dashcam (relative path, size, modified time) without copying anything - reuses the existing find_video_files() walk, pointed at the dashcam's mount point instead of the local destination folder.
• Delete support: individual file delete and a "delete all" bulk action, each requiring the mount to be briefly remounted read-write (currently always ro) - only for the duration of the delete call, then back to read-only (or unmounted) afterward, to minimize risk of filesystem corruption from an unclean power loss while writable.
• UI: a /dash card showing total file count + total size on the dashcam, a scrollable file list with a delete button per row, a "Delete all" button behind a confirmation dialog (real data loss risk - no undo).
• Explicitly deferred/out of scope: no in-browser preview/playback of files still on the dashcam itself (only after import - see Phase 16). No changes to the USB select-mode/data-communication toggle behavior - the user replugs power manually to return the camera to recording mode, as they already do.

Phase 16: Local imported-video browser + player — 🚧 PLANNED

Same file-browser treatment for videos already imported to the Pi's own disk (dash/videos/, populated by import_dashcam()), built on the Phase 14 shared list component:

• List endpoint (reuses find_video_files() against app_dash.DEFAULT_DESTINATION) showing relative path, size, modified time, and upload status (cross-check against app_sql.dashcam_file_marked_uploaded() - so it's visible at a glance which imported clips have already reached the remote server and which are still pending the Pi's own background uploader).
• Delete support: local Pi copy only - remote server copies are intentionally left alone as the safe backup/archive (explicit user decision - see 2026-08-09 discussion). No "also delete from server" option is being built for now.
• In-browser video player: since the dashcam records MP4 (confirmed with the user), this is a plain HTML5 <video> element - no server-side transcoding needed. Reuses the same Range-request-capable file-serving pattern already proven in the (now-removed) relay's /api/dash/file/<path> endpoint (needed for correct seeking/scrubbing, not just playback).
• UI: a /dash card (or a dedicated sub-page if the list gets long) with a file list, a "Play" action that opens the video inline, and a delete button per row with a confirmation prompt.
• QOL: video thumbnails. Scrolling a list of VID_00234.MOV-style filenames to find a specific clip is rough. Grab one low-res frame per file via ffmpeg (cheap, done once per file and cached, not on every page load) so the browser shows an actual thumbnail per row instead of just a filename.
• QOL: retention/auto-cleanup policy. Once a file is confirmed uploaded to the remote server, it's safe to free local space. Add an optional automatic cleanup (configurable via the Phase 21 Settings page - e.g. "delete local copies older than N days once uploaded", off by default) so the Pi's limited storage doesn't need to be babysat manually as the dashcam backlog grows.

Phase 17: OBD real-hardware readiness (code hardening) — 🚧 PLANNED

Independent of everything else in this plan — only touches app_obd.py/app_events.py, no UI dependency for the backend part, so this can be done anytime/in parallel with the phases above. Nothing has been installed or tested in the actual 2013 Yaris yet - modules/app_obd.py has only ever been tested standalone on the Pi with nothing plugged into the OBD-II port (correctly reports "no connection to car"). Per explicit user decision, this phase is code hardening only - the user will handle the physical install/bring-up testing themselves. Planned fixes, all in modules/app_obd.py / modules/app_events.py:

• Add a genuine background reconnect loop. Today, connect_obd()/obd_connection() only ever run once per browser tab's socket connect event (app_events.py's connect_event()) - there is no periodic retry. In the real car, this means: if the page happens to load before the adapter/ignition is ready, the connection attempt fails once and never retries until another browser tab connects. Add a small polling loop (mirroring the pattern already proven in app_gps.py's _poll_loop()) that periodically calls connect_obd() while disconnected, independent of any browser being open.
• Catch exceptions from connection.query(). Currently a communication hiccup (adapter timeout, malformed response, USB blip) would raise inside obd_speed()/obd_rpm() uncaught, silently killing that polling thread forever with no recovery - not just returning a null response, an actual crash. Wrap query calls, treat exceptions like a lost connection (stop polling, let the reconnect loop above pick it back up) instead of dying silently.
• Add an OBD_PROTOCOL env override, mirroring the existing OBD_PORT override pattern - in case the 2013 Yaris's ECU doesn't cleanly auto-negotiate via ELM327's ATSP0 (auto) and needs an explicit protocol code forced (Toyotas this age are normally ISO 15765-4 CAN 11-bit 500kbaud, but this has never been confirmed against this specific car).
• Improve connection-failure diagnostics. Distinguish and log clearly: no adapter detected at all (USB/port issue) vs adapter detected but ELM327 handshake failed vs handshake OK but no response from the car's ECU (car off/ignition not on) - currently all three collapse into the same generic "OBD connection failed" message, which will make first-real-car troubleshooting much harder than it needs to be.
• Add a small connect-time diagnostic log (ELM327 firmware version + negotiated protocol, both available from python-obd's connection object) written once per successful connect - useful evidence to sanity-check against Toyota's expected protocol on the very first real test.
• Expand the data pulled from the car beyond speed/RPM (backend only - the dashboard cards displaying this land in Phase 22 once the redesign reaches them): coolant temperature, fuel level, throttle position, and DTC/MIL status (check-engine light on/off + stored trouble codes, reusing/wiring up the already-scaffolded but currently-empty /dtc page and GET_DTC command). Odometer/mileage is not guaranteed - the standard OBD-II mode 1 PID for total odometer (0xA6) was only added to the SAE J1979 spec in later revisions and is inconsistently supported across manufacturers/years; whether the 2013 Yaris's ECU exposes it at all can only be confirmed once tested against the real car (falls under "code hardening only" for now - add the query with graceful "unsupported" handling rather than assuming it will work).

Phase 18: Trip history & replay — 🚧 PLANNED

New phase. Right now the tracking table (Phase 7) is just a raw stream of GPS/speed points with no concept of a discrete "trip" - there's no way to look back and see "trip #47, left home 8:03am, arrived work 8:31am, 22km." Add:

• Trip boundary detection: group existing tracking rows into discrete trips based on speed>0 gaps (a simple, dependency-free heuristic - e.g. a gap of N minutes with no motion ends a trip), computed either live as rows are logged or in a batch pass over existing data. Does not need to wait on Phase 17's OBD hardening - can ship with GPS-only boundary detection first, refine with ignition-on/off signals from OBD later if useful.
• A trips table (start/end time, start/end coordinates, distance, duration) in modules/app_sql.py.
• A new history page/list (built on the Phase 14 shared list component) showing past trips, and a map replay view (reusing the existing offline-tile Leaflet setup from Phase 3/4) that draws the recorded route for a selected trip.

Phase 19: Pi health monitoring card — 🚧 PLANNED

New phase, small and standalone - no dependencies. CPU temperature, free disk space, uptime, and RAM usage are all cheap to read (vcgencmd measure_temp, psutil//proc stats) and cheap to display, but currently invisible - the only way to notice a problem (thermal throttling in a hot car, an SD card filling up) today is to SSH in and check manually. Add a small backend endpoint and a dashboard card (built on the Phase 14 component set) surfacing these at a glance. Also feeds directly into Phase 20's alerting.

Phase 20: Alerts/notifications — 🚧 PLANNED

New phase - depends on Phases 16/17/19 existing first, since it needs real data sources to alert on. Right now, finding out about a problem (check-engine light on, dashcam SD card nearly full, upload backlog growing unbounded, Pi overheating, GPS fix lost for a long stretch) requires manually opening the web UI. Add a lightweight push-notification integration - no new infrastructure needed, just an outbound webhook to something like ntfy.sh or a Telegram bot - fired on a handful of conditions:

• DTC/MIL status changes to "on" (from Phase 17's expanded OBD data).
• Dashcam USB storage or local dash/videos/ disk usage crossing a configurable threshold.
• Upload backlog (pending tracking rows or dashcam files) growing past a configurable size, suggesting the Pi's been offline/uploading too slowly for a while.
• Pi CPU temperature crossing a configurable threshold (from Phase 19).
• GPS fix lost for longer than a configurable duration while the car appears to be moving (OBD speed > 0 with no fix - needs Phase 17's data). Webhook target/thresholds configurable via the Phase 21 Settings page rather than hardcoded.

Phase 21: Settings page — 🚧 PLANNED

New phase - depends on Phase 14's shared components, and benefits from Phases 16/20 existing so there's real configuration to consolidate. Several settings currently only exist as SSH-edited files or scattered individual toggles (dashcam upload pause switch on /dash, cellular APN in db/upload_creds.env, hotspot SSID/password via nmcli directly) or don't exist yet (retention days from Phase 16, alert webhook/thresholds from Phase 20). Add one dedicated Settings page consolidating anything reasonably safe to expose over the local-only web UI: upload toggle, cellular APN, local video retention policy, alert webhook URL + thresholds. WiFi SSID/password changes likely stay out of scope (higher-risk to expose over a UI that could lock the Pi out of its own network if fat-fingered) unless it turns out to be easy/safe to add later.

Phase 22: Finish web UI redesign — 🚧 PLANNED

The current templates//static/ UI is an old, mostly-unmodified admin dashboard template (SB Admin 2-style Bootstrap theme) with leftover placeholder cruft never replaced with real content - e.g. index.html's "Placeholder button", dropdown menus full of "Action" / "Another action" / "Something else here" stub links, and a /dtc page that's just an empty "Output" card with nothing wired up to it at all. Phase 14 already pulled the foundational piece of this work forward (shared layout/list component, used by every phase since); this phase is the remaining cleanup once all the functional phases above have landed:

• Replace/remove all remaining placeholder buttons, dropdown menus, and stub links that don't do anything.
• Wire up the actually-empty /dtc page to show real DTC/MIL data from Phase 17.
• Add the new dashboard cards for Phase 17's expanded OBD data (coolant temp, fuel level, throttle) and Phase 19's Pi health card, using the Phase 14 component patterns.
• Final visual polish pass across every page now that the full feature set (dashcam/video browsers, trip history, settings, alerts) exists, rather than redesigning against a moving target.
• Not yet scoped: whether to keep Bootstrap 5 + the current theme framework and just clean it up, or replace the frontend stack entirely - decide once everything above has landed.

Phase 23: Native Android companion app — 🚧 PLANNED (low priority, do last)

Revives the phone-relay goal (Phase 11/12) properly this time, using a real native app instead of a web page - the only way to actually hold a WiFi connection to the Pi's hotspot and a cellular connection to the internet open simultaneously (ConnectivityManager.bindProcessToNetwork(), unavailable to browser JS - see the Phase 11 update). Explicitly not urgent - deprioritized to the end of the roadmap, after all web-side work (Phases 12-22) is done. Scope, at a high level (not yet designed in detail):

• Dashcam file transfer: browse/select/delete dashcam clips already imported to the Pi (same data as the Phase 16 local video browser, just a native client instead of/alongside the web UI), download them to the phone, and upload to the remote server over cellular while the phone stays on the Pi's hotspot for the download leg - the true simultaneous dual-network relay the web version could never deliver.
• Trip/location info: surface live GPS position, speed, and trip history (Phase 18) pulled from the Pi while connected to its hotspot (reusing the existing /api/gps-style data feeds), so the app is useful as a real "check on the car" companion, not just a file-transfer tool.
• Not yet scoped: distribution (sideload vs Play Store), whether it talks to the Pi directly over the hotspot only or also to the remote server directly for trip history when away from the car, and how much of the existing REST API surface can be reused as-is vs needs new mobile-friendly endpoints. To be designed once Phases 12-22 (web-side cleanup + hardening + redesign) are done.

Working Notes / Environment Gotchas

• Pi credentials/SSH: ssh yaris@<pi-ip>. Confirm current IP before connecting (varies by network — check locally, not recorded here) — usually reachable via yaris.local too (mDNS, see Phase 6).
• Local dev repo: work happens on a separate dev machine, not on the Pi. Push to origin/shameless-aislop, the repo's current default branch (repo is public).
• unraid server SSH: ssh [email protected] (alias unraid). Real git checkout lives at /mnt/user/app/git/yaris-obd (not /mnt/user/appdata/..., which is only the docker-managed data volume for the db/dashcam containers). Deploy with cd /mnt/user/app/git/yaris-obd && git pull --ff-only && cd server && docker compose up -d --build. Container names: yaris-obd-db (Postgres 17), yaris-obd-api.
• Pi deploy loop: ssh yaris@<pi-ip> (alias yarisobd) -> cd ~/app/yaris-obd && git pull --ff-only -> sudo systemctl restart yaris-obd.service (passwordless sudo confirmed via yaris ALL=(ALL) NOPASSWD: ALL). App stdout/stderr goes to ~/app/yaris-obd/yaris-obd.log (not journald - the systemd unit redirects both streams there), so use tail/grep on that file, not journalctl, to see application-level prints (upload progress, OBD/GNSS errors, etc.) - journalctl -u yaris-obd.service only shows service start/stop/crash lifecycle events. Update 2026-08-09: plain print() calls were silently invisible in that log for months because Python fully-buffers stdout when it's not a TTY (a file, in this case) — logging-based lines (e.g. the obd library's own logger, which uses stderr) showed up fine, but bare print() diagnostics from app_gps.py never did. Fixed by adding Environment=PYTHONUNBUFFERED=1 to the systemd unit (deploy/yaris-obd.service) — remember to also copy the updated unit file to /etc/systemd/system/yaris-obd.service and run sudo systemctl daemon-reload on the Pi, a plain git pull + restart does NOT pick up systemd unit file changes (the live /etc/systemd/system/ copy is independent of the repo's deploy/ copy).
• Pi deploy loop, requirements.txt changes: after any requirements.txt change (e.g. adding RPi.GPIO), also run ~/app/yaris-obd/venv/bin/pip install -r requirements.txt on the Pi — a git pull alone doesn't install new dependencies into the venv.
• Pi memory budget: ~900MB usable RAM, no swap. Any code path that might buffer a whole dashcam-sized file (hundreds of MB) in memory is unsafe on this hardware - stream, don't buffer (see Phase 10).
• Git push auth: if git push fails with a credential/auth error, run gh auth login (device-code browser flow) then gh auth setup-git to wire git to use the gh CLI's stored credentials — this fixed push auth issues in this environment without needing SSH keys or PATs.
• Pi environment gap: Debian Bookworm blocks system-wide pip install (PEP 668, "externally-managed-environment"). Use a venv for the real app dependency install (Phase 5) rather than --break-system-packages everywhere. (Resolved in Phase 5 — venv at /home/yaris/app/yaris-obd/venv.)
• Pi → GitHub auth for git pull (private repo): the Pi's pre-existing ~/.ssh/id_ed25519.pub is an inbound key only (private half is on the dev machine) — can't be reused for the Pi's own outbound git auth. Generated a dedicated keypair on the Pi (~/.ssh/yaris_deploy_key), added it as a GitHub read-only deploy key on this repo, and added an ~/.ssh/config entry pinning github.com to that key. [email protected]:... (SSH remote) then works standalone on the Pi, no PAT/interactive login needed.
• piwheels: pip on the Pi automatically uses piwheels.org as an extra index (Raspberry Pi OS default pip.conf) — this is why gevent (which normally needs a C compile) installed in seconds from a prebuilt ARM wheel instead of taking many minutes to build from source.
• python-obd note: obd.OBD() (or with portstr=) performs the ELM327 handshake at connection time; if there's no car on the other end, it raises rather than silently returning a dead handle — app_obd.py's connect_obd() already catches this and sets connection = None.