yaris-obd/PLAN.md

yaris-obd Revival Plan

This document tracks the plan to revive and finish this project: a Raspberry Pi 3-based OBD-II + GPS car tracker for a 2013 Toyota Yaris, hosting a local web UI over a self-broadcast WiFi hotspot, with dashcam file retrieval and remote upload of itinerary/dashcam data.

Keep this file up to date as phases complete or plans change, so any future session/agent can pick up the project without needing prior chat history.

Hardware

• Board: Raspberry Pi 3 Model B Rev 1.2, 906MB RAM. Confirmed too weak to run an AI coding agent — all agent-assisted dev work happens on a separate local machine; the Pi is only used for hardware bring-up/testing/deployment via SSH (yaris@<pi-ip>).
• GPS/GSM: Waveshare SIM868 GSM/GPRS/GNSS/Bluetooth HAT. Also has unused cellular (GSM/GPRS) data capability, earmarked as a fallback internet uplink (see Remote Server phase).
• OBD-II adapter: a cheap unbranded USB adapter using a PL2303 USB-to-serial chip, presumed ELM327-compatible (works with the python-obd library). Originally the project used a Bluetooth OBD adapter; this has been switched to USB.
• Car: 2013 Toyota Yaris.

SIM868 HAT bring-up notes (important gotchas)

• UART routing is a 3-position jumper: A = USB↔SIM868 (via onboard CP2102), B = Pi GPIO↔SIM868, C = USB↔Pi console. Must be on B for the Pi to talk to the module over its own GPIO UART.
• Pi-side UART setup required: enable_uart=1 and dtoverlay=disable-bt in /boot/firmware/config.txt (frees the full PL011 UART — /dev/ttyAMA0//dev/serial0 — from the onboard Bluetooth controller), raspi-config nonint do_serial_cons 1 (stops a login shell from claiming the serial port), and disable the hciuart systemd service.
• The SIM868 module chip itself needs its PWRKEY button pressed and held (~1-2s) to power on — this is separate from the board's 5V/PWR LED, which can be lit while the module chip is fully off. AT commands get zero response if the module chip isn't powered, even with perfect UART wiring/config. Update 2026-08-09: this HAT revision DOES have the "P4" GPIO-power-on mod wire installed (per the Waveshare FAQ/GSM_PWRKEY.rar script) — confirmed by pulsing physical header pin 7 (BCM GPIO4) LOW for 4s then HIGH while the module was unresponsive, after which it immediately answered AT. This is now automated in modules/app_gps.py (_ensure_module_powered()), which checks AT first and only pulses PWRKEY if the module isn't already responding (PWRKEY is a toggle — pulsing an already-on module would turn it back off, so this check-first behavior is required). No more manual button-pressing needed after a Pi power cycle.
• Confirmed AT command baud rate: 115200.
• GNSS workflow: AT+CGNSPWR? to check GNSS power state, AT+CGNSPWR=1 to power on, AT+CGNSINF to poll for a fix. Response format: +CGNSINF: run,fix,utc_time,lat,lon,alt,speed,course,fixmode,reserved1,hdop,pdop,vdop,reserved2,gps_sat_view,gnss_sat_used,gnss_sat_view,cn0,hpa,vpa — field 1 (fix) must be '1' for a valid fix.
• The GPS/GNSS antenna is separate from the GSM antenna — both need to be physically connected to their respective u.FL connectors on the HAT. A real fix was obtained only after discovering the GPS antenna wasn't plugged in (only GSM was).
• Confirmed real fix obtained near a window: ~80s cold-start TTFF, 5-7 satellites, HDOP ~1.2-1.6.

Software Architecture Decisions

1. Offline maps: Pre-cache/download offline map tiles for the car's usual driving area so the live map works fully in the car with no internet (not just numeric GPS readout). Tooling: MapTiler Cloud free tier (raster XYZ tiles, streets-v2 style) — one-time download on the dev machine via scripts/download_map_tiles.py, output shipped to the Pi as static PNGs. Bulk-downloading directly from tile.openstreetmap.org was ruled out — it violates the OSM tile usage policy. Bounding box is a required CLI arg (the user's own driving area, not committed here), zoom 8-15 by default (~14k tiles for a city-plus-suburbs-sized area, well under MapTiler's 100k/month free quota).
2. WiFi hotspot mode: Auto-switch — the Pi joins home WiFi when in range (for syncing/updates), and falls back to broadcasting its own AP otherwise. Important scope clarification: the hotspot is only for local web-UI viewing on a phone/laptop while driving — it is not part of the upload path. Remote-server uploads happen independently via home WiFi (when in range) or the SIM868 cellular fallback (decision #5) — the phone connecting to the hotspot never relays/carries upload traffic.
3. Local/remote persistence: Keep SQLite locally (reliable local write path per trip). The original plan paired this with MongoDB Atlas for remote sync (modules/app_db.py, fixed in Phase 7) — superseded by decision #4/Phase 9: the actual remote store is the self-hosted Postgres server, uploaded via modules/app_upload.py. app_db.py's Mongo path is not wired into app.py and is effectively dead code at this point, kept only for reference.
4. Remote server: Self-hosted — a custom server/API (e.g. on a home server or VPS) receives itinerary and dashcam uploads, rather than 3rd-party managed cloud services.
5. Remote upload uplink: Use the SIM868's GSM/GPRS cellular connection as a fallback uplink for uploads when no WiFi is available (e.g. while driving), not just "sync when on home WiFi."

Phases

Phase 1: Real GPS integration — ✅ DONE

modules/app_gps.py rewritten to read real SIM868 GNSS data via serial AT commands (/dev/serial0 @ 115200 baud) instead of hardcoded fake coordinates. Background daemon thread polls AT+CGNSINF every 2s, parses fix data, emits gps_location via socketio only on an actual fix. Verified against real hardware (see bring-up notes above).

Phase 2: OBD fixes for USB adapter — ✅ DONE

modules/app_obd.py fixed (commit "Fix app_obd.py bugs and adapt for USB OBD-II adapter", pushed to origin/shameless-aislop):

• connection is never left undefined on failed connect (was a NameError crash).
• Added _find_obd_port()/connect_obd(): auto-detects /dev/ttyUSB* (the PL2303 adapter), honors an OBD_PORT env override, falls back to python-obd's own port auto-scan (still covers the old /dev/rfcomm* Bluetooth path).
• Fixed stale-response bug: obd_speed()/obd_rpm() now re-query every loop iteration instead of reusing one response forever.
• Fixed obd_rpm() using the wrong lock, never updating the global rpm, and emitting to the wrong socketio event (obd_speed instead of obd_rpm).
• Fixed kill_obd() being a no-op — now uses threading.Event flags actually checked in the polling loops.
• Fixed inverted thread-start logic (threads were starting on failed connection, not success).
• Fixed obm_error → obd_error typo (frontend listens for obd_error).
• Added obd (python-obd) to requirements.txt.
• Still untested: a real connection to the actual car (adapter has only been tested standalone on the Pi so far — correctly reports "no connection to car" with nothing attached).

Phase 3: Web UI must work fully offline — ✅ DONE

The whole point of the project is a Pi hotspot with no internet in the car, but the current UI silently depended on the internet for core functionality:

• templates/index.html/dash.html/dtc.html loaded Google Fonts, FontAwesome, Leaflet JS/CSS, and the Socket.IO client from public CDNs — none of this loads offline, breaking layout, the websocket connection itself, and the map entirely.
• static/js/yaris.js loaded live map tiles from tile.openstreetmap.org (also needs internet).
• A local static/js/chart.min.js existed but was unused — index.html loaded Chart.js from cdn.jsdelivr.net instead (turned out the local copy was a stale Chart.js v2.8.0 while yaris.js's config uses v3/v4-style scales: {x:{...}, y:{...}} syntax — that mismatch is exactly why the CDN was used instead. Replaced the local file with a current v4.5.0 UMD build).

What was done:

• Vendored Socket.IO client (4.8.1, matches Flask-SocketIO 5.6.1's Engine.IO v4 protocol) under static/vendor/socketio/, and Leaflet (1.9.4 JS/CSS + marker/layers images) under static/vendor/leaflet/.
• FontAwesome 5.12.0 was already vendored locally at static/fonts/ (self-consistent, just never referenced) — repointed all three templates to it instead of the CDN.
• Dropped the Google Fonts (Nunito) <link> entirely rather than self-hosting 12 font-weight files — Bootstrap's --bs-font-sans-serif variable already falls back cleanly to system fonts (-apple-system, Segoe UI, Roboto, ...) after Nunito, so this is a clean no-cost simplification.
• Replaced static/js/chart.min.js with a current Chart.js v4.5.0 UMD build (auto-registering, same drop-in behavior as the old CDN <script src="https://cdn.jsdelivr.net/npm/chart.js">).
• Offline map tiles: added scripts/download_map_tiles.py, a dev-machine-only one-time tile downloader using the MapTiler Cloud free tier (raster XYZ, streets-v2 style, 256px tiles). Reads MAPTILER_KEY from an env var/CLI arg only — the key itself is never committed or deployed, only the resulting static PNG tiles are. Bounding box is a required CLI arg (your own driving area's coordinates - not committed/hardcoded) at zoom 8-15 (~14k tiles for a city-plus-suburbs-sized area, well under MapTiler's 100k tiles/month free quota). Output goes to static/vendor/tiles/{z}/{x}/{y}.png (gitignored — regenerate locally, rsync to the Pi as part of deployment, don't bloat the git repo with binary tile data).
  • Tiles populated: user signed up for a free MapTiler key and the full zoom 8-15 download ran to completion — 14,241 downloaded, 24 skipped (out-of-bounds), 0 failed (14,265 tiles total, ~285MB). Rsynced to the Pi's static/vendor/tiles/ (tile counts verified identical on both sides). Live Pi instance re-confirmed serving HTTP 200 with the complete offline tile set.
• Updated static/js/yaris.js's Leaflet init to point at /static/vendor/tiles/{z}/{x}/{y}.png (minZoom/maxZoom 8/15 to match). The initial map center is just a generic placeholder (not a real location) - it re-centers automatically to the real live GPS fix once one comes in, and the marker is exposed as carMarker so Phase 4 can move it on live fixes.
• Added a repo-root .gitignore (Python caches, venvs, .vscode/, static/vendor/tiles/, *.env except the already-tracked db/creds.env placeholder) — there wasn't one before, which is why stray __pycache__/.pyc files kept showing up in source control.
• Verified: ran the Flask app in a local venv, confirmed all three pages (/, /dtc, /dash) render with zero external URLs in the served HTML, every local static asset (JS, CSS, fonts, Leaflet images) returns HTTP 200, and all vendored JS files pass a Node.js syntax check.

Phase 4: Map/GPS display bug — ✅ DONE

yaris.js's Leaflet marker (carMarker) was never moved on incoming gps_location socket events — only two text spans (gpsLocationX/Y) got updated. Fixed: carMarker.setLatLng(...) is now called on every gps_location event, and the map re-centers (zoom 15) on the first fix only (mapCentered flag), so the user isn't fighting the map re-centering under them while panning/zooming during a trip.

Verified: also used this as an opportunity to test Phase 3 end-to-end — ran scripts/download_map_tiles.py for real with the user's actual MapTiler API key (smoke-tested zoom 8-10 first, confirmed idempotent re-run skips existing tiles, confirmed a downloaded tile is a real map image of the target driving area), then started the full zoom 8-15 download. Took a headless Chromium screenshot of the running app's / page and confirmed: sidebar/icons/fonts render correctly (FontAwesome working), the Speed Graph (Chart.js v4) renders, and the Leaflet map renders real offline map tiles with the marker in place — full offline end-to-end proof, not just "no CDN URLs in the HTML".

• Fixed (was noted here as out of scope, now addressed in Phase 8): /dash and /dtc used to render near-identical body content (both showed "DTC Codes") — dash.html now has real dashcam-specific content.
• Full tile download completed (see Phase 3 update above) and rsynced to the Pi — Phase 3/4's offline map is now 100% functional end-to-end on the actual deployed device, not just the dev machine.

Phase 5: Pi deployment / autostart — ✅ DONE

Set up a proper venv (/home/yaris/app/yaris-obd/venv), installed all requirements (all found as prebuilt ARM wheels via piwheels — no slow source compiles needed, even for gevent). Added deploy/yaris-obd.service (systemd unit: runs as the yaris user, Restart=on-failure, 5s ExecStartPre sleep to let USB/serial devices enumerate after boot, logs to yaris-obd.log) and deploy/install_service.sh (idempotent one-time install helper — also creates the venv if missing). Installed and enabled on the Pi (systemctl enable — survives reboot).

Also had to fix Pi → GitHub auth for git pull: the repo is private, and the Pi's existing ~/.ssh/id_ed25519.pub turned out to be an inbound SSH key (its private half lives on the dev machine, not the Pi) — useless for the Pi to authenticate outbound. Generated a dedicated ed25519 keypair on the Pi itself (~/.ssh/yaris_deploy_key), registered it as a read-only deploy key on the GitHub repo (least-privilege — the Pi only ever needs to pull), and added an ~/.ssh/config entry so git transparently uses it for github.com. Switched the Pi's origin remote to SSH. git pull now works standalone on the Pi with no PAT/interactive auth needed.

Verified end-to-end: pulled latest code onto the Pi (was 5 commits stale), confirmed all Python deps import cleanly, ran the app manually first (HTTP 200, all local static assets — Leaflet/Socket.IO/Chart.js/FontAwesome — served correctly from the Pi itself), then installed + started the systemd service and confirmed active (running) + enabled. Tested crash recovery: sent SIGKILL to the running process, confirmed systemd auto-restarted it (through the 5s ExecStartPre delay) and the app was serving HTTP 200 again within ~10s, with zero manual intervention.

Phase 6: WiFi hotspot mode — ✅ DONE

Used NetworkManager (already the network manager in use, not hostapd/dnsmasq directly — those were leftover/unused) since it natively supports AP mode via nmcli. Created two connection profiles on the Pi: home-wifi (your real home WiFi network, autoconnect-priority 10) and Hotspot (AP mode, ipv4.method shared, its own dedicated SSID/password set locally on the Pi - not committed here, WPA2-PSK, autoconnect-priority -10). NM's own autoconnect logic handles "leaving home WiFi range -> fall back to hotspot" automatically since home-wifi has higher priority and NM always tries it first.

For the reverse direction (home WiFi back in range while sitting in AP mode) added deploy/yaris-netwatch.sh + .service/.timer: since the Pi's onboard BCM43438 radio can't be an AP and background-scan for client networks simultaneously, a systemd timer runs every 3 minutes, briefly attempts reconnecting to home-wifi, and restores the hotspot if home WiFi still isn't in range. Installed via deploy/install_netwatch.sh.

Root-caused and fixed a real bug during testing: the hotspot's SSID broadcast worked immediately, but its DHCP server initially failed (dnsmasq: failed to create listening socket for 10.42.0.1: Address already in use) — a pre-existing orphaned system-wide dnsmasq.service (noted as "likely unused" in earlier investigation) was already bound to the address space NetworkManager's own internal dnsmasq needs for the hotspot. Confirmed the system dnsmasq's config was genuinely empty/default (nothing in this project depends on it) and disabled it (systemctl disable --now dnsmasq).

Also added easy access without needing to know the Pi's IP:

• Renamed the Pi's hostname from yarisobd to yaris — avahi-daemon (already installed and running) now publishes it as yaris.local via mDNS.
• Port 80 -> 5000 redirect via nftables (deploy/nftables-yaris.conf + install_port_redirect.sh, loaded through an include in /etc/nftables.conf), so the app is reachable at plain http://yaris.local/ (no port number) instead of :5000.

Verified on real hardware, very carefully since the SSH session used to do all this testing itself runs over wlan0: used a "dead-man's-switch" pattern for every risky test — a detached background command (setsid nohup bash -c 'sleep N && nmcli con up home-wifi' & disown) armed before each hotspot activation, so the Pi automatically reverts to home WiFi (and SSH access comes back) even if something goes wrong, no physical access needed. Confirmed: hotspot broadcasts and a real phone gets a genuine DHCP lease from NetworkManager's own DHCP server and can load the web UI; yaris-netwatch correctly detects AP mode and reconnects to home WiFi when in range (and is a no-op when already on home WiFi); http://yaris.local/ and http://<pi-lan-ip>/ both return HTTP 200 with no port number from another machine on the network. (Real hotspot SSID/password and the Pi's LAN IP are intentionally not recorded in this file — see local, non-committed notes.)

Update — real-hardware bug found + fixed (hotspot->home reconnect was flaky): while testing the phone-relay feature live (see Phase 11 update below), yaris-netwatch repeatedly logged "home WiFi not found" 4 times in a row over ~13 minutes while sitting right next to the router — a real regression from the original design, not a one-off fluke. Root cause: a plain nmcli con up home-wifi straight out of AP mode relies on NM's scan cache for wlan0, which is stale/empty coming out of AP mode (the radio can't scan while acting as an AP - see above). A single --wait 20 connect attempt gave up before a fresh scan could land. Fixed yaris-netwatch.sh to explicitly drop the AP connection first (releases the radio), force nmcli device wifi rescan, wait for it to settle, then retry the connect (longer 30s wait) up to 2 times before conceding and restoring the hotspot. Bumped yaris-netwatch.service's TimeoutStartSec to 180s (retry loop can take up to ~80s, default systemd 90s start timeout was too tight). Verified fixed on real hardware: both hotspot->home transitions after the fix recovered in ~7 seconds (previously failing outright). Deployed: sudo cp deploy/yaris-netwatch.sh /usr/local/bin/... && cp .service ... && daemon-reload.

Phase 7: app_db.py / app_sql.py fixes — ✅ DONE

Neither file was called from app.py or anywhere else - orphaned from a prior iteration. Fixed:

• app_db.py: applied a prior-session stash fix - resolves .env path via __file__ (was cwd-relative), fixes os.getenv() calls to pass the actual key names (was called with zero args, an immediate TypeError if ever invoked), added a clear RuntimeError guard if credentials are still empty placeholders, returns client[mDB].
• app_sql.py: rewritten - resolves db/sql/ via __file__ + os.makedirs (was cwd-relative), fixed create_conn() closing the connection immediately and returning None (so table creation never ran and every caller got a dead connection), added get_conn() (lazy singleton
  • table creation for both tracking and dtc tables), insert_tracking(), and a background start_logging()/_log_loop() thread (5s interval) that snapshots GPS (app_gps.LocationX/Y)
  • OBD speed (app_obd.speed) into the local tracking table. Used a deferred (function-local) import of app_obd inside _log_loop() to keep the pre-existing app_obd<->app_sql top-level circular import safe/explicit.
• Also fixed a Python 3.12+/3.14 compat bug found during testing: sqlite3.version (the sqlite3 module's version) was removed/deprecated - switched to sqlite3.sqlite_version (the actual SQLite library version), which is what was intended.
• Wired app_sql.start_logging() into app.py alongside the other startup calls (app_gps.start_gps(), app_obd.OBD()).
• db/sql/ added to .gitignore (per-device local trip data, not source).
• Verified locally: get_conn() creates db/sql/<account>_<car>.db with both tables; insert_tracking() writes a real row; start_logging() thread starts and logs "SQL tracking-logger thread started, alive: True"; full app.py run confirms the thread starts cleanly and (with no real GPS/OBD hardware on the dev machine) correctly logs read errors instead of crashing; app_db.get_database() raises the new clear RuntimeError instead of crashing, since db/creds.env still holds empty placeholder credentials (real Mongo credentials are a user-provided follow-up, out of scope here).

Phase 8: Dashcam retrieval (app_dash.py) — ✅ DONE, verified on real hardware

Rewrote app_dash.py to be brand/device-agnostic: detection via lsblk -J, matching partitions whose parent disk reports TRAN == "usb" (this is what safely distinguishes a plugged-in dashcam/USB stick from the Pi's own onboard storage, which is always mmc transport - no more hardcoded /dev/sda). Walks the whole mounted filesystem for video file extensions (.mp4/.mov/.avi/.ts/.mkv) instead of assuming a specific brand's folder layout (DCIM, MOVIE, NORM/EVENT/PARKING, etc. all just work), preserves each file's relative sub-folder path when copying (avoids filename collisions across dashcam event/date folders), and skips files already copied (same relative path + size) so re-running an import is cheap. Mounts read-only to avoid any risk of corrupting the dashcam's SD card. Dropped the unused/uninstalled tqdm CLI-progress dependency (never actually in requirements.txt) since this now runs headless inside the always- on web server, not interactively.

Wired into app.py: GET /api/dash/devices (list detected USB storage) and POST /api/dash/import (run the mount->copy->unmount pipeline), both JSON. Fixed the pre-existing /dash vs /dtc duplicate-template bug — dash.html now has real dashcam content (scan/import buttons + live status) instead of a copy-pasted "DTC Codes" body. Added a dedicated dash.js (not the shared yaris.js, which unconditionally initializes the /index-only map/chart and would throw/crash on /dash and /dtc since those pages don't have those DOM elements - a pre-existing bug, worked around by not depending on that shared file for this page's new JS). Added dash/videos/ to .gitignore (imported footage is per-device data, not source).

Tested thoroughly without the physical dashcam (user didn't have it on hand this session):

1. Unit-tested the lsblk-JSON detection/filtering logic against a realistic synthetic sample (Pi's own mmcblk0 boot+root partitions plus a simulated USB dashcam partition) - confirmed it correctly finds only the USB partition and ignores the Pi's own storage.
2. Integration-tested the real mount -> find-videos -> copy (preserving structure) -> dedupe -> unmount pipeline on the actual Pi using a loop-device-backed FAT32 image (mkfs.vfat + losetup) populated with fake nested dashcam video files (DCIM/100EVENT/..., DCIM/100NORMAL/...) - confirmed real mount/copy/unmount all work, confirmed a second import run correctly skips already-copied files, confirmed non-video files (e.g. thumbs.db) are ignored. All simulation artifacts (loop device, test image, temp files) cleaned up afterward.
3. Verified the full /dash page + both new API routes locally (Flask dev run + headless browser check): auto-scan on page load correctly reports "No USB dashcam/storage device detected" (accurate, since none is plugged into the dev machine), Import button correctly stays disabled until a device is found, no JS console errors, no regression on / or /dtc.

Verified against the real physical dashcam (Jieli-chipset USB ID 1224:2725, 29.7GB SD card, DCIM/1/VID_###.MOV layout, 87 files, ~28GB total). This first live hardware test found and fixed three real bugs the loop-device simulation couldn't have caught:

1. Superfloppy SD card - the card has no partition table at all (lsblk reports the whole /dev/sda disk directly as fstype: vfat, no child partition) - the original detection assumed a partition would always exist. Fixed to also match a TRAN == "usb" disk with its own fstype set directly, not just partitions under it.
2. Mount permission error - mount_device() called os.makedirs() directly, but the service runs as unprivileged yaris and /mnt is root-owned. Fixed to sudo mkdir -p <mount_point> before the existing sudo mount, mirroring that pattern.
3. OOM crash during upload (see Phase 10 below) - not a copy-path bug, but discovered immediately after the first successful real import.

End-to-end result: full 28GB/87-file import from the SD card completed successfully (resumed cleanly after being interrupted once by the OOM crash below - copy_files()'s skip-if-already- copied logic meant re-running /api/dash/import just picked up the remaining files, no data loss or re-copy of the first 19 files). SD card unmounted cleanly afterward.

Phase 9: Remote server for itinerary/dashcam upload — ✅ DONE

Per architecture decisions #4 and #5: self-hosted server/API to receive itinerary (from the SQLite pipeline) and dashcam file uploads.

Decided/built: Flask API + dedicated Postgres, deployed as a docker-compose stack on the user's unraid server (separate from any existing containers/DBs there), living in server/ in this same repo. Auth: simple X-Device-Name/X-Api-Key header pair per device, key hash stored server-side, real key never committed (raw key printed once by server/scripts/create_device.py, saved locally on the Pi only, in gitignored db/upload_creds.env). External access: exposed at https://yaris.ceaf.dev via the user's existing Cloudflare Tunnel (public-hostname rule added in the Cloudflare dashboard, pointing at 192.168.10.10:8098) — router port-forward + the user's Unifi firewall is the documented fallback if ever needed instead. Large dashcam uploads can still prefer a local-network path (e.g. an AdGuard DNS override to bypass the tunnel while on home WiFi) since tunnels typically cap per-request upload size — noted in server/README.md, not yet needed/built.

Built (server/ folder):

• server/sql/init.sql — Postgres schema: devices (name + sha256 api-key-hash), tracking_points (device_id FK, GPS/speed/odometer + recorded_at, unique per device+timestamp so retried batches can't duplicate rows), dashcam_files (device_id FK, relative_path + sha256, unique per device+path+hash so re-uploads are no-ops).
• server/modules/db.py — psycopg3 connection pool + query helpers (bulk tracking insert with ON CONFLICT DO NOTHING, dashcam dedupe-check-before-write).
• server/modules/auth.py — require_device_auth decorator checking the two headers against the stored key hash.
• server/app.py — Flask app: GET /health (unauthenticated), POST /api/v1/tracking/batch (bulk JSON insert, capped at 5000 points/request as a sanity guard), POST /api/v1/dashcam/upload (multipart, streamed to disk with sha256 verification + path-traversal guard on relative_path).
• server/Dockerfile + server/docker-compose.yml — api (gunicorn, published on host port 8098 by default) + db (Postgres 17, internal-only, not published to the LAN) services, data volume defaults to server/data/ but pointed at /mnt/user/appdata/yaris-obd-server on unraid.
• server/.env.example — template for POSTGRES_PASSWORD/API_PORT/DATA_DIR/upload size cap.
• server/scripts/create_device.py — run once per device inside the api container to register it and print its one-time API key. Ran once for device yaris-2013.
• server/README.md — full unraid deployment + Cloudflare Tunnel exposure + device-registration instructions.

Pi-side upload client (modules/app_upload.py):

• Background thread, uploads unsent tracking rows (batches of 500) and not-yet-uploaded dashcam files (sha256-verified) via the API above, with exponential retry/backoff on failure.
• Independent dashcam-upload-only pause toggle (persisted settings table + a switch on the /dash web UI) — trip/GPS logs always keep uploading regardless of the toggle, per explicit user request (useful on a slow/metered connection).
• Backlog-drain tuning: since tracking rows log every 5s locally, an offline/stale device can queue up a large backlog. The loop drains back-to-back (short 1s pause between batches) while a batch comes back full, only settling into the slow 30s idle cadence once actually caught up — a simulated 1250-row backlog now drains in 3 batches (~0.35s) instead of 13 batches over ~6.5 minutes with the original fixed 100-row/30s-sleep design.
• modules/app_sql.py extended: UPLOADED column + migration on tracking, settings table, dashcam_uploads dedupe table, and helper functions backing all of the above.

Tested end-to-end against the real deployed server (tracking batch + dedupe, dashcam upload + dedupe + path-traversal rejection, bad-API-key rejection, toggle-off correctly pausing only dashcam uploads) and on the real Pi (its actual ~1800-row pre-feature backlog drained and confirmed matching row counts on the server via direct psql query, live upload status visible on /dash). Committed+pushed (4ff99f0, c4db1ab, 8dc304e), deployed on both unraid and the Pi. Exposed externally at https://yaris.ceaf.dev via the user's Cloudflare Tunnel (public hostname added in the Cloudflare dashboard, pointing at 192.168.10.10:8098).

Phase 10: Dashcam upload memory/size fixes — ✅ DONE, verified on real hardware

Two real-hardware-only bugs surfaced once actual ~300MB+ video files (not small test fixtures) hit the upload path for the first time:

1. OOM crash from in-memory multipart uploads. modules/app_upload.py originally uploaded dashcam files via requests.post(..., files={"file": f}). The requests library builds the entire multipart/form-data body in memory before sending when using files= - a well-known gotcha, not specific to this codebase. On the Pi 3's ~900MB RAM, uploading one ~320MB video this way spiked the process past 600MB+ RSS, triggering the kernel OOM killer within minutes (confirmed via dmesg: repeated Out of memory: Killed process ... anon-rss:823928kB, and via pmap -x showing a single ~362MB anonymous mapping matching the file size almost exactly). Fixed: added requests-toolbelt and rewrote the upload body construction to use MultipartEncoder, which streams the file in chunks as the request is sent instead of materializing it up front. Confirmed via an isolated 300MB test (628MB RSS old way -> 29MB new way, byte-identical result) and live on the Pi (RSS stayed flat ~55-59MB during real uploads, vs. previously crashing every few minutes).
2. Cloudflare Tunnel's hard ~100MB request-body cap. Even after the memory fix, real uploads still failed once routed through https://yaris.ceaf.dev - a bare 413 Payload Too Large from Cloudflare itself (confirmed: the identical file succeeds direct-LAN, 413s only through the tunnel). This is a free-tier platform limit, unrelated to MAX_DASHCAM_UPLOAD_MB on the Flask side. Fixed via a chunked/resumable upload redesign (well under the 100MB cap):
   • Client (modules/app_upload.py): DASHCAM_CHUNK_SIZE = 40MB; new _upload_video_chunked() checks /status first (enables resume across restarts/failures), then POSTs missing chunks one at a time via MultipartEncoder.
   • Server (server/app.py): new GET /api/v1/dashcam/upload/status (reports already_uploaded or received_chunk_indices) and POST /api/v1/dashcam/upload/chunk (stages chunks under STORAGE_DIR/.upload_tmp/<device>/<sha256>/, reassembles + verifies SHA256 once all chunks arrive, records via the existing db.insert_dashcam_file, cleans up the staging dir). A .total_chunks marker guards against stale/mismatched partial sets if the chunk size ever changes between client versions. The original single-shot /api/v1/dashcam/upload endpoint is untouched, kept for reference/compatibility.

Verified end-to-end against the real deployed server through the real Cloudflare tunnel, first with a synthetic interrupted-then-resumed 250MB test (status transitions not_started -> partial upload -> in_progress w/ correct received_chunk_indices -> resume -> complete -> already_uploaded, SHA256 match confirmed on disk), then with real dashcam footage: the Pi copied all 87 real files locally and the background upload thread drained the real backlog one file at a time, each one verified byte-for-byte (SHA256 match Pi vs. server) and correctly recorded in Postgres, while the Pi's service ran 50+ minutes straight with RSS staying in a bounded ~55-141MB range (no OOM kills) - a stark contrast to the previous crash-every-few-minutes behavior. Given home-upload-bandwidth constraints, the full 28GB backlog drains at roughly one ~350MB file per 4-5 minutes and is expected to take several hours total - this runs unattended in the background with no user action needed once started.

Not yet built: parallel/concurrent chunk uploads (currently strictly sequential, one chunk then one file at a time) - would speed up the multi-hour full-backlog drain but adds complexity; not requested by the user yet, flagged here as a possible future optimization if the sequential pace becomes annoying in practice.

Known manual step (not yet automated): this dashcam model boots into a "select mode" menu (dashcam / mass-storage / webcam) when it senses USB data lines are live, and needs USB data communication physically absent to boot straight into recording mode, but present when plugging into the Pi for import. No code in this repo handles that switch - it's a manual physical step by the user (e.g. a data-blocking USB cable/adapter, or plugging/unplugging appropriately) each time before recording vs. before importing. Worth revisiting if it becomes tedious in daily use (e.g. a Pi-controlled USB data-line switch), but out of scope for now.

SIM868 cellular fallback for tracking uploads — ⚠️ built, UNTESTED (no SIM card available)

modules/app_cellular.py gives the tracking-upload path a fallback over the SIM868's cellular data connection when WiFi/Ethernet is unreachable. Design notes:

• Uses the SIM868's built-in AT-command HTTP(S) client (AT+HTTPINIT/HTTPPARA/HTTPACTION/ HTTPREAD) rather than a real ppp0 network interface. Reason: the SIM868's GNSS (polled continuously by modules/app_gps.py) and its GSM/GPRS radio share the same physical UART (/dev/serial0) — a classic pppd/chat dial-up would put that port into raw PPP framing for as long as the link is up, starving the GNSS polling thread. Staying in AT-command mode lets cellular HTTP calls interleave safely through the same serial lock (modules.app_gps._serial_lock) — each exchange just briefly delays the next GPS poll.
• Only carries small JSON tracking batches, never dashcam video — cellular data is far too slow/limited for multi-MB video files, and the modem's HTTP AT interface has a small internal buffer unsuited to them anyway. Dashcam uploads remain WiFi/Ethernet-only, unchanged.
• modules/app_upload.py's tracking-batch upload always tries WiFi/Ethernet (requests) first; only on a connection-level failure (no route/DNS/timeout — as opposed to an HTTP error status, which means the server WAS reached) does it retry the same batch over cellular.
• Feature-flagged off by default: YARIS_CELLULAR_ENABLED/YARIS_CELLULAR_APN/ YARIS_CELLULAR_APN_USER/YARIS_CELLULAR_APN_PASS added to db/upload_creds.env, all blank by default so this has zero effect on the already-verified WiFi upload path until the user fills in an APN once they get a SIM card. Regression-tested (against the real server) that normal WiFi uploads are completely unaffected while cellular is unconfigured.
• /dash upload-status card now shows which path (wifi/cellular) the last successful batch used, and whether cellular fallback is configured.
• Needs validation once a SIM card is available — exact AT command dialect/timing (SIMCOM firmware revisions vary slightly), and whether AT+HTTPSSL works for the HTTPS yaris.ceaf.dev endpoint on this specific module's firmware (if not, a documented fallback is a plain-HTTP router port-forward path, avoiding the need for the modem's TLS stack to validate a cert chain).

Phase 11: Phone-relay dashcam upload — ⛔ REMOVED (see update below - hit a hard platform limitation)

Root cause investigation into "why is the transfer from the Pi to the server so slow?" (real dashcam backlog draining at only ~350MB/4-5min) found the bottleneck is the Pi's own onboard BCM43438 WiFi chip (2.4GHz-only, congested/interfered channel) — confirmed via iperf3 raw-TCP tests showing only ~14.7Mbit/s with active retransmits even over a same-subnet direct-LAN hop that never touches the internet/Cloudflare, ruling out ISP/Cloudflare limits. Since the Pi 3 B has no 5GHz radio and can't be fixed in software, added a way to bypass its weak WiFi for the internet-bound leg entirely: a phone opens the same /dash page (over the Pi's hotspot/home WiFi) and manually relays pending dashcam files straight to the remote server using the phone's own network (WiFi or cellular data), instead of routing everything through the Pi.

Zero new remote-server upload logic — the phone's browser JS calls the exact same /api/v1/dashcam/upload/status + /api/v1/dashcam/upload/chunk endpoints the Pi's own Python client already uses, in the same 40MB-chunk protocol, just orchestrated by fetch()/FormData instead of requests/MultipartEncoder. The server dedupes purely by SHA256 regardless of which device sends the bytes, so no server-side changes were needed for the upload path itself — only a CORS after_request hook (server/app.py) was added, since the phone's browser calls yaris.ceaf.dev from a page served by a different origin (the Pi), which browsers block without Access-Control-Allow-* headers on both the real request and its preflight OPTIONS. Wildcard origin is fine here — auth is header-based (X-Device-Name/X-Api-Key), not cookie/session-based.

New Pi-side endpoints (app.py):

• GET /api/dash/pending — lists not-yet-uploaded dashcam files (relative_path, sha256, size) via a new app_upload.list_pending_dashcam_files().
• GET /api/dash/file/<path:relpath> — serves a dashcam file's bytes for the phone to download, via send_file(..., conditional=True) for Range-request support (206 Partial Content) so the phone downloads (and immediately re-uploads) one 40MB slice at a time instead of ever holding a whole 300+MB file in the browser's memory — the exact same memory-safety lesson learned the hard way in Phase 10, applied symmetrically to the download side. Path-traversal safe via Werkzeug's safe_join against app_dash.DEFAULT_DESTINATION.
• GET /api/dash/relay/config — hands the phone the same SERVER_URL/DEVICE_NAME/API_KEY the Pi's own client uses, so its JS can auth directly with the remote server. Same trust boundary as the rest of this project's unauthenticated local UI (anyone on the Pi's own hotspot/LAN is already implicitly trusted).
• POST /api/dash/relay/heartbeat / POST /api/dash/relay/stop — a lease-based pause (RELAY_LEASE_SECONDS = 45) for the Pi's own background dashcam uploader, chosen over a plain on/off toggle specifically to self-heal if the phone's browser tab closes/loses connection uncleanly (a simple toggle could get stuck "paused" forever). The phone page heartbeats every 15s while relaying; _upload_dashcam_files() skips entirely while a lease is active, re-checked between files so it stops promptly if a relay starts mid-drain. Also released via navigator.sendBeacon() on beforeunload as a best-effort immediate release.
• POST /api/dash/relay/mark_uploaded — lets the phone tell the Pi "I just uploaded this file" right after a successful relay, so the local dedupe table (and the pending list / Pi's own background loop) reflect it immediately instead of waiting for the next full backlog scan.

Performance side-fix: modules/app_upload.py's dashcam loop previously recomputed SHA256 for every candidate file on every loop iteration, including files long since uploaded — wasteful CPU/IO against a growing, possibly 20GB+, local backlog. Added a dashcam_hash_cache SQLite table (modules/app_sql.py, keyed by relative_path+size+mtime) so a file's hash is only computed once; both the existing background uploader and the new pending-files endpoint benefit.

UI (templates/dash.html / static/js/dash.js): a new "Relay via phone" card on /dash lists pending files with a per-file "Relay" button and a "Relay all pending" button (manual trigger only — chosen deliberately over an automatic one, per explicit user preference), plus a "Stop relay" button. Chunk downloads/uploads retry up to 3 times with backoff before surfacing an error, and a resumed relay re-checks /status first so a previously-interrupted file picks up from its first missing chunk instead of restarting.

Tested: a full local end-to-end simulation (no physical phone/network needed) drove the real Pi app.py and real server/app.py Flask apps side by side (Postgres mocked at the db module boundary only — all HTTP routing, chunking, auth, and CORS logic is real), reproducing the exact sequence the phone's browser JS performs: list pending -> check remote /status -> download each chunk from the Pi via Range requests (verified real 206 Partial Content responses) -> upload each chunk to the mock-backed server -> verify the reassembled file matches the original byte-for-byte via SHA256 -> mark uploaded on the Pi -> confirm it drops out of the pending list. Also verified: path-traversal attempts on /api/dash/file/ correctly 404, the relay heartbeat/stop lease correctly pauses/resumes _upload_dashcam_files(), and the sha256 cache table is populated and reused across repeated /api/dash/pending calls.

Deploy gotcha found: the CORS fix in server/app.py was initially deployed to the Pi but not rebuilt on the actual remote server (the server's Docker image bakes app.py/modules/ in at build time via COPY, so a plain git pull + container restart isn't enough — needs docker compose build api && docker compose up -d api). Without it, the phone's cross-origin fetch() calls to yaris.ceaf.dev were silently blocked by the browser's CORS policy, which surfaces in JS as a generic, unhelpful TypeError: Failed to fetch — worth remembering as the first thing to check if this error reappears after a future server-side change.

Tested for real against a real phone and the real deployed server (first end-to-end proof beyond the local simulation above): the initial /api/dash/pending call took ~25 minutes for a one-time full-backlog SHA256 hash of all 87 files (~28GB, no cache existed yet) — after that, repeat calls returned in ~0.13s. The user then successfully relayed a real file from their phone (home WiFi) partway through (10/~12 chunks) before stopping to check something important: is the relay actually faster, given the phone downloads from the Pi over the same congested radio diagnosed as the bottleneck? Measured directly from chunk-arrival timestamps on the server: ~1.11MB/s (~8.9Mbit/s) for the phone-relay path — statistically the same as the ~9-10Mbit/s measured for the Pi's own direct Cloudflare upload earlier this session. Conclusion: when the phone is on the same home WiFi as the Pi, the relay provides no speed benefit — the download-to-phone leg pays the exact same slow-radio cost as a direct Pi upload would, and the phone's reupload leg then goes out through the same home internet connection anyway, adding an extra hop for no gain. The relay only has genuine value when the phone's second leg differs meaningfully from the Pi's own path — e.g. cellular data with better upload than the home ISP, or uploading after physically leaving home with files already grabbed locally. Stopped the test relay (POST /api/dash/relay/stop) and confirmed the Pi's own background uploader resumed the same file's chunked upload exactly where the phone left off (shared /status/.upload_tmp state on the server made this seamless, no wasted re-transfer).

Next idea (not yet built): the user wants a way to browse/preview individual dashcam clips on the phone and select specific ones to view or upload (as opposed to bulk-relaying the entire backlog) — useful for "just grab this one important clip" rather than a wholesale speed optimization. Not scoped/designed yet.

Update — tested the "away from home" hotspot+cellular scenario for real, found a real phone-OS limitation (not a code bug): the user asked to test the specific scenario the relay feature was actually designed for — phone connects to the Pi's own hotspot (not home WiFi) to download pending files, then re-uploads using the phone's cellular data, bypassing the Pi's own (in hotspot mode, nonexistent) internet connection entirely. Confirmed via code review this needs zero changes: /api/dash/relay/config already hands the phone the public https://yaris.ceaf.dev SERVER_URL (not a LAN-only address), so the re-upload leg was always going to be internet-bound regardless of which physical network carries it.

Live-tested with a real dummy file (dashcam upload toggle paused first so the Pi's own uploader couldn't grab it before the phone did): got TypeError: Failed to fetch trying to relay. Confirmed via a direct test (loading https://yaris.ceaf.dev/health as a plain page navigation, not a fetch(), ruling out CORS) that the phone's own OS was declining to route internet-bound traffic over cellular while still associated with the Pi's no-internet-gateway hotspot WiFi — it only succeeded once the phone auto-fell-back to the home WiFi network (which does have internet) instead. This is a phone/OS-level "smart network switching" behavior (common on both iOS and Android to avoid using a WiFi network with no internet gateway) that cannot be fixed in this project's code — the architecture is sound and was already proven correct in the local end-to-end simulation and the byte-for-byte reassembly tests earlier in this phase.

Practical implication: testing this exact scenario at home is inherently confounded — the phone always has a known-good home WiFi it can silently fall back to, defeating the test. A true validation needs either (a) actually being away from home with no known WiFi in range, forcing the phone to use cellular for internet traffic, or (b) manually disabling the phone's WiFi (or its "switch to mobile data automatically" setting) while connected to the Pi's hotspot, to force cellular use for this one operation.

Update — retested more rigorously, confirmed this is a fundamental architecture-level dead end, not just an untested corner case. Re-ran the same live test multiple times with longer/managed "dead-man's-switch" windows (yaris-netwatch.timer explicitly stopped during each test run so it wouldn't yank the Pi back to home WiFi mid-test — an earlier run's failure was actually caused by netwatch reconnecting home WiFi under the phone, see Phase 6 update, not a phone-network issue). With netwatch correctly disabled and the Pi confirmed to stay in AP mode for the whole test window, still got Failed to fetch. Directly tested a plain page navigation (not fetch()) to https://yaris.ceaf.dev/health with WiFi fully turned off (pure cellular) — that also failed, while every other website loaded fine over the same cellular connection with WiFi off. This narrows the cause precisely: the browser/phone genuinely cannot reach the public internet at all while wlan0/the phone's WiFi radio is associated with the Pi's hotspot, full stop — this is not a settings toggle problem (checked Pixel-specific "switch to mobile data automatically" options, not present on this device/Android version) but a fundamental OS-level "default network" routing decision: a device's default route for all IP traffic (including a web page's fetch() calls) is a single, whole-device choice made by the OS's connectivity manager, not something a browser page's JavaScript can override per-request. Real per-request network binding (e.g. Android's ConnectivityManager.bindProcessToNetwork()) is an API only available to native apps, not exposed to web pages at all, in any browser, on any phone. Given this, the phone-relay feature as designed (a web page doing simultaneous WiFi-to-Pi + cellular-to-server transfers) cannot ever work while the phone is connected to the Pi's no-internet hotspot — this isn't a bug to fix, it's a hard platform ceiling.

Decision: removed the phone-relay feature entirely (was Phase 11, now reverted - see below). Two ways to actually deliver on the original "upload from the road without the Pi's slow WiFi" goal were discussed: (1) restructure as two distinct, non-simultaneous phases (fully download to the phone's local storage first while on the Pi's hotspot, then have the user manually switch to cellular, then upload from local storage) - no native app needed but a more awkward two-step user flow, or (2) a real native Android app, which can hold both networks open at once via bindProcessToNetwork() for a true simultaneous relay. Neither has been built - shelved for now in favor of simplifying the dashcam retrieval UI (see the new phases below). The Pi's own background uploader (plain WiFi/Ethernet, optionally SIM868 cellular fallback for tracking data only) remains the only upload path; it's just slower when the Pi's own WiFi is weak, with no workaround currently in place for that specific case.

Side-fix from this test session: found and fixed a real, unrelated bug in yaris-netwatch.sh during this same test — see the Phase 6 update above.

Phase 12: Remove phone-relay feature — ✅ DONE

Undo Phase 11 cleanly given the Phase 11 update above (fundamental browser platform limitation, not fixable). Removed:

• app.py: /api/dash/pending, /api/dash/file/<path>, /api/dash/relay/config, /api/dash/relay/heartbeat, /api/dash/relay/stop, /api/dash/relay/mark_uploaded routes (and the now-unused os/send_file/abort/safe_join imports they needed).
• modules/app_upload.py: list_pending_dashcam_files(), relay_heartbeat()/relay_stop()/ is_relay_active(), RELAY_LEASE_SECONDS/_relay_lease_until, and the is_relay_active() check inside _upload_dashcam_files()/get_status().
• templates/dash.html: the "Relay via phone" card.
• static/js/dash.js: all relay JS (relayOneFile, fetchChunkWithRetry, uploadChunkWithRetry, heartbeat timers, pending-file list rendering, etc).
• server/app.py: the CORS after_request hook, added solely for the phone's cross-origin browser calls (the Pi's own Python requests-based client was never subject to CORS - that's a browser-only enforcement mechanism).
• Kept untouched, as planned: the chunked upload protocol itself (/api/v1/dashcam/upload/chunk + /status), dashcam_hash_cache table + _hash_dashcam_file() (still used by the Pi's own background uploader to avoid rehashing its 20GB+ backlog every loop), DASHCAM_CHUNK_SIZE/Cloudflare 100MB workaround - none of that is relay-specific.

Verified: python3 -m py_compile on all touched Python files, node --check on dash.js, grepped all five touched files for any leftover relay/RELAY reference (none found). Smoke- tested by actually importing app.py in a scratch venv and hitting it with Flask's test client: app imports cleanly (GPS/OBD/SQL/upload startup all log their normal "no hardware" messages, no crash), app.url_map confirms every relay route is gone while /, /dash, /dtc, and the remaining dashcam/upload API routes are still registered, and both / and /dash render with HTTP 200 (/dash's HTML contains zero occurrences of "relay"). Deployed to both the Pi (git pull --ff-only + systemctl restart yaris-obd.service, confirmed active and /dash returns HTTP 200 with no "relay" text over the real network) and the remote server (needed a full docker compose build api && docker compose up -d api on unraid, not just a restart, since the CORS-hook removal is baked into the image at build time - confirmed /health still returns {"status":"ok"} and the response no longer carries any Access-Control-* header).

Update 2026-08-09: Phase 13 (OBD hardening) was completed next; see its section below. Phase 15 is the recommended next software phase because Phases 16/17 and the later list-based features depend on its shared UI foundation. Phase 14 remains independent but requires a careful hardware/filesystem deployment pass rather than ordinary application deployment.

Phase 13: OBD real-hardware readiness (code hardening) — ✅ DONE

Reordered from Phase 17 on 2026-08-09 because it was completed directly after Phase 12; the phase numbers now follow actual execution order instead of leaving completed work below several still-planned phases. Independent of everything else in this plan — only touches app_obd.py/app_events.py, no UI dependency for the backend part, so this can be done anytime/in parallel with the phases above. Nothing has been installed or tested in the actual 2013 Yaris yet - modules/app_obd.py has only ever been tested standalone on the Pi with nothing plugged into the OBD-II port (correctly reports "no connection to car"). Per explicit user decision, this phase is code hardening only - the user will handle the physical install/bring-up testing themselves. Planned fixes, all in modules/app_obd.py / modules/app_events.py:

• Add a genuine background reconnect loop. Today, connect_obd()/obd_connection() only ever run once per browser tab's socket connect event (app_events.py's connect_event()) - there is no periodic retry. In the real car, this means: if the page happens to load before the adapter/ignition is ready, the connection attempt fails once and never retries until another browser tab connects. Add a small polling loop (mirroring the pattern already proven in app_gps.py's _poll_loop()) that periodically calls connect_obd() while disconnected, independent of any browser being open.
• Catch exceptions from connection.query(). Currently a communication hiccup (adapter timeout, malformed response, USB blip) would raise inside obd_speed()/obd_rpm() uncaught, silently killing that polling thread forever with no recovery - not just returning a null response, an actual crash. Wrap query calls, treat exceptions like a lost connection (stop polling, let the reconnect loop above pick it back up) instead of dying silently.
• Add an OBD_PROTOCOL env override, mirroring the existing OBD_PORT override pattern - in case the 2013 Yaris's ECU doesn't cleanly auto-negotiate via ELM327's ATSP0 (auto) and needs an explicit protocol code forced (Toyotas this age are normally ISO 15765-4 CAN 11-bit 500kbaud, but this has never been confirmed against this specific car).
• Improve connection-failure diagnostics. Distinguish and log clearly: no adapter detected at all (USB/port issue) vs adapter detected but ELM327 handshake failed vs handshake OK but no response from the car's ECU (car off/ignition not on) - currently all three collapse into the same generic "OBD connection failed" message, which will make first-real-car troubleshooting much harder than it needs to be.
• Add a small connect-time diagnostic log (ELM327 firmware version + negotiated protocol, both available from python-obd's connection object) written once per successful connect - useful evidence to sanity-check against Toyota's expected protocol on the very first real test.
• Expand the data pulled from the car beyond speed/RPM (backend only - the dashboard cards displaying this land in Phase 23 once the redesign reaches them): coolant temperature, fuel level, throttle position, and DTC/MIL status (check-engine light on/off + stored trouble codes, reusing/wiring up the already-scaffolded but currently-empty /dtc page and GET_DTC command). Odometer/mileage is not guaranteed - the standard OBD-II mode 1 PID for total odometer (0xA6) was only added to the SAE J1979 spec in later revisions and is inconsistently supported across manufacturers/years; whether the 2013 Yaris's ECU exposes it at all can only be confirmed once tested against the real car (falls under "code hardening only" for now - add the query with graceful "unsupported" handling rather than assuming it will work).

Implemented 2026-08-09:

• Added an idempotent, browser-independent reconnect daemon started from app.py; it retries every 10 seconds and restarts any dead polling workers after reconnect.
• Serialized all python-obd queries so speed, RPM, and the expanded sensor worker cannot issue overlapping serial exchanges. Query exceptions and silent connection-state changes now close and discard the stale handle, emit disconnected state, and hand recovery back to the reconnect loop instead of permanently killing a worker.
• Added OBD_PROTOCOL support and distinct logs for no serial adapter, failed ELM327 handshake, and ELM327 connected but no ECU response. Successful connections log port, ELM327 firmware, negotiated protocol ID, and protocol name once.
• Added backend polling/emits for coolant temperature, fuel level, throttle position, MIL/DTC status, stored DTCs, and SAE PID 0xA6 odometer. Odometer uses a custom python-obd command with a null/unsupported result left as None; support is not assumed for the 2013 Yaris.
• Verified against the Pi's installed python-obd 0.7.3 API, plus focused mocked checks covering successful connection, forced protocol, ECU-off status, silent disconnect cleanup, query exceptions, and odometer decoding. The Pi currently has /dev/ttyUSB0 present but is not connected to the car, so actual PID availability and negotiated Toyota protocol still require the user-planned first in-car test.
• Committed as 8d7909d, pushed, and deployed to the Pi. After restart the service remained active, / returned HTTP 200, and the log showed four autonomous reconnect attempts with the correct "ELM327 handshake succeeded, but the car ECU did not respond" diagnosis and no traceback.

Phase 14: Power-loss resilience & watchdog — 🟡 PARTIAL (watchdog done; filesystem/shutdown deferred)

New phase, not code-dependent on anything else — OS/hardware-level, can be done anytime, ideally before the Pi lives in the car full-time. A car cuts power abruptly at every key-off; nothing in the project currently protects against that, and repeated unclean power loss is a classic cause of SD card corruption on embedded Pi projects. Two complementary angles:

• Filesystem hardening: convert the Pi's root filesystem to read-only with a writable overlay (overlayroot/raspi-config's built-in overlay FS option), so an abrupt power cut can no longer corrupt the OS itself. App data that must persist (db/sql/, dash/videos/, logs) needs to live on an explicitly writable partition/bind-mount carved out of the overlay, since those still need real writes.
• Clean shutdown before power loss: either (a) a small delayed-shutdown power module/circuit (holds 5V for a few seconds after ignition-off so the Pi can run shutdown -h now cleanly), or (b) if the SIM868 HAT or another ADC/GPIO source can sense ignition/voltage state, a software voltage-drop watcher that triggers a clean shutdown proactively. Exact approach TBD - depends on what's easiest to wire into the existing power setup.
• Hardware watchdog: enable the Pi's built-in bcm2835_wdt watchdog (via systemd's RuntimeWatchdogSec/watchdog-device support) so a fully hung app or kernel gets a hard reboot automatically instead of sitting dead in the car until someone notices.

Implemented 2026-08-09:

• Added deploy/yaris-watchdog.conf with systemd's hardware watchdog set to 10 seconds and deploy/install_watchdog.sh to install, activate, and verify it. Deployed to the Pi; confirmed /dev/watchdog0 is the Broadcom BCM2835 watchdog and systemd reports RuntimeWatchdogUSec=10s.
• Rebooted the Pi and verified the watchdog setting persisted, systemd returned to running, yaris-obd.service returned to active, and / returned HTTP 200 after startup.
• Explicitly deferred by user decision: read-only root/filesystem hardening and clean-shutdown hardware. The current 59GB root partition fills the SD card and contains 28GB of dashcam data, so overlay root needs a separate persistent writable storage decision first; shrinking the live mounted root filesystem is not safe. Clean shutdown likewise needs delayed-power or ignition-sense hardware before software can implement it.

Phase 15: UI design-system foundation — ✅ DONE

New phase, pulled forward from the old "do the whole redesign at the end" plan. Building the Phase 16/17 file browsers (and later the trip-history/Pi-health/settings pages) against the current old admin-template style just means rebuilding the same UI twice once the full redesign (Phase 23) happens. Instead, do a small, scoped slice of the redesign right now: a shared page layout/shell plus one reusable list/table component (sortable columns, per-row action buttons, delete-with-confirmation pattern, empty-state messaging) that every subsequent list-based feature in this plan (dashcam browser, video browser, trip history, settings) can be built on directly, instead of writing bespoke list/delete UI code from scratch each time. Not a full redesign of every existing page yet - just establishes the pattern once so nothing built after this point needs to be redone in Phase 23.

Implemented 2026-08-09:

• Added a shared Jinja shell (templates/base.html) for navigation, top bar, footer, common styles/scripts, page titles, and per-route active navigation. Converted /, /dash, and /dtc to extend it, removing their duplicated shell markup while preserving page content and page-specific assets.
• Added a reusable data-table macro (templates/components.html) and generic static/js/data-list.js behavior for dynamic empty states, numeric/text sorting, accessible sort direction, and capture-phase confirmation that can stop destructive row handlers before they execute. static/yaris.css supplies the shared list/action styling.
• Used the empty /dtc page as the first component consumer without pulling its Phase 23 OBD wiring forward. Also stopped loading dashboard-only yaris.js there and guarded its optional #obd-error target after browser testing exposed an existing null dereference.
• Verified Jinja rendering and one active nav item on all three routes, JavaScript syntax, and live Pi HTTP 200 responses. Browser-tested all pages at 1280px and 390px with no horizontal overflow; injected sample rows confirmed ascending sort, aria-sort, empty-state switching, and cancellation of a destructive handler.

Phase 16: Dashcam USB file browser (list + delete) — ✅ DONE

Replace the current all-or-nothing "Import videos" flow's blind spot (no visibility into what's actually on the camera) with a real file browser for the mounted USB dashcam device, built on the Phase 15 shared list component:

• New endpoint(s) in modules/app_dash.py to list every video file on the mounted dashcam (relative path, size, modified time) without copying anything - reuses the existing find_video_files() walk, pointed at the dashcam's mount point instead of the local destination folder.
• Delete support: individual file delete and a "delete all" bulk action, each requiring the mount to be briefly remounted read-write (currently always ro) - only for the duration of the delete call, then back to read-only (or unmounted) afterward, to minimize risk of filesystem corruption from an unclean power loss while writable.
• UI: a /dash card showing total file count + total size on the dashcam, a scrollable file list with a delete button per row, a "Delete all" button behind a confirmation dialog (real data loss risk - no undo).
• Explicitly deferred/out of scope: no in-browser preview/playback of files still on the dashcam itself (only after import - see Phase 17). No changes to the USB select-mode/data-communication toggle behavior - the user replugs power manually to return the camera to recording mode, as they already do.

Implemented 2026-08-09:

• Added list, single-delete, and delete-all backend operations using the existing USB-only storage detection. Each list/import/delete operation shares one lock so mount and file operations cannot race; listings include relative path, byte size, modified timestamp, count, and total size.
• Deletes accept only recognized video extensions, resolve and re-check every target beneath the selected mount, and reject absolute paths, traversal, missing files, and symlink escapes.
• Delete operations mount or remount the dashcam read-write only for the operation, then unmount or restore it read-only. If unmount fails, the cleanup path immediately attempts a read-only remount and reports a cleanup failure if neither safety action succeeds.
• Added /api/dash/files and /api/dash/file routes plus a /dash browser built on Phase 15's shared component: file/size/time sorting, totals, refresh, per-row confirmed delete, confirmed delete-all, empty/no-device states, and error feedback.
• Verified metadata, traversal rejection, one/all deletion, non-video preservation, import regression behavior, and cleanup-failure reporting against temporary filesystem fixtures. Deployed to the Pi; the live API correctly returned no storage devices while ignoring the attached PL2303 OBD adapter, and the responsive browser UI passed no-device and mocked populated list/sort/confirmation/delete tests.
• Real-hardware deletion verification (2026-08-09): connected the 29.7GB FAT dashcam and listed all 87 original videos (29,615,111,758 bytes) through the live API. The first controlled delete exposed that FAT files were root-owned despite the read-write remount, so unprivileged os.remove() failed; added a privileged sudo rm -- <validated-path> fallback only after the existing extension, realpath, and mount-boundary checks pass. Created and deleted a tiny disposable YARIS_OBD_DELETE_TEST.MP4 twice through the real API. Both cycles returned exactly to the original 87-file count and byte total, left every real video untouched, unmounted /mnt/dashcam, and kept the service active. Bulk delete uses the same validated per-file removal path but was not run against the real card because it would intentionally erase all footage.
• Selected import enhancement and real-hardware verification (2026-08-09): added per-row Import buttons, checkboxes, Select all/Clear selection, and Import selected so importing is no longer all-or-nothing. The backend validates every selected path with the same mount-boundary protections as deletion, de-duplicates selections, preserves relative folders, tolerates a stale/missing file while importing the remaining valid selection, and keeps the existing same-size skip behavior. Before committing, paused background dashcam uploads, created a disposable YARIS_OBD_IMPORT_TEST.MP4 on the physical card, selected it in the real mobile-width UI, and confirmed exactly one file copied with identical contents. A row-level repeat correctly skipped the existing copy; Select all selected all 88 visible rows and Clear selection reset the action. Removed both disposable copies, restored uploads, and verified the card returned to its original 87 files/29,615,111,758 bytes unmounted. The refined UI was then re-tested against a real existing video: it reported one skip, cleared the checkbox, and re-enabled all controls.

Phase 17: Local imported-video browser + player — ✅ DONE

Same file-browser treatment for videos already imported to the Pi's own disk (dash/videos/, populated by import_dashcam()), built on the Phase 15 shared list component:

• List endpoint (reuses find_video_files() against app_dash.DEFAULT_DESTINATION) showing relative path, size, modified time, and upload status (cross-check against app_sql.dashcam_file_marked_uploaded() - so it's visible at a glance which imported clips have already reached the remote server and which are still pending the Pi's own background uploader).
• Delete support: local Pi copy only - remote server copies are intentionally left alone as the safe backup/archive (explicit user decision - see 2026-08-09 discussion). No "also delete from server" option is being built for now.
• In-browser video player: since the dashcam records MP4 (confirmed with the user), this is a plain HTML5 <video> element - no server-side transcoding needed. Reuses the same Range-request-capable file-serving pattern already proven in the (now-removed) relay's /api/dash/file/<path> endpoint (needed for correct seeking/scrubbing, not just playback).
• UI: a /dash card (or a dedicated sub-page if the list gets long) with a file list, a "Play" action that opens the video inline, and a delete button per row with a confirmation prompt.
• QOL: video thumbnails. Scrolling a list of VID_00234.MOV-style filenames to find a specific clip is rough. Grab one low-res frame per file via ffmpeg (cheap, done once per file and cached, not on every page load) so the browser shows an actual thumbnail per row instead of just a filename.
• QOL: retention/auto-cleanup policy. Once a file is confirmed uploaded to the remote server, it's safe to free local space. Add an optional automatic cleanup (configurable via the Phase 21 Settings page - e.g. "delete local copies older than N days once uploaded", off by default) so the Pi's limited storage doesn't need to be babysat manually as the dashcam backlog grows.

Implemented and real-hardware verified 2026-08-09:

• Added local-video list, playback, thumbnail, and local-only delete APIs. Listing reports path, size, modified time, and exact upload status without rehashing: one SQLite join matches the existing (relative path, size, mtime) -> SHA256 cache against confirmed uploads.
• Added a second /dash browser using the Phase 15 table component, with lazy thumbnails, sortable file/size/time/upload columns, Uploaded/Pending badges, inline HTML5 video playback, confirmed "Delete local" actions, totals, and retention status. Remote copies are never deleted.
• Playback uses Flask conditional file serving; a real Range: bytes=0-1023 request returned 206, the exact Content-Range, and 1,024 bytes. Browser-tested a real H.264 1080p MOV at mobile width: the player reached readyState 4 with its duration loaded and no media error.
• Added cached ffmpeg JPEG thumbnails plus deploy/install_video_tools.sh; installed ffmpeg 5.1.9 on the Pi. Real clips exposed and fixed the edge case where ffmpeg exits 0 without output at a late seek, so generation now retries frame zero and verifies a non-empty file. A normal 366MB clip generated in ~4.1s and its cached request returned in ~81ms. Generation uses a cooperative gevent semaphore/poll loop so cold thumbnails do not freeze the web server.
• Added off-by-default local_video_retention_days support. The upload loop checks it after sync; only files older than N days whose exact current content is confirmed uploaded are eligible. The Phase 21 Settings page will expose the control; 0/unset remains disabled. Temporary filesystem tests confirmed disabled retention deletes nothing and enabled retention preserves pending files.
• Replaced a long-held upload/delete lock with per-path in-use/deleting state and reference counts: delete now returns HTTP 409 while that exact file is uploading or generating a thumbnail instead of blocking gevent and freezing every request.
• Deployed uncommitted code before committing and tested against the real 87-file/28GB Pi library: all 87 reported uploaded, total bytes matched 29,615,111,758, and retention reported off. Copied a tiny real clip to YARIS_OBD_LOCAL_TEST.MOV, verified it appeared Pending, played it in the browser, and deleted it through the confirmed UI action; the player closed and the library returned to exactly 87 uploaded files. Background uploads were paused for the disposable test, restored afterward, and the service remained active.

Phase 18: Trip history & replay — ✅ DONE

Added a derived, idempotently rebuilt trips index over the original non-destructive tracking stream, plus /api/trips, bounded /api/trip/<id> route responses, and a responsive /history page using the Phase 15 sortable list and the existing offline Leaflet bundle. Trip rows and replay details show distance in kilometres/miles and average/maximum speed in km/h and mph; speed metrics prefer OBD wheel speed and fall back to plausible GPS-derived segment speed. The map replay includes a timestamped scrubber and play/pause timeline that moves a vehicle-position marker through every recorded point while showing its date, time, and recorded speed.

Motion-boundary decision after auditing the real Pi data: the logger records every five seconds for as long as GPS has a fix, including while parked, so timestamp gaps alone are not trip boundaries. OBD wheel speed above zero is the primary motion signal. When OBD is absent or stuck at zero, three sustained GPS displacements of at least eight metres at normal logger cadence act as the fallback; isolated jumps, implausible jumps, malformed coordinates, stationary jitter, clock rollback, and five-minute motion gaps are handled explicitly. Trip summaries contain start/end time and coordinates, Haversine distance, duration, and point count. Route responses retain both endpoints, filter implausible GPS outliers, and downsample only above 2,000 points. OBD samples are used only while their last successful query is fresh, preventing a disconnected adapter's stale last-known speed from creating a false trip. Rebuilds process an overlap after the persisted high-water mark plus the latest trip rather than loading the unbounded full history every time.

Implemented and verified 2026-08-09:

• Audited the live database before choosing the heuristic: 5,132 initial rows covering about 25.5 hours, no null coordinates, all stored speeds zero, only 114 five-decimal coordinate positions, and one old NUL-prefixed latitude. Future coordinates are normalized before insertion; invalid snapshots are logged and rejected instead of poisoning trip calculations.
• Replaced unconditional five-second database writes with motion-aware persistence. Sensors are still checked every five seconds, but rows are written only for the first fix, fresh OBD motion, at least eight metres of accumulated GPS displacement, or a one-minute stationary heartbeat. The Pi normally runs only while the car has power, so this is not presented as a full-day parked storage problem; it reduces needless writes during bench testing and long idling while retaining enough stopped points for traffic pauses in the replay timeline.
• Nine focused detector tests cover sustained GPS fallback, parked jitter/malformed coordinates, inactivity splitting, OBD wheel-speed detection and slow-distance calculation, clock rollback, route outlier filtering, and route downsampling. SQLite incremental-build integration was also exercised against a disposable database, and five logger tests cover first-fix, parked, wheel-speed, GPS-fallback, and heartbeat decisions. A copy of the real database produced zero false trips across 5,179 parked rows.
• Deployed the uncommitted implementation to the Pi and built the schema against the real database without modifying any original tracking rows. Inserted an eight-point disposable GPS route while the service was stopped; the live API detected one 0.078km/35-second trip and returned all eight ordered replay points. Restored the exact pre-test database afterward, confirmed zero synthetic rows, PRAGMA quick_check=ok, zero derived trips, and an active service.
• Browser-tested http://yaris.local/history through the deployed mDNS/port-80 redirect at a 390x844 mobile viewport. Empty state, populated sortable row, replay action, offline route polyline following every route point, start/end markers, metric/imperial distance and speed summary, and no-horizontal-overflow behavior passed. Offline tiles now attach only after selecting a route, avoiding irrelevant missing-tile requests around the generic empty-map center.
• With user approval, removed the accumulated stationary bench backlog after verification: 5,454 local SQLite rows and 2,907 remotely synced Postgres rows, all with zero/null speed and no detected trip. Preserved a dated pre-cleanup SQLite backup under /home/yaris/backups/, vacuumed and integrity-checked SQLite, vacuum-analyzed Postgres, and left unrelated settings, upload markers, DTCs, and video data untouched. The deployed motion-aware logger then proved two consecutive stationary heartbeats were 60.02 seconds apart rather than five seconds.

Phase 19: Pi health monitoring card — ✅ DONE

Added a dependency-free /api/health endpoint and responsive dashboard card reporting CPU temperature, usable disk pressure/free bytes, RAM usage, human-readable uptime, and Raspberry Pi firmware power/throttling flags. Data comes from thermal sysfs, /proc, shutil.disk_usage, and vcgencmd get_throttled; individual unavailable sources are reported explicitly without hiding the remaining readings. The endpoint exposes structured current and historical flags for Phase 20 alerts, plus warning/degraded/ok status. The browser refreshes every 30 seconds.

Implemented and verified 2026-08-09:

• Five focused tests cover memory parsing, missing memory fields, current/historical throttling bits, malformed firmware output, and ext4 reserved-block-aware disk pressure. Local non-Pi validation correctly reports thermal/firmware sources as unavailable while retaining disk, memory, and uptime instead of failing the endpoint.
• Removed the old browser-triggered thread that emitted uptime once per second indefinitely; the health poll now provides uptime cheaply with the rest of the system snapshot.
• Deployed uncommitted code to the real Pi. The API returned complete data with no source errors: about 47-49°C, 19-20% RAM, 59% usable disk pressure with 22.8GB free, and roughly two hours uptime during testing. SQLite/app service behavior remained active.
• Real hardware monitoring immediately found an actionable condition: vcgencmd get_throttled alternated between 0x50005 (current undervoltage and CPU throttling, both previously occurred) and 0x50000 (historical occurrences after the current condition cleared). The dashboard prominently distinguishes current warnings from prior events. The Pi power source/cable should be corrected before sustained in-car use.
• Browser-tested the real dashboard through http://yaris.local/ at 390x844 and 1280x900: all four metrics, warning state, timestamp, responsive columns, and 30-second data path rendered without overflow or new console errors. Also fixed disconnected OBD speed showing undefined and deferred offline map tiles until a real GPS fix, avoiding irrelevant placeholder requests.

Phase 20: Alerts/notifications — ✅ DONE

Added an offline-first notification system. A shared bell in the top bar on every web page shows active local alerts with no modem, SIM card, or internet connection. Optional ntfy delivery mirrors the same conditions only when explicitly enabled and configured; it defaults off until Phase 21 exposes its topic URL, optional access token, and thresholds.

Monitored conditions:

• MIL/check-engine state and DTC codes.
• Pi filesystem pressure and the physical dashcam USB filesystem, checked read-only and cached for five minutes to avoid repeated mounts.
• Pending tracking-row or local dashcam-video upload backlog.
• Pi CPU temperature and Raspberry Pi firmware undervoltage/throttling flags.
• GPS fix loss for a configured duration while fresh OBD wheel speed confirms movement.

The alert engine runs independently every minute and persists active state, first-active time, last attempt/success, delivery errors, recovery state, and last value in SQLite. It sends once on a transition, repeats only after the configured cooldown, retries failed deliveries after five minutes, and sends a recovery notification after a delivered condition clears. GPS cold starts must remain lost for the full duration before alerting, and stale MIL/DTC values are suppressed after OBD communication stops. Firmware current flags alert immediately; new historical bits remain visible for five minutes, and cleared bits are reconciled so the same fault can alert again after reboot. Failed transient deliveries retry every minute during that window. The bell explicitly reports a dead/stale monitor instead of silently showing old conditions. /api/alerts/status exposes sanitized diagnostics without the ntfy URL, token, or request errors that could contain a private topic URL.

Implemented and verified 2026-08-09:

• Thirteen focused alert tests cover ntfy request shape/authentication/error redaction, transition deduplication, cooldown repeats, recovery, failed-delivery retry, disabled delivery, GPS duration/reset, firmware-event recurrence, diagnostic freshness, and dashcam-usage caching. The combined targeted suite is 33 tests.
• Deployed uncommitted code and verified the alert thread, state table, status API, and service on the Pi. enabled=false and configured=false produced no external attempt while local alerts continued working.
• Tested the ntfy transport against a temporary HTTP receiver bound only to 127.0.0.1 on the Pi: path, UTF-8 body, title, priority, tags, and bearer Authorization header all arrived correctly; no third-party request or real token was used.
• The real read-only physical-dashcam check found 96.1% usage/1.23GB free, unmounted afterward, and the original library remained exactly 87 files/29,615,111,758 bytes. The live bell correctly showed that storage alert alongside the real Pi undervoltage/throttling alert.
• Browser-tested the real local bell/dropdown at 390x844 on the dashboard and trip-history page: badge counts, alert titles/details, no-alert state, synthetic monitor-failure state, shared-shell availability, and no-horizontal-overflow behavior passed with no new console errors.

Phase 21: Settings page — ✅ DONE

New phase - depends on Phase 15's shared components, and benefits from Phases 17/20 existing so there's real configuration to consolidate. Several settings currently only exist as SSH-edited files or scattered individual toggles (dashcam upload pause switch on /dash, cellular APN in db/upload_creds.env, hotspot SSID/password via nmcli directly) or don't exist yet (retention days from Phase 17, alert webhook/thresholds from Phase 20). Add one dedicated Settings page consolidating anything reasonably safe to expose over the local-only web UI: upload toggle, cellular APN, local video retention policy, alert webhook URL + thresholds. WiFi SSID/password changes likely stay out of scope (higher-risk to expose over a UI that could lock the Pi out of its own network if fat-fingered) unless it turns out to be easy/safe to add later.

Implemented and hardware-tested 2026-08-09. Added a dedicated local /settings page and validated /api/settings endpoint backed by the existing SQLite settings table. It consolidates the dashcam-upload toggle, uploaded-video retention days, SIM868 cellular enable/APN, optional ntfy enable/URL/token, and alert temperature/disk/backlog/GPS-loss/repeat thresholds. Updates are validated as a complete group and committed atomically; invalid requests leave the prior configuration untouched. Cellular enable/APN reads persisted settings live, so those changes do not need an app restart. Sensitive ntfy URL/token values are write-only through the API: the page only reports whether each is configured and supports explicit removal, so saved secrets are never sent back to browser JavaScript. Local Web UI alerts remain active independently of ntfy.

When cellular is configured, ntfy delivery now uses the SIM868's small-body HTTP client as the primary transport with the notification title, priority, tags, and optional bearer token. WiFi/Ethernet is the fallback if the modem path fails. The generalized modem POST path rejects quotes/newlines in AT-command parameters before touching hardware. This is mock/Pi-code tested but remains explicitly unverified over a real carrier until a SIM is available; Phase 25 must prove authenticated HTTPS, fallback, retry, and recovery behavior or remove ntfy entirely if the SIM868 cannot deliver it reliably.

Moved the dashcam-upload control from /dash into Settings rather than keeping duplicate controls, while retaining pending-row, last-success/path, and last-error upload health in the new page. WiFi credentials, remote-server device/API credentials, cellular username/password, and other lockout/high-risk configuration remain file/SSH managed by design. Focused settings/alert tests pass locally and on the Pi; the real Pi rejected an invalid update with HTTP 400 while preserving its exact configuration. The live page loaded through yaris.local with the Pi's existing defaults and shared alert/navigation shell. Browser testing at 390x844 found and fixed an initial top-bar overlap by adding a dedicated bottom mobile save action; the real save then completed with no horizontal overflow or console errors.

Phase 22: Phone-hotspot internet uplink — ✅ DONE

Add a no-SIM internet path where the Pi joins a user's phone hotspot as a WiFi client and uses the phone's cellular data for outbound synchronization. This must work with an ordinary iPhone or Android hotspot and support tracking rows, alert events/state, software/server checks, and large dashcam-video uploads without requiring a modem SIM.

The Pi currently creates its own hotspot for the local Web UI, so this phase must explicitly solve the single-radio networking tradeoff rather than assuming it can be an access point and hotspot client simultaneously:

• Add a Settings workflow to save/replace/remove a phone-hotspot SSID and password without exposing the saved password back to browser JavaScript. Validate before applying and provide a recovery path so a bad credential cannot permanently lock the user out of yaris.local.
• Prefer a deterministic NetworkManager design: when the configured phone hotspot is available, connect as a client for internet access; when it is unavailable or association fails, restore the Pi-hosted hotspot automatically within a bounded timeout. Preserve the existing boot/network watchdog behavior and document the exact priority/retry rules.
• Because a Pi 3 single WiFi interface cannot reliably host and join separate networks at once, test and document how the user reaches the local Web UI while connected to the phone hotspot. Prefer http://yaris.local/ on the shared phone LAN when the phone permits peer traffic. If iOS/Android client isolation or mDNS behavior prevents that, keep the automatic Pi-hotspot fallback and present clear current-mode/status guidance rather than promising simultaneous access.
• Surface current uplink mode, connected SSID without password, IP address, internet reachability, last transition/error, and whether large video upload is permitted. Never mistake association with the phone for proven internet reachability.
• Keep dashcam video off SIM868 cellular, but allow it through the phone hotspot's WiFi path. Tracking, server alert synchronization, and other queued data must resume idempotently after the phone hotspot appears. Existing cellular-first ntfy behavior remains separate.
• Hardware-test with at least one real phone: cold boot with hotspot present/absent, hotspot enable/disable while running, wrong credentials, phone leaving/re-entering range, internet loss while still associated, automatic Pi-hotspot recovery, local yaris.local access, tracking catch-up, and a disposable chunked video upload. Record phone/OS limitations before committing.

Implemented and hardware-tested 2026-08-09 with a real Pixel hotspot.

• Added a root-owned yaris-uplink NetworkManager helper, systemd timer/wait service, dispatcher transition recorder, restricted sudoers entries, and Settings APIs/UI. Priority is phone-hotspot (20), home-wifi (10), then the Pi Hotspot (-10). A non-disruptive scan checks that the phone SSID is actually visible before replacing a working home connection.
• The intended single-phone workflow is save credentials first → select Wait for hotspot → enable the phone hotspot. The wait service gives the user ten seconds before dropping the Pi AP, scans for up to two minutes, and always restarts the periodic watcher. If the phone never appears or later disappears, NetworkManager/helper recovery selects home WiFi when available and otherwise restores the Pi hotspot. Multiple dead-man tests independently forced home WiFi recovery.
• Phone credentials go from browser to the privileged helper over stdin and into an atomic root-only (0600) NetworkManager keyfile. Passwords never enter SQLite, command arguments, status files, HTML/JavaScript responses, or read APIs. The UI exposes only configured state and SSID, supports replace/remove, and preserves in-progress edits during status polling.
• Status reports actual connection/mode/IP, NetworkManager-proven internet connectivity, whether large uploads are currently possible, last transition/error, and the last phone-assigned direct address. The non-secret transition record persists across reboot under /var/lib/yaris-obd.
• Pi-hosted hotspot testing initially found that the real phone could open 10.42.0.1 but not yaris.local even though Avahi correctly joined/published on wlan0. Added a NetworkManager shared-dnsmasq unicast fallback for yaris.local; the repeated dead-man test then passed.
• Pixel tethering has a different limitation: the hotspot-host phone can reach the Pi by its DHCP address, but relays neither mDNS (yaris.local) nor the DHCP hostname (yaris). Settings therefore retains and links the last direct phone-network address with a DHCP-change warning. The tested Pixel repeatedly assigned the same address, but the implementation does not assume it is permanent.
• Proved cold boot both with Pixel absent (selected home WiFi) and present (selected Pixel directly, Web UI/service started, and the server received a fresh heartbeat). Turning Pixel off repeatedly restored home WiFi in about 16 seconds. The same fallback code is covered for restoring the Pi hotspot when home WiFi is unavailable.
• Proved real tracking uploads through Pixel data against remote Postgres. The first disposable 1 MiB video exposed repeated TLS EOF failures over tethering despite small requests working; setting only the phone profile MTU to 1400 fixed the path, after which a second 1 MiB disposable video reached the real server over Pixel data. Both named local/server files and exact database upload/hash rows were removed afterward, and dashcam uploads were restored enabled.
• The future Phase 24 alert-event synchronization queue will automatically use this normal IP uplink; Phase 22 does not prematurely implement that server schema/API.

Phase 23: Finish web UI redesign — ✅ DONE

Completed 2026-08-10 (commit 981b24b). Full responsive redesign of the Pi web UI.

What was done:

• Replaced the SB Admin 2-style fixed left sidebar with a purpose-built layout:
  • Desktop (≥768 px): dark fixed sidebar (var(--sidebar-width) = 14 rem) with collapsible icon-only mode (4 rem). State persisted via localStorage. Logo is a functional home link.
  • Mobile (<768 px): no sidebar. Full-width content with a sticky bottom tab bar (5 items: Dashboard, Dashcam, History, DTC, Settings). Logo shown in the topbar.
• Removed every placeholder element: "Placeholder button", stub dropdown menus ("Action" / "Another action"), inline style colour hacks, and meaningless card-header dropdowns.
• Dashboard now shows 6 OBD stat cards: OBD status, speed, RPM, coolant temp, throttle, fuel level — all wired to live socket events (obd_coolant_temp, obd_fuel_level, obd_throttle added to yaris.js).
• GPS info section replaced with clean lat/lon display.
• DTC page fully wired to a new /api/dtc route (returns {mil, codes} from app_obd.get_fresh_mil() / get_fresh_dtc_codes()). MIL indicator pill + DTC table populated from the API on load and on manual refresh.
• yaris.css completely rewritten: CSS custom properties, sidebar, bottom nav, OBD cards, border-start-* utilities.
• theme.js rewritten: localStorage-backed collapse toggle; no Bootstrap Collapse dependency.
• All 63 tests still pass. Verified on Pi at both 390×844 (mobile) and 1280×800 (desktop).

Phase 24: Remote trip + dashcam portal with shared web UI — 🚧 PLANNED

Added 2026-08-09; intentionally scheduled after Phase 23. Phase 23 first establishes the final visual language and complete in-car interface. Building the remote portal afterward avoids duplicating the current unfinished theme and lets both deployments deliberately share the same finished shell, components, trip list, route map, timeline replay, dashcam library, units, and responsive behavior.

Extend the existing self-hosted server/ deployment from an upload-only API into a secure remote web portal backed by its Postgres tracking data and uploaded dashcam archive, so trip history, recorded trip details, and footage can be accessed away from the car:

• Add server-side trip boundary/index data equivalent to Phase 18, generated from uploaded tracking points without changing or deleting the raw records. Expose trip summaries, full/downsampled routes, timestamps, distance in km/mi, average/max speed in km/h and mph, and timeline replay.
• Reuse the redesigned Phase 23 Jinja/Bootstrap UI rather than creating a visually separate app. Extract/share presentation assets where practical so local and remote pages do not drift, while keeping their data providers and available actions explicit.
• Make the uploaded dashcam archive a required first-class remote page, matching the redesigned in-car imported-video library: sortable/searchable file list, relative path, size, timestamp, thumbnails, inline H.264 playback with Range/seeking support, and an explicit download action for individual original files. Downloads must stream from server storage with safe Content-Disposition, bounded/traversal-safe paths, and no whole-video memory buffering. Remote deletion is not included unless it is designed later with separate safeguards; the server remains the archive/backup.
• Put an always-visible environment indicator in the shared shell: In-car / Local Pi on yaris.local, and Remote server on the hosted portal. Page title, navigation, and status styling must also make the context unambiguous so the user cannot mistake remote historical data for a live connection to the car.
• The remote portal is historical/read-only. It should feel like the same product, but render only server-relevant navigation and data—not redacted/empty imitations of unavailable hardware. Physical dashcam mount/import/delete, live OBD/GPS, cellular status/configuration, Pi health, and local Pi settings are omitted entirely. Trip history/replay and the uploaded dashcam archive are the required remote scope.
• Add real human-user authentication and secure sessions for the internet-facing UI. Existing device X-Device-Name/X-Api-Key upload authentication remains separate and must not become a browser login mechanism. Preserve Cloudflare Tunnel compatibility and do not expose secrets in HTML/JavaScript.
• Support selecting a vehicle/device if more than one is registered later, while defaulting cleanly to the current Yaris. Keep trip APIs scoped to the authenticated user/device.
• Synchronize alert transitions, recoveries, timestamps, severity/details, and current state from the Pi to the server through an authenticated, idempotent offline queue. The Pi remains authoritative for vehicle/hardware detection and its local bell must never depend on internet access. The server stores per-vehicle alert history/current state, reconciles reconnects without duplicates, and exposes it in the remote portal; remote acknowledgement must not silently clear the underlying Pi condition.
• Synchronize alert transitions, recoveries, timestamps, severity/details, and current state from the Pi to the server through an authenticated, idempotent offline queue. The Pi remains authoritative for vehicle/hardware detection and its local bell must never depend on internet access. The server stores per-vehicle alert history/current state, reconciles reconnects without duplicates, and exposes it in the remote portal; remote acknowledgement must not silently clear the underlying Pi condition.
• Deploy only after Phase 23 is complete: rebuild the unraid Docker stack, verify migrations against copied data first, compare remote trip calculations with uploaded Pi records, verify playback, Range requests, and downloads against disposable copies of real server footage, and browser-test desktop/mobile through the real hosted endpoint before committing.

Phase 25: Pre-install Yaris 2013 compatibility test — 🚧 PLANNED (core-system gate)

Added 2026-08-09; intentionally immediately before the Android app. Bench tests, synthetic routes, attached-hardware tests, and stationary Pi runs cannot prove the complete system works when temporarily connected to and powered by the actual 2013 Toyota Yaris. This phase is the pre-installation acceptance gate for the complete core Pi/web/server system after Phase 24. It is verification and targeted correction—not permanent installation or another redesign—and the system is not considered ready to install until the results are recorded here. Permanent mounting, cable routing, trim work, and final installation are explicitly outside the roadmap and will be handled by the owner after this phase passes.

Prepare a temporary, removable test setup before driving:

• Connect the Pi, SIM868 HAT/antennas, USB OBD adapter, dashcam, and representative intended power source without permanently mounting or routing anything. Secure every temporary component and cable so it cannot interfere with pedals, steering, airbags, visibility, occupants, or loose-item safety during the controlled drive; remove the setup after testing.
• Resolve the Phase 19/20 undervoltage/throttling warning before the test; confirm vcgencmd get_throttled shows no current power fault at idle and under load with the temporary representative power setup. This validates electrical compatibility, not final cable routing.
• Back up the Pi SQLite database and record baseline row/trip/video/upload counts on the Pi and remote server. Use a uniquely timestamped test window so generated records can be reconciled without deleting unrelated history.
• Verify the service, watchdog, storage mounts, hotspot, http://yaris.local/, alert engine, GPS antenna view, dashcam recording mode, free space, and remote server health before moving.
• Use a second person for UI/log observations while the vehicle is moving, or record diagnostics for review afterward. The driver must not operate or troubleshoot the UI while driving.

Run a controlled route containing ignition-off/on, stationary idle, low-speed streets, normal road speed, several complete stops, turns, a short GPS-obstructed segment if safely available, and a clean trip end:

• OBD/ECU: confirm the ELM327 reaches CAR_CONNECTED, record negotiated Toyota protocol and adapter firmware, and verify real speed/RPM/coolant/fuel/throttle/MIL/DTC values. Compare wheel speed against the instrument cluster at several safe speeds. Record unsupported PIDs (especially odometer PID A6) as capabilities, not failures, and verify reconnect after ignition cycling and a deliberate USB disconnect/reconnect.
• GPS and motion logging: confirm fixes while stationary and moving, route coordinates on the offline map, OBD-speed-first motion detection, GPS displacement fallback when OBD is deliberately unavailable, one-minute idle heartbeat behavior, and no false trip from stationary GPS drift.
• Trip boundary/replay: after the inactivity timeout, verify exactly one trip (or the expected number for deliberate ignition gaps), correct start/end time and coordinates, plausible distance/average/max speed, km/mi and km/h/mph values, route polyline, start/end markers, timeline scrubber, play/pause marker position, and stopped portions of the route.
• Local web UI: test the final Phase 23 interface on the actual phone over the Pi/phone network at http://yaris.local/; verify the explicit In-car / Local Pi identity, dashboard telemetry, DTC page, trip history, dashcam library, settings, alerts, responsive layout, reconnect behavior, and no new browser console/API errors.
• Dashcam workflow: create a known short real clip, switch the camera safely into USB storage mode, detect/list it without confusing the OBD adapter, import only that selected file, verify thumbnail/playback/Range seeking, preserve the source until checks finish, and confirm the camera returns to recording afterward. Test deletion only with the disposable clip.
• Alerts: safely exercise conditions that can be simulated without damaging hardware (upload backlog, temporary GPS loss while OBD reports motion, and disposable threshold overrides). Confirm local bell transition/recovery/cooldown behavior and offline alert-state reconciliation with the Phase 24 server after connectivity returns. Do not intentionally overheat, undervolt, fill storage, or trigger a real MIL. With a real SIM, explicitly determine whether the SIM868 transport can reliably deliver ntfy notifications while WiFi/Ethernet is unavailable, including authentication, timeout, retry, and recovery. If that cannot be made reliable on the actual modem/network, remove the ntfy feature and its settings rather than retain a transport that fails in the only situation where it is useful.
• Power/reliability: perform multiple normal ignition/power cycles, confirm automatic boot, service/watchdog startup, SIM868 power-on, OBD/GPS recovery, no SQLite corruption, and no current undervoltage/throttling. The deferred Phase 14 clean-shutdown limitation must be explicitly accepted or resolved with representative delayed-power/ignition-sense hardware before declaring the design ready for permanent installation.

Verify the complete remote path after connectivity returns:

• Confirm every test-window tracking row uploads exactly once and the selected dashcam clip resumes or completes without corruption; compare counts, timestamps, hashes, and byte sizes locally and remotely.
• On the Phase 24 portal, verify the explicit Remote server identity, authenticated vehicle-scoped trip summary/replay, route and timeline agreement with the local Pi, archived clip thumbnail/playback/Range seeking/download, synchronized alert history/current state without duplicate transitions after an offline interval, and absence of local-only OBD/cellular/Pi controls.
• The Android companion is intentionally outside this core acceptance gate and remains the final, low-priority Phase 26. Its own hotspot/cellular binding, trip display, and disposable clip relay require separate real-phone/in-car acceptance when that phase is implemented.

Acceptance requires a written result for every item, captured negotiated protocol/PID support, measured trip/count/hash comparisons, relevant logs/screenshots, no unexplained tracebacks or data loss, and cleanup of every disposable record/file while preserving the real drive. Any defect found must be fixed, redeployed uncommitted, and the affected in-car test repeated before the final commit and ready-to-install sign-off. Afterward, the owner handles permanent installation outside the phase roadmap.

Phase 26: Native Android companion app — 🚧 PLANNED (low priority, always last)

Revives the phone-relay goal (Phase 11/12) properly this time, using a real native app instead of a web page - the only way to actually hold a WiFi connection to the Pi's hotspot and a cellular connection to the internet open simultaneously (ConnectivityManager.bindProcessToNetwork(), unavailable to browser JS - see the Phase 11 update). Explicitly not urgent and intentionally kept at the end of the roadmap, after all web/server work and the Phase 25 core in-car acceptance test are done. Permanent vehicle installation remains owner-handled and is not an implementation phase. Scope, at a high level (not yet designed in detail):

• Dashcam file transfer: browse/select/delete dashcam clips already imported to the Pi (same data as the Phase 17 local video browser, just a native client instead of/alongside the web UI), download them to the phone, and upload to the remote server over cellular while the phone stays on the Pi's hotspot for the download leg - the true simultaneous dual-network relay the web version could never deliver.
• Trip/location info: surface live GPS position, speed, and trip history (Phase 18) pulled from the Pi while connected to its hotspot (reusing the existing /api/gps-style data feeds), so the app is useful as a real "check on the car" companion, not just a file-transfer tool.
• Not yet scoped: distribution (sideload vs Play Store), whether it talks to the Pi directly over the hotspot only or also to the remote server directly for trip history when away from the car, and how much of the existing REST API surface can be reused as-is vs needs new mobile-friendly endpoints. To be designed once Phases 12-24 (web-side cleanup + hardening, redesign, remote portal, and core in-car acceptance) are done.

Working Notes / Environment Gotchas

• Pi credentials/SSH: ssh yaris@<pi-ip>. Confirm current IP before connecting (varies by network — check locally, not recorded here) — usually reachable via yaris.local too (mDNS, see Phase 6).
• Local dev repo: work happens on a separate dev machine, not on the Pi. Push to origin/shameless-aislop, the repo's current default branch (repo is public).
• unraid server SSH: ssh [email protected] (alias unraid). Real git checkout lives at /mnt/user/app/git/yaris-obd (not /mnt/user/appdata/..., which is only the docker-managed data volume for the db/dashcam containers). Deploy with cd /mnt/user/app/git/yaris-obd && git pull --ff-only && cd server && docker compose up -d --build. Container names: yaris-obd-db (Postgres 17), yaris-obd-api.
• Pi deploy loop: ssh yaris@<pi-ip> (alias yarisobd) -> cd ~/app/yaris-obd && git pull --ff-only -> sudo systemctl restart yaris-obd.service (passwordless sudo confirmed via yaris ALL=(ALL) NOPASSWD: ALL). App stdout/stderr goes to ~/app/yaris-obd/yaris-obd.log (not journald - the systemd unit redirects both streams there), so use tail/grep on that file, not journalctl, to see application-level prints (upload progress, OBD/GNSS errors, etc.) - journalctl -u yaris-obd.service only shows service start/stop/crash lifecycle events. Update 2026-08-09: plain print() calls were silently invisible in that log for months because Python fully-buffers stdout when it's not a TTY (a file, in this case) — logging-based lines (e.g. the obd library's own logger, which uses stderr) showed up fine, but bare print() diagnostics from app_gps.py never did. Fixed by adding Environment=PYTHONUNBUFFERED=1 to the systemd unit (deploy/yaris-obd.service) — remember to also copy the updated unit file to /etc/systemd/system/yaris-obd.service and run sudo systemctl daemon-reload on the Pi, a plain git pull + restart does NOT pick up systemd unit file changes (the live /etc/systemd/system/ copy is independent of the repo's deploy/ copy).
• Pi deploy loop, requirements.txt changes: after any requirements.txt change (e.g. adding RPi.GPIO), also run ~/app/yaris-obd/venv/bin/pip install -r requirements.txt on the Pi — a git pull alone doesn't install new dependencies into the venv.
• Pi memory budget: ~900MB usable RAM, no swap. Any code path that might buffer a whole dashcam-sized file (hundreds of MB) in memory is unsafe on this hardware - stream, don't buffer (see Phase 10).
• Git push auth: if git push fails with a credential/auth error, run gh auth login (device-code browser flow) then gh auth setup-git to wire git to use the gh CLI's stored credentials — this fixed push auth issues in this environment without needing SSH keys or PATs.
• Pi environment gap: Debian Bookworm blocks system-wide pip install (PEP 668, "externally-managed-environment"). Use a venv for the real app dependency install (Phase 5) rather than --break-system-packages everywhere. (Resolved in Phase 5 — venv at /home/yaris/app/yaris-obd/venv.)
• Pi → GitHub auth for git pull (private repo): the Pi's pre-existing ~/.ssh/id_ed25519.pub is an inbound key only (private half is on the dev machine) — can't be reused for the Pi's own outbound git auth. Generated a dedicated keypair on the Pi (~/.ssh/yaris_deploy_key), added it as a GitHub read-only deploy key on this repo, and added an ~/.ssh/config entry pinning github.com to that key. [email protected]:... (SSH remote) then works standalone on the Pi, no PAT/interactive login needed.
• piwheels: pip on the Pi automatically uses piwheels.org as an extra index (Raspberry Pi OS default pip.conf) — this is why gevent (which normally needs a C compile) installed in seconds from a prebuilt ARM wheel instead of taking many minutes to build from source.
• python-obd note: obd.OBD() (or with portstr=) performs the ELM327 handshake at connection time; if there's no car on the other end, it raises rather than silently returning a dead handle — app_obd.py's connect_obd() already catches this and sets connection = None.