yaris-obd/PLAN.md

yaris-obd Revival Plan

This document tracks the plan to revive and finish this project: a Raspberry Pi 3-based OBD-II + GPS car tracker for a 2013 Toyota Yaris, hosting a local web UI over a self-broadcast WiFi hotspot, with dashcam file retrieval and remote upload of itinerary/dashcam data.

Keep this file up to date as phases complete or plans change, so any future session/agent can pick up the project without needing prior chat history.

Hardware

• Board: Raspberry Pi 3 Model B Rev 1.2, 906MB RAM. Confirmed too weak to run an AI coding agent — all agent-assisted dev work happens on a separate local machine; the Pi is only used for hardware bring-up/testing/deployment via SSH (yaris@<pi-ip>).
• GPS/GSM: Waveshare SIM868 GSM/GPRS/GNSS/Bluetooth HAT. Also has unused cellular (GSM/GPRS) data capability, earmarked as a fallback internet uplink (see Remote Server phase).
• OBD-II adapter: a cheap unbranded USB adapter using a PL2303 USB-to-serial chip, presumed ELM327-compatible (works with the python-obd library). Originally the project used a Bluetooth OBD adapter; this has been switched to USB.
• Car: 2013 Toyota Yaris.

SIM868 HAT bring-up notes (important gotchas)

• UART routing is a 3-position jumper: A = USB↔SIM868 (via onboard CP2102), B = Pi GPIO↔SIM868, C = USB↔Pi console. Must be on B for the Pi to talk to the module over its own GPIO UART.
• Pi-side UART setup required: enable_uart=1 and dtoverlay=disable-bt in /boot/firmware/config.txt (frees the full PL011 UART — /dev/ttyAMA0//dev/serial0 — from the onboard Bluetooth controller), raspi-config nonint do_serial_cons 1 (stops a login shell from claiming the serial port), and disable the hciuart systemd service.
• The SIM868 module chip itself needs its PWRKEY button pressed and held (~1-2s) to power on — this is separate from the board's 5V/PWR LED, which can be lit while the module chip is fully off. AT commands get zero response if the module chip isn't powered, even with perfect UART wiring/config. This is a manual step needed on every power cycle unless a P4 GPIO-power-on hardware mod (per Waveshare FAQ) is done — not pursued so far.
• Confirmed AT command baud rate: 115200.
• GNSS workflow: AT+CGNSPWR? to check GNSS power state, AT+CGNSPWR=1 to power on, AT+CGNSINF to poll for a fix. Response format: +CGNSINF: run,fix,utc_time,lat,lon,alt,speed,course,fixmode,reserved1,hdop,pdop,vdop,reserved2,gps_sat_view,gnss_sat_used,gnss_sat_view,cn0,hpa,vpa — field 1 (fix) must be '1' for a valid fix.
• The GPS/GNSS antenna is separate from the GSM antenna — both need to be physically connected to their respective u.FL connectors on the HAT. A real fix was obtained only after discovering the GPS antenna wasn't plugged in (only GSM was).
• Confirmed real fix obtained near a window: ~80s cold-start TTFF, 5-7 satellites, HDOP ~1.2-1.6.

Software Architecture Decisions

1. Offline maps: Pre-cache/download offline map tiles for the car's usual driving area so the live map works fully in the car with no internet (not just numeric GPS readout). Tooling: MapTiler Cloud free tier (raster XYZ tiles, streets-v2 style) — one-time download on the dev machine via scripts/download_map_tiles.py, output shipped to the Pi as static PNGs. Bulk-downloading directly from tile.openstreetmap.org was ruled out — it violates the OSM tile usage policy. Bounding box is a required CLI arg (the user's own driving area, not committed here), zoom 8-15 by default (~14k tiles for a city-plus-suburbs-sized area, well under MapTiler's 100k/month free quota).
2. WiFi hotspot mode: Auto-switch — the Pi joins home WiFi when in range (for syncing/updates), and falls back to broadcasting its own AP otherwise. Important scope clarification: the hotspot is only for local web-UI viewing on a phone/laptop while driving — it is not part of the upload path. Remote-server uploads happen independently via home WiFi (when in range) or the SIM868 cellular fallback (decision #5) — the phone connecting to the hotspot never relays/carries upload traffic.
3. Local/remote persistence: Keep SQLite locally (reliable local write path per trip). The original plan paired this with MongoDB Atlas for remote sync (modules/app_db.py, fixed in Phase 7) — superseded by decision #4/Phase 9: the actual remote store is the self-hosted Postgres server, uploaded via modules/app_upload.py. app_db.py's Mongo path is not wired into app.py and is effectively dead code at this point, kept only for reference.
4. Remote server: Self-hosted — a custom server/API (e.g. on a home server or VPS) receives itinerary and dashcam uploads, rather than 3rd-party managed cloud services.
5. Remote upload uplink: Use the SIM868's GSM/GPRS cellular connection as a fallback uplink for uploads when no WiFi is available (e.g. while driving), not just "sync when on home WiFi."

Phases

Phase 1: Real GPS integration — ✅ DONE

modules/app_gps.py rewritten to read real SIM868 GNSS data via serial AT commands (/dev/serial0 @ 115200 baud) instead of hardcoded fake coordinates. Background daemon thread polls AT+CGNSINF every 2s, parses fix data, emits gps_location via socketio only on an actual fix. Verified against real hardware (see bring-up notes above).

Phase 2: OBD fixes for USB adapter — ✅ DONE

modules/app_obd.py fixed (commit "Fix app_obd.py bugs and adapt for USB OBD-II adapter", pushed to origin/shameless-aislop):

• connection is never left undefined on failed connect (was a NameError crash).
• Added _find_obd_port()/connect_obd(): auto-detects /dev/ttyUSB* (the PL2303 adapter), honors an OBD_PORT env override, falls back to python-obd's own port auto-scan (still covers the old /dev/rfcomm* Bluetooth path).
• Fixed stale-response bug: obd_speed()/obd_rpm() now re-query every loop iteration instead of reusing one response forever.
• Fixed obd_rpm() using the wrong lock, never updating the global rpm, and emitting to the wrong socketio event (obd_speed instead of obd_rpm).
• Fixed kill_obd() being a no-op — now uses threading.Event flags actually checked in the polling loops.
• Fixed inverted thread-start logic (threads were starting on failed connection, not success).
• Fixed obm_error → obd_error typo (frontend listens for obd_error).
• Added obd (python-obd) to requirements.txt.
• Still untested: a real connection to the actual car (adapter has only been tested standalone on the Pi so far — correctly reports "no connection to car" with nothing attached).

Phase 3: Web UI must work fully offline — ✅ DONE

The whole point of the project is a Pi hotspot with no internet in the car, but the current UI silently depended on the internet for core functionality:

• templates/index.html/dash.html/dtc.html loaded Google Fonts, FontAwesome, Leaflet JS/CSS, and the Socket.IO client from public CDNs — none of this loads offline, breaking layout, the websocket connection itself, and the map entirely.
• static/js/yaris.js loaded live map tiles from tile.openstreetmap.org (also needs internet).
• A local static/js/chart.min.js existed but was unused — index.html loaded Chart.js from cdn.jsdelivr.net instead (turned out the local copy was a stale Chart.js v2.8.0 while yaris.js's config uses v3/v4-style scales: {x:{...}, y:{...}} syntax — that mismatch is exactly why the CDN was used instead. Replaced the local file with a current v4.5.0 UMD build).

What was done:

• Vendored Socket.IO client (4.8.1, matches Flask-SocketIO 5.6.1's Engine.IO v4 protocol) under static/vendor/socketio/, and Leaflet (1.9.4 JS/CSS + marker/layers images) under static/vendor/leaflet/.
• FontAwesome 5.12.0 was already vendored locally at static/fonts/ (self-consistent, just never referenced) — repointed all three templates to it instead of the CDN.
• Dropped the Google Fonts (Nunito) <link> entirely rather than self-hosting 12 font-weight files — Bootstrap's --bs-font-sans-serif variable already falls back cleanly to system fonts (-apple-system, Segoe UI, Roboto, ...) after Nunito, so this is a clean no-cost simplification.
• Replaced static/js/chart.min.js with a current Chart.js v4.5.0 UMD build (auto-registering, same drop-in behavior as the old CDN <script src="https://cdn.jsdelivr.net/npm/chart.js">).
• Offline map tiles: added scripts/download_map_tiles.py, a dev-machine-only one-time tile downloader using the MapTiler Cloud free tier (raster XYZ, streets-v2 style, 256px tiles). Reads MAPTILER_KEY from an env var/CLI arg only — the key itself is never committed or deployed, only the resulting static PNG tiles are. Bounding box is a required CLI arg (your own driving area's coordinates - not committed/hardcoded) at zoom 8-15 (~14k tiles for a city-plus-suburbs-sized area, well under MapTiler's 100k tiles/month free quota). Output goes to static/vendor/tiles/{z}/{x}/{y}.png (gitignored — regenerate locally, rsync to the Pi as part of deployment, don't bloat the git repo with binary tile data).
  • Tiles populated: user signed up for a free MapTiler key and the full zoom 8-15 download ran to completion — 14,241 downloaded, 24 skipped (out-of-bounds), 0 failed (14,265 tiles total, ~285MB). Rsynced to the Pi's static/vendor/tiles/ (tile counts verified identical on both sides). Live Pi instance re-confirmed serving HTTP 200 with the complete offline tile set.
• Updated static/js/yaris.js's Leaflet init to point at /static/vendor/tiles/{z}/{x}/{y}.png (minZoom/maxZoom 8/15 to match). The initial map center is just a generic placeholder (not a real location) - it re-centers automatically to the real live GPS fix once one comes in, and the marker is exposed as carMarker so Phase 4 can move it on live fixes.
• Added a repo-root .gitignore (Python caches, venvs, .vscode/, static/vendor/tiles/, *.env except the already-tracked db/creds.env placeholder) — there wasn't one before, which is why stray __pycache__/.pyc files kept showing up in source control.
• Verified: ran the Flask app in a local venv, confirmed all three pages (/, /dtc, /dash) render with zero external URLs in the served HTML, every local static asset (JS, CSS, fonts, Leaflet images) returns HTTP 200, and all vendored JS files pass a Node.js syntax check.

Phase 4: Map/GPS display bug — ✅ DONE

yaris.js's Leaflet marker (carMarker) was never moved on incoming gps_location socket events — only two text spans (gpsLocationX/Y) got updated. Fixed: carMarker.setLatLng(...) is now called on every gps_location event, and the map re-centers (zoom 15) on the first fix only (mapCentered flag), so the user isn't fighting the map re-centering under them while panning/zooming during a trip.

Verified: also used this as an opportunity to test Phase 3 end-to-end — ran scripts/download_map_tiles.py for real with the user's actual MapTiler API key (smoke-tested zoom 8-10 first, confirmed idempotent re-run skips existing tiles, confirmed a downloaded tile is a real map image of the target driving area), then started the full zoom 8-15 download. Took a headless Chromium screenshot of the running app's / page and confirmed: sidebar/icons/fonts render correctly (FontAwesome working), the Speed Graph (Chart.js v4) renders, and the Leaflet map renders real offline map tiles with the marker in place — full offline end-to-end proof, not just "no CDN URLs in the HTML".

• Fixed (was noted here as out of scope, now addressed in Phase 8): /dash and /dtc used to render near-identical body content (both showed "DTC Codes") — dash.html now has real dashcam-specific content.
• Full tile download completed (see Phase 3 update above) and rsynced to the Pi — Phase 3/4's offline map is now 100% functional end-to-end on the actual deployed device, not just the dev machine.

Phase 5: Pi deployment / autostart — ✅ DONE

Set up a proper venv (/home/yaris/app/yaris-obd/venv), installed all requirements (all found as prebuilt ARM wheels via piwheels — no slow source compiles needed, even for gevent). Added deploy/yaris-obd.service (systemd unit: runs as the yaris user, Restart=on-failure, 5s ExecStartPre sleep to let USB/serial devices enumerate after boot, logs to yaris-obd.log) and deploy/install_service.sh (idempotent one-time install helper — also creates the venv if missing). Installed and enabled on the Pi (systemctl enable — survives reboot).

Also had to fix Pi → GitHub auth for git pull: the repo is private, and the Pi's existing ~/.ssh/id_ed25519.pub turned out to be an inbound SSH key (its private half lives on the dev machine, not the Pi) — useless for the Pi to authenticate outbound. Generated a dedicated ed25519 keypair on the Pi itself (~/.ssh/yaris_deploy_key), registered it as a read-only deploy key on the GitHub repo (least-privilege — the Pi only ever needs to pull), and added an ~/.ssh/config entry so git transparently uses it for github.com. Switched the Pi's origin remote to SSH. git pull now works standalone on the Pi with no PAT/interactive auth needed.

Verified end-to-end: pulled latest code onto the Pi (was 5 commits stale), confirmed all Python deps import cleanly, ran the app manually first (HTTP 200, all local static assets — Leaflet/Socket.IO/Chart.js/FontAwesome — served correctly from the Pi itself), then installed + started the systemd service and confirmed active (running) + enabled. Tested crash recovery: sent SIGKILL to the running process, confirmed systemd auto-restarted it (through the 5s ExecStartPre delay) and the app was serving HTTP 200 again within ~10s, with zero manual intervention.

Phase 6: WiFi hotspot mode — ✅ DONE

Used NetworkManager (already the network manager in use, not hostapd/dnsmasq directly — those were leftover/unused) since it natively supports AP mode via nmcli. Created two connection profiles on the Pi: home-wifi (your real home WiFi network, autoconnect-priority 10) and Hotspot (AP mode, ipv4.method shared, its own dedicated SSID/password set locally on the Pi - not committed here, WPA2-PSK, autoconnect-priority -10). NM's own autoconnect logic handles "leaving home WiFi range -> fall back to hotspot" automatically since home-wifi has higher priority and NM always tries it first.

For the reverse direction (home WiFi back in range while sitting in AP mode) added deploy/yaris-netwatch.sh + .service/.timer: since the Pi's onboard BCM43438 radio can't be an AP and background-scan for client networks simultaneously, a systemd timer runs every 3 minutes, briefly attempts reconnecting to home-wifi, and restores the hotspot if home WiFi still isn't in range. Installed via deploy/install_netwatch.sh.

Root-caused and fixed a real bug during testing: the hotspot's SSID broadcast worked immediately, but its DHCP server initially failed (dnsmasq: failed to create listening socket for 10.42.0.1: Address already in use) — a pre-existing orphaned system-wide dnsmasq.service (noted as "likely unused" in earlier investigation) was already bound to the address space NetworkManager's own internal dnsmasq needs for the hotspot. Confirmed the system dnsmasq's config was genuinely empty/default (nothing in this project depends on it) and disabled it (systemctl disable --now dnsmasq).

Also added easy access without needing to know the Pi's IP:

• Renamed the Pi's hostname from yarisobd to yaris — avahi-daemon (already installed and running) now publishes it as yaris.local via mDNS.
• Port 80 -> 5000 redirect via nftables (deploy/nftables-yaris.conf + install_port_redirect.sh, loaded through an include in /etc/nftables.conf), so the app is reachable at plain http://yaris.local/ (no port number) instead of :5000.

Verified on real hardware, very carefully since the SSH session used to do all this testing itself runs over wlan0: used a "dead-man's-switch" pattern for every risky test — a detached background command (setsid nohup bash -c 'sleep N && nmcli con up home-wifi' & disown) armed before each hotspot activation, so the Pi automatically reverts to home WiFi (and SSH access comes back) even if something goes wrong, no physical access needed. Confirmed: hotspot broadcasts and a real phone gets a genuine DHCP lease from NetworkManager's own DHCP server and can load the web UI; yaris-netwatch correctly detects AP mode and reconnects to home WiFi when in range (and is a no-op when already on home WiFi); http://yaris.local/ and http://<pi-lan-ip>/ both return HTTP 200 with no port number from another machine on the network. (Real hotspot SSID/password and the Pi's LAN IP are intentionally not recorded in this file — see local, non-committed notes.)

Phase 7: app_db.py / app_sql.py fixes — DONE

Neither file was called from app.py or anywhere else - orphaned from a prior iteration. Fixed:

• app_db.py: applied a prior-session stash fix - resolves .env path via __file__ (was cwd-relative), fixes os.getenv() calls to pass the actual key names (was called with zero args, an immediate TypeError if ever invoked), added a clear RuntimeError guard if credentials are still empty placeholders, returns client[mDB].
• app_sql.py: rewritten - resolves db/sql/ via __file__ + os.makedirs (was cwd-relative), fixed create_conn() closing the connection immediately and returning None (so table creation never ran and every caller got a dead connection), added get_conn() (lazy singleton
  • table creation for both tracking and dtc tables), insert_tracking(), and a background start_logging()/_log_loop() thread (5s interval) that snapshots GPS (app_gps.LocationX/Y)
  • OBD speed (app_obd.speed) into the local tracking table. Used a deferred (function-local) import of app_obd inside _log_loop() to keep the pre-existing app_obd<->app_sql top-level circular import safe/explicit.
• Also fixed a Python 3.12+/3.14 compat bug found during testing: sqlite3.version (the sqlite3 module's version) was removed/deprecated - switched to sqlite3.sqlite_version (the actual SQLite library version), which is what was intended.
• Wired app_sql.start_logging() into app.py alongside the other startup calls (app_gps.start_gps(), app_obd.OBD()).
• db/sql/ added to .gitignore (per-device local trip data, not source).
• Verified locally: get_conn() creates db/sql/<account>_<car>.db with both tables; insert_tracking() writes a real row; start_logging() thread starts and logs "SQL tracking-logger thread started, alive: True"; full app.py run confirms the thread starts cleanly and (with no real GPS/OBD hardware on the dev machine) correctly logs read errors instead of crashing; app_db.get_database() raises the new clear RuntimeError instead of crashing, since db/creds.env still holds empty placeholder credentials (real Mongo credentials are a user-provided follow-up, out of scope here).

Phase 8: Dashcam retrieval (app_dash.py) — ✅ DONE, verified on real hardware

Rewrote app_dash.py to be brand/device-agnostic: detection via lsblk -J, matching partitions whose parent disk reports TRAN == "usb" (this is what safely distinguishes a plugged-in dashcam/USB stick from the Pi's own onboard storage, which is always mmc transport - no more hardcoded /dev/sda). Walks the whole mounted filesystem for video file extensions (.mp4/.mov/.avi/.ts/.mkv) instead of assuming a specific brand's folder layout (DCIM, MOVIE, NORM/EVENT/PARKING, etc. all just work), preserves each file's relative sub-folder path when copying (avoids filename collisions across dashcam event/date folders), and skips files already copied (same relative path + size) so re-running an import is cheap. Mounts read-only to avoid any risk of corrupting the dashcam's SD card. Dropped the unused/uninstalled tqdm CLI-progress dependency (never actually in requirements.txt) since this now runs headless inside the always- on web server, not interactively.

Wired into app.py: GET /api/dash/devices (list detected USB storage) and POST /api/dash/import (run the mount->copy->unmount pipeline), both JSON. Fixed the pre-existing /dash vs /dtc duplicate-template bug — dash.html now has real dashcam content (scan/import buttons + live status) instead of a copy-pasted "DTC Codes" body. Added a dedicated dash.js (not the shared yaris.js, which unconditionally initializes the /index-only map/chart and would throw/crash on /dash and /dtc since those pages don't have those DOM elements - a pre-existing bug, worked around by not depending on that shared file for this page's new JS). Added dash/videos/ to .gitignore (imported footage is per-device data, not source).

Tested thoroughly without the physical dashcam (user didn't have it on hand this session):

1. Unit-tested the lsblk-JSON detection/filtering logic against a realistic synthetic sample (Pi's own mmcblk0 boot+root partitions plus a simulated USB dashcam partition) - confirmed it correctly finds only the USB partition and ignores the Pi's own storage.
2. Integration-tested the real mount -> find-videos -> copy (preserving structure) -> dedupe -> unmount pipeline on the actual Pi using a loop-device-backed FAT32 image (mkfs.vfat + losetup) populated with fake nested dashcam video files (DCIM/100EVENT/..., DCIM/100NORMAL/...) - confirmed real mount/copy/unmount all work, confirmed a second import run correctly skips already-copied files, confirmed non-video files (e.g. thumbs.db) are ignored. All simulation artifacts (loop device, test image, temp files) cleaned up afterward.
3. Verified the full /dash page + both new API routes locally (Flask dev run + headless browser check): auto-scan on page load correctly reports "No USB dashcam/storage device detected" (accurate, since none is plugged into the dev machine), Import button correctly stays disabled until a device is found, no JS console errors, no regression on / or /dtc.

Verified against the real physical dashcam (Jieli-chipset USB ID 1224:2725, 29.7GB SD card, DCIM/1/VID_###.MOV layout, 87 files, ~28GB total). This first live hardware test found and fixed three real bugs the loop-device simulation couldn't have caught:

1. Superfloppy SD card - the card has no partition table at all (lsblk reports the whole /dev/sda disk directly as fstype: vfat, no child partition) - the original detection assumed a partition would always exist. Fixed to also match a TRAN == "usb" disk with its own fstype set directly, not just partitions under it.
2. Mount permission error - mount_device() called os.makedirs() directly, but the service runs as unprivileged yaris and /mnt is root-owned. Fixed to sudo mkdir -p <mount_point> before the existing sudo mount, mirroring that pattern.
3. OOM crash during upload (see Phase 10 below) - not a copy-path bug, but discovered immediately after the first successful real import.

End-to-end result: full 28GB/87-file import from the SD card completed successfully (resumed cleanly after being interrupted once by the OOM crash below - copy_files()'s skip-if-already- copied logic meant re-running /api/dash/import just picked up the remaining files, no data loss or re-copy of the first 19 files). SD card unmounted cleanly afterward.

Phase 9: Remote server for itinerary/dashcam upload — DONE

Per architecture decisions #4 and #5: self-hosted server/API to receive itinerary (from the SQLite pipeline) and dashcam file uploads.

Decided/built: Flask API + dedicated Postgres, deployed as a docker-compose stack on the user's unraid server (separate from any existing containers/DBs there), living in server/ in this same repo. Auth: simple X-Device-Name/X-Api-Key header pair per device, key hash stored server-side, real key never committed (raw key printed once by server/scripts/create_device.py, saved locally on the Pi only, in gitignored db/upload_creds.env). External access: exposed at https://yaris.ceaf.dev via the user's existing Cloudflare Tunnel (public-hostname rule added in the Cloudflare dashboard, pointing at 192.168.10.10:8098) — router port-forward + the user's Unifi firewall is the documented fallback if ever needed instead. Large dashcam uploads can still prefer a local-network path (e.g. an AdGuard DNS override to bypass the tunnel while on home WiFi) since tunnels typically cap per-request upload size — noted in server/README.md, not yet needed/built.

Built (server/ folder):

• server/sql/init.sql — Postgres schema: devices (name + sha256 api-key-hash), tracking_points (device_id FK, GPS/speed/odometer + recorded_at, unique per device+timestamp so retried batches can't duplicate rows), dashcam_files (device_id FK, relative_path + sha256, unique per device+path+hash so re-uploads are no-ops).
• server/modules/db.py — psycopg3 connection pool + query helpers (bulk tracking insert with ON CONFLICT DO NOTHING, dashcam dedupe-check-before-write).
• server/modules/auth.py — require_device_auth decorator checking the two headers against the stored key hash.
• server/app.py — Flask app: GET /health (unauthenticated), POST /api/v1/tracking/batch (bulk JSON insert, capped at 5000 points/request as a sanity guard), POST /api/v1/dashcam/upload (multipart, streamed to disk with sha256 verification + path-traversal guard on relative_path).
• server/Dockerfile + server/docker-compose.yml — api (gunicorn, published on host port 8098 by default) + db (Postgres 17, internal-only, not published to the LAN) services, data volume defaults to server/data/ but pointed at /mnt/user/appdata/yaris-obd-server on unraid.
• server/.env.example — template for POSTGRES_PASSWORD/API_PORT/DATA_DIR/upload size cap.
• server/scripts/create_device.py — run once per device inside the api container to register it and print its one-time API key. Ran once for device yaris-2013.
• server/README.md — full unraid deployment + Cloudflare Tunnel exposure + device-registration instructions.

Pi-side upload client (modules/app_upload.py):

• Background thread, uploads unsent tracking rows (batches of 500) and not-yet-uploaded dashcam files (sha256-verified) via the API above, with exponential retry/backoff on failure.
• Independent dashcam-upload-only pause toggle (persisted settings table + a switch on the /dash web UI) — trip/GPS logs always keep uploading regardless of the toggle, per explicit user request (useful on a slow/metered connection).
• Backlog-drain tuning: since tracking rows log every 5s locally, an offline/stale device can queue up a large backlog. The loop drains back-to-back (short 1s pause between batches) while a batch comes back full, only settling into the slow 30s idle cadence once actually caught up — a simulated 1250-row backlog now drains in 3 batches (~0.35s) instead of 13 batches over ~6.5 minutes with the original fixed 100-row/30s-sleep design.
• modules/app_sql.py extended: UPLOADED column + migration on tracking, settings table, dashcam_uploads dedupe table, and helper functions backing all of the above.

Tested end-to-end against the real deployed server (tracking batch + dedupe, dashcam upload + dedupe + path-traversal rejection, bad-API-key rejection, toggle-off correctly pausing only dashcam uploads) and on the real Pi (its actual ~1800-row pre-feature backlog drained and confirmed matching row counts on the server via direct psql query, live upload status visible on /dash). Committed+pushed (4ff99f0, c4db1ab, 8dc304e), deployed on both unraid and the Pi. Exposed externally at https://yaris.ceaf.dev via the user's Cloudflare Tunnel (public hostname added in the Cloudflare dashboard, pointing at 192.168.10.10:8098).

Phase 10: Dashcam upload memory/size fixes — ✅ DONE, verified on real hardware

Two real-hardware-only bugs surfaced once actual ~300MB+ video files (not small test fixtures) hit the upload path for the first time:

1. OOM crash from in-memory multipart uploads. modules/app_upload.py originally uploaded dashcam files via requests.post(..., files={"file": f}). The requests library builds the entire multipart/form-data body in memory before sending when using files= - a well-known gotcha, not specific to this codebase. On the Pi 3's ~900MB RAM, uploading one ~320MB video this way spiked the process past 600MB+ RSS, triggering the kernel OOM killer within minutes (confirmed via dmesg: repeated Out of memory: Killed process ... anon-rss:823928kB, and via pmap -x showing a single ~362MB anonymous mapping matching the file size almost exactly). Fixed: added requests-toolbelt and rewrote the upload body construction to use MultipartEncoder, which streams the file in chunks as the request is sent instead of materializing it up front. Confirmed via an isolated 300MB test (628MB RSS old way -> 29MB new way, byte-identical result) and live on the Pi (RSS stayed flat ~55-59MB during real uploads, vs. previously crashing every few minutes).
2. Cloudflare Tunnel's hard ~100MB request-body cap. Even after the memory fix, real uploads still failed once routed through https://yaris.ceaf.dev - a bare 413 Payload Too Large from Cloudflare itself (confirmed: the identical file succeeds direct-LAN, 413s only through the tunnel). This is a free-tier platform limit, unrelated to MAX_DASHCAM_UPLOAD_MB on the Flask side. Fixed via a chunked/resumable upload redesign (well under the 100MB cap):
   • Client (modules/app_upload.py): DASHCAM_CHUNK_SIZE = 40MB; new _upload_video_chunked() checks /status first (enables resume across restarts/failures), then POSTs missing chunks one at a time via MultipartEncoder.
   • Server (server/app.py): new GET /api/v1/dashcam/upload/status (reports already_uploaded or received_chunk_indices) and POST /api/v1/dashcam/upload/chunk (stages chunks under STORAGE_DIR/.upload_tmp/<device>/<sha256>/, reassembles + verifies SHA256 once all chunks arrive, records via the existing db.insert_dashcam_file, cleans up the staging dir). A .total_chunks marker guards against stale/mismatched partial sets if the chunk size ever changes between client versions. The original single-shot /api/v1/dashcam/upload endpoint is untouched, kept for reference/compatibility.

Verified end-to-end against the real deployed server through the real Cloudflare tunnel, first with a synthetic interrupted-then-resumed 250MB test (status transitions not_started -> partial upload -> in_progress w/ correct received_chunk_indices -> resume -> complete -> already_uploaded, SHA256 match confirmed on disk), then with real dashcam footage: the Pi copied all 87 real files locally and the background upload thread drained the real backlog one file at a time, each one verified byte-for-byte (SHA256 match Pi vs. server) and correctly recorded in Postgres, while the Pi's service ran 50+ minutes straight with RSS staying in a bounded ~55-141MB range (no OOM kills) - a stark contrast to the previous crash-every-few-minutes behavior. Given home-upload-bandwidth constraints, the full 28GB backlog drains at roughly one ~350MB file per 4-5 minutes and is expected to take several hours total - this runs unattended in the background with no user action needed once started.

Not yet built: parallel/concurrent chunk uploads (currently strictly sequential, one chunk then one file at a time) - would speed up the multi-hour full-backlog drain but adds complexity; not requested by the user yet, flagged here as a possible future optimization if the sequential pace becomes annoying in practice.

Known manual step (not yet automated): this dashcam model boots into a "select mode" menu (dashcam / mass-storage / webcam) when it senses USB data lines are live, and needs USB data communication physically absent to boot straight into recording mode, but present when plugging into the Pi for import. No code in this repo handles that switch - it's a manual physical step by the user (e.g. a data-blocking USB cable/adapter, or plugging/unplugging appropriately) each time before recording vs. before importing. Worth revisiting if it becomes tedious in daily use (e.g. a Pi-controlled USB data-line switch), but out of scope for now.

SIM868 cellular fallback for tracking uploads — built, UNTESTED (no SIM card available)

modules/app_cellular.py gives the tracking-upload path a fallback over the SIM868's cellular data connection when WiFi/Ethernet is unreachable. Design notes:

• Uses the SIM868's built-in AT-command HTTP(S) client (AT+HTTPINIT/HTTPPARA/HTTPACTION/ HTTPREAD) rather than a real ppp0 network interface. Reason: the SIM868's GNSS (polled continuously by modules/app_gps.py) and its GSM/GPRS radio share the same physical UART (/dev/serial0) — a classic pppd/chat dial-up would put that port into raw PPP framing for as long as the link is up, starving the GNSS polling thread. Staying in AT-command mode lets cellular HTTP calls interleave safely through the same serial lock (modules.app_gps._serial_lock) — each exchange just briefly delays the next GPS poll.
• Only carries small JSON tracking batches, never dashcam video — cellular data is far too slow/limited for multi-MB video files, and the modem's HTTP AT interface has a small internal buffer unsuited to them anyway. Dashcam uploads remain WiFi/Ethernet-only, unchanged.
• modules/app_upload.py's tracking-batch upload always tries WiFi/Ethernet (requests) first; only on a connection-level failure (no route/DNS/timeout — as opposed to an HTTP error status, which means the server WAS reached) does it retry the same batch over cellular.
• Feature-flagged off by default: YARIS_CELLULAR_ENABLED/YARIS_CELLULAR_APN/ YARIS_CELLULAR_APN_USER/YARIS_CELLULAR_APN_PASS added to db/upload_creds.env, all blank by default so this has zero effect on the already-verified WiFi upload path until the user fills in an APN once they get a SIM card. Regression-tested (against the real server) that normal WiFi uploads are completely unaffected while cellular is unconfigured.
• /dash upload-status card now shows which path (wifi/cellular) the last successful batch used, and whether cellular fallback is configured.
• Needs validation once a SIM card is available — exact AT command dialect/timing (SIMCOM firmware revisions vary slightly), and whether AT+HTTPSSL works for the HTTPS yaris.ceaf.dev endpoint on this specific module's firmware (if not, a documented fallback is a plain-HTTP router port-forward path, avoiding the need for the modem's TLS stack to validate a cert chain).

Phase 11: Phone-relay dashcam upload — ✅ DONE

Root cause investigation into "why is the transfer from the Pi to the server so slow?" (real dashcam backlog draining at only ~350MB/4-5min) found the bottleneck is the Pi's own onboard BCM43438 WiFi chip (2.4GHz-only, congested/interfered channel) — confirmed via iperf3 raw-TCP tests showing only ~14.7Mbit/s with active retransmits even over a same-subnet direct-LAN hop that never touches the internet/Cloudflare, ruling out ISP/Cloudflare limits. Since the Pi 3 B has no 5GHz radio and can't be fixed in software, added a way to bypass its weak WiFi for the internet-bound leg entirely: a phone opens the same /dash page (over the Pi's hotspot/home WiFi) and manually relays pending dashcam files straight to the remote server using the phone's own network (WiFi or cellular data), instead of routing everything through the Pi.

Zero new remote-server upload logic — the phone's browser JS calls the exact same /api/v1/dashcam/upload/status + /api/v1/dashcam/upload/chunk endpoints the Pi's own Python client already uses, in the same 40MB-chunk protocol, just orchestrated by fetch()/FormData instead of requests/MultipartEncoder. The server dedupes purely by SHA256 regardless of which device sends the bytes, so no server-side changes were needed for the upload path itself — only a CORS after_request hook (server/app.py) was added, since the phone's browser calls yaris.ceaf.dev from a page served by a different origin (the Pi), which browsers block without Access-Control-Allow-* headers on both the real request and its preflight OPTIONS. Wildcard origin is fine here — auth is header-based (X-Device-Name/X-Api-Key), not cookie/session-based.

New Pi-side endpoints (app.py):

• GET /api/dash/pending — lists not-yet-uploaded dashcam files (relative_path, sha256, size) via a new app_upload.list_pending_dashcam_files().
• GET /api/dash/file/<path:relpath> — serves a dashcam file's bytes for the phone to download, via send_file(..., conditional=True) for Range-request support (206 Partial Content) so the phone downloads (and immediately re-uploads) one 40MB slice at a time instead of ever holding a whole 300+MB file in the browser's memory — the exact same memory-safety lesson learned the hard way in Phase 10, applied symmetrically to the download side. Path-traversal safe via Werkzeug's safe_join against app_dash.DEFAULT_DESTINATION.
• GET /api/dash/relay/config — hands the phone the same SERVER_URL/DEVICE_NAME/API_KEY the Pi's own client uses, so its JS can auth directly with the remote server. Same trust boundary as the rest of this project's unauthenticated local UI (anyone on the Pi's own hotspot/LAN is already implicitly trusted).
• POST /api/dash/relay/heartbeat / POST /api/dash/relay/stop — a lease-based pause (RELAY_LEASE_SECONDS = 45) for the Pi's own background dashcam uploader, chosen over a plain on/off toggle specifically to self-heal if the phone's browser tab closes/loses connection uncleanly (a simple toggle could get stuck "paused" forever). The phone page heartbeats every 15s while relaying; _upload_dashcam_files() skips entirely while a lease is active, re-checked between files so it stops promptly if a relay starts mid-drain. Also released via navigator.sendBeacon() on beforeunload as a best-effort immediate release.
• POST /api/dash/relay/mark_uploaded — lets the phone tell the Pi "I just uploaded this file" right after a successful relay, so the local dedupe table (and the pending list / Pi's own background loop) reflect it immediately instead of waiting for the next full backlog scan.

Performance side-fix: modules/app_upload.py's dashcam loop previously recomputed SHA256 for every candidate file on every loop iteration, including files long since uploaded — wasteful CPU/IO against a growing, possibly 20GB+, local backlog. Added a dashcam_hash_cache SQLite table (modules/app_sql.py, keyed by relative_path+size+mtime) so a file's hash is only computed once; both the existing background uploader and the new pending-files endpoint benefit.

UI (templates/dash.html / static/js/dash.js): a new "Relay via phone" card on /dash lists pending files with a per-file "Relay" button and a "Relay all pending" button (manual trigger only — chosen deliberately over an automatic one, per explicit user preference), plus a "Stop relay" button. Chunk downloads/uploads retry up to 3 times with backoff before surfacing an error, and a resumed relay re-checks /status first so a previously-interrupted file picks up from its first missing chunk instead of restarting.

Tested: a full local end-to-end simulation (no physical phone/network needed) drove the real Pi app.py and real server/app.py Flask apps side by side (Postgres mocked at the db module boundary only — all HTTP routing, chunking, auth, and CORS logic is real), reproducing the exact sequence the phone's browser JS performs: list pending -> check remote /status -> download each chunk from the Pi via Range requests (verified real 206 Partial Content responses) -> upload each chunk to the mock-backed server -> verify the reassembled file matches the original byte-for-byte via SHA256 -> mark uploaded on the Pi -> confirm it drops out of the pending list. Also verified: path-traversal attempts on /api/dash/file/ correctly 404, the relay heartbeat/stop lease correctly pauses/resumes _upload_dashcam_files(), and the sha256 cache table is populated and reused across repeated /api/dash/pending calls.

Deploy gotcha found: the CORS fix in server/app.py was initially deployed to the Pi but not rebuilt on the actual remote server (the server's Docker image bakes app.py/modules/ in at build time via COPY, so a plain git pull + container restart isn't enough — needs docker compose build api && docker compose up -d api). Without it, the phone's cross-origin fetch() calls to yaris.ceaf.dev were silently blocked by the browser's CORS policy, which surfaces in JS as a generic, unhelpful TypeError: Failed to fetch — worth remembering as the first thing to check if this error reappears after a future server-side change.

Tested for real against a real phone and the real deployed server (first end-to-end proof beyond the local simulation above): the initial /api/dash/pending call took ~25 minutes for a one-time full-backlog SHA256 hash of all 87 files (~28GB, no cache existed yet) — after that, repeat calls returned in ~0.13s. The user then successfully relayed a real file from their phone (home WiFi) partway through (10/~12 chunks) before stopping to check something important: is the relay actually faster, given the phone downloads from the Pi over the same congested radio diagnosed as the bottleneck? Measured directly from chunk-arrival timestamps on the server: ~1.11MB/s (~8.9Mbit/s) for the phone-relay path — statistically the same as the ~9-10Mbit/s measured for the Pi's own direct Cloudflare upload earlier this session. Conclusion: when the phone is on the same home WiFi as the Pi, the relay provides no speed benefit — the download-to-phone leg pays the exact same slow-radio cost as a direct Pi upload would, and the phone's reupload leg then goes out through the same home internet connection anyway, adding an extra hop for no gain. The relay only has genuine value when the phone's second leg differs meaningfully from the Pi's own path — e.g. cellular data with better upload than the home ISP, or uploading after physically leaving home with files already grabbed locally. Stopped the test relay (POST /api/dash/relay/stop) and confirmed the Pi's own background uploader resumed the same file's chunked upload exactly where the phone left off (shared /status/.upload_tmp state on the server made this seamless, no wasted re-transfer).

Next idea (not yet built): the user wants a way to browse/preview individual dashcam clips on the phone and select specific ones to view or upload (as opposed to bulk-relaying the entire backlog) — useful for "just grab this one important clip" rather than a wholesale speed optimization. Not scoped/designed yet.

Working Notes / Environment Gotchas

• Pi credentials/SSH: ssh yaris@<pi-ip>. Confirm current IP before connecting (varies by network — check locally, not recorded here) — usually reachable via yaris.local too (mDNS, see Phase 6).
• Local dev repo: work happens on a separate dev machine, not on the Pi. Push to origin/shameless-aislop, the repo's current default branch (repo is public).
• unraid server SSH: ssh [email protected] (alias unraid). Real git checkout lives at /mnt/user/app/git/yaris-obd (not /mnt/user/appdata/..., which is only the docker-managed data volume for the db/dashcam containers). Deploy with cd /mnt/user/app/git/yaris-obd && git pull --ff-only && cd server && docker compose up -d --build. Container names: yaris-obd-db (Postgres 17), yaris-obd-api.
• Pi deploy loop: ssh yaris@<pi-ip> (alias yarisobd) -> cd ~/app/yaris-obd && git pull --ff-only -> sudo systemctl restart yaris-obd.service (passwordless sudo confirmed via yaris ALL=(ALL) NOPASSWD: ALL). App stdout/stderr goes to ~/app/yaris-obd/yaris-obd.log (not journald - the systemd unit redirects both streams there), so use tail/grep on that file, not journalctl, to see application-level prints (upload progress, OBD/GNSS errors, etc.) - journalctl -u yaris-obd.service only shows service start/stop/crash lifecycle events.
• Pi memory budget: ~900MB usable RAM, no swap. Any code path that might buffer a whole dashcam-sized file (hundreds of MB) in memory is unsafe on this hardware - stream, don't buffer (see Phase 10).
• Git push auth: if git push fails with a credential/auth error, run gh auth login (device-code browser flow) then gh auth setup-git to wire git to use the gh CLI's stored credentials — this fixed push auth issues in this environment without needing SSH keys or PATs.
• Pi environment gap: Debian Bookworm blocks system-wide pip install (PEP 668, "externally-managed-environment"). Use a venv for the real app dependency install (Phase 5) rather than --break-system-packages everywhere. (Resolved in Phase 5 — venv at /home/yaris/app/yaris-obd/venv.)
• Pi → GitHub auth for git pull (private repo): the Pi's pre-existing ~/.ssh/id_ed25519.pub is an inbound key only (private half is on the dev machine) — can't be reused for the Pi's own outbound git auth. Generated a dedicated keypair on the Pi (~/.ssh/yaris_deploy_key), added it as a GitHub read-only deploy key on this repo, and added an ~/.ssh/config entry pinning github.com to that key. [email protected]:... (SSH remote) then works standalone on the Pi, no PAT/interactive login needed.
• piwheels: pip on the Pi automatically uses piwheels.org as an extra index (Raspberry Pi OS default pip.conf) — this is why gevent (which normally needs a C compile) installed in seconds from a prebuilt ARM wheel instead of taking many minutes to build from source.
• python-obd note: obd.OBD() (or with portstr=) performs the ELM327 handshake at connection time; if there's no car on the other end, it raises rather than silently returning a dead handle — app_obd.py's connect_obd() already catches this and sets connection = None.