1400 lines
110 KiB
Markdown
1400 lines
110 KiB
Markdown
# yaris-obd Revival Plan
|
||
|
||
This document tracks the plan to revive and finish this project: a Raspberry Pi 3-based
|
||
OBD-II + GPS car tracker for a 2013 Toyota Yaris, hosting a local web UI over a self-broadcast
|
||
WiFi hotspot, with dashcam file retrieval and remote upload of itinerary/dashcam data.
|
||
|
||
Keep this file up to date as phases complete or plans change, so any future session/agent can
|
||
pick up the project without needing prior chat history.
|
||
|
||
## Hardware
|
||
|
||
- **Board**: Raspberry Pi 3 Model B Rev 1.2, 906MB RAM. Confirmed too weak to run an AI coding
|
||
agent — all agent-assisted dev work happens on a separate local machine; the Pi is only used
|
||
for hardware bring-up/testing/deployment via SSH (`yaris@<pi-ip>`).
|
||
- **GPS/GSM**: Waveshare **SIM868 GSM/GPRS/GNSS/Bluetooth HAT**. Also has unused cellular
|
||
(GSM/GPRS) data capability, earmarked as a fallback internet uplink (see Remote Server phase).
|
||
- **OBD-II adapter**: a cheap unbranded USB adapter using a **PL2303** USB-to-serial chip,
|
||
presumed ELM327-compatible (works with the `python-obd` library). Originally the project used
|
||
a Bluetooth OBD adapter; this has been switched to USB.
|
||
- **Car**: 2013 Toyota Yaris.
|
||
|
||
### SIM868 HAT bring-up notes (important gotchas)
|
||
- UART routing is a 3-position jumper: **A** = USB↔SIM868 (via onboard CP2102), **B** = Pi
|
||
GPIO↔SIM868, **C** = USB↔Pi console. Must be on **B** for the Pi to talk to the module over
|
||
its own GPIO UART.
|
||
- Pi-side UART setup required: `enable_uart=1` and `dtoverlay=disable-bt` in
|
||
`/boot/firmware/config.txt` (frees the full PL011 UART — `/dev/ttyAMA0`/`/dev/serial0` — from
|
||
the onboard Bluetooth controller), `raspi-config nonint do_serial_cons 1` (stops a login shell
|
||
from claiming the serial port), and disable the `hciuart` systemd service.
|
||
- **The SIM868 module chip itself needs its PWRKEY button pressed and held (~1-2s) to power on**
|
||
— this is separate from the board's 5V/PWR LED, which can be lit while the module chip is
|
||
fully off. AT commands get zero response if the module chip isn't powered, even with perfect
|
||
UART wiring/config. **Update 2026-08-09: this HAT revision DOES have the "P4" GPIO-power-on
|
||
mod wire installed** (per the Waveshare FAQ/`GSM_PWRKEY.rar` script) — confirmed by pulsing
|
||
physical header pin 7 (BCM GPIO4) LOW for 4s then HIGH while the module was unresponsive, after
|
||
which it immediately answered `AT`. This is now automated in `modules/app_gps.py`
|
||
(`_ensure_module_powered()`), which checks `AT` first and only pulses PWRKEY if the module
|
||
isn't already responding (PWRKEY is a toggle — pulsing an already-on module would turn it back
|
||
off, so this check-first behavior is required). No more manual button-pressing needed after a
|
||
Pi power cycle.
|
||
- Confirmed AT command baud rate: **115200**.
|
||
- GNSS workflow: `AT+CGNSPWR?` to check GNSS power state, `AT+CGNSPWR=1` to power on,
|
||
`AT+CGNSINF` to poll for a fix. Response format:
|
||
`+CGNSINF: run,fix,utc_time,lat,lon,alt,speed,course,fixmode,reserved1,hdop,pdop,vdop,reserved2,gps_sat_view,gnss_sat_used,gnss_sat_view,cn0,hpa,vpa`
|
||
— field 1 (`fix`) must be `'1'` for a valid fix.
|
||
- **The GPS/GNSS antenna is separate from the GSM antenna** — both need to be physically
|
||
connected to their respective u.FL connectors on the HAT. A real fix was obtained only after
|
||
discovering the GPS antenna wasn't plugged in (only GSM was).
|
||
- Confirmed real fix obtained near a window: ~80s cold-start TTFF, 5-7 satellites, HDOP ~1.2-1.6.
|
||
|
||
## Software Architecture Decisions
|
||
|
||
1. **Offline maps**: Pre-cache/download offline map tiles for the car's usual driving area so
|
||
the live map works fully in the car with no internet (not just numeric GPS readout).
|
||
Tooling: MapTiler Cloud free tier (raster XYZ tiles, `streets-v2` style) — one-time download
|
||
on the dev machine via `scripts/download_map_tiles.py`, output shipped to the Pi as static
|
||
PNGs. Bulk-downloading directly from `tile.openstreetmap.org` was ruled out — it violates the
|
||
OSM tile usage policy. Bounding box is a required CLI arg (the user's own driving area, not
|
||
committed here), zoom 8-15 by default (~14k tiles for a city-plus-suburbs-sized area, well
|
||
under MapTiler's 100k/month free quota).
|
||
2. **WiFi hotspot mode**: Auto-switch — the Pi joins home WiFi when in range (for
|
||
syncing/updates), and falls back to broadcasting its own AP otherwise. **Important scope
|
||
clarification**: the hotspot is *only* for local web-UI viewing on a phone/laptop while
|
||
driving — it is not part of the upload path. Remote-server uploads happen independently via
|
||
home WiFi (when in range) or the SIM868 cellular fallback (decision #5) — the phone connecting
|
||
to the hotspot never relays/carries upload traffic.
|
||
3. **Local/remote persistence**: Keep SQLite locally (reliable local write path per trip). The
|
||
original plan paired this with MongoDB Atlas for remote sync (`modules/app_db.py`, fixed in
|
||
Phase 7) — **superseded by decision #4/Phase 9**: the actual remote store is the self-hosted
|
||
Postgres server, uploaded via `modules/app_upload.py`. `app_db.py`'s Mongo path is not wired
|
||
into `app.py` and is effectively dead code at this point, kept only for reference.
|
||
4. **Remote server**: Self-hosted — a custom server/API (e.g. on a home server or VPS) receives
|
||
itinerary and dashcam uploads, rather than 3rd-party managed cloud services.
|
||
5. **Remote upload uplink**: Use the SIM868's GSM/GPRS cellular connection as a fallback uplink
|
||
for uploads when no WiFi is available (e.g. while driving), not just "sync when on home WiFi."
|
||
|
||
## Phases
|
||
|
||
### Phase 1: Real GPS integration — ✅ DONE
|
||
`modules/app_gps.py` rewritten to read real SIM868 GNSS data via serial AT commands
|
||
(`/dev/serial0` @ 115200 baud) instead of hardcoded fake coordinates. Background daemon thread
|
||
polls `AT+CGNSINF` every 2s, parses fix data, emits `gps_location` via socketio only on an actual
|
||
fix. Verified against real hardware (see bring-up notes above).
|
||
|
||
### Phase 2: OBD fixes for USB adapter — ✅ DONE
|
||
`modules/app_obd.py` fixed (commit "Fix app_obd.py bugs and adapt for USB OBD-II adapter",
|
||
pushed to `origin/shameless-aislop`):
|
||
- `connection` is never left undefined on failed connect (was a `NameError` crash).
|
||
- Added `_find_obd_port()`/`connect_obd()`: auto-detects `/dev/ttyUSB*` (the PL2303 adapter),
|
||
honors an `OBD_PORT` env override, falls back to python-obd's own port auto-scan (still covers
|
||
the old `/dev/rfcomm*` Bluetooth path).
|
||
- Fixed stale-response bug: `obd_speed()`/`obd_rpm()` now re-query every loop iteration instead
|
||
of reusing one response forever.
|
||
- Fixed `obd_rpm()` using the wrong lock, never updating the global `rpm`, and emitting to the
|
||
wrong socketio event (`obd_speed` instead of `obd_rpm`).
|
||
- Fixed `kill_obd()` being a no-op — now uses `threading.Event` flags actually checked in the
|
||
polling loops.
|
||
- Fixed inverted thread-start logic (threads were starting on failed connection, not success).
|
||
- Fixed `obm_error` → `obd_error` typo (frontend listens for `obd_error`).
|
||
- Added `obd` (python-obd) to `requirements.txt`.
|
||
- **Still untested**: a real connection to the actual car (adapter has only been tested
|
||
standalone on the Pi so far — correctly reports "no connection to car" with nothing attached).
|
||
|
||
### Phase 3: Web UI must work fully offline — ✅ DONE
|
||
The whole point of the project is a Pi hotspot with **no internet in the car**, but the current
|
||
UI silently depended on the internet for core functionality:
|
||
- `templates/index.html`/`dash.html`/`dtc.html` loaded Google Fonts, FontAwesome, Leaflet JS/CSS,
|
||
and the Socket.IO client from public CDNs — none of this loads offline, breaking layout, the
|
||
websocket connection itself, and the map entirely.
|
||
- `static/js/yaris.js` loaded live map tiles from `tile.openstreetmap.org` (also needs internet).
|
||
- A local `static/js/chart.min.js` existed but was unused — `index.html` loaded Chart.js from
|
||
`cdn.jsdelivr.net` instead (turned out the local copy was a stale Chart.js **v2.8.0** while
|
||
`yaris.js`'s config uses v3/v4-style `scales: {x:{...}, y:{...}}` syntax — that mismatch is
|
||
exactly why the CDN was used instead. Replaced the local file with a current v4.5.0 UMD build).
|
||
|
||
**What was done**:
|
||
- Vendored Socket.IO client (4.8.1, matches Flask-SocketIO 5.6.1's Engine.IO v4 protocol) under
|
||
`static/vendor/socketio/`, and Leaflet (1.9.4 JS/CSS + marker/layers images) under
|
||
`static/vendor/leaflet/`.
|
||
- FontAwesome 5.12.0 was already vendored locally at `static/fonts/` (self-consistent, just never
|
||
referenced) — repointed all three templates to it instead of the CDN.
|
||
- Dropped the Google Fonts (Nunito) `<link>` entirely rather than self-hosting 12 font-weight
|
||
files — Bootstrap's `--bs-font-sans-serif` variable already falls back cleanly to system fonts
|
||
(`-apple-system, Segoe UI, Roboto, ...`) after Nunito, so this is a clean no-cost simplification.
|
||
- Replaced `static/js/chart.min.js` with a current Chart.js v4.5.0 UMD build (auto-registering,
|
||
same drop-in behavior as the old CDN `<script src="https://cdn.jsdelivr.net/npm/chart.js">`).
|
||
- Offline map tiles: added `scripts/download_map_tiles.py`, a dev-machine-only one-time tile
|
||
downloader using the MapTiler Cloud free tier (raster XYZ, `streets-v2` style, 256px tiles).
|
||
Reads `MAPTILER_KEY` from an env var/CLI arg only — the key itself is never committed or
|
||
deployed, only the resulting static PNG tiles are. Bounding box is a required CLI arg (your own
|
||
driving area's coordinates - not committed/hardcoded) at zoom 8-15 (~14k tiles for a
|
||
city-plus-suburbs-sized area, well under MapTiler's 100k tiles/month free quota). Output goes to
|
||
`static/vendor/tiles/{z}/{x}/{y}.png` (gitignored — regenerate locally, rsync to the Pi as part
|
||
of deployment, don't bloat the git repo with binary tile data).
|
||
- **Tiles populated**: user signed up for a free MapTiler key and the full zoom 8-15 download
|
||
ran to completion — 14,241 downloaded, 24 skipped (out-of-bounds), 0 failed (14,265 tiles
|
||
total, ~285MB). Rsynced to the Pi's `static/vendor/tiles/` (tile counts verified identical
|
||
on both sides). Live Pi instance re-confirmed serving HTTP 200 with the complete offline
|
||
tile set.
|
||
- Updated `static/js/yaris.js`'s Leaflet init to point at `/static/vendor/tiles/{z}/{x}/{y}.png`
|
||
(`minZoom`/`maxZoom` 8/15 to match). The initial map center is just a generic placeholder (not
|
||
a real location) - it re-centers automatically to the real live GPS fix once one comes in, and
|
||
the marker is exposed as `carMarker` so Phase 4 can move it on live fixes.
|
||
- Added a repo-root `.gitignore` (Python caches, venvs, `.vscode/`, `static/vendor/tiles/`,
|
||
`*.env` except the already-tracked `db/creds.env` placeholder) — there wasn't one before, which
|
||
is why stray `__pycache__`/`.pyc` files kept showing up in source control.
|
||
- **Verified**: ran the Flask app in a local venv, confirmed all three pages (`/`, `/dtc`, `/dash`)
|
||
render with **zero external URLs** in the served HTML, every local static asset (JS, CSS, fonts,
|
||
Leaflet images) returns HTTP 200, and all vendored JS files pass a Node.js syntax check.
|
||
|
||
### Phase 4: Map/GPS display bug — ✅ DONE
|
||
`yaris.js`'s Leaflet marker (`carMarker`) was never moved on incoming `gps_location` socket
|
||
events — only two text spans (`gpsLocationX`/`Y`) got updated. Fixed: `carMarker.setLatLng(...)`
|
||
is now called on every `gps_location` event, and the map re-centers (zoom 15) on the first fix
|
||
only (`mapCentered` flag), so the user isn't fighting the map re-centering under them while
|
||
panning/zooming during a trip.
|
||
|
||
**Verified**: also used this as an opportunity to test Phase 3 end-to-end — ran
|
||
`scripts/download_map_tiles.py` for real with the user's actual MapTiler API key (smoke-tested
|
||
zoom 8-10 first, confirmed idempotent re-run skips existing tiles, confirmed a downloaded tile
|
||
is a real map image of the target driving area), then started the full zoom 8-15
|
||
download. Took a **headless Chromium screenshot** of the running app's `/` page and confirmed:
|
||
sidebar/icons/fonts render correctly (FontAwesome working), the Speed Graph (Chart.js v4) renders,
|
||
and **the Leaflet map renders real offline map tiles with the marker in place** — full offline
|
||
end-to-end proof, not just "no CDN URLs in the HTML".
|
||
- Fixed (was noted here as out of scope, now addressed in Phase 8): `/dash` and `/dtc` used to
|
||
render near-identical body content (both showed "DTC Codes") — `dash.html` now has real
|
||
dashcam-specific content.
|
||
- **Full tile download completed** (see Phase 3 update above) and rsynced to the Pi — Phase
|
||
3/4's offline map is now 100% functional end-to-end on the actual deployed device, not just
|
||
the dev machine.
|
||
|
||
### Phase 5: Pi deployment / autostart — ✅ DONE
|
||
Set up a proper venv (`/home/yaris/app/yaris-obd/venv`), installed all requirements (all found
|
||
as prebuilt ARM wheels via piwheels — no slow source compiles needed, even for gevent).
|
||
Added `deploy/yaris-obd.service` (systemd unit: runs as the `yaris` user, `Restart=on-failure`,
|
||
5s `ExecStartPre` sleep to let USB/serial devices enumerate after boot, logs to
|
||
`yaris-obd.log`) and `deploy/install_service.sh` (idempotent one-time install helper — also
|
||
creates the venv if missing). Installed and enabled on the Pi (`systemctl enable` — survives
|
||
reboot).
|
||
|
||
**Also had to fix Pi → GitHub auth for `git pull`**: the repo is private, and the Pi's existing
|
||
`~/.ssh/id_ed25519.pub` turned out to be an *inbound* SSH key (its private half lives on the dev
|
||
machine, not the Pi) — useless for the Pi to authenticate outbound. Generated a dedicated
|
||
ed25519 keypair **on the Pi itself** (`~/.ssh/yaris_deploy_key`), registered it as a **read-only
|
||
deploy key** on the GitHub repo (least-privilege — the Pi only ever needs to pull), and added an
|
||
`~/.ssh/config` entry so git transparently uses it for `github.com`. Switched the Pi's `origin`
|
||
remote to SSH. `git pull` now works standalone on the Pi with no PAT/interactive auth needed.
|
||
|
||
**Verified end-to-end**: pulled latest code onto the Pi (was 5 commits stale), confirmed all
|
||
Python deps import cleanly, ran the app manually first (HTTP 200, all local static assets —
|
||
Leaflet/Socket.IO/Chart.js/FontAwesome — served correctly from the Pi itself), then installed +
|
||
started the systemd service and confirmed `active (running)` + `enabled`. **Tested crash
|
||
recovery**: sent SIGKILL to the running process, confirmed systemd auto-restarted it (through the
|
||
5s `ExecStartPre` delay) and the app was serving HTTP 200 again within ~10s, with zero manual
|
||
intervention.
|
||
|
||
### Phase 6: WiFi hotspot mode — ✅ DONE
|
||
Used NetworkManager (already the network manager in use, not hostapd/dnsmasq directly — those
|
||
were leftover/unused) since it natively supports AP mode via `nmcli`. Created two connection
|
||
profiles on the Pi: `home-wifi` (your real home WiFi network, `autoconnect-priority 10`) and
|
||
`Hotspot` (AP mode, `ipv4.method shared`, its own dedicated SSID/password set locally on the Pi -
|
||
not committed here, WPA2-PSK, `autoconnect-priority -10`). NM's own autoconnect logic handles
|
||
"leaving home WiFi range -> fall back to hotspot" automatically since `home-wifi` has higher
|
||
priority and NM always tries it first.
|
||
|
||
For the reverse direction (home WiFi back in range while sitting in AP mode) added
|
||
`deploy/yaris-netwatch.sh` + `.service`/`.timer`: since the Pi's onboard BCM43438 radio can't be
|
||
an AP and background-scan for client networks simultaneously, a systemd timer runs every 3
|
||
minutes, briefly attempts reconnecting to `home-wifi`, and restores the hotspot if home WiFi still
|
||
isn't in range. Installed via `deploy/install_netwatch.sh`.
|
||
|
||
**Root-caused and fixed a real bug during testing**: the hotspot's SSID broadcast worked
|
||
immediately, but its DHCP server initially failed (`dnsmasq: failed to create listening socket
|
||
for 10.42.0.1: Address already in use`) — a pre-existing **orphaned system-wide `dnsmasq.service`**
|
||
(noted as "likely unused" in earlier investigation) was already bound to the address space
|
||
NetworkManager's own internal dnsmasq needs for the hotspot. Confirmed the system dnsmasq's
|
||
config was genuinely empty/default (nothing in this project depends on it) and disabled it
|
||
(`systemctl disable --now dnsmasq`).
|
||
|
||
Also added easy access without needing to know the Pi's IP:
|
||
- **Renamed the Pi's hostname** from `yarisobd` to `yaris` — `avahi-daemon` (already installed
|
||
and running) now publishes it as `yaris.local` via mDNS.
|
||
- **Port 80 -> 5000 redirect** via nftables (`deploy/nftables-yaris.conf` +
|
||
`install_port_redirect.sh`, loaded through an `include` in `/etc/nftables.conf`), so the app is
|
||
reachable at plain `http://yaris.local/` (no port number) instead of `:5000`.
|
||
|
||
**Verified on real hardware, very carefully** since the SSH session used to do all this testing
|
||
itself runs over `wlan0`: used a "dead-man's-switch" pattern for every risky test — a detached
|
||
background command (`setsid nohup bash -c 'sleep N && nmcli con up home-wifi' & disown`) armed
|
||
*before* each hotspot activation, so the Pi automatically reverts to home WiFi (and SSH access
|
||
comes back) even if something goes wrong, no physical access needed. Confirmed: hotspot broadcasts
|
||
and a real phone gets a genuine DHCP lease from NetworkManager's own DHCP server and can load the
|
||
web UI; `yaris-netwatch` correctly detects AP mode and reconnects to home WiFi when in range (and
|
||
is a no-op when already on home WiFi); `http://yaris.local/` and `http://<pi-lan-ip>/` both return
|
||
HTTP 200 with no port number from another machine on the network. (Real hotspot SSID/password and
|
||
the Pi's LAN IP are intentionally not recorded in this file — see local, non-committed notes.)
|
||
|
||
**Update — real-hardware bug found + fixed (hotspot->home reconnect was flaky)**: while testing
|
||
the phone-relay feature live (see Phase 11 update below), `yaris-netwatch` repeatedly logged "home
|
||
WiFi not found" 4 times in a row over ~13 minutes while sitting right next to the router — a real
|
||
regression from the original design, not a one-off fluke. Root cause: a plain `nmcli con up
|
||
home-wifi` straight out of AP mode relies on NM's scan cache for `wlan0`, which is stale/empty
|
||
coming out of AP mode (the radio can't scan while acting as an AP - see above). A single
|
||
`--wait 20` connect attempt gave up before a fresh scan could land. Fixed `yaris-netwatch.sh` to
|
||
explicitly drop the AP connection first (releases the radio), force `nmcli device wifi rescan`,
|
||
wait for it to settle, then retry the connect (longer 30s wait) up to 2 times before conceding and
|
||
restoring the hotspot. Bumped `yaris-netwatch.service`'s `TimeoutStartSec` to 180s (retry loop can
|
||
take up to ~80s, default systemd 90s start timeout was too tight). Verified fixed on real hardware:
|
||
both hotspot->home transitions after the fix recovered in ~7 seconds (previously failing outright).
|
||
Deployed: `sudo cp deploy/yaris-netwatch.sh /usr/local/bin/... && cp .service ... && daemon-reload`.
|
||
|
||
### Phase 7: app_db.py / app_sql.py fixes — ✅ DONE
|
||
Neither file was called from `app.py` or anywhere else - orphaned from a prior iteration. Fixed:
|
||
- `app_db.py`: applied a prior-session stash fix - resolves `.env` path via `__file__` (was
|
||
cwd-relative), fixes `os.getenv()` calls to pass the actual key names (was called with zero
|
||
args, an immediate `TypeError` if ever invoked), added a clear `RuntimeError` guard if
|
||
credentials are still empty placeholders, returns `client[mDB]`.
|
||
- `app_sql.py`: rewritten - resolves `db/sql/` via `__file__` + `os.makedirs` (was cwd-relative),
|
||
fixed `create_conn()` closing the connection immediately and returning `None` (so table
|
||
creation never ran and every caller got a dead connection), added `get_conn()` (lazy singleton
|
||
+ table creation for both `tracking` and `dtc` tables), `insert_tracking()`, and a background
|
||
`start_logging()`/`_log_loop()` thread (5s interval) that snapshots GPS (`app_gps.LocationX/Y`)
|
||
+ OBD speed (`app_obd.speed`) into the local `tracking` table. Used a deferred (function-local)
|
||
import of `app_obd` inside `_log_loop()` to keep the pre-existing `app_obd`<->`app_sql`
|
||
top-level circular import safe/explicit.
|
||
- Also fixed a Python 3.12+/3.14 compat bug found during testing: `sqlite3.version` (the
|
||
sqlite3 *module's* version) was removed/deprecated - switched to `sqlite3.sqlite_version`
|
||
(the actual SQLite library version), which is what was intended.
|
||
- Wired `app_sql.start_logging()` into `app.py` alongside the other startup calls
|
||
(`app_gps.start_gps()`, `app_obd.OBD()`).
|
||
- `db/sql/` added to `.gitignore` (per-device local trip data, not source).
|
||
- Verified locally: `get_conn()` creates `db/sql/<account>_<car>.db` with both tables;
|
||
`insert_tracking()` writes a real row; `start_logging()` thread starts and logs
|
||
"SQL tracking-logger thread started, alive: True"; full `app.py` run confirms the thread
|
||
starts cleanly and (with no real GPS/OBD hardware on the dev machine) correctly logs read
|
||
errors instead of crashing; `app_db.get_database()` raises the new clear `RuntimeError` instead
|
||
of crashing, since `db/creds.env` still holds empty placeholder credentials (real Mongo
|
||
credentials are a user-provided follow-up, out of scope here).
|
||
|
||
### Phase 8: Dashcam retrieval (app_dash.py) — ✅ DONE, verified on real hardware
|
||
Rewrote `app_dash.py` to be brand/device-agnostic: detection via `lsblk -J`, matching partitions
|
||
whose parent disk reports `TRAN == "usb"` (this is what safely distinguishes a plugged-in
|
||
dashcam/USB stick from the Pi's own onboard storage, which is always `mmc` transport - no more
|
||
hardcoded `/dev/sda`). Walks the whole mounted filesystem for video file extensions
|
||
(`.mp4/.mov/.avi/.ts/.mkv`) instead of assuming a specific brand's folder layout (DCIM, MOVIE,
|
||
NORM/EVENT/PARKING, etc. all just work), preserves each file's relative sub-folder path when
|
||
copying (avoids filename collisions across dashcam event/date folders), and skips files already
|
||
copied (same relative path + size) so re-running an import is cheap. Mounts read-only to avoid
|
||
any risk of corrupting the dashcam's SD card. Dropped the unused/uninstalled `tqdm` CLI-progress
|
||
dependency (never actually in `requirements.txt`) since this now runs headless inside the always-
|
||
on web server, not interactively.
|
||
|
||
Wired into `app.py`: `GET /api/dash/devices` (list detected USB storage) and
|
||
`POST /api/dash/import` (run the mount->copy->unmount pipeline), both JSON. Fixed the pre-existing
|
||
`/dash` vs `/dtc` duplicate-template bug — `dash.html` now has real dashcam content (scan/import
|
||
buttons + live status) instead of a copy-pasted "DTC Codes" body. Added a dedicated `dash.js`
|
||
(not the shared `yaris.js`, which unconditionally initializes the `/index`-only map/chart and
|
||
would throw/crash on `/dash` and `/dtc` since those pages don't have those DOM elements - a
|
||
pre-existing bug, worked around by not depending on that shared file for this page's new JS).
|
||
Added `dash/videos/` to `.gitignore` (imported footage is per-device data, not source).
|
||
|
||
**Tested thoroughly without the physical dashcam** (user didn't have it on hand this session):
|
||
1. Unit-tested the `lsblk`-JSON detection/filtering logic against a realistic synthetic sample
|
||
(Pi's own `mmcblk0` boot+root partitions plus a simulated USB dashcam partition) - confirmed
|
||
it correctly finds only the USB partition and ignores the Pi's own storage.
|
||
2. Integration-tested the real mount -> find-videos -> copy (preserving structure) -> dedupe ->
|
||
unmount pipeline on the actual Pi using a loop-device-backed FAT32 image (`mkfs.vfat` +
|
||
`losetup`) populated with fake nested dashcam video files (`DCIM/100EVENT/...`,
|
||
`DCIM/100NORMAL/...`) - confirmed real mount/copy/unmount all work, confirmed a second import
|
||
run correctly skips already-copied files, confirmed non-video files (e.g. `thumbs.db`) are
|
||
ignored. All simulation artifacts (loop device, test image, temp files) cleaned up afterward.
|
||
3. Verified the full `/dash` page + both new API routes locally (Flask dev run + headless
|
||
browser check): auto-scan on page load correctly reports "No USB dashcam/storage device
|
||
detected" (accurate, since none is plugged into the dev machine), Import button correctly
|
||
stays disabled until a device is found, no JS console errors, no regression on `/` or `/dtc`.
|
||
|
||
**Verified against the real physical dashcam** (Jieli-chipset USB ID `1224:2725`, 29.7GB SD card,
|
||
`DCIM/1/VID_###.MOV` layout, 87 files, ~28GB total). This first live hardware test found and fixed
|
||
three real bugs the loop-device simulation couldn't have caught:
|
||
1. **Superfloppy SD card** - the card has no partition table at all (`lsblk` reports the whole
|
||
`/dev/sda` disk directly as `fstype: vfat`, no child partition) - the original detection assumed
|
||
a partition would always exist. Fixed to also match a `TRAN == "usb"` disk with its own
|
||
`fstype` set directly, not just partitions under it.
|
||
2. **Mount permission error** - `mount_device()` called `os.makedirs()` directly, but the service
|
||
runs as unprivileged `yaris` and `/mnt` is root-owned. Fixed to `sudo mkdir -p <mount_point>`
|
||
before the existing `sudo mount`, mirroring that pattern.
|
||
3. **OOM crash during upload** (see Phase 10 below) - not a copy-path bug, but discovered
|
||
immediately after the first successful real import.
|
||
|
||
End-to-end result: full 28GB/87-file import from the SD card completed successfully (resumed
|
||
cleanly after being interrupted once by the OOM crash below - `copy_files()`'s skip-if-already-
|
||
copied logic meant re-running `/api/dash/import` just picked up the remaining files, no data loss
|
||
or re-copy of the first 19 files). SD card unmounted cleanly afterward.
|
||
|
||
### Phase 9: Remote server for itinerary/dashcam upload — ✅ DONE
|
||
Per architecture decisions #4 and #5: self-hosted server/API to receive itinerary (from the
|
||
SQLite pipeline) and dashcam file uploads.
|
||
|
||
Decided/built: Flask API + dedicated Postgres, deployed as a docker-compose stack on the user's
|
||
unraid server (separate from any existing containers/DBs there), living in `server/` in this
|
||
same repo. Auth: simple `X-Device-Name`/`X-Api-Key` header pair per device, key hash stored
|
||
server-side, real key never committed (raw key printed once by `server/scripts/create_device.py`,
|
||
saved locally on the Pi only, in gitignored `db/upload_creds.env`). External access: exposed at
|
||
`https://yaris.ceaf.dev` via the user's existing Cloudflare Tunnel (public-hostname rule added in
|
||
the Cloudflare dashboard, pointing at `192.168.10.10:8098`) — router port-forward + the user's
|
||
Unifi firewall is the documented fallback if ever needed instead. Large dashcam uploads can
|
||
still prefer a local-network path (e.g. an AdGuard DNS override to bypass the tunnel while on
|
||
home WiFi) since tunnels typically cap per-request upload size — noted in `server/README.md`,
|
||
not yet needed/built.
|
||
|
||
Built (`server/` folder):
|
||
- `server/sql/init.sql` — Postgres schema: `devices` (name + sha256 api-key-hash), `tracking_points`
|
||
(device_id FK, GPS/speed/odometer + `recorded_at`, unique per device+timestamp so retried
|
||
batches can't duplicate rows), `dashcam_files` (device_id FK, relative_path + sha256, unique per
|
||
device+path+hash so re-uploads are no-ops).
|
||
- `server/modules/db.py` — psycopg3 connection pool + query helpers (bulk tracking insert with
|
||
`ON CONFLICT DO NOTHING`, dashcam dedupe-check-before-write).
|
||
- `server/modules/auth.py` — `require_device_auth` decorator checking the two headers against
|
||
the stored key hash.
|
||
- `server/app.py` — Flask app: `GET /health` (unauthenticated), `POST /api/v1/tracking/batch`
|
||
(bulk JSON insert, capped at 5000 points/request as a sanity guard), `POST /api/v1/dashcam/upload`
|
||
(multipart, streamed to disk with sha256 verification + path-traversal guard on `relative_path`).
|
||
- `server/Dockerfile` + `server/docker-compose.yml` — `api` (gunicorn, published on host port
|
||
8098 by default) + `db` (Postgres 17, internal-only, not published to the LAN) services, data
|
||
volume defaults to `server/data/` but pointed at `/mnt/user/appdata/yaris-obd-server` on unraid.
|
||
- `server/.env.example` — template for `POSTGRES_PASSWORD`/`API_PORT`/`DATA_DIR`/upload size cap.
|
||
- `server/scripts/create_device.py` — run once per device inside the `api` container to register
|
||
it and print its one-time API key. Ran once for device `yaris-2013`.
|
||
- `server/README.md` — full unraid deployment + Cloudflare Tunnel exposure + device-registration
|
||
instructions.
|
||
|
||
Pi-side upload client (`modules/app_upload.py`):
|
||
- Background thread, uploads unsent `tracking` rows (batches of 500) and not-yet-uploaded
|
||
dashcam files (sha256-verified) via the API above, with exponential retry/backoff on failure.
|
||
- Independent dashcam-upload-only pause toggle (persisted `settings` table + a switch on the
|
||
`/dash` web UI) — trip/GPS logs always keep uploading regardless of the toggle, per explicit
|
||
user request (useful on a slow/metered connection).
|
||
- **Backlog-drain tuning**: since tracking rows log every 5s locally, an offline/stale device can
|
||
queue up a large backlog. The loop drains back-to-back (short 1s pause between batches) while a
|
||
batch comes back full, only settling into the slow 30s idle cadence once actually caught up —
|
||
a simulated 1250-row backlog now drains in 3 batches (~0.35s) instead of 13 batches over ~6.5
|
||
minutes with the original fixed 100-row/30s-sleep design.
|
||
- `modules/app_sql.py` extended: `UPLOADED` column + migration on `tracking`, `settings` table,
|
||
`dashcam_uploads` dedupe table, and helper functions backing all of the above.
|
||
|
||
Tested end-to-end against the real deployed server (tracking batch + dedupe, dashcam upload +
|
||
dedupe + path-traversal rejection, bad-API-key rejection, toggle-off correctly pausing only
|
||
dashcam uploads) and on the real Pi (its actual ~1800-row pre-feature backlog drained and
|
||
confirmed matching row counts on the server via direct psql query, live upload status visible on
|
||
`/dash`). Committed+pushed (`4ff99f0`, `c4db1ab`, `8dc304e`), deployed on both unraid and the Pi.
|
||
Exposed externally at `https://yaris.ceaf.dev` via the user's Cloudflare Tunnel (public hostname
|
||
added in the Cloudflare dashboard, pointing at `192.168.10.10:8098`).
|
||
|
||
### Phase 10: Dashcam upload memory/size fixes — ✅ DONE, verified on real hardware
|
||
Two real-hardware-only bugs surfaced once actual ~300MB+ video files (not small test fixtures) hit
|
||
the upload path for the first time:
|
||
|
||
1. **OOM crash from in-memory multipart uploads.** `modules/app_upload.py` originally uploaded
|
||
dashcam files via `requests.post(..., files={"file": f})`. The `requests` library builds the
|
||
*entire* multipart/form-data body in memory before sending when using `files=` - a well-known
|
||
gotcha, not specific to this codebase. On the Pi 3's ~900MB RAM, uploading one ~320MB video this
|
||
way spiked the process past 600MB+ RSS, triggering the kernel OOM killer within minutes
|
||
(confirmed via `dmesg`: repeated `Out of memory: Killed process ... anon-rss:823928kB`, and via
|
||
`pmap -x` showing a single ~362MB anonymous mapping matching the file size almost exactly).
|
||
**Fixed**: added `requests-toolbelt` and rewrote the upload body construction to use
|
||
`MultipartEncoder`, which streams the file in chunks as the request is sent instead of
|
||
materializing it up front. Confirmed via an isolated 300MB test (628MB RSS old way -> 29MB new
|
||
way, byte-identical result) and live on the Pi (RSS stayed flat ~55-59MB during real uploads,
|
||
vs. previously crashing every few minutes).
|
||
2. **Cloudflare Tunnel's hard ~100MB request-body cap.** Even after the memory fix, real uploads
|
||
still failed once routed through `https://yaris.ceaf.dev` - a bare `413 Payload Too Large` from
|
||
Cloudflare itself (confirmed: the identical file succeeds direct-LAN, 413s only through the
|
||
tunnel). This is a free-tier platform limit, unrelated to `MAX_DASHCAM_UPLOAD_MB` on the Flask
|
||
side. **Fixed** via a chunked/resumable upload redesign (well under the 100MB cap):
|
||
- Client (`modules/app_upload.py`): `DASHCAM_CHUNK_SIZE = 40MB`; new
|
||
`_upload_video_chunked()` checks `/status` first (enables resume across restarts/failures),
|
||
then POSTs missing chunks one at a time via `MultipartEncoder`.
|
||
- Server (`server/app.py`): new `GET /api/v1/dashcam/upload/status` (reports
|
||
`already_uploaded` or `received_chunk_indices`) and `POST /api/v1/dashcam/upload/chunk`
|
||
(stages chunks under `STORAGE_DIR/.upload_tmp/<device>/<sha256>/`, reassembles + verifies
|
||
SHA256 once all chunks arrive, records via the existing `db.insert_dashcam_file`, cleans up
|
||
the staging dir). A `.total_chunks` marker guards against stale/mismatched partial sets if the
|
||
chunk size ever changes between client versions. The original single-shot
|
||
`/api/v1/dashcam/upload` endpoint is untouched, kept for reference/compatibility.
|
||
|
||
**Verified end-to-end against the real deployed server through the real Cloudflare tunnel**,
|
||
first with a synthetic interrupted-then-resumed 250MB test (status transitions
|
||
`not_started` -> partial upload -> `in_progress` w/ correct `received_chunk_indices` -> resume ->
|
||
`complete` -> `already_uploaded`, SHA256 match confirmed on disk), then with real dashcam footage:
|
||
the Pi copied all 87 real files locally and the background upload thread drained the real backlog
|
||
one file at a time, each one verified byte-for-byte (SHA256 match Pi vs. server) and correctly
|
||
recorded in Postgres, while the Pi's service ran 50+ minutes straight with RSS staying in a bounded
|
||
~55-141MB range (no OOM kills) - a stark contrast to the previous crash-every-few-minutes behavior.
|
||
Given home-upload-bandwidth constraints, the full 28GB backlog drains at roughly one ~350MB file
|
||
per 4-5 minutes and is expected to take several hours total - this runs unattended in the
|
||
background with no user action needed once started.
|
||
|
||
**Not yet built**: parallel/concurrent chunk uploads (currently strictly sequential, one chunk
|
||
then one file at a time) - would speed up the multi-hour full-backlog drain but adds complexity;
|
||
not requested by the user yet, flagged here as a possible future optimization if the sequential
|
||
pace becomes annoying in practice.
|
||
|
||
**Known manual step (not yet automated)**: this dashcam model boots into a "select mode" menu
|
||
(dashcam / mass-storage / webcam) when it senses USB data lines are live, and needs USB data
|
||
communication physically absent to boot straight into recording mode, but present when plugging
|
||
into the Pi for import. No code in this repo handles that switch - it's a manual physical step by
|
||
the user (e.g. a data-blocking USB cable/adapter, or plugging/unplugging appropriately) each time
|
||
before recording vs. before importing. Worth revisiting if it becomes tedious in daily use (e.g. a
|
||
Pi-controlled USB data-line switch), but out of scope for now.
|
||
|
||
### SIM868 cellular fallback for tracking uploads — ⚠️ built, UNTESTED (no SIM card available)
|
||
`modules/app_cellular.py` gives the tracking-upload path a fallback over the SIM868's cellular
|
||
data connection when WiFi/Ethernet is unreachable. Design notes:
|
||
- Uses the SIM868's built-in AT-command HTTP(S) client (`AT+HTTPINIT`/`HTTPPARA`/`HTTPACTION`/
|
||
`HTTPREAD`) rather than a real `ppp0` network interface. Reason: the SIM868's GNSS (polled
|
||
continuously by `modules/app_gps.py`) and its GSM/GPRS radio share the *same* physical UART
|
||
(`/dev/serial0`) — a classic `pppd`/`chat` dial-up would put that port into raw PPP framing for
|
||
as long as the link is up, starving the GNSS polling thread. Staying in AT-command mode lets
|
||
cellular HTTP calls interleave safely through the same serial lock
|
||
(`modules.app_gps._serial_lock`) — each exchange just briefly delays the next GPS poll.
|
||
- Only carries small JSON tracking batches, **never** dashcam video — cellular data is far too
|
||
slow/limited for multi-MB video files, and the modem's HTTP AT interface has a small internal
|
||
buffer unsuited to them anyway. Dashcam uploads remain WiFi/Ethernet-only, unchanged.
|
||
- `modules/app_upload.py`'s tracking-batch upload always tries WiFi/Ethernet (`requests`) first;
|
||
only on a connection-level failure (no route/DNS/timeout — as opposed to an HTTP error status,
|
||
which means the server WAS reached) does it retry the same batch over cellular.
|
||
- Feature-flagged off by default: `YARIS_CELLULAR_ENABLED`/`YARIS_CELLULAR_APN`/
|
||
`YARIS_CELLULAR_APN_USER`/`YARIS_CELLULAR_APN_PASS` added to `db/upload_creds.env`, all blank
|
||
by default so this has zero effect on the already-verified WiFi upload path until the user
|
||
fills in an APN once they get a SIM card. Regression-tested (against the real server) that
|
||
normal WiFi uploads are completely unaffected while cellular is unconfigured.
|
||
- `/dash` upload-status card now shows which path (`wifi`/`cellular`) the last successful batch
|
||
used, and whether cellular fallback is configured.
|
||
- **Needs validation once a SIM card is available** — exact AT command dialect/timing (SIMCOM
|
||
firmware revisions vary slightly), and whether `AT+HTTPSSL` works for the HTTPS
|
||
`yaris.ceaf.dev` endpoint on this specific module's firmware (if not, a documented fallback is
|
||
a plain-HTTP router port-forward path, avoiding the need for the modem's TLS stack to validate
|
||
a cert chain).
|
||
|
||
### Phase 11: Phone-relay dashcam upload — ⛔ REMOVED (see update below - hit a hard platform limitation)
|
||
Root cause investigation into "why is the transfer from the Pi to the server so slow?" (real
|
||
dashcam backlog draining at only ~350MB/4-5min) found the bottleneck is the **Pi's own onboard
|
||
BCM43438 WiFi chip** (2.4GHz-only, congested/interfered channel) — confirmed via `iperf3`
|
||
raw-TCP tests showing only ~14.7Mbit/s with active retransmits even over a same-subnet direct-LAN
|
||
hop that never touches the internet/Cloudflare, ruling out ISP/Cloudflare limits. Since the Pi 3 B
|
||
has no 5GHz radio and can't be fixed in software, added a way to bypass its weak WiFi for the
|
||
internet-bound leg entirely: a phone opens the same `/dash` page (over the Pi's hotspot/home WiFi)
|
||
and manually relays pending dashcam files straight to the remote server using the phone's own
|
||
network (WiFi or cellular data), instead of routing everything through the Pi.
|
||
|
||
**Zero new remote-server upload logic** — the phone's browser JS calls the *exact same*
|
||
`/api/v1/dashcam/upload/status` + `/api/v1/dashcam/upload/chunk` endpoints the Pi's own Python
|
||
client already uses, in the same 40MB-chunk protocol, just orchestrated by `fetch()`/`FormData`
|
||
instead of `requests`/`MultipartEncoder`. The server dedupes purely by SHA256 regardless of which
|
||
device sends the bytes, so no server-side changes were needed for the upload path itself — only a
|
||
CORS `after_request` hook (`server/app.py`) was added, since the phone's browser calls
|
||
`yaris.ceaf.dev` from a page served by a *different* origin (the Pi), which browsers block without
|
||
`Access-Control-Allow-*` headers on both the real request and its preflight `OPTIONS`. Wildcard
|
||
origin is fine here — auth is header-based (`X-Device-Name`/`X-Api-Key`), not cookie/session-based.
|
||
|
||
**New Pi-side endpoints** (`app.py`):
|
||
- `GET /api/dash/pending` — lists not-yet-uploaded dashcam files (`relative_path`, `sha256`,
|
||
`size`) via a new `app_upload.list_pending_dashcam_files()`.
|
||
- `GET /api/dash/file/<path:relpath>` — serves a dashcam file's bytes for the phone to download,
|
||
via `send_file(..., conditional=True)` for **Range-request support** (206 Partial Content) so
|
||
the phone downloads (and immediately re-uploads) one 40MB slice at a time instead of ever
|
||
holding a whole 300+MB file in the browser's memory — the exact same memory-safety lesson
|
||
learned the hard way in Phase 10, applied symmetrically to the download side. Path-traversal
|
||
safe via Werkzeug's `safe_join` against `app_dash.DEFAULT_DESTINATION`.
|
||
- `GET /api/dash/relay/config` — hands the phone the same `SERVER_URL`/`DEVICE_NAME`/`API_KEY`
|
||
the Pi's own client uses, so its JS can auth directly with the remote server. Same trust
|
||
boundary as the rest of this project's unauthenticated local UI (anyone on the Pi's own
|
||
hotspot/LAN is already implicitly trusted).
|
||
- `POST /api/dash/relay/heartbeat` / `POST /api/dash/relay/stop` — a **lease-based pause**
|
||
(`RELAY_LEASE_SECONDS = 45`) for the Pi's own background dashcam uploader, chosen over a plain
|
||
on/off toggle specifically to self-heal if the phone's browser tab closes/loses connection
|
||
uncleanly (a simple toggle could get stuck "paused" forever). The phone page heartbeats every
|
||
15s while relaying; `_upload_dashcam_files()` skips entirely while a lease is active, re-checked
|
||
between files so it stops promptly if a relay starts mid-drain. Also released via
|
||
`navigator.sendBeacon()` on `beforeunload` as a best-effort immediate release.
|
||
- `POST /api/dash/relay/mark_uploaded` — lets the phone tell the Pi "I just uploaded this file"
|
||
right after a successful relay, so the local dedupe table (and the pending list / Pi's own
|
||
background loop) reflect it immediately instead of waiting for the next full backlog scan.
|
||
|
||
**Performance side-fix**: `modules/app_upload.py`'s dashcam loop previously recomputed SHA256 for
|
||
*every* candidate file on *every* loop iteration, including files long since uploaded — wasteful
|
||
CPU/IO against a growing, possibly 20GB+, local backlog. Added a `dashcam_hash_cache` SQLite table
|
||
(`modules/app_sql.py`, keyed by `relative_path`+`size`+`mtime`) so a file's hash is only computed
|
||
once; both the existing background uploader and the new pending-files endpoint benefit.
|
||
|
||
UI (`templates/dash.html` / `static/js/dash.js`): a new "Relay via phone" card on `/dash` lists
|
||
pending files with a per-file "Relay" button and a "Relay all pending" button (manual trigger only
|
||
— chosen deliberately over an automatic one, per explicit user preference), plus a "Stop relay"
|
||
button. Chunk downloads/uploads retry up to 3 times with backoff before surfacing an error, and a
|
||
resumed relay re-checks `/status` first so a previously-interrupted file picks up from its first
|
||
missing chunk instead of restarting.
|
||
|
||
**Tested**: a full local end-to-end simulation (no physical phone/network needed) drove the real
|
||
Pi `app.py` and real `server/app.py` Flask apps side by side (Postgres mocked at the `db` module
|
||
boundary only — all HTTP routing, chunking, auth, and CORS logic is real), reproducing the exact
|
||
sequence the phone's browser JS performs: list pending -> check remote `/status` -> download each
|
||
chunk from the Pi via `Range` requests (verified real `206 Partial Content` responses) -> upload
|
||
each chunk to the mock-backed server -> verify the reassembled file matches the original
|
||
byte-for-byte via SHA256 -> mark uploaded on the Pi -> confirm it drops out of the pending list.
|
||
Also verified: path-traversal attempts on `/api/dash/file/` correctly 404, the relay
|
||
heartbeat/stop lease correctly pauses/resumes `_upload_dashcam_files()`, and the sha256 cache
|
||
table is populated and reused across repeated `/api/dash/pending` calls.
|
||
|
||
**Deploy gotcha found**: the CORS fix in `server/app.py` was initially deployed to the Pi but not
|
||
rebuilt on the actual remote server (the server's Docker image bakes `app.py`/`modules/` in at
|
||
build time via `COPY`, so a plain `git pull` + container restart isn't enough — needs
|
||
`docker compose build api && docker compose up -d api`). Without it, the phone's cross-origin
|
||
`fetch()` calls to `yaris.ceaf.dev` were silently blocked by the browser's CORS policy, which
|
||
surfaces in JS as a generic, unhelpful `TypeError: Failed to fetch` — worth remembering as the
|
||
first thing to check if this error reappears after a future server-side change.
|
||
|
||
**Tested for real against a real phone and the real deployed server** (first end-to-end proof
|
||
beyond the local simulation above): the initial `/api/dash/pending` call took ~25 minutes for a
|
||
one-time full-backlog SHA256 hash of all 87 files (~28GB, no cache existed yet) — after that,
|
||
repeat calls returned in ~0.13s. The user then successfully relayed a real file from their phone
|
||
(home WiFi) partway through (10/~12 chunks) before stopping to check something important: **is
|
||
the relay actually faster, given the phone downloads from the Pi over the same congested radio
|
||
diagnosed as the bottleneck?** Measured directly from chunk-arrival timestamps on the server:
|
||
**~1.11MB/s (~8.9Mbit/s) for the phone-relay path — statistically the same as the ~9-10Mbit/s
|
||
measured for the Pi's own direct Cloudflare upload** earlier this session. **Conclusion: when the
|
||
phone is on the same home WiFi as the Pi, the relay provides no speed benefit** — the
|
||
download-to-phone leg pays the exact same slow-radio cost as a direct Pi upload would, and the
|
||
phone's reupload leg then goes out through the same home internet connection anyway, adding an
|
||
extra hop for no gain. The relay only has genuine value when the phone's *second* leg differs
|
||
meaningfully from the Pi's own path — e.g. cellular data with better upload than the home ISP, or
|
||
uploading after physically leaving home with files already grabbed locally. Stopped the test relay
|
||
(`POST /api/dash/relay/stop`) and confirmed the Pi's own background uploader resumed the same
|
||
file's chunked upload exactly where the phone left off (shared `/status`/`.upload_tmp` state on
|
||
the server made this seamless, no wasted re-transfer).
|
||
|
||
**Next idea (not yet built)**: the user wants a way to browse/preview individual dashcam clips on
|
||
the phone and select specific ones to view or upload (as opposed to bulk-relaying the entire
|
||
backlog) — useful for "just grab this one important clip" rather than a wholesale speed
|
||
optimization. Not scoped/designed yet.
|
||
|
||
**Update — tested the "away from home" hotspot+cellular scenario for real, found a real phone-OS
|
||
limitation (not a code bug)**: the user asked to test the specific scenario the relay feature was
|
||
actually designed for — phone connects to the **Pi's own hotspot** (not home WiFi) to download
|
||
pending files, then re-uploads using the phone's **cellular data**, bypassing the Pi's own (in
|
||
hotspot mode, nonexistent) internet connection entirely. Confirmed via code review this needs zero
|
||
changes: `/api/dash/relay/config` already hands the phone the public `https://yaris.ceaf.dev`
|
||
`SERVER_URL` (not a LAN-only address), so the re-upload leg was always going to be internet-bound
|
||
regardless of which physical network carries it.
|
||
|
||
Live-tested with a real dummy file (dashcam upload toggle paused first so the Pi's own uploader
|
||
couldn't grab it before the phone did): got `TypeError: Failed to fetch` trying to relay. Confirmed
|
||
via a direct test (loading `https://yaris.ceaf.dev/health` as a plain page navigation, not a
|
||
`fetch()`, ruling out CORS) that **the phone's own OS was declining to route internet-bound traffic
|
||
over cellular while still associated with the Pi's no-internet-gateway hotspot WiFi** — it only
|
||
succeeded once the phone auto-fell-back to the home WiFi network (which does have internet)
|
||
instead. This is a phone/OS-level "smart network switching" behavior (common on both iOS and
|
||
Android to avoid using a WiFi network with no internet gateway) that cannot be fixed in this
|
||
project's code — the architecture is sound and was already proven correct in the local end-to-end
|
||
simulation and the byte-for-byte reassembly tests earlier in this phase.
|
||
|
||
**Practical implication**: testing this exact scenario **at home is inherently confounded** — the
|
||
phone always has a known-good home WiFi it can silently fall back to, defeating the test. A true
|
||
validation needs either (a) actually being away from home with no known WiFi in range, forcing the
|
||
phone to use cellular for internet traffic, or (b) manually disabling the phone's WiFi (or its
|
||
"switch to mobile data automatically" setting) while connected to the Pi's hotspot, to force
|
||
cellular use for this one operation.
|
||
|
||
**Update — retested more rigorously, confirmed this is a fundamental architecture-level dead end,
|
||
not just an untested corner case.** Re-ran the same live test multiple times with longer/managed
|
||
"dead-man's-switch" windows (`yaris-netwatch.timer` explicitly stopped during each test run so it
|
||
wouldn't yank the Pi back to home WiFi mid-test — an earlier run's failure was actually caused by
|
||
netwatch reconnecting home WiFi under the phone, see Phase 6 update, not a phone-network issue).
|
||
With netwatch correctly disabled and the Pi confirmed to stay in AP mode for the whole test window,
|
||
still got `Failed to fetch`. Directly tested a plain page navigation (not `fetch()`) to
|
||
`https://yaris.ceaf.dev/health` with WiFi *fully turned off* (pure cellular) — **that also failed**,
|
||
while every other website loaded fine over the same cellular connection with WiFi off. This
|
||
narrows the cause precisely: **the browser/phone genuinely cannot reach the public internet at all
|
||
while `wlan0`/the phone's WiFi radio is associated with the Pi's hotspot, full stop** — this is not
|
||
a settings toggle problem (checked Pixel-specific "switch to mobile data automatically" options,
|
||
not present on this device/Android version) but a fundamental **OS-level "default network" routing
|
||
decision**: a device's default route for *all* IP traffic (including a web page's `fetch()` calls)
|
||
is a single, whole-device choice made by the OS's connectivity manager, not something a browser
|
||
page's JavaScript can override per-request. Real per-request network binding (e.g. Android's
|
||
`ConnectivityManager.bindProcessToNetwork()`) is an API only available to **native
|
||
apps**, not exposed to web pages at all, in any browser, on any phone. Given this, the phone-relay
|
||
feature as designed (a web page doing simultaneous WiFi-to-Pi + cellular-to-server transfers)
|
||
**cannot ever work while the phone is connected to the Pi's no-internet hotspot** — this isn't a
|
||
bug to fix, it's a hard platform ceiling.
|
||
|
||
**Decision: removed the phone-relay feature entirely** (was Phase 11, now reverted - see below).
|
||
Two ways to actually deliver on the original "upload from the road without the Pi's slow WiFi"
|
||
goal were discussed: (1) restructure as two distinct, non-simultaneous phases (fully download to
|
||
the phone's local storage first while on the Pi's hotspot, then have the user manually switch to
|
||
cellular, then upload from local storage) - no native app needed but a more awkward two-step user
|
||
flow, or (2) a real native Android app, which *can* hold both networks open at once via
|
||
`bindProcessToNetwork()` for a true simultaneous relay. Neither has been built - shelved for now
|
||
in favor of simplifying the dashcam retrieval UI (see the new phases below). The Pi's own
|
||
background uploader (plain WiFi/Ethernet, optionally SIM868 cellular fallback for tracking data
|
||
only) remains the only upload path; it's just slower when the Pi's own WiFi is weak, with no
|
||
workaround currently in place for that specific case.
|
||
|
||
**Side-fix from this test session**: found and fixed a real, unrelated bug in `yaris-netwatch.sh`
|
||
during this same test — see the Phase 6 update above.
|
||
|
||
### Phase 12: Remove phone-relay feature — ✅ DONE
|
||
Undo Phase 11 cleanly given the Phase 11 update above (fundamental browser platform limitation,
|
||
not fixable). Removed:
|
||
- `app.py`: `/api/dash/pending`, `/api/dash/file/<path>`, `/api/dash/relay/config`,
|
||
`/api/dash/relay/heartbeat`, `/api/dash/relay/stop`, `/api/dash/relay/mark_uploaded` routes
|
||
(and the now-unused `os`/`send_file`/`abort`/`safe_join` imports they needed).
|
||
- `modules/app_upload.py`: `list_pending_dashcam_files()`, `relay_heartbeat()`/`relay_stop()`/
|
||
`is_relay_active()`, `RELAY_LEASE_SECONDS`/`_relay_lease_until`, and the `is_relay_active()`
|
||
check inside `_upload_dashcam_files()`/`get_status()`.
|
||
- `templates/dash.html`: the "Relay via phone" card.
|
||
- `static/js/dash.js`: all relay JS (`relayOneFile`, `fetchChunkWithRetry`,
|
||
`uploadChunkWithRetry`, heartbeat timers, pending-file list rendering, etc).
|
||
- `server/app.py`: the CORS `after_request` hook, added solely for the phone's cross-origin
|
||
browser calls (the Pi's own Python `requests`-based client was never subject to CORS - that's
|
||
a browser-only enforcement mechanism).
|
||
- Kept untouched, as planned: the chunked upload protocol itself
|
||
(`/api/v1/dashcam/upload/chunk` + `/status`), `dashcam_hash_cache` table + `_hash_dashcam_file()`
|
||
(still used by the Pi's own background uploader to avoid rehashing its 20GB+ backlog every
|
||
loop), `DASHCAM_CHUNK_SIZE`/Cloudflare 100MB workaround - none of that is relay-specific.
|
||
|
||
**Verified**: `python3 -m py_compile` on all touched Python files, `node --check` on `dash.js`,
|
||
grepped all five touched files for any leftover `relay`/`RELAY` reference (none found). Smoke-
|
||
tested by actually importing `app.py` in a scratch venv and hitting it with Flask's test client:
|
||
app imports cleanly (GPS/OBD/SQL/upload startup all log their normal "no hardware" messages, no
|
||
crash), `app.url_map` confirms every relay route is gone while `/`, `/dash`, `/dtc`, and the
|
||
remaining dashcam/upload API routes are still registered, and both `/` and `/dash` render with
|
||
HTTP 200 (`/dash`'s HTML contains zero occurrences of "relay"). Deployed to both the Pi
|
||
(`git pull --ff-only` + `systemctl restart yaris-obd.service`, confirmed `active` and `/dash`
|
||
returns HTTP 200 with no "relay" text over the real network) and the remote server (needed a full
|
||
`docker compose build api && docker compose up -d api` on unraid, not just a restart, since the
|
||
CORS-hook removal is baked into the image at build time - confirmed `/health` still returns
|
||
`{"status":"ok"}` and the response no longer carries any `Access-Control-*` header).
|
||
|
||
**Update 2026-08-09**: Phase 13 (OBD hardening) was completed next; see its section below.
|
||
Phase 15 is the recommended next software phase because Phases 16/17 and the later list-based
|
||
features depend on its shared UI foundation. Phase 14 remains independent but requires a careful
|
||
hardware/filesystem deployment pass rather than ordinary application deployment.
|
||
|
||
### Phase 13: OBD real-hardware readiness (code hardening) — ✅ DONE
|
||
**Reordered from Phase 17 on 2026-08-09** because it was completed directly after Phase 12;
|
||
the phase numbers now follow actual execution order instead of leaving completed work below
|
||
several still-planned phases.
|
||
**Independent of everything else in this plan — only touches `app_obd.py`/`app_events.py`, no UI
|
||
dependency for the backend part, so this can be done anytime/in parallel with the phases above.**
|
||
Nothing has been installed or tested in the actual 2013 Yaris yet - `modules/app_obd.py` has
|
||
only ever been tested standalone on the Pi with nothing plugged into the OBD-II port (correctly
|
||
reports "no connection to car"). Per explicit user decision, this phase is **code hardening
|
||
only** - the user will handle the physical install/bring-up testing themselves. Planned fixes,
|
||
all in `modules/app_obd.py` / `modules/app_events.py`:
|
||
- **Add a genuine background reconnect loop.** Today, `connect_obd()`/`obd_connection()` only
|
||
ever run once per browser tab's socket `connect` event (`app_events.py`'s
|
||
`connect_event()`) - there is no periodic retry. In the real car, this means: if the page
|
||
happens to load before the adapter/ignition is ready, the connection attempt fails once and
|
||
never retries until another browser tab connects. Add a small polling loop (mirroring the
|
||
pattern already proven in `app_gps.py`'s `_poll_loop()`) that periodically calls
|
||
`connect_obd()` while disconnected, independent of any browser being open.
|
||
- **Catch exceptions from `connection.query()`.** Currently a communication hiccup
|
||
(adapter timeout, malformed response, USB blip) would raise inside `obd_speed()`/`obd_rpm()`
|
||
uncaught, silently killing that polling thread forever with no recovery - not just returning
|
||
a null response, an actual crash. Wrap query calls, treat exceptions like a lost connection
|
||
(stop polling, let the reconnect loop above pick it back up) instead of dying silently.
|
||
- **Add an `OBD_PROTOCOL` env override**, mirroring the existing `OBD_PORT` override pattern -
|
||
in case the 2013 Yaris's ECU doesn't cleanly auto-negotiate via ELM327's `ATSP0` (auto) and
|
||
needs an explicit protocol code forced (Toyotas this age are normally ISO 15765-4 CAN 11-bit
|
||
500kbaud, but this has never been confirmed against this specific car).
|
||
- **Improve connection-failure diagnostics.** Distinguish and log clearly: no adapter detected
|
||
at all (USB/port issue) vs adapter detected but ELM327 handshake failed vs handshake OK but no
|
||
response from the car's ECU (car off/ignition not on) - currently all three collapse into the
|
||
same generic "OBD connection failed" message, which will make first-real-car troubleshooting
|
||
much harder than it needs to be.
|
||
- **Add a small connect-time diagnostic log** (ELM327 firmware version + negotiated protocol,
|
||
both available from `python-obd`'s connection object) written once per successful connect -
|
||
useful evidence to sanity-check against Toyota's expected protocol on the very first real test.
|
||
- **Expand the data pulled from the car beyond speed/RPM** (backend only - the dashboard cards
|
||
displaying this land in Phase 23 once the redesign reaches them): coolant temperature, fuel
|
||
level, throttle position, and DTC/MIL status (check-engine light on/off + stored trouble codes,
|
||
reusing/wiring up the already-scaffolded but currently-empty `/dtc` page and `GET_DTC` command).
|
||
**Odometer/mileage is not guaranteed** - the standard OBD-II mode 1 PID for total odometer
|
||
(0xA6) was only added to the SAE J1979 spec in later revisions and is inconsistently supported
|
||
across manufacturers/years; whether the 2013 Yaris's ECU exposes it at all can only be
|
||
confirmed once tested against the real car (falls under "code hardening only" for now - add
|
||
the query with graceful "unsupported" handling rather than assuming it will work).
|
||
|
||
**Implemented 2026-08-09**:
|
||
- Added an idempotent, browser-independent reconnect daemon started from `app.py`; it retries
|
||
every 10 seconds and restarts any dead polling workers after reconnect.
|
||
- Serialized all python-obd queries so speed, RPM, and the expanded sensor worker cannot issue
|
||
overlapping serial exchanges. Query exceptions and silent connection-state changes now close
|
||
and discard the stale handle, emit disconnected state, and hand recovery back to the reconnect
|
||
loop instead of permanently killing a worker.
|
||
- Added `OBD_PROTOCOL` support and distinct logs for no serial adapter, failed ELM327 handshake,
|
||
and ELM327 connected but no ECU response. Successful connections log port, ELM327 firmware,
|
||
negotiated protocol ID, and protocol name once.
|
||
- Added backend polling/emits for coolant temperature, fuel level, throttle position, MIL/DTC
|
||
status, stored DTCs, and SAE PID `0xA6` odometer. Odometer uses a custom python-obd command with
|
||
a null/unsupported result left as `None`; support is not assumed for the 2013 Yaris.
|
||
- Verified against the Pi's installed python-obd 0.7.3 API, plus focused mocked checks covering
|
||
successful connection, forced protocol, ECU-off status, silent disconnect cleanup, query
|
||
exceptions, and odometer decoding. The Pi currently has `/dev/ttyUSB0` present but is not
|
||
connected to the car, so actual PID availability and negotiated Toyota protocol still require
|
||
the user-planned first in-car test.
|
||
- Committed as `8d7909d`, pushed, and deployed to the Pi. After restart the service remained
|
||
`active`, `/` returned HTTP 200, and the log showed four autonomous reconnect attempts with the
|
||
correct "ELM327 handshake succeeded, but the car ECU did not respond" diagnosis and no traceback.
|
||
|
||
### Phase 14: Power-loss resilience & watchdog — 🟡 PARTIAL (watchdog done; filesystem/shutdown deferred)
|
||
**New phase, not code-dependent on anything else — OS/hardware-level, can be done anytime,
|
||
ideally before the Pi lives in the car full-time.** A car cuts power abruptly at every key-off;
|
||
nothing in the project currently protects against that, and repeated unclean power loss is a
|
||
classic cause of SD card corruption on embedded Pi projects. Two complementary angles:
|
||
- **Filesystem hardening**: convert the Pi's root filesystem to read-only with a writable overlay
|
||
(`overlayroot`/`raspi-config`'s built-in overlay FS option), so an abrupt power cut can no longer
|
||
corrupt the OS itself. App data that must persist (`db/sql/`, `dash/videos/`, logs) needs to
|
||
live on an explicitly writable partition/bind-mount carved out of the overlay, since those
|
||
still need real writes.
|
||
- **Clean shutdown before power loss**: either (a) a small delayed-shutdown power module/circuit
|
||
(holds 5V for a few seconds after ignition-off so the Pi can run `shutdown -h now` cleanly), or
|
||
(b) if the SIM868 HAT or another ADC/GPIO source can sense ignition/voltage state, a software
|
||
voltage-drop watcher that triggers a clean shutdown proactively. Exact approach TBD - depends
|
||
on what's easiest to wire into the existing power setup.
|
||
- **Hardware watchdog**: enable the Pi's built-in `bcm2835_wdt` watchdog (via `systemd`'s
|
||
`RuntimeWatchdogSec`/`watchdog-device` support) so a fully hung app or kernel gets a hard reboot
|
||
automatically instead of sitting dead in the car until someone notices.
|
||
|
||
**Implemented 2026-08-09**:
|
||
- Added `deploy/yaris-watchdog.conf` with systemd's hardware watchdog set to 10 seconds and
|
||
`deploy/install_watchdog.sh` to install, activate, and verify it. Deployed to the Pi; confirmed
|
||
`/dev/watchdog0` is the Broadcom BCM2835 watchdog and systemd reports
|
||
`RuntimeWatchdogUSec=10s`.
|
||
- Rebooted the Pi and verified the watchdog setting persisted, systemd returned to `running`,
|
||
`yaris-obd.service` returned to `active`, and `/` returned HTTP 200 after startup.
|
||
- **Explicitly deferred by user decision**: read-only root/filesystem hardening and clean-shutdown
|
||
hardware. The current 59GB root partition fills the SD card and contains 28GB of dashcam data,
|
||
so overlay root needs a separate persistent writable storage decision first; shrinking the live
|
||
mounted root filesystem is not safe. Clean shutdown likewise needs delayed-power or
|
||
ignition-sense hardware before software can implement it.
|
||
|
||
### Phase 15: UI design-system foundation — ✅ DONE
|
||
**New phase, pulled forward from the old "do the whole redesign at the end" plan.** Building the
|
||
Phase 16/17 file browsers (and later the trip-history/Pi-health/settings pages) against the
|
||
current old admin-template style just means rebuilding the same UI twice once the full redesign
|
||
(Phase 23) happens. Instead, do a *small, scoped* slice of the redesign right now: a shared page
|
||
layout/shell plus one reusable list/table component (sortable columns, per-row action buttons,
|
||
delete-with-confirmation pattern, empty-state messaging) that every subsequent list-based feature
|
||
in this plan (dashcam browser, video browser, trip history, settings) can be built on directly,
|
||
instead of writing bespoke list/delete UI code from scratch each time. Not a full redesign of
|
||
every existing page yet - just establishes the pattern once so nothing built after this point
|
||
needs to be redone in Phase 23.
|
||
|
||
**Implemented 2026-08-09**:
|
||
- Added a shared Jinja shell (`templates/base.html`) for navigation, top bar, footer, common
|
||
styles/scripts, page titles, and per-route active navigation. Converted `/`, `/dash`, and
|
||
`/dtc` to extend it, removing their duplicated shell markup while preserving page content and
|
||
page-specific assets.
|
||
- Added a reusable data-table macro (`templates/components.html`) and generic
|
||
`static/js/data-list.js` behavior for dynamic empty states, numeric/text sorting, accessible
|
||
sort direction, and capture-phase confirmation that can stop destructive row handlers before
|
||
they execute. `static/yaris.css` supplies the shared list/action styling.
|
||
- Used the empty `/dtc` page as the first component consumer without pulling its Phase 23 OBD
|
||
wiring forward. Also stopped loading dashboard-only `yaris.js` there and guarded its optional
|
||
`#obd-error` target after browser testing exposed an existing null dereference.
|
||
- Verified Jinja rendering and one active nav item on all three routes, JavaScript syntax, and
|
||
live Pi HTTP 200 responses. Browser-tested all pages at 1280px and 390px with no horizontal
|
||
overflow; injected sample rows confirmed ascending sort, `aria-sort`, empty-state switching,
|
||
and cancellation of a destructive handler.
|
||
|
||
### Phase 16: Dashcam USB file browser (list + delete) — ✅ DONE
|
||
Replace the current all-or-nothing "Import videos" flow's blind spot (no visibility into what's
|
||
actually on the camera) with a real file browser for the mounted USB dashcam device, built on the
|
||
Phase 15 shared list component:
|
||
- New endpoint(s) in `modules/app_dash.py` to list every video file on the mounted dashcam
|
||
(relative path, size, modified time) without copying anything - reuses the existing
|
||
`find_video_files()` walk, pointed at the dashcam's mount point instead of the local
|
||
destination folder.
|
||
- Delete support: individual file delete and a "delete all" bulk action, each requiring the
|
||
mount to be briefly remounted read-write (currently always `ro`) - only for the duration of
|
||
the delete call, then back to read-only (or unmounted) afterward, to minimize risk of
|
||
filesystem corruption from an unclean power loss while writable.
|
||
- UI: a `/dash` card showing total file count + total size on the dashcam, a scrollable file
|
||
list with a delete button per row, a "Delete all" button behind a confirmation dialog (real
|
||
data loss risk - no undo).
|
||
- Explicitly deferred/out of scope: no in-browser preview/playback of files still on the
|
||
dashcam itself (only after import - see Phase 17). No changes to the USB
|
||
select-mode/data-communication toggle behavior - the user replugs power manually to return the
|
||
camera to recording mode, as they already do.
|
||
|
||
**Implemented 2026-08-09**:
|
||
- Added list, single-delete, and delete-all backend operations using the existing USB-only storage
|
||
detection. Each list/import/delete operation shares one lock so mount and file operations cannot
|
||
race; listings include relative path, byte size, modified timestamp, count, and total size.
|
||
- Deletes accept only recognized video extensions, resolve and re-check every target beneath the
|
||
selected mount, and reject absolute paths, traversal, missing files, and symlink escapes.
|
||
- Delete operations mount or remount the dashcam read-write only for the operation, then unmount
|
||
or restore it read-only. If unmount fails, the cleanup path immediately attempts a read-only
|
||
remount and reports a cleanup failure if neither safety action succeeds.
|
||
- Added `/api/dash/files` and `/api/dash/file` routes plus a `/dash` browser built on Phase 15's
|
||
shared component: file/size/time sorting, totals, refresh, per-row confirmed delete, confirmed
|
||
delete-all, empty/no-device states, and error feedback.
|
||
- Verified metadata, traversal rejection, one/all deletion, non-video preservation, import
|
||
regression behavior, and cleanup-failure reporting against temporary filesystem fixtures.
|
||
Deployed to the Pi; the live API correctly returned no storage devices while ignoring the
|
||
attached PL2303 OBD adapter, and the responsive browser UI passed no-device and mocked populated
|
||
list/sort/confirmation/delete tests.
|
||
- **Real-hardware deletion verification (2026-08-09)**: connected the 29.7GB FAT dashcam and listed
|
||
all 87 original videos (29,615,111,758 bytes) through the live API. The first controlled delete
|
||
exposed that FAT files were root-owned despite the read-write remount, so unprivileged
|
||
`os.remove()` failed; added a privileged `sudo rm -- <validated-path>` fallback only after the
|
||
existing extension, realpath, and mount-boundary checks pass. Created and deleted a tiny
|
||
disposable `YARIS_OBD_DELETE_TEST.MP4` twice through the real API. Both cycles returned exactly
|
||
to the original 87-file count and byte total, left every real video untouched, unmounted
|
||
`/mnt/dashcam`, and kept the service active. Bulk delete uses the same validated per-file removal
|
||
path but was not run against the real card because it would intentionally erase all footage.
|
||
- **Selected import enhancement and real-hardware verification (2026-08-09)**: added per-row
|
||
Import buttons, checkboxes, Select all/Clear selection, and Import selected so importing is no
|
||
longer all-or-nothing. The backend validates every selected path with the same mount-boundary
|
||
protections as deletion, de-duplicates selections, preserves relative folders, tolerates a
|
||
stale/missing file while importing the remaining valid selection, and keeps the existing
|
||
same-size skip behavior. Before committing, paused background dashcam uploads, created a
|
||
disposable `YARIS_OBD_IMPORT_TEST.MP4` on the physical card, selected it in the real mobile-width
|
||
UI, and confirmed exactly one file copied with identical contents. A row-level repeat correctly
|
||
skipped the existing copy; Select all selected all 88 visible rows and Clear selection reset the
|
||
action. Removed both disposable copies, restored uploads, and verified the card returned to its
|
||
original 87 files/29,615,111,758 bytes unmounted. The refined UI was then re-tested against a
|
||
real existing video: it reported one skip, cleared the checkbox, and re-enabled all controls.
|
||
|
||
### Phase 17: Local imported-video browser + player — ✅ DONE
|
||
Same file-browser treatment for videos already imported to the Pi's own disk
|
||
(`dash/videos/`, populated by `import_dashcam()`), built on the Phase 15 shared list component:
|
||
- List endpoint (reuses `find_video_files()` against `app_dash.DEFAULT_DESTINATION`) showing
|
||
relative path, size, modified time, and upload status (cross-check against
|
||
`app_sql.dashcam_file_marked_uploaded()` - so it's visible at a glance which imported clips
|
||
have already reached the remote server and which are still pending the Pi's own background
|
||
uploader).
|
||
- Delete support: **local Pi copy only** - remote server copies are intentionally left alone
|
||
as the safe backup/archive (explicit user decision - see 2026-08-09 discussion). No
|
||
"also delete from server" option is being built for now.
|
||
- In-browser video player: since the dashcam records **MP4** (confirmed with the user), this
|
||
is a plain HTML5 `<video>` element - no server-side transcoding needed. Reuses the same
|
||
Range-request-capable file-serving pattern already proven in the (now-removed) relay's
|
||
`/api/dash/file/<path>` endpoint (needed for correct seeking/scrubbing, not just playback).
|
||
- UI: a `/dash` card (or a dedicated sub-page if the list gets long) with a file list, a
|
||
"Play" action that opens the video inline, and a delete button per row with a confirmation
|
||
prompt.
|
||
- **QOL: video thumbnails.** Scrolling a list of `VID_00234.MOV`-style filenames to find a
|
||
specific clip is rough. Grab one low-res frame per file via `ffmpeg` (cheap, done once per file
|
||
and cached, not on every page load) so the browser shows an actual thumbnail per row instead of
|
||
just a filename.
|
||
- **QOL: retention/auto-cleanup policy.** Once a file is confirmed uploaded to the remote server,
|
||
it's safe to free local space. Add an optional automatic cleanup (configurable via the Phase 21
|
||
Settings page - e.g. "delete local copies older than N days once uploaded", off by default)
|
||
so the Pi's limited storage doesn't need to be babysat manually as the dashcam backlog grows.
|
||
|
||
**Implemented and real-hardware verified 2026-08-09**:
|
||
- Added local-video list, playback, thumbnail, and local-only delete APIs. Listing reports path,
|
||
size, modified time, and exact upload status without rehashing: one SQLite join matches the
|
||
existing `(relative path, size, mtime) -> SHA256` cache against confirmed uploads.
|
||
- Added a second `/dash` browser using the Phase 15 table component, with lazy thumbnails, sortable
|
||
file/size/time/upload columns, Uploaded/Pending badges, inline HTML5 video playback, confirmed
|
||
"Delete local" actions, totals, and retention status. Remote copies are never deleted.
|
||
- Playback uses Flask conditional file serving; a real `Range: bytes=0-1023` request returned 206,
|
||
the exact `Content-Range`, and 1,024 bytes. Browser-tested a real H.264 1080p MOV at mobile width:
|
||
the player reached readyState 4 with its duration loaded and no media error.
|
||
- Added cached ffmpeg JPEG thumbnails plus `deploy/install_video_tools.sh`; installed ffmpeg 5.1.9
|
||
on the Pi. Real clips exposed and fixed the edge case where ffmpeg exits 0 without output at a
|
||
late seek, so generation now retries frame zero and verifies a non-empty file. A normal 366MB
|
||
clip generated in ~4.1s and its cached request returned in ~81ms. Generation uses a cooperative
|
||
gevent semaphore/poll loop so cold thumbnails do not freeze the web server.
|
||
- Added off-by-default `local_video_retention_days` support. The upload loop checks it after sync;
|
||
only files older than N days whose exact current content is confirmed uploaded are eligible.
|
||
The Phase 21 Settings page will expose the control; `0`/unset remains disabled. Temporary
|
||
filesystem tests confirmed disabled retention deletes nothing and enabled retention preserves
|
||
pending files.
|
||
- Replaced a long-held upload/delete lock with per-path in-use/deleting state and reference counts:
|
||
delete now returns HTTP 409 while that exact file is uploading or generating a thumbnail instead
|
||
of blocking gevent and freezing every request.
|
||
- Deployed uncommitted code before committing and tested against the real 87-file/28GB Pi library:
|
||
all 87 reported uploaded, total bytes matched 29,615,111,758, and retention reported off. Copied
|
||
a tiny real clip to `YARIS_OBD_LOCAL_TEST.MOV`, verified it appeared Pending, played it in the
|
||
browser, and deleted it through the confirmed UI action; the player closed and the library
|
||
returned to exactly 87 uploaded files. Background uploads were paused for the disposable test,
|
||
restored afterward, and the service remained active.
|
||
|
||
### Phase 18: Trip history & replay — ✅ DONE
|
||
Added a derived, idempotently rebuilt `trips` index over the original non-destructive `tracking`
|
||
stream, plus `/api/trips`, bounded `/api/trip/<id>` route responses, and a responsive `/history`
|
||
page using the Phase 15 sortable list and the existing offline Leaflet bundle. Trip rows and replay
|
||
details show distance in kilometres/miles and average/maximum speed in km/h and mph; speed metrics
|
||
prefer OBD wheel speed and fall back to plausible GPS-derived segment speed. The map replay includes
|
||
a timestamped scrubber and play/pause timeline that moves a vehicle-position marker through every
|
||
recorded point while showing its date, time, and recorded speed.
|
||
|
||
**Motion-boundary decision after auditing the real Pi data:** the logger records every five seconds
|
||
for as long as GPS has a fix, including while parked, so timestamp gaps alone are not trip
|
||
boundaries. OBD wheel speed above zero is the primary motion signal. When OBD is absent or stuck at
|
||
zero, three sustained GPS displacements of at least eight metres at normal logger cadence act as
|
||
the fallback; isolated jumps, implausible jumps, malformed coordinates, stationary jitter, clock
|
||
rollback, and five-minute motion gaps are handled explicitly. Trip summaries contain start/end
|
||
time and coordinates, Haversine distance, duration, and point count. Route responses retain both
|
||
endpoints, filter implausible GPS outliers, and downsample only above 2,000 points. OBD samples are
|
||
used only while their last successful query is fresh, preventing a disconnected adapter's stale
|
||
last-known speed from creating a false trip. Rebuilds process an overlap after the persisted
|
||
high-water mark plus the latest trip rather than loading the unbounded full history every time.
|
||
|
||
**Implemented and verified 2026-08-09:**
|
||
- Audited the live database before choosing the heuristic: 5,132 initial rows covering about 25.5
|
||
hours, no null coordinates, all stored speeds zero, only 114 five-decimal coordinate positions,
|
||
and one old NUL-prefixed latitude. Future coordinates are normalized before insertion; invalid
|
||
snapshots are logged and rejected instead of poisoning trip calculations.
|
||
- Replaced unconditional five-second database writes with motion-aware persistence. Sensors are
|
||
still checked every five seconds, but rows are written only for the first fix, fresh OBD motion,
|
||
at least eight metres of accumulated GPS displacement, or a one-minute stationary heartbeat.
|
||
The Pi normally runs only while the car has power, so this is not presented as a full-day parked
|
||
storage problem; it reduces needless writes during bench testing and long idling while retaining
|
||
enough stopped points for traffic pauses in the replay timeline.
|
||
- Nine focused detector tests cover sustained GPS fallback, parked jitter/malformed coordinates,
|
||
inactivity splitting, OBD wheel-speed detection and slow-distance calculation, clock rollback,
|
||
route outlier filtering, and route downsampling. SQLite incremental-build integration was also
|
||
exercised against a disposable database, and five logger tests cover first-fix, parked,
|
||
wheel-speed, GPS-fallback, and heartbeat decisions. A copy of the real database
|
||
produced zero false trips across 5,179 parked rows.
|
||
- Deployed the uncommitted implementation to the Pi and built the schema against the real database
|
||
without modifying any original tracking rows. Inserted an eight-point disposable GPS route while
|
||
the service was stopped; the live API detected one 0.078km/35-second trip and returned all eight
|
||
ordered replay points. Restored the exact pre-test database afterward, confirmed zero synthetic
|
||
rows, `PRAGMA quick_check=ok`, zero derived trips, and an active service.
|
||
- Browser-tested `http://yaris.local/history` through the deployed mDNS/port-80 redirect at a
|
||
390x844 mobile viewport. Empty state, populated sortable row, replay action, offline route
|
||
polyline following every route point, start/end markers, metric/imperial distance and speed
|
||
summary, and no-horizontal-overflow behavior passed. Offline tiles
|
||
now attach only after selecting a route, avoiding irrelevant missing-tile requests around the
|
||
generic empty-map center.
|
||
- With user approval, removed the accumulated stationary bench backlog after verification:
|
||
5,454 local SQLite rows and 2,907 remotely synced Postgres rows, all with zero/null speed and no
|
||
detected trip. Preserved a dated pre-cleanup SQLite backup under `/home/yaris/backups/`, vacuumed
|
||
and integrity-checked SQLite, vacuum-analyzed Postgres, and left unrelated settings, upload
|
||
markers, DTCs, and video data untouched. The deployed motion-aware logger then proved two
|
||
consecutive stationary heartbeats were 60.02 seconds apart rather than five seconds.
|
||
|
||
### Phase 19: Pi health monitoring card — ✅ DONE
|
||
Added a dependency-free `/api/health` endpoint and responsive dashboard card reporting CPU
|
||
temperature, usable disk pressure/free bytes, RAM usage, human-readable uptime, and Raspberry Pi
|
||
firmware power/throttling flags. Data comes from thermal sysfs, `/proc`, `shutil.disk_usage`, and
|
||
`vcgencmd get_throttled`; individual unavailable sources are reported explicitly without hiding
|
||
the remaining readings. The endpoint exposes structured current and historical flags for Phase
|
||
20 alerts, plus warning/degraded/ok status. The browser refreshes every 30 seconds.
|
||
|
||
**Implemented and verified 2026-08-09:**
|
||
- Five focused tests cover memory parsing, missing memory fields, current/historical throttling
|
||
bits, malformed firmware output, and ext4 reserved-block-aware disk pressure. Local non-Pi
|
||
validation correctly reports thermal/firmware sources as unavailable while retaining disk,
|
||
memory, and uptime instead of failing the endpoint.
|
||
- Removed the old browser-triggered thread that emitted uptime once per second indefinitely; the
|
||
health poll now provides uptime cheaply with the rest of the system snapshot.
|
||
- Deployed uncommitted code to the real Pi. The API returned complete data with no source errors:
|
||
about 47-49°C, 19-20% RAM, 59% usable disk pressure with 22.8GB free, and roughly two hours
|
||
uptime during testing. SQLite/app service behavior remained active.
|
||
- Real hardware monitoring immediately found an actionable condition: `vcgencmd get_throttled`
|
||
alternated between `0x50005` (current undervoltage and CPU throttling, both previously occurred)
|
||
and `0x50000` (historical occurrences after the current condition cleared). The dashboard
|
||
prominently distinguishes current warnings from prior events. The Pi power source/cable should
|
||
be corrected before sustained in-car use.
|
||
- Browser-tested the real dashboard through `http://yaris.local/` at 390x844 and 1280x900:
|
||
all four metrics, warning state, timestamp, responsive columns, and 30-second data path rendered
|
||
without overflow or new console errors. Also fixed disconnected OBD speed showing `undefined`
|
||
and deferred offline map tiles until a real GPS fix, avoiding irrelevant placeholder requests.
|
||
|
||
### Phase 20: Alerts/notifications — ✅ DONE
|
||
Added an offline-first notification system. A shared bell in the top bar on every web page shows
|
||
active local alerts with no modem, SIM card, or internet connection. Optional ntfy delivery mirrors
|
||
the same conditions only when explicitly enabled and configured; it defaults off until Phase 21
|
||
exposes its topic URL, optional access token, and thresholds.
|
||
|
||
Monitored conditions:
|
||
- MIL/check-engine state and DTC codes.
|
||
- Pi filesystem pressure and the physical dashcam USB filesystem, checked read-only and cached for
|
||
five minutes to avoid repeated mounts.
|
||
- Pending tracking-row or local dashcam-video upload backlog.
|
||
- Pi CPU temperature and Raspberry Pi firmware undervoltage/throttling flags.
|
||
- GPS fix loss for a configured duration while fresh OBD wheel speed confirms movement.
|
||
|
||
The alert engine runs independently every minute and persists active state, first-active time,
|
||
last attempt/success, delivery errors, recovery state, and last value in SQLite. It sends once on a
|
||
transition, repeats only after the configured cooldown, retries failed deliveries after five
|
||
minutes, and sends a recovery notification after a delivered condition clears. GPS cold starts
|
||
must remain lost for the full duration before alerting, and stale MIL/DTC values are suppressed
|
||
after OBD communication stops. Firmware current flags alert immediately; new historical bits
|
||
remain visible for five minutes, and cleared bits are reconciled so the same fault can alert again
|
||
after reboot. Failed transient deliveries retry every minute during that window. The bell
|
||
explicitly reports a dead/stale monitor instead of silently showing old conditions.
|
||
`/api/alerts/status` exposes sanitized diagnostics without the ntfy URL, token, or request errors
|
||
that could contain a private topic URL.
|
||
|
||
**Implemented and verified 2026-08-09:**
|
||
- Thirteen focused alert tests cover ntfy request shape/authentication/error redaction, transition
|
||
deduplication, cooldown repeats, recovery, failed-delivery retry, disabled delivery, GPS
|
||
duration/reset, firmware-event recurrence, diagnostic freshness, and dashcam-usage caching. The
|
||
combined targeted suite is 33 tests.
|
||
- Deployed uncommitted code and verified the alert thread, state table, status API, and service on
|
||
the Pi. `enabled=false` and `configured=false` produced no external attempt while local alerts
|
||
continued working.
|
||
- Tested the ntfy transport against a temporary HTTP receiver bound only to `127.0.0.1` on the Pi:
|
||
path, UTF-8 body, title, priority, tags, and bearer Authorization header all arrived correctly;
|
||
no third-party request or real token was used.
|
||
- The real read-only physical-dashcam check found 96.1% usage/1.23GB free, unmounted afterward, and
|
||
the original library remained exactly 87 files/29,615,111,758 bytes. The live bell correctly
|
||
showed that storage alert alongside the real Pi undervoltage/throttling alert.
|
||
- Browser-tested the real local bell/dropdown at 390x844 on the dashboard and trip-history page:
|
||
badge counts, alert titles/details, no-alert state, synthetic monitor-failure state, shared-shell
|
||
availability, and no-horizontal-overflow behavior passed with no new console errors.
|
||
|
||
### Phase 21: Settings page — ✅ DONE
|
||
**New phase - depends on Phase 15's shared components, and benefits from Phases 17/20 existing**
|
||
so there's real configuration to consolidate. Several settings currently only exist as
|
||
SSH-edited files or scattered individual toggles (dashcam upload pause switch on `/dash`,
|
||
cellular APN in `db/upload_creds.env`, hotspot SSID/password via `nmcli` directly) or don't exist
|
||
yet (retention days from Phase 17, alert webhook/thresholds from Phase 20). Add one dedicated
|
||
Settings page consolidating anything reasonably safe to expose over the local-only web UI:
|
||
upload toggle, cellular APN, local video retention policy, alert webhook URL + thresholds. WiFi
|
||
SSID/password changes likely stay out of scope (higher-risk to expose over a UI that could lock
|
||
the Pi out of its own network if fat-fingered) unless it turns out to be easy/safe to add later.
|
||
|
||
**Implemented and hardware-tested 2026-08-09.** Added a dedicated local `/settings` page and
|
||
validated `/api/settings` endpoint backed by the existing SQLite settings table. It consolidates
|
||
the dashcam-upload toggle, uploaded-video retention days, SIM868 cellular enable/APN, optional
|
||
ntfy enable/URL/token, and alert temperature/disk/backlog/GPS-loss/repeat thresholds. Updates are
|
||
validated as a complete group and committed atomically; invalid requests leave the prior
|
||
configuration untouched. Cellular enable/APN reads persisted settings live, so those changes do
|
||
not need an app restart. Sensitive ntfy URL/token values are write-only through the API: the page
|
||
only reports whether each is configured and supports explicit removal, so saved secrets are never
|
||
sent back to browser JavaScript. Local Web UI alerts remain active independently of ntfy.
|
||
|
||
When cellular is configured, ntfy delivery now uses the SIM868's small-body HTTP client as the
|
||
primary transport with the notification title, priority, tags, and optional bearer token.
|
||
WiFi/Ethernet is the fallback if the modem path fails. The generalized modem POST path rejects
|
||
quotes/newlines in AT-command parameters before touching hardware. This is mock/Pi-code tested but
|
||
remains explicitly unverified over a real carrier until a SIM is available; Phase 25 must prove
|
||
authenticated HTTPS, fallback, retry, and recovery behavior or remove ntfy entirely if the SIM868
|
||
cannot deliver it reliably.
|
||
|
||
Moved the dashcam-upload control from `/dash` into Settings rather than keeping duplicate controls,
|
||
while retaining pending-row, last-success/path, and last-error upload health in the new page.
|
||
WiFi credentials, remote-server device/API credentials, cellular username/password, and other
|
||
lockout/high-risk configuration remain file/SSH managed by design. Focused settings/alert tests
|
||
pass locally and on the Pi; the real Pi rejected an invalid update with HTTP 400 while preserving
|
||
its exact configuration. The live page loaded through `yaris.local` with the Pi's existing
|
||
defaults and shared alert/navigation shell. Browser testing at 390x844 found and fixed an initial
|
||
top-bar overlap by adding a dedicated bottom mobile save action; the real save then completed with
|
||
no horizontal overflow or console errors.
|
||
|
||
### Phase 22: Phone-hotspot internet uplink — ✅ DONE
|
||
Add a no-SIM internet path where the Pi joins a user's phone hotspot as a WiFi client and uses the
|
||
phone's cellular data for outbound synchronization. This must work with an ordinary iPhone or
|
||
Android hotspot and support tracking rows, alert events/state, software/server checks, and large
|
||
dashcam-video uploads without requiring a modem SIM.
|
||
|
||
The Pi currently creates its own hotspot for the local Web UI, so this phase must explicitly solve
|
||
the single-radio networking tradeoff rather than assuming it can be an access point and hotspot
|
||
client simultaneously:
|
||
- Add a Settings workflow to save/replace/remove a phone-hotspot SSID and password without exposing
|
||
the saved password back to browser JavaScript. Validate before applying and provide a recovery
|
||
path so a bad credential cannot permanently lock the user out of `yaris.local`.
|
||
- Prefer a deterministic NetworkManager design: when the configured phone hotspot is available,
|
||
connect as a client for internet access; when it is unavailable or association fails, restore the
|
||
Pi-hosted hotspot automatically within a bounded timeout. Preserve the existing boot/network
|
||
watchdog behavior and document the exact priority/retry rules.
|
||
- Because a Pi 3 single WiFi interface cannot reliably host and join separate networks at once,
|
||
test and document how the user reaches the local Web UI while connected to the phone hotspot.
|
||
Prefer `http://yaris.local/` on the shared phone LAN when the phone permits peer traffic. If
|
||
iOS/Android client isolation or mDNS behavior prevents that, keep the automatic Pi-hotspot
|
||
fallback and present clear current-mode/status guidance rather than promising simultaneous access.
|
||
- Surface current uplink mode, connected SSID without password, IP address, internet reachability,
|
||
last transition/error, and whether large video upload is permitted. Never mistake association
|
||
with the phone for proven internet reachability.
|
||
- Keep dashcam video off SIM868 cellular, but allow it through the phone hotspot's WiFi path.
|
||
Tracking, server alert synchronization, and other queued data must resume idempotently after the
|
||
phone hotspot appears. Existing cellular-first ntfy behavior remains separate.
|
||
- Hardware-test with at least one real phone: cold boot with hotspot present/absent, hotspot
|
||
enable/disable while running, wrong credentials, phone leaving/re-entering range, internet loss
|
||
while still associated, automatic Pi-hotspot recovery, local `yaris.local` access, tracking
|
||
catch-up, and a disposable chunked video upload. Record phone/OS limitations before committing.
|
||
|
||
**Implemented and hardware-tested 2026-08-09 with a real Pixel hotspot.**
|
||
- Added a root-owned `yaris-uplink` NetworkManager helper, systemd timer/wait service, dispatcher
|
||
transition recorder, restricted sudoers entries, and Settings APIs/UI. Priority is
|
||
`phone-hotspot` (20), `home-wifi` (10), then the Pi `Hotspot` (-10). A non-disruptive scan checks
|
||
that the phone SSID is actually visible before replacing a working home connection.
|
||
- The intended single-phone workflow is **save credentials first → select Wait for hotspot → enable
|
||
the phone hotspot**. The wait service gives the user ten seconds before dropping the Pi AP, scans
|
||
for up to two minutes, and always restarts the periodic watcher. If the phone never appears or
|
||
later disappears, NetworkManager/helper recovery selects home WiFi when available and otherwise
|
||
restores the Pi hotspot. Multiple dead-man tests independently forced home WiFi recovery.
|
||
- Phone credentials go from browser to the privileged helper over stdin and into an atomic
|
||
root-only (`0600`) NetworkManager keyfile. Passwords never enter SQLite, command arguments,
|
||
status files, HTML/JavaScript responses, or read APIs. The UI exposes only configured state and
|
||
SSID, supports replace/remove, and preserves in-progress edits during status polling.
|
||
- Status reports actual connection/mode/IP, NetworkManager-proven internet connectivity, whether
|
||
large uploads are currently possible, last transition/error, and the last phone-assigned direct
|
||
address. The non-secret transition record persists across reboot under `/var/lib/yaris-obd`.
|
||
- Pi-hosted hotspot testing initially found that the real phone could open `10.42.0.1` but not
|
||
`yaris.local` even though Avahi correctly joined/published on `wlan0`. Added a NetworkManager
|
||
shared-dnsmasq unicast fallback for `yaris.local`; the repeated dead-man test then passed.
|
||
- Pixel tethering has a different limitation: the hotspot-host phone can reach the Pi by its DHCP
|
||
address, but relays neither mDNS (`yaris.local`) nor the DHCP hostname (`yaris`). Settings
|
||
therefore retains and links the last direct phone-network address with a DHCP-change warning.
|
||
The tested Pixel repeatedly assigned the same address, but the implementation does not assume it
|
||
is permanent.
|
||
- Proved cold boot both with Pixel absent (selected home WiFi) and present (selected Pixel directly,
|
||
Web UI/service started, and the server received a fresh heartbeat). Turning Pixel off repeatedly
|
||
restored home WiFi in about 16 seconds. The same fallback code is covered for restoring the Pi
|
||
hotspot when home WiFi is unavailable.
|
||
- Proved real tracking uploads through Pixel data against remote Postgres. The first disposable
|
||
1 MiB video exposed repeated TLS EOF failures over tethering despite small requests working;
|
||
setting only the phone profile MTU to 1400 fixed the path, after which a second 1 MiB disposable
|
||
video reached the real server over Pixel data. Both named local/server files and exact database
|
||
upload/hash rows were removed afterward, and dashcam uploads were restored enabled.
|
||
- The future Phase 24 alert-event synchronization queue will automatically use this normal IP
|
||
uplink; Phase 22 does not prematurely implement that server schema/API.
|
||
|
||
### Phase 23: Finish web UI redesign — ✅ DONE
|
||
**Completed 2026-08-10 (commit 981b24b).** Full responsive redesign of the Pi web UI.
|
||
|
||
**What was done:**
|
||
- Replaced the SB Admin 2-style fixed left sidebar with a purpose-built layout:
|
||
- **Desktop (≥768 px):** dark fixed sidebar (`var(--sidebar-width) = 14 rem`) with collapsible
|
||
icon-only mode (`4 rem`). State persisted via `localStorage`. Logo is a functional home link.
|
||
- **Mobile (<768 px):** no sidebar. Full-width content with a sticky **bottom tab bar** (5 items:
|
||
Dashboard, Dashcam, History, DTC, Settings). Logo shown in the topbar.
|
||
- Removed every placeholder element: "Placeholder button", stub dropdown menus ("Action" /
|
||
"Another action"), inline `style` colour hacks, and meaningless card-header dropdowns.
|
||
- Dashboard now shows **6 OBD stat cards**: OBD status, speed, RPM, coolant temp, throttle,
|
||
fuel level — all wired to live socket events (`obd_coolant_temp`, `obd_fuel_level`,
|
||
`obd_throttle` added to `yaris.js`).
|
||
- GPS info section replaced with clean lat/lon display.
|
||
- DTC page fully wired to a new **`/api/dtc`** route (returns `{mil, codes}` from
|
||
`app_obd.get_fresh_mil()` / `get_fresh_dtc_codes()`). MIL indicator pill + DTC table
|
||
populated from the API on load and on manual refresh.
|
||
- `yaris.css` completely rewritten: CSS custom properties, sidebar, bottom nav, OBD cards,
|
||
`border-start-*` utilities.
|
||
- `theme.js` rewritten: localStorage-backed collapse toggle; no Bootstrap Collapse dependency.
|
||
- All 63 tests still pass. Verified on Pi at both 390×844 (mobile) and 1280×800 (desktop).
|
||
|
||
### Phase 24: Remote trip + dashcam portal with shared web UI — 🚧 PLANNED
|
||
**Added 2026-08-09; intentionally scheduled after Phase 23.** Phase 23 first establishes the final
|
||
visual language and complete in-car interface. Building the remote portal afterward avoids
|
||
duplicating the current unfinished theme and lets both deployments deliberately share the same
|
||
finished shell, components, trip list, route map, timeline replay, dashcam library, units, and
|
||
responsive behavior.
|
||
|
||
Extend the existing self-hosted `server/` deployment from an upload-only API into a secure remote
|
||
web portal backed by its Postgres tracking data and uploaded dashcam archive, so trip history,
|
||
recorded trip details, and footage can be accessed away from the car:
|
||
- Add server-side trip boundary/index data equivalent to Phase 18, generated from uploaded tracking
|
||
points without changing or deleting the raw records. Expose trip summaries, full/downsampled
|
||
routes, timestamps, distance in km/mi, average/max speed in km/h and mph, and timeline replay.
|
||
- Reuse the redesigned Phase 23 Jinja/Bootstrap UI rather than creating a visually separate app.
|
||
Extract/share presentation assets where practical so local and remote pages do not drift, while
|
||
keeping their data providers and available actions explicit.
|
||
- Make the uploaded dashcam archive a required first-class remote page, matching the redesigned
|
||
in-car imported-video library: sortable/searchable file list, relative path, size, timestamp,
|
||
thumbnails, inline H.264 playback with Range/seeking support, and an explicit download action for
|
||
individual original files. Downloads must stream from server storage with safe
|
||
`Content-Disposition`, bounded/traversal-safe paths, and no whole-video memory buffering. Remote
|
||
deletion is not included unless it is designed later with separate safeguards; the server remains
|
||
the archive/backup.
|
||
- Put an always-visible environment indicator in the shared shell: **In-car / Local Pi** on
|
||
`yaris.local`, and **Remote server** on the hosted portal. Page title, navigation, and status
|
||
styling must also make the context unambiguous so the user cannot mistake remote historical data
|
||
for a live connection to the car.
|
||
- The remote portal is historical/read-only. It should feel like the same product, but render only
|
||
server-relevant navigation and data—not redacted/empty imitations of unavailable hardware.
|
||
Physical dashcam mount/import/delete, live OBD/GPS, cellular status/configuration, Pi health, and
|
||
local Pi settings are omitted entirely. Trip history/replay and the uploaded dashcam archive are
|
||
the required remote scope.
|
||
- Add real human-user authentication and secure sessions for the internet-facing UI. Existing
|
||
device `X-Device-Name`/`X-Api-Key` upload authentication remains separate and must not become a
|
||
browser login mechanism. Preserve Cloudflare Tunnel compatibility and do not expose secrets in
|
||
HTML/JavaScript.
|
||
- Support selecting a vehicle/device if more than one is registered later, while defaulting
|
||
cleanly to the current Yaris. Keep trip APIs scoped to the authenticated user/device.
|
||
- Synchronize alert transitions, recoveries, timestamps, severity/details, and current state from
|
||
the Pi to the server through an authenticated, idempotent offline queue. The Pi remains
|
||
authoritative for vehicle/hardware detection and its local bell must never depend on internet
|
||
access. The server stores per-vehicle alert history/current state, reconciles reconnects without
|
||
duplicates, and exposes it in the remote portal; remote acknowledgement must not silently clear
|
||
the underlying Pi condition.
|
||
- Synchronize alert transitions, recoveries, timestamps, severity/details, and current state from
|
||
the Pi to the server through an authenticated, idempotent offline queue. The Pi remains
|
||
authoritative for vehicle/hardware detection and its local bell must never depend on internet
|
||
access. The server stores per-vehicle alert history/current state, reconciles reconnects without
|
||
duplicates, and exposes it in the remote portal; remote acknowledgement must not silently clear
|
||
the underlying Pi condition.
|
||
- Deploy only after Phase 23 is complete: rebuild the unraid Docker stack, verify migrations against
|
||
copied data first, compare remote trip calculations with uploaded Pi records, verify playback,
|
||
Range requests, and downloads against disposable copies of real server footage, and browser-test
|
||
desktop/mobile through the real hosted endpoint before committing.
|
||
|
||
### Phase 25: Pre-install Yaris 2013 compatibility test — 🚧 PLANNED (core-system gate)
|
||
**Added 2026-08-09; intentionally immediately before the Android app.** Bench tests, synthetic
|
||
routes, attached-hardware tests, and stationary Pi runs cannot prove the complete system works
|
||
when temporarily connected to and powered by the actual 2013 Toyota Yaris. This phase is the
|
||
pre-installation acceptance gate for the complete core Pi/web/server system after Phase 24. It is
|
||
verification and targeted correction—not permanent installation or another redesign—and the
|
||
system is not considered ready to install until the results are recorded here. Permanent mounting,
|
||
cable routing, trim work, and final installation are explicitly outside the roadmap and will be
|
||
handled by the owner after this phase passes.
|
||
|
||
Prepare a temporary, removable test setup before driving:
|
||
- Connect the Pi, SIM868 HAT/antennas, USB OBD adapter, dashcam, and representative intended power
|
||
source without permanently mounting or routing anything. Secure every temporary component and
|
||
cable so it cannot interfere with pedals, steering, airbags, visibility, occupants, or loose-item
|
||
safety during the controlled drive; remove the setup after testing.
|
||
- Resolve the Phase 19/20 undervoltage/throttling warning before the test; confirm
|
||
`vcgencmd get_throttled` shows no current power fault at idle and under load with the temporary
|
||
representative power setup. This validates electrical compatibility, not final cable routing.
|
||
- Back up the Pi SQLite database and record baseline row/trip/video/upload counts on the Pi and
|
||
remote server. Use a uniquely timestamped test window so generated records can be reconciled
|
||
without deleting unrelated history.
|
||
- Verify the service, watchdog, storage mounts, hotspot, `http://yaris.local/`, alert engine, GPS
|
||
antenna view, dashcam recording mode, free space, and remote server health before moving.
|
||
- Use a second person for UI/log observations while the vehicle is moving, or record diagnostics
|
||
for review afterward. The driver must not operate or troubleshoot the UI while driving.
|
||
|
||
Run a controlled route containing ignition-off/on, stationary idle, low-speed streets, normal road
|
||
speed, several complete stops, turns, a short GPS-obstructed segment if safely available, and a
|
||
clean trip end:
|
||
- **OBD/ECU:** confirm the ELM327 reaches `CAR_CONNECTED`, record negotiated Toyota protocol and
|
||
adapter firmware, and verify real speed/RPM/coolant/fuel/throttle/MIL/DTC values. Compare wheel
|
||
speed against the instrument cluster at several safe speeds. Record unsupported PIDs (especially
|
||
odometer PID A6) as capabilities, not failures, and verify reconnect after ignition cycling and a
|
||
deliberate USB disconnect/reconnect.
|
||
- **GPS and motion logging:** confirm fixes while stationary and moving, route coordinates on the
|
||
offline map, OBD-speed-first motion detection, GPS displacement fallback when OBD is deliberately
|
||
unavailable, one-minute idle heartbeat behavior, and no false trip from stationary GPS drift.
|
||
- **Trip boundary/replay:** after the inactivity timeout, verify exactly one trip (or the expected
|
||
number for deliberate ignition gaps), correct start/end time and coordinates, plausible
|
||
distance/average/max speed, km/mi and km/h/mph values, route polyline, start/end markers, timeline
|
||
scrubber, play/pause marker position, and stopped portions of the route.
|
||
- **Local web UI:** test the final Phase 23 interface on the actual phone over the Pi/phone network at
|
||
`http://yaris.local/`; verify the explicit **In-car / Local Pi** identity, dashboard telemetry,
|
||
DTC page, trip history, dashcam library, settings, alerts, responsive layout, reconnect behavior,
|
||
and no new browser console/API errors.
|
||
- **Dashcam workflow:** create a known short real clip, switch the camera safely into USB storage
|
||
mode, detect/list it without confusing the OBD adapter, import only that selected file, verify
|
||
thumbnail/playback/Range seeking, preserve the source until checks finish, and confirm the camera
|
||
returns to recording afterward. Test deletion only with the disposable clip.
|
||
- **Alerts:** safely exercise conditions that can be simulated without damaging hardware (upload
|
||
backlog, temporary GPS loss while OBD reports motion, and disposable threshold overrides).
|
||
Confirm local bell transition/recovery/cooldown behavior and offline alert-state reconciliation
|
||
with the Phase 24 server after connectivity returns. Do not intentionally overheat, undervolt,
|
||
fill storage, or trigger a real MIL. With a real SIM, explicitly determine whether the SIM868
|
||
transport can reliably deliver ntfy notifications while WiFi/Ethernet is unavailable, including
|
||
authentication, timeout, retry, and recovery. If that cannot be made reliable on the actual
|
||
modem/network, remove the ntfy feature and its settings rather than retain a transport that fails
|
||
in the only situation where it is useful.
|
||
- **Power/reliability:** perform multiple normal ignition/power cycles, confirm automatic boot,
|
||
service/watchdog startup, SIM868 power-on, OBD/GPS recovery, no SQLite corruption, and no current
|
||
undervoltage/throttling. The deferred Phase 14 clean-shutdown limitation must be explicitly
|
||
accepted or resolved with representative delayed-power/ignition-sense hardware before declaring
|
||
the design ready for permanent installation.
|
||
|
||
Verify the complete remote path after connectivity returns:
|
||
- Confirm every test-window tracking row uploads exactly once and the selected dashcam clip resumes
|
||
or completes without corruption; compare counts, timestamps, hashes, and byte sizes locally and
|
||
remotely.
|
||
- On the Phase 24 portal, verify the explicit **Remote server** identity, authenticated
|
||
vehicle-scoped trip summary/replay, route and timeline agreement with the local Pi, archived clip
|
||
thumbnail/playback/Range seeking/download, synchronized alert history/current state without
|
||
duplicate transitions after an offline interval, and absence of local-only OBD/cellular/Pi
|
||
controls.
|
||
- The Android companion is intentionally outside this core acceptance gate and remains the final,
|
||
low-priority Phase 26. Its own hotspot/cellular binding, trip display, and disposable clip relay
|
||
require separate real-phone/in-car acceptance when that phase is implemented.
|
||
|
||
Acceptance requires a written result for every item, captured negotiated protocol/PID support,
|
||
measured trip/count/hash comparisons, relevant logs/screenshots, no unexplained tracebacks or data
|
||
loss, and cleanup of every disposable record/file while preserving the real drive. Any defect found
|
||
must be fixed, redeployed uncommitted, and the affected in-car test repeated before the final commit
|
||
and **ready-to-install** sign-off. Afterward, the owner handles permanent installation outside the
|
||
phase roadmap.
|
||
|
||
### Phase 26: Native Android companion app — 🚧 PLANNED (low priority, always last)
|
||
Revives the phone-relay goal (Phase 11/12) properly this time, using a real native app instead of
|
||
a web page - the only way to actually hold a WiFi connection to the Pi's hotspot and a cellular
|
||
connection to the internet open simultaneously (`ConnectivityManager.bindProcessToNetwork()`,
|
||
unavailable to browser JS - see the Phase 11 update). Explicitly **not urgent** and intentionally
|
||
kept at the end of the roadmap, after all web/server work and the Phase 25 core in-car acceptance
|
||
test are done. Permanent vehicle installation remains owner-handled and is not an implementation
|
||
phase. Scope, at a high level (not yet designed in detail):
|
||
- **Dashcam file transfer**: browse/select/delete dashcam clips already imported to the Pi (same
|
||
data as the Phase 17 local video browser, just a native client instead of/alongside the web UI),
|
||
download them to the phone, and upload to the remote server over cellular while the phone stays
|
||
on the Pi's hotspot for the download leg - the true simultaneous dual-network relay the web
|
||
version could never deliver.
|
||
- **Trip/location info**: surface live GPS position, speed, and trip history (Phase 18) pulled
|
||
from the Pi while connected to its hotspot (reusing the existing `/api/gps`-style data feeds),
|
||
so the app is useful as a real "check on the car" companion, not just a file-transfer tool.
|
||
- Not yet scoped: distribution (sideload vs Play Store), whether it talks to the Pi directly over
|
||
the hotspot only or also to the remote server directly for trip history when away from the car,
|
||
and how much of the existing REST API surface can be reused as-is vs needs new
|
||
mobile-friendly endpoints. To be designed once Phases 12-24 (web-side cleanup + hardening,
|
||
redesign, remote portal, and core in-car acceptance) are done.
|
||
|
||
## Working Notes / Environment Gotchas
|
||
|
||
- **Pi credentials/SSH**: `ssh yaris@<pi-ip>`. Confirm current IP before connecting (varies by
|
||
network — check locally, not recorded here) — usually reachable via `yaris.local` too (mDNS,
|
||
see Phase 6).
|
||
- **Local dev repo**: work happens on a separate dev machine, not on the Pi. Push to
|
||
`origin/shameless-aislop`, the repo's current default branch (repo is public).
|
||
- **unraid server SSH**: `ssh [email protected]` (alias `unraid`). Real git checkout lives at
|
||
`/mnt/user/app/git/yaris-obd` (not `/mnt/user/appdata/...`, which is only the docker-managed data
|
||
volume for the `db`/`dashcam` containers). Deploy with
|
||
`cd /mnt/user/app/git/yaris-obd && git pull --ff-only && cd server && docker compose up -d --build`.
|
||
Container names: `yaris-obd-db` (Postgres 17), `yaris-obd-api`.
|
||
- **Pi deploy loop**: `ssh yaris@<pi-ip>` (alias `yarisobd`) ->
|
||
`cd ~/app/yaris-obd && git pull --ff-only` -> `sudo systemctl restart yaris-obd.service`
|
||
(passwordless sudo confirmed via `yaris ALL=(ALL) NOPASSWD: ALL`). App stdout/stderr goes to
|
||
`~/app/yaris-obd/yaris-obd.log` (not journald - the systemd unit redirects both streams there),
|
||
so use `tail`/`grep` on that file, not `journalctl`, to see application-level prints (upload
|
||
progress, OBD/GNSS errors, etc.) - `journalctl -u yaris-obd.service` only shows service
|
||
start/stop/crash lifecycle events. **Update 2026-08-09**: plain `print()` calls were silently
|
||
invisible in that log for months because Python fully-buffers stdout when it's not a TTY (a
|
||
file, in this case) — `logging`-based lines (e.g. the `obd` library's own logger, which uses
|
||
stderr) showed up fine, but bare `print()` diagnostics from `app_gps.py` never did. Fixed by
|
||
adding `Environment=PYTHONUNBUFFERED=1` to the systemd unit
|
||
(`deploy/yaris-obd.service`) — **remember to also copy the updated unit file to
|
||
`/etc/systemd/system/yaris-obd.service` and run `sudo systemctl daemon-reload`** on the Pi, a
|
||
plain `git pull` + restart does NOT pick up systemd unit file changes (the live
|
||
`/etc/systemd/system/` copy is independent of the repo's `deploy/` copy).
|
||
- **Pi deploy loop, requirements.txt changes**: after any `requirements.txt` change (e.g. adding
|
||
`RPi.GPIO`), also run `~/app/yaris-obd/venv/bin/pip install -r requirements.txt` on the Pi — a
|
||
`git pull` alone doesn't install new dependencies into the venv.
|
||
- **Pi memory budget**: ~900MB usable RAM, no swap. Any code path that might buffer a whole
|
||
dashcam-sized file (hundreds of MB) in memory is unsafe on this hardware - stream, don't buffer
|
||
(see Phase 10).
|
||
- **Git push auth**: if `git push` fails with a credential/auth error, run `gh auth login`
|
||
(device-code browser flow) then `gh auth setup-git` to wire git to use the `gh` CLI's stored
|
||
credentials — this fixed push auth issues in this environment without needing SSH keys or PATs.
|
||
- **Pi environment gap**: Debian Bookworm blocks system-wide `pip install` (PEP 668,
|
||
"externally-managed-environment"). Use a venv for the real app dependency install (Phase 5)
|
||
rather than `--break-system-packages` everywhere. (Resolved in Phase 5 — venv at
|
||
`/home/yaris/app/yaris-obd/venv`.)
|
||
- **Pi → GitHub auth for `git pull` (private repo)**: the Pi's pre-existing
|
||
`~/.ssh/id_ed25519.pub` is an *inbound* key only (private half is on the dev machine) — can't
|
||
be reused for the Pi's own outbound git auth. Generated a dedicated keypair on the Pi
|
||
(`~/.ssh/yaris_deploy_key`), added it as a GitHub **read-only deploy key** on this repo, and
|
||
added an `~/.ssh/config` entry pinning `github.com` to that key. `[email protected]:...` (SSH
|
||
remote) then works standalone on the Pi, no PAT/interactive login needed.
|
||
- **piwheels**: pip on the Pi automatically uses `piwheels.org` as an extra index (Raspberry Pi
|
||
OS default `pip.conf`) — this is why `gevent` (which normally needs a C compile) installed in
|
||
seconds from a prebuilt ARM wheel instead of taking many minutes to build from source.
|
||
- **python-obd note**: `obd.OBD()` (or with `portstr=`) performs the ELM327 handshake at
|
||
connection time; if there's no car on the other end, it raises rather than silently returning
|
||
a dead handle — `app_obd.py`'s `connect_obd()` already catches this and sets `connection = None`.
|