1038 lines
80 KiB
Markdown
1038 lines
80 KiB
Markdown
# yaris-obd Revival Plan
|
|
|
|
This document tracks the plan to revive and finish this project: a Raspberry Pi 3-based
|
|
OBD-II + GPS car tracker for a 2013 Toyota Yaris, hosting a local web UI over a self-broadcast
|
|
WiFi hotspot, with dashcam file retrieval and remote upload of itinerary/dashcam data.
|
|
|
|
Keep this file up to date as phases complete or plans change, so any future session/agent can
|
|
pick up the project without needing prior chat history.
|
|
|
|
## Hardware
|
|
|
|
- **Board**: Raspberry Pi 3 Model B Rev 1.2, 906MB RAM. Confirmed too weak to run an AI coding
|
|
agent — all agent-assisted dev work happens on a separate local machine; the Pi is only used
|
|
for hardware bring-up/testing/deployment via SSH (`yaris@<pi-ip>`).
|
|
- **GPS/GSM**: Waveshare **SIM868 GSM/GPRS/GNSS/Bluetooth HAT**. Also has unused cellular
|
|
(GSM/GPRS) data capability, earmarked as a fallback internet uplink (see Remote Server phase).
|
|
- **OBD-II adapter**: a cheap unbranded USB adapter using a **PL2303** USB-to-serial chip,
|
|
presumed ELM327-compatible (works with the `python-obd` library). Originally the project used
|
|
a Bluetooth OBD adapter; this has been switched to USB.
|
|
- **Car**: 2013 Toyota Yaris.
|
|
|
|
### SIM868 HAT bring-up notes (important gotchas)
|
|
- UART routing is a 3-position jumper: **A** = USB↔SIM868 (via onboard CP2102), **B** = Pi
|
|
GPIO↔SIM868, **C** = USB↔Pi console. Must be on **B** for the Pi to talk to the module over
|
|
its own GPIO UART.
|
|
- Pi-side UART setup required: `enable_uart=1` and `dtoverlay=disable-bt` in
|
|
`/boot/firmware/config.txt` (frees the full PL011 UART — `/dev/ttyAMA0`/`/dev/serial0` — from
|
|
the onboard Bluetooth controller), `raspi-config nonint do_serial_cons 1` (stops a login shell
|
|
from claiming the serial port), and disable the `hciuart` systemd service.
|
|
- **The SIM868 module chip itself needs its PWRKEY button pressed and held (~1-2s) to power on**
|
|
— this is separate from the board's 5V/PWR LED, which can be lit while the module chip is
|
|
fully off. AT commands get zero response if the module chip isn't powered, even with perfect
|
|
UART wiring/config. **Update 2026-08-09: this HAT revision DOES have the "P4" GPIO-power-on
|
|
mod wire installed** (per the Waveshare FAQ/`GSM_PWRKEY.rar` script) — confirmed by pulsing
|
|
physical header pin 7 (BCM GPIO4) LOW for 4s then HIGH while the module was unresponsive, after
|
|
which it immediately answered `AT`. This is now automated in `modules/app_gps.py`
|
|
(`_ensure_module_powered()`), which checks `AT` first and only pulses PWRKEY if the module
|
|
isn't already responding (PWRKEY is a toggle — pulsing an already-on module would turn it back
|
|
off, so this check-first behavior is required). No more manual button-pressing needed after a
|
|
Pi power cycle.
|
|
- Confirmed AT command baud rate: **115200**.
|
|
- GNSS workflow: `AT+CGNSPWR?` to check GNSS power state, `AT+CGNSPWR=1` to power on,
|
|
`AT+CGNSINF` to poll for a fix. Response format:
|
|
`+CGNSINF: run,fix,utc_time,lat,lon,alt,speed,course,fixmode,reserved1,hdop,pdop,vdop,reserved2,gps_sat_view,gnss_sat_used,gnss_sat_view,cn0,hpa,vpa`
|
|
— field 1 (`fix`) must be `'1'` for a valid fix.
|
|
- **The GPS/GNSS antenna is separate from the GSM antenna** — both need to be physically
|
|
connected to their respective u.FL connectors on the HAT. A real fix was obtained only after
|
|
discovering the GPS antenna wasn't plugged in (only GSM was).
|
|
- Confirmed real fix obtained near a window: ~80s cold-start TTFF, 5-7 satellites, HDOP ~1.2-1.6.
|
|
|
|
## Software Architecture Decisions
|
|
|
|
1. **Offline maps**: Pre-cache/download offline map tiles for the car's usual driving area so
|
|
the live map works fully in the car with no internet (not just numeric GPS readout).
|
|
Tooling: MapTiler Cloud free tier (raster XYZ tiles, `streets-v2` style) — one-time download
|
|
on the dev machine via `scripts/download_map_tiles.py`, output shipped to the Pi as static
|
|
PNGs. Bulk-downloading directly from `tile.openstreetmap.org` was ruled out — it violates the
|
|
OSM tile usage policy. Bounding box is a required CLI arg (the user's own driving area, not
|
|
committed here), zoom 8-15 by default (~14k tiles for a city-plus-suburbs-sized area, well
|
|
under MapTiler's 100k/month free quota).
|
|
2. **WiFi hotspot mode**: Auto-switch — the Pi joins home WiFi when in range (for
|
|
syncing/updates), and falls back to broadcasting its own AP otherwise. **Important scope
|
|
clarification**: the hotspot is *only* for local web-UI viewing on a phone/laptop while
|
|
driving — it is not part of the upload path. Remote-server uploads happen independently via
|
|
home WiFi (when in range) or the SIM868 cellular fallback (decision #5) — the phone connecting
|
|
to the hotspot never relays/carries upload traffic.
|
|
3. **Local/remote persistence**: Keep SQLite locally (reliable local write path per trip). The
|
|
original plan paired this with MongoDB Atlas for remote sync (`modules/app_db.py`, fixed in
|
|
Phase 7) — **superseded by decision #4/Phase 9**: the actual remote store is the self-hosted
|
|
Postgres server, uploaded via `modules/app_upload.py`. `app_db.py`'s Mongo path is not wired
|
|
into `app.py` and is effectively dead code at this point, kept only for reference.
|
|
4. **Remote server**: Self-hosted — a custom server/API (e.g. on a home server or VPS) receives
|
|
itinerary and dashcam uploads, rather than 3rd-party managed cloud services.
|
|
5. **Remote upload uplink**: Use the SIM868's GSM/GPRS cellular connection as a fallback uplink
|
|
for uploads when no WiFi is available (e.g. while driving), not just "sync when on home WiFi."
|
|
|
|
## Phases
|
|
|
|
### Phase 1: Real GPS integration — ✅ DONE
|
|
`modules/app_gps.py` rewritten to read real SIM868 GNSS data via serial AT commands
|
|
(`/dev/serial0` @ 115200 baud) instead of hardcoded fake coordinates. Background daemon thread
|
|
polls `AT+CGNSINF` every 2s, parses fix data, emits `gps_location` via socketio only on an actual
|
|
fix. Verified against real hardware (see bring-up notes above).
|
|
|
|
### Phase 2: OBD fixes for USB adapter — ✅ DONE
|
|
`modules/app_obd.py` fixed (commit "Fix app_obd.py bugs and adapt for USB OBD-II adapter",
|
|
pushed to `origin/shameless-aislop`):
|
|
- `connection` is never left undefined on failed connect (was a `NameError` crash).
|
|
- Added `_find_obd_port()`/`connect_obd()`: auto-detects `/dev/ttyUSB*` (the PL2303 adapter),
|
|
honors an `OBD_PORT` env override, falls back to python-obd's own port auto-scan (still covers
|
|
the old `/dev/rfcomm*` Bluetooth path).
|
|
- Fixed stale-response bug: `obd_speed()`/`obd_rpm()` now re-query every loop iteration instead
|
|
of reusing one response forever.
|
|
- Fixed `obd_rpm()` using the wrong lock, never updating the global `rpm`, and emitting to the
|
|
wrong socketio event (`obd_speed` instead of `obd_rpm`).
|
|
- Fixed `kill_obd()` being a no-op — now uses `threading.Event` flags actually checked in the
|
|
polling loops.
|
|
- Fixed inverted thread-start logic (threads were starting on failed connection, not success).
|
|
- Fixed `obm_error` → `obd_error` typo (frontend listens for `obd_error`).
|
|
- Added `obd` (python-obd) to `requirements.txt`.
|
|
- **Still untested**: a real connection to the actual car (adapter has only been tested
|
|
standalone on the Pi so far — correctly reports "no connection to car" with nothing attached).
|
|
|
|
### Phase 3: Web UI must work fully offline — ✅ DONE
|
|
The whole point of the project is a Pi hotspot with **no internet in the car**, but the current
|
|
UI silently depended on the internet for core functionality:
|
|
- `templates/index.html`/`dash.html`/`dtc.html` loaded Google Fonts, FontAwesome, Leaflet JS/CSS,
|
|
and the Socket.IO client from public CDNs — none of this loads offline, breaking layout, the
|
|
websocket connection itself, and the map entirely.
|
|
- `static/js/yaris.js` loaded live map tiles from `tile.openstreetmap.org` (also needs internet).
|
|
- A local `static/js/chart.min.js` existed but was unused — `index.html` loaded Chart.js from
|
|
`cdn.jsdelivr.net` instead (turned out the local copy was a stale Chart.js **v2.8.0** while
|
|
`yaris.js`'s config uses v3/v4-style `scales: {x:{...}, y:{...}}` syntax — that mismatch is
|
|
exactly why the CDN was used instead. Replaced the local file with a current v4.5.0 UMD build).
|
|
|
|
**What was done**:
|
|
- Vendored Socket.IO client (4.8.1, matches Flask-SocketIO 5.6.1's Engine.IO v4 protocol) under
|
|
`static/vendor/socketio/`, and Leaflet (1.9.4 JS/CSS + marker/layers images) under
|
|
`static/vendor/leaflet/`.
|
|
- FontAwesome 5.12.0 was already vendored locally at `static/fonts/` (self-consistent, just never
|
|
referenced) — repointed all three templates to it instead of the CDN.
|
|
- Dropped the Google Fonts (Nunito) `<link>` entirely rather than self-hosting 12 font-weight
|
|
files — Bootstrap's `--bs-font-sans-serif` variable already falls back cleanly to system fonts
|
|
(`-apple-system, Segoe UI, Roboto, ...`) after Nunito, so this is a clean no-cost simplification.
|
|
- Replaced `static/js/chart.min.js` with a current Chart.js v4.5.0 UMD build (auto-registering,
|
|
same drop-in behavior as the old CDN `<script src="https://cdn.jsdelivr.net/npm/chart.js">`).
|
|
- Offline map tiles: added `scripts/download_map_tiles.py`, a dev-machine-only one-time tile
|
|
downloader using the MapTiler Cloud free tier (raster XYZ, `streets-v2` style, 256px tiles).
|
|
Reads `MAPTILER_KEY` from an env var/CLI arg only — the key itself is never committed or
|
|
deployed, only the resulting static PNG tiles are. Bounding box is a required CLI arg (your own
|
|
driving area's coordinates - not committed/hardcoded) at zoom 8-15 (~14k tiles for a
|
|
city-plus-suburbs-sized area, well under MapTiler's 100k tiles/month free quota). Output goes to
|
|
`static/vendor/tiles/{z}/{x}/{y}.png` (gitignored — regenerate locally, rsync to the Pi as part
|
|
of deployment, don't bloat the git repo with binary tile data).
|
|
- **Tiles populated**: user signed up for a free MapTiler key and the full zoom 8-15 download
|
|
ran to completion — 14,241 downloaded, 24 skipped (out-of-bounds), 0 failed (14,265 tiles
|
|
total, ~285MB). Rsynced to the Pi's `static/vendor/tiles/` (tile counts verified identical
|
|
on both sides). Live Pi instance re-confirmed serving HTTP 200 with the complete offline
|
|
tile set.
|
|
- Updated `static/js/yaris.js`'s Leaflet init to point at `/static/vendor/tiles/{z}/{x}/{y}.png`
|
|
(`minZoom`/`maxZoom` 8/15 to match). The initial map center is just a generic placeholder (not
|
|
a real location) - it re-centers automatically to the real live GPS fix once one comes in, and
|
|
the marker is exposed as `carMarker` so Phase 4 can move it on live fixes.
|
|
- Added a repo-root `.gitignore` (Python caches, venvs, `.vscode/`, `static/vendor/tiles/`,
|
|
`*.env` except the already-tracked `db/creds.env` placeholder) — there wasn't one before, which
|
|
is why stray `__pycache__`/`.pyc` files kept showing up in source control.
|
|
- **Verified**: ran the Flask app in a local venv, confirmed all three pages (`/`, `/dtc`, `/dash`)
|
|
render with **zero external URLs** in the served HTML, every local static asset (JS, CSS, fonts,
|
|
Leaflet images) returns HTTP 200, and all vendored JS files pass a Node.js syntax check.
|
|
|
|
### Phase 4: Map/GPS display bug — ✅ DONE
|
|
`yaris.js`'s Leaflet marker (`carMarker`) was never moved on incoming `gps_location` socket
|
|
events — only two text spans (`gpsLocationX`/`Y`) got updated. Fixed: `carMarker.setLatLng(...)`
|
|
is now called on every `gps_location` event, and the map re-centers (zoom 15) on the first fix
|
|
only (`mapCentered` flag), so the user isn't fighting the map re-centering under them while
|
|
panning/zooming during a trip.
|
|
|
|
**Verified**: also used this as an opportunity to test Phase 3 end-to-end — ran
|
|
`scripts/download_map_tiles.py` for real with the user's actual MapTiler API key (smoke-tested
|
|
zoom 8-10 first, confirmed idempotent re-run skips existing tiles, confirmed a downloaded tile
|
|
is a real map image of the target driving area), then started the full zoom 8-15
|
|
download. Took a **headless Chromium screenshot** of the running app's `/` page and confirmed:
|
|
sidebar/icons/fonts render correctly (FontAwesome working), the Speed Graph (Chart.js v4) renders,
|
|
and **the Leaflet map renders real offline map tiles with the marker in place** — full offline
|
|
end-to-end proof, not just "no CDN URLs in the HTML".
|
|
- Fixed (was noted here as out of scope, now addressed in Phase 8): `/dash` and `/dtc` used to
|
|
render near-identical body content (both showed "DTC Codes") — `dash.html` now has real
|
|
dashcam-specific content.
|
|
- **Full tile download completed** (see Phase 3 update above) and rsynced to the Pi — Phase
|
|
3/4's offline map is now 100% functional end-to-end on the actual deployed device, not just
|
|
the dev machine.
|
|
|
|
### Phase 5: Pi deployment / autostart — ✅ DONE
|
|
Set up a proper venv (`/home/yaris/app/yaris-obd/venv`), installed all requirements (all found
|
|
as prebuilt ARM wheels via piwheels — no slow source compiles needed, even for gevent).
|
|
Added `deploy/yaris-obd.service` (systemd unit: runs as the `yaris` user, `Restart=on-failure`,
|
|
5s `ExecStartPre` sleep to let USB/serial devices enumerate after boot, logs to
|
|
`yaris-obd.log`) and `deploy/install_service.sh` (idempotent one-time install helper — also
|
|
creates the venv if missing). Installed and enabled on the Pi (`systemctl enable` — survives
|
|
reboot).
|
|
|
|
**Also had to fix Pi → GitHub auth for `git pull`**: the repo is private, and the Pi's existing
|
|
`~/.ssh/id_ed25519.pub` turned out to be an *inbound* SSH key (its private half lives on the dev
|
|
machine, not the Pi) — useless for the Pi to authenticate outbound. Generated a dedicated
|
|
ed25519 keypair **on the Pi itself** (`~/.ssh/yaris_deploy_key`), registered it as a **read-only
|
|
deploy key** on the GitHub repo (least-privilege — the Pi only ever needs to pull), and added an
|
|
`~/.ssh/config` entry so git transparently uses it for `github.com`. Switched the Pi's `origin`
|
|
remote to SSH. `git pull` now works standalone on the Pi with no PAT/interactive auth needed.
|
|
|
|
**Verified end-to-end**: pulled latest code onto the Pi (was 5 commits stale), confirmed all
|
|
Python deps import cleanly, ran the app manually first (HTTP 200, all local static assets —
|
|
Leaflet/Socket.IO/Chart.js/FontAwesome — served correctly from the Pi itself), then installed +
|
|
started the systemd service and confirmed `active (running)` + `enabled`. **Tested crash
|
|
recovery**: sent SIGKILL to the running process, confirmed systemd auto-restarted it (through the
|
|
5s `ExecStartPre` delay) and the app was serving HTTP 200 again within ~10s, with zero manual
|
|
intervention.
|
|
|
|
### Phase 6: WiFi hotspot mode — ✅ DONE
|
|
Used NetworkManager (already the network manager in use, not hostapd/dnsmasq directly — those
|
|
were leftover/unused) since it natively supports AP mode via `nmcli`. Created two connection
|
|
profiles on the Pi: `home-wifi` (your real home WiFi network, `autoconnect-priority 10`) and
|
|
`Hotspot` (AP mode, `ipv4.method shared`, its own dedicated SSID/password set locally on the Pi -
|
|
not committed here, WPA2-PSK, `autoconnect-priority -10`). NM's own autoconnect logic handles
|
|
"leaving home WiFi range -> fall back to hotspot" automatically since `home-wifi` has higher
|
|
priority and NM always tries it first.
|
|
|
|
For the reverse direction (home WiFi back in range while sitting in AP mode) added
|
|
`deploy/yaris-netwatch.sh` + `.service`/`.timer`: since the Pi's onboard BCM43438 radio can't be
|
|
an AP and background-scan for client networks simultaneously, a systemd timer runs every 3
|
|
minutes, briefly attempts reconnecting to `home-wifi`, and restores the hotspot if home WiFi still
|
|
isn't in range. Installed via `deploy/install_netwatch.sh`.
|
|
|
|
**Root-caused and fixed a real bug during testing**: the hotspot's SSID broadcast worked
|
|
immediately, but its DHCP server initially failed (`dnsmasq: failed to create listening socket
|
|
for 10.42.0.1: Address already in use`) — a pre-existing **orphaned system-wide `dnsmasq.service`**
|
|
(noted as "likely unused" in earlier investigation) was already bound to the address space
|
|
NetworkManager's own internal dnsmasq needs for the hotspot. Confirmed the system dnsmasq's
|
|
config was genuinely empty/default (nothing in this project depends on it) and disabled it
|
|
(`systemctl disable --now dnsmasq`).
|
|
|
|
Also added easy access without needing to know the Pi's IP:
|
|
- **Renamed the Pi's hostname** from `yarisobd` to `yaris` — `avahi-daemon` (already installed
|
|
and running) now publishes it as `yaris.local` via mDNS.
|
|
- **Port 80 -> 5000 redirect** via nftables (`deploy/nftables-yaris.conf` +
|
|
`install_port_redirect.sh`, loaded through an `include` in `/etc/nftables.conf`), so the app is
|
|
reachable at plain `http://yaris.local/` (no port number) instead of `:5000`.
|
|
|
|
**Verified on real hardware, very carefully** since the SSH session used to do all this testing
|
|
itself runs over `wlan0`: used a "dead-man's-switch" pattern for every risky test — a detached
|
|
background command (`setsid nohup bash -c 'sleep N && nmcli con up home-wifi' & disown`) armed
|
|
*before* each hotspot activation, so the Pi automatically reverts to home WiFi (and SSH access
|
|
comes back) even if something goes wrong, no physical access needed. Confirmed: hotspot broadcasts
|
|
and a real phone gets a genuine DHCP lease from NetworkManager's own DHCP server and can load the
|
|
web UI; `yaris-netwatch` correctly detects AP mode and reconnects to home WiFi when in range (and
|
|
is a no-op when already on home WiFi); `http://yaris.local/` and `http://<pi-lan-ip>/` both return
|
|
HTTP 200 with no port number from another machine on the network. (Real hotspot SSID/password and
|
|
the Pi's LAN IP are intentionally not recorded in this file — see local, non-committed notes.)
|
|
|
|
**Update — real-hardware bug found + fixed (hotspot->home reconnect was flaky)**: while testing
|
|
the phone-relay feature live (see Phase 11 update below), `yaris-netwatch` repeatedly logged "home
|
|
WiFi not found" 4 times in a row over ~13 minutes while sitting right next to the router — a real
|
|
regression from the original design, not a one-off fluke. Root cause: a plain `nmcli con up
|
|
home-wifi` straight out of AP mode relies on NM's scan cache for `wlan0`, which is stale/empty
|
|
coming out of AP mode (the radio can't scan while acting as an AP - see above). A single
|
|
`--wait 20` connect attempt gave up before a fresh scan could land. Fixed `yaris-netwatch.sh` to
|
|
explicitly drop the AP connection first (releases the radio), force `nmcli device wifi rescan`,
|
|
wait for it to settle, then retry the connect (longer 30s wait) up to 2 times before conceding and
|
|
restoring the hotspot. Bumped `yaris-netwatch.service`'s `TimeoutStartSec` to 180s (retry loop can
|
|
take up to ~80s, default systemd 90s start timeout was too tight). Verified fixed on real hardware:
|
|
both hotspot->home transitions after the fix recovered in ~7 seconds (previously failing outright).
|
|
Deployed: `sudo cp deploy/yaris-netwatch.sh /usr/local/bin/... && cp .service ... && daemon-reload`.
|
|
|
|
### Phase 7: app_db.py / app_sql.py fixes — ✅ DONE
|
|
Neither file was called from `app.py` or anywhere else - orphaned from a prior iteration. Fixed:
|
|
- `app_db.py`: applied a prior-session stash fix - resolves `.env` path via `__file__` (was
|
|
cwd-relative), fixes `os.getenv()` calls to pass the actual key names (was called with zero
|
|
args, an immediate `TypeError` if ever invoked), added a clear `RuntimeError` guard if
|
|
credentials are still empty placeholders, returns `client[mDB]`.
|
|
- `app_sql.py`: rewritten - resolves `db/sql/` via `__file__` + `os.makedirs` (was cwd-relative),
|
|
fixed `create_conn()` closing the connection immediately and returning `None` (so table
|
|
creation never ran and every caller got a dead connection), added `get_conn()` (lazy singleton
|
|
+ table creation for both `tracking` and `dtc` tables), `insert_tracking()`, and a background
|
|
`start_logging()`/`_log_loop()` thread (5s interval) that snapshots GPS (`app_gps.LocationX/Y`)
|
|
+ OBD speed (`app_obd.speed`) into the local `tracking` table. Used a deferred (function-local)
|
|
import of `app_obd` inside `_log_loop()` to keep the pre-existing `app_obd`<->`app_sql`
|
|
top-level circular import safe/explicit.
|
|
- Also fixed a Python 3.12+/3.14 compat bug found during testing: `sqlite3.version` (the
|
|
sqlite3 *module's* version) was removed/deprecated - switched to `sqlite3.sqlite_version`
|
|
(the actual SQLite library version), which is what was intended.
|
|
- Wired `app_sql.start_logging()` into `app.py` alongside the other startup calls
|
|
(`app_gps.start_gps()`, `app_obd.OBD()`).
|
|
- `db/sql/` added to `.gitignore` (per-device local trip data, not source).
|
|
- Verified locally: `get_conn()` creates `db/sql/<account>_<car>.db` with both tables;
|
|
`insert_tracking()` writes a real row; `start_logging()` thread starts and logs
|
|
"SQL tracking-logger thread started, alive: True"; full `app.py` run confirms the thread
|
|
starts cleanly and (with no real GPS/OBD hardware on the dev machine) correctly logs read
|
|
errors instead of crashing; `app_db.get_database()` raises the new clear `RuntimeError` instead
|
|
of crashing, since `db/creds.env` still holds empty placeholder credentials (real Mongo
|
|
credentials are a user-provided follow-up, out of scope here).
|
|
|
|
### Phase 8: Dashcam retrieval (app_dash.py) — ✅ DONE, verified on real hardware
|
|
Rewrote `app_dash.py` to be brand/device-agnostic: detection via `lsblk -J`, matching partitions
|
|
whose parent disk reports `TRAN == "usb"` (this is what safely distinguishes a plugged-in
|
|
dashcam/USB stick from the Pi's own onboard storage, which is always `mmc` transport - no more
|
|
hardcoded `/dev/sda`). Walks the whole mounted filesystem for video file extensions
|
|
(`.mp4/.mov/.avi/.ts/.mkv`) instead of assuming a specific brand's folder layout (DCIM, MOVIE,
|
|
NORM/EVENT/PARKING, etc. all just work), preserves each file's relative sub-folder path when
|
|
copying (avoids filename collisions across dashcam event/date folders), and skips files already
|
|
copied (same relative path + size) so re-running an import is cheap. Mounts read-only to avoid
|
|
any risk of corrupting the dashcam's SD card. Dropped the unused/uninstalled `tqdm` CLI-progress
|
|
dependency (never actually in `requirements.txt`) since this now runs headless inside the always-
|
|
on web server, not interactively.
|
|
|
|
Wired into `app.py`: `GET /api/dash/devices` (list detected USB storage) and
|
|
`POST /api/dash/import` (run the mount->copy->unmount pipeline), both JSON. Fixed the pre-existing
|
|
`/dash` vs `/dtc` duplicate-template bug — `dash.html` now has real dashcam content (scan/import
|
|
buttons + live status) instead of a copy-pasted "DTC Codes" body. Added a dedicated `dash.js`
|
|
(not the shared `yaris.js`, which unconditionally initializes the `/index`-only map/chart and
|
|
would throw/crash on `/dash` and `/dtc` since those pages don't have those DOM elements - a
|
|
pre-existing bug, worked around by not depending on that shared file for this page's new JS).
|
|
Added `dash/videos/` to `.gitignore` (imported footage is per-device data, not source).
|
|
|
|
**Tested thoroughly without the physical dashcam** (user didn't have it on hand this session):
|
|
1. Unit-tested the `lsblk`-JSON detection/filtering logic against a realistic synthetic sample
|
|
(Pi's own `mmcblk0` boot+root partitions plus a simulated USB dashcam partition) - confirmed
|
|
it correctly finds only the USB partition and ignores the Pi's own storage.
|
|
2. Integration-tested the real mount -> find-videos -> copy (preserving structure) -> dedupe ->
|
|
unmount pipeline on the actual Pi using a loop-device-backed FAT32 image (`mkfs.vfat` +
|
|
`losetup`) populated with fake nested dashcam video files (`DCIM/100EVENT/...`,
|
|
`DCIM/100NORMAL/...`) - confirmed real mount/copy/unmount all work, confirmed a second import
|
|
run correctly skips already-copied files, confirmed non-video files (e.g. `thumbs.db`) are
|
|
ignored. All simulation artifacts (loop device, test image, temp files) cleaned up afterward.
|
|
3. Verified the full `/dash` page + both new API routes locally (Flask dev run + headless
|
|
browser check): auto-scan on page load correctly reports "No USB dashcam/storage device
|
|
detected" (accurate, since none is plugged into the dev machine), Import button correctly
|
|
stays disabled until a device is found, no JS console errors, no regression on `/` or `/dtc`.
|
|
|
|
**Verified against the real physical dashcam** (Jieli-chipset USB ID `1224:2725`, 29.7GB SD card,
|
|
`DCIM/1/VID_###.MOV` layout, 87 files, ~28GB total). This first live hardware test found and fixed
|
|
three real bugs the loop-device simulation couldn't have caught:
|
|
1. **Superfloppy SD card** - the card has no partition table at all (`lsblk` reports the whole
|
|
`/dev/sda` disk directly as `fstype: vfat`, no child partition) - the original detection assumed
|
|
a partition would always exist. Fixed to also match a `TRAN == "usb"` disk with its own
|
|
`fstype` set directly, not just partitions under it.
|
|
2. **Mount permission error** - `mount_device()` called `os.makedirs()` directly, but the service
|
|
runs as unprivileged `yaris` and `/mnt` is root-owned. Fixed to `sudo mkdir -p <mount_point>`
|
|
before the existing `sudo mount`, mirroring that pattern.
|
|
3. **OOM crash during upload** (see Phase 10 below) - not a copy-path bug, but discovered
|
|
immediately after the first successful real import.
|
|
|
|
End-to-end result: full 28GB/87-file import from the SD card completed successfully (resumed
|
|
cleanly after being interrupted once by the OOM crash below - `copy_files()`'s skip-if-already-
|
|
copied logic meant re-running `/api/dash/import` just picked up the remaining files, no data loss
|
|
or re-copy of the first 19 files). SD card unmounted cleanly afterward.
|
|
|
|
### Phase 9: Remote server for itinerary/dashcam upload — ✅ DONE
|
|
Per architecture decisions #4 and #5: self-hosted server/API to receive itinerary (from the
|
|
SQLite pipeline) and dashcam file uploads.
|
|
|
|
Decided/built: Flask API + dedicated Postgres, deployed as a docker-compose stack on the user's
|
|
unraid server (separate from any existing containers/DBs there), living in `server/` in this
|
|
same repo. Auth: simple `X-Device-Name`/`X-Api-Key` header pair per device, key hash stored
|
|
server-side, real key never committed (raw key printed once by `server/scripts/create_device.py`,
|
|
saved locally on the Pi only, in gitignored `db/upload_creds.env`). External access: exposed at
|
|
`https://yaris.ceaf.dev` via the user's existing Cloudflare Tunnel (public-hostname rule added in
|
|
the Cloudflare dashboard, pointing at `192.168.10.10:8098`) — router port-forward + the user's
|
|
Unifi firewall is the documented fallback if ever needed instead. Large dashcam uploads can
|
|
still prefer a local-network path (e.g. an AdGuard DNS override to bypass the tunnel while on
|
|
home WiFi) since tunnels typically cap per-request upload size — noted in `server/README.md`,
|
|
not yet needed/built.
|
|
|
|
Built (`server/` folder):
|
|
- `server/sql/init.sql` — Postgres schema: `devices` (name + sha256 api-key-hash), `tracking_points`
|
|
(device_id FK, GPS/speed/odometer + `recorded_at`, unique per device+timestamp so retried
|
|
batches can't duplicate rows), `dashcam_files` (device_id FK, relative_path + sha256, unique per
|
|
device+path+hash so re-uploads are no-ops).
|
|
- `server/modules/db.py` — psycopg3 connection pool + query helpers (bulk tracking insert with
|
|
`ON CONFLICT DO NOTHING`, dashcam dedupe-check-before-write).
|
|
- `server/modules/auth.py` — `require_device_auth` decorator checking the two headers against
|
|
the stored key hash.
|
|
- `server/app.py` — Flask app: `GET /health` (unauthenticated), `POST /api/v1/tracking/batch`
|
|
(bulk JSON insert, capped at 5000 points/request as a sanity guard), `POST /api/v1/dashcam/upload`
|
|
(multipart, streamed to disk with sha256 verification + path-traversal guard on `relative_path`).
|
|
- `server/Dockerfile` + `server/docker-compose.yml` — `api` (gunicorn, published on host port
|
|
8098 by default) + `db` (Postgres 17, internal-only, not published to the LAN) services, data
|
|
volume defaults to `server/data/` but pointed at `/mnt/user/appdata/yaris-obd-server` on unraid.
|
|
- `server/.env.example` — template for `POSTGRES_PASSWORD`/`API_PORT`/`DATA_DIR`/upload size cap.
|
|
- `server/scripts/create_device.py` — run once per device inside the `api` container to register
|
|
it and print its one-time API key. Ran once for device `yaris-2013`.
|
|
- `server/README.md` — full unraid deployment + Cloudflare Tunnel exposure + device-registration
|
|
instructions.
|
|
|
|
Pi-side upload client (`modules/app_upload.py`):
|
|
- Background thread, uploads unsent `tracking` rows (batches of 500) and not-yet-uploaded
|
|
dashcam files (sha256-verified) via the API above, with exponential retry/backoff on failure.
|
|
- Independent dashcam-upload-only pause toggle (persisted `settings` table + a switch on the
|
|
`/dash` web UI) — trip/GPS logs always keep uploading regardless of the toggle, per explicit
|
|
user request (useful on a slow/metered connection).
|
|
- **Backlog-drain tuning**: since tracking rows log every 5s locally, an offline/stale device can
|
|
queue up a large backlog. The loop drains back-to-back (short 1s pause between batches) while a
|
|
batch comes back full, only settling into the slow 30s idle cadence once actually caught up —
|
|
a simulated 1250-row backlog now drains in 3 batches (~0.35s) instead of 13 batches over ~6.5
|
|
minutes with the original fixed 100-row/30s-sleep design.
|
|
- `modules/app_sql.py` extended: `UPLOADED` column + migration on `tracking`, `settings` table,
|
|
`dashcam_uploads` dedupe table, and helper functions backing all of the above.
|
|
|
|
Tested end-to-end against the real deployed server (tracking batch + dedupe, dashcam upload +
|
|
dedupe + path-traversal rejection, bad-API-key rejection, toggle-off correctly pausing only
|
|
dashcam uploads) and on the real Pi (its actual ~1800-row pre-feature backlog drained and
|
|
confirmed matching row counts on the server via direct psql query, live upload status visible on
|
|
`/dash`). Committed+pushed (`4ff99f0`, `c4db1ab`, `8dc304e`), deployed on both unraid and the Pi.
|
|
Exposed externally at `https://yaris.ceaf.dev` via the user's Cloudflare Tunnel (public hostname
|
|
added in the Cloudflare dashboard, pointing at `192.168.10.10:8098`).
|
|
|
|
### Phase 10: Dashcam upload memory/size fixes — ✅ DONE, verified on real hardware
|
|
Two real-hardware-only bugs surfaced once actual ~300MB+ video files (not small test fixtures) hit
|
|
the upload path for the first time:
|
|
|
|
1. **OOM crash from in-memory multipart uploads.** `modules/app_upload.py` originally uploaded
|
|
dashcam files via `requests.post(..., files={"file": f})`. The `requests` library builds the
|
|
*entire* multipart/form-data body in memory before sending when using `files=` - a well-known
|
|
gotcha, not specific to this codebase. On the Pi 3's ~900MB RAM, uploading one ~320MB video this
|
|
way spiked the process past 600MB+ RSS, triggering the kernel OOM killer within minutes
|
|
(confirmed via `dmesg`: repeated `Out of memory: Killed process ... anon-rss:823928kB`, and via
|
|
`pmap -x` showing a single ~362MB anonymous mapping matching the file size almost exactly).
|
|
**Fixed**: added `requests-toolbelt` and rewrote the upload body construction to use
|
|
`MultipartEncoder`, which streams the file in chunks as the request is sent instead of
|
|
materializing it up front. Confirmed via an isolated 300MB test (628MB RSS old way -> 29MB new
|
|
way, byte-identical result) and live on the Pi (RSS stayed flat ~55-59MB during real uploads,
|
|
vs. previously crashing every few minutes).
|
|
2. **Cloudflare Tunnel's hard ~100MB request-body cap.** Even after the memory fix, real uploads
|
|
still failed once routed through `https://yaris.ceaf.dev` - a bare `413 Payload Too Large` from
|
|
Cloudflare itself (confirmed: the identical file succeeds direct-LAN, 413s only through the
|
|
tunnel). This is a free-tier platform limit, unrelated to `MAX_DASHCAM_UPLOAD_MB` on the Flask
|
|
side. **Fixed** via a chunked/resumable upload redesign (well under the 100MB cap):
|
|
- Client (`modules/app_upload.py`): `DASHCAM_CHUNK_SIZE = 40MB`; new
|
|
`_upload_video_chunked()` checks `/status` first (enables resume across restarts/failures),
|
|
then POSTs missing chunks one at a time via `MultipartEncoder`.
|
|
- Server (`server/app.py`): new `GET /api/v1/dashcam/upload/status` (reports
|
|
`already_uploaded` or `received_chunk_indices`) and `POST /api/v1/dashcam/upload/chunk`
|
|
(stages chunks under `STORAGE_DIR/.upload_tmp/<device>/<sha256>/`, reassembles + verifies
|
|
SHA256 once all chunks arrive, records via the existing `db.insert_dashcam_file`, cleans up
|
|
the staging dir). A `.total_chunks` marker guards against stale/mismatched partial sets if the
|
|
chunk size ever changes between client versions. The original single-shot
|
|
`/api/v1/dashcam/upload` endpoint is untouched, kept for reference/compatibility.
|
|
|
|
**Verified end-to-end against the real deployed server through the real Cloudflare tunnel**,
|
|
first with a synthetic interrupted-then-resumed 250MB test (status transitions
|
|
`not_started` -> partial upload -> `in_progress` w/ correct `received_chunk_indices` -> resume ->
|
|
`complete` -> `already_uploaded`, SHA256 match confirmed on disk), then with real dashcam footage:
|
|
the Pi copied all 87 real files locally and the background upload thread drained the real backlog
|
|
one file at a time, each one verified byte-for-byte (SHA256 match Pi vs. server) and correctly
|
|
recorded in Postgres, while the Pi's service ran 50+ minutes straight with RSS staying in a bounded
|
|
~55-141MB range (no OOM kills) - a stark contrast to the previous crash-every-few-minutes behavior.
|
|
Given home-upload-bandwidth constraints, the full 28GB backlog drains at roughly one ~350MB file
|
|
per 4-5 minutes and is expected to take several hours total - this runs unattended in the
|
|
background with no user action needed once started.
|
|
|
|
**Not yet built**: parallel/concurrent chunk uploads (currently strictly sequential, one chunk
|
|
then one file at a time) - would speed up the multi-hour full-backlog drain but adds complexity;
|
|
not requested by the user yet, flagged here as a possible future optimization if the sequential
|
|
pace becomes annoying in practice.
|
|
|
|
**Known manual step (not yet automated)**: this dashcam model boots into a "select mode" menu
|
|
(dashcam / mass-storage / webcam) when it senses USB data lines are live, and needs USB data
|
|
communication physically absent to boot straight into recording mode, but present when plugging
|
|
into the Pi for import. No code in this repo handles that switch - it's a manual physical step by
|
|
the user (e.g. a data-blocking USB cable/adapter, or plugging/unplugging appropriately) each time
|
|
before recording vs. before importing. Worth revisiting if it becomes tedious in daily use (e.g. a
|
|
Pi-controlled USB data-line switch), but out of scope for now.
|
|
|
|
### SIM868 cellular fallback for tracking uploads — ⚠️ built, UNTESTED (no SIM card available)
|
|
`modules/app_cellular.py` gives the tracking-upload path a fallback over the SIM868's cellular
|
|
data connection when WiFi/Ethernet is unreachable. Design notes:
|
|
- Uses the SIM868's built-in AT-command HTTP(S) client (`AT+HTTPINIT`/`HTTPPARA`/`HTTPACTION`/
|
|
`HTTPREAD`) rather than a real `ppp0` network interface. Reason: the SIM868's GNSS (polled
|
|
continuously by `modules/app_gps.py`) and its GSM/GPRS radio share the *same* physical UART
|
|
(`/dev/serial0`) — a classic `pppd`/`chat` dial-up would put that port into raw PPP framing for
|
|
as long as the link is up, starving the GNSS polling thread. Staying in AT-command mode lets
|
|
cellular HTTP calls interleave safely through the same serial lock
|
|
(`modules.app_gps._serial_lock`) — each exchange just briefly delays the next GPS poll.
|
|
- Only carries small JSON tracking batches, **never** dashcam video — cellular data is far too
|
|
slow/limited for multi-MB video files, and the modem's HTTP AT interface has a small internal
|
|
buffer unsuited to them anyway. Dashcam uploads remain WiFi/Ethernet-only, unchanged.
|
|
- `modules/app_upload.py`'s tracking-batch upload always tries WiFi/Ethernet (`requests`) first;
|
|
only on a connection-level failure (no route/DNS/timeout — as opposed to an HTTP error status,
|
|
which means the server WAS reached) does it retry the same batch over cellular.
|
|
- Feature-flagged off by default: `YARIS_CELLULAR_ENABLED`/`YARIS_CELLULAR_APN`/
|
|
`YARIS_CELLULAR_APN_USER`/`YARIS_CELLULAR_APN_PASS` added to `db/upload_creds.env`, all blank
|
|
by default so this has zero effect on the already-verified WiFi upload path until the user
|
|
fills in an APN once they get a SIM card. Regression-tested (against the real server) that
|
|
normal WiFi uploads are completely unaffected while cellular is unconfigured.
|
|
- `/dash` upload-status card now shows which path (`wifi`/`cellular`) the last successful batch
|
|
used, and whether cellular fallback is configured.
|
|
- **Needs validation once a SIM card is available** — exact AT command dialect/timing (SIMCOM
|
|
firmware revisions vary slightly), and whether `AT+HTTPSSL` works for the HTTPS
|
|
`yaris.ceaf.dev` endpoint on this specific module's firmware (if not, a documented fallback is
|
|
a plain-HTTP router port-forward path, avoiding the need for the modem's TLS stack to validate
|
|
a cert chain).
|
|
|
|
### Phase 11: Phone-relay dashcam upload — ⛔ REMOVED (see update below - hit a hard platform limitation)
|
|
Root cause investigation into "why is the transfer from the Pi to the server so slow?" (real
|
|
dashcam backlog draining at only ~350MB/4-5min) found the bottleneck is the **Pi's own onboard
|
|
BCM43438 WiFi chip** (2.4GHz-only, congested/interfered channel) — confirmed via `iperf3`
|
|
raw-TCP tests showing only ~14.7Mbit/s with active retransmits even over a same-subnet direct-LAN
|
|
hop that never touches the internet/Cloudflare, ruling out ISP/Cloudflare limits. Since the Pi 3 B
|
|
has no 5GHz radio and can't be fixed in software, added a way to bypass its weak WiFi for the
|
|
internet-bound leg entirely: a phone opens the same `/dash` page (over the Pi's hotspot/home WiFi)
|
|
and manually relays pending dashcam files straight to the remote server using the phone's own
|
|
network (WiFi or cellular data), instead of routing everything through the Pi.
|
|
|
|
**Zero new remote-server upload logic** — the phone's browser JS calls the *exact same*
|
|
`/api/v1/dashcam/upload/status` + `/api/v1/dashcam/upload/chunk` endpoints the Pi's own Python
|
|
client already uses, in the same 40MB-chunk protocol, just orchestrated by `fetch()`/`FormData`
|
|
instead of `requests`/`MultipartEncoder`. The server dedupes purely by SHA256 regardless of which
|
|
device sends the bytes, so no server-side changes were needed for the upload path itself — only a
|
|
CORS `after_request` hook (`server/app.py`) was added, since the phone's browser calls
|
|
`yaris.ceaf.dev` from a page served by a *different* origin (the Pi), which browsers block without
|
|
`Access-Control-Allow-*` headers on both the real request and its preflight `OPTIONS`. Wildcard
|
|
origin is fine here — auth is header-based (`X-Device-Name`/`X-Api-Key`), not cookie/session-based.
|
|
|
|
**New Pi-side endpoints** (`app.py`):
|
|
- `GET /api/dash/pending` — lists not-yet-uploaded dashcam files (`relative_path`, `sha256`,
|
|
`size`) via a new `app_upload.list_pending_dashcam_files()`.
|
|
- `GET /api/dash/file/<path:relpath>` — serves a dashcam file's bytes for the phone to download,
|
|
via `send_file(..., conditional=True)` for **Range-request support** (206 Partial Content) so
|
|
the phone downloads (and immediately re-uploads) one 40MB slice at a time instead of ever
|
|
holding a whole 300+MB file in the browser's memory — the exact same memory-safety lesson
|
|
learned the hard way in Phase 10, applied symmetrically to the download side. Path-traversal
|
|
safe via Werkzeug's `safe_join` against `app_dash.DEFAULT_DESTINATION`.
|
|
- `GET /api/dash/relay/config` — hands the phone the same `SERVER_URL`/`DEVICE_NAME`/`API_KEY`
|
|
the Pi's own client uses, so its JS can auth directly with the remote server. Same trust
|
|
boundary as the rest of this project's unauthenticated local UI (anyone on the Pi's own
|
|
hotspot/LAN is already implicitly trusted).
|
|
- `POST /api/dash/relay/heartbeat` / `POST /api/dash/relay/stop` — a **lease-based pause**
|
|
(`RELAY_LEASE_SECONDS = 45`) for the Pi's own background dashcam uploader, chosen over a plain
|
|
on/off toggle specifically to self-heal if the phone's browser tab closes/loses connection
|
|
uncleanly (a simple toggle could get stuck "paused" forever). The phone page heartbeats every
|
|
15s while relaying; `_upload_dashcam_files()` skips entirely while a lease is active, re-checked
|
|
between files so it stops promptly if a relay starts mid-drain. Also released via
|
|
`navigator.sendBeacon()` on `beforeunload` as a best-effort immediate release.
|
|
- `POST /api/dash/relay/mark_uploaded` — lets the phone tell the Pi "I just uploaded this file"
|
|
right after a successful relay, so the local dedupe table (and the pending list / Pi's own
|
|
background loop) reflect it immediately instead of waiting for the next full backlog scan.
|
|
|
|
**Performance side-fix**: `modules/app_upload.py`'s dashcam loop previously recomputed SHA256 for
|
|
*every* candidate file on *every* loop iteration, including files long since uploaded — wasteful
|
|
CPU/IO against a growing, possibly 20GB+, local backlog. Added a `dashcam_hash_cache` SQLite table
|
|
(`modules/app_sql.py`, keyed by `relative_path`+`size`+`mtime`) so a file's hash is only computed
|
|
once; both the existing background uploader and the new pending-files endpoint benefit.
|
|
|
|
UI (`templates/dash.html` / `static/js/dash.js`): a new "Relay via phone" card on `/dash` lists
|
|
pending files with a per-file "Relay" button and a "Relay all pending" button (manual trigger only
|
|
— chosen deliberately over an automatic one, per explicit user preference), plus a "Stop relay"
|
|
button. Chunk downloads/uploads retry up to 3 times with backoff before surfacing an error, and a
|
|
resumed relay re-checks `/status` first so a previously-interrupted file picks up from its first
|
|
missing chunk instead of restarting.
|
|
|
|
**Tested**: a full local end-to-end simulation (no physical phone/network needed) drove the real
|
|
Pi `app.py` and real `server/app.py` Flask apps side by side (Postgres mocked at the `db` module
|
|
boundary only — all HTTP routing, chunking, auth, and CORS logic is real), reproducing the exact
|
|
sequence the phone's browser JS performs: list pending -> check remote `/status` -> download each
|
|
chunk from the Pi via `Range` requests (verified real `206 Partial Content` responses) -> upload
|
|
each chunk to the mock-backed server -> verify the reassembled file matches the original
|
|
byte-for-byte via SHA256 -> mark uploaded on the Pi -> confirm it drops out of the pending list.
|
|
Also verified: path-traversal attempts on `/api/dash/file/` correctly 404, the relay
|
|
heartbeat/stop lease correctly pauses/resumes `_upload_dashcam_files()`, and the sha256 cache
|
|
table is populated and reused across repeated `/api/dash/pending` calls.
|
|
|
|
**Deploy gotcha found**: the CORS fix in `server/app.py` was initially deployed to the Pi but not
|
|
rebuilt on the actual remote server (the server's Docker image bakes `app.py`/`modules/` in at
|
|
build time via `COPY`, so a plain `git pull` + container restart isn't enough — needs
|
|
`docker compose build api && docker compose up -d api`). Without it, the phone's cross-origin
|
|
`fetch()` calls to `yaris.ceaf.dev` were silently blocked by the browser's CORS policy, which
|
|
surfaces in JS as a generic, unhelpful `TypeError: Failed to fetch` — worth remembering as the
|
|
first thing to check if this error reappears after a future server-side change.
|
|
|
|
**Tested for real against a real phone and the real deployed server** (first end-to-end proof
|
|
beyond the local simulation above): the initial `/api/dash/pending` call took ~25 minutes for a
|
|
one-time full-backlog SHA256 hash of all 87 files (~28GB, no cache existed yet) — after that,
|
|
repeat calls returned in ~0.13s. The user then successfully relayed a real file from their phone
|
|
(home WiFi) partway through (10/~12 chunks) before stopping to check something important: **is
|
|
the relay actually faster, given the phone downloads from the Pi over the same congested radio
|
|
diagnosed as the bottleneck?** Measured directly from chunk-arrival timestamps on the server:
|
|
**~1.11MB/s (~8.9Mbit/s) for the phone-relay path — statistically the same as the ~9-10Mbit/s
|
|
measured for the Pi's own direct Cloudflare upload** earlier this session. **Conclusion: when the
|
|
phone is on the same home WiFi as the Pi, the relay provides no speed benefit** — the
|
|
download-to-phone leg pays the exact same slow-radio cost as a direct Pi upload would, and the
|
|
phone's reupload leg then goes out through the same home internet connection anyway, adding an
|
|
extra hop for no gain. The relay only has genuine value when the phone's *second* leg differs
|
|
meaningfully from the Pi's own path — e.g. cellular data with better upload than the home ISP, or
|
|
uploading after physically leaving home with files already grabbed locally. Stopped the test relay
|
|
(`POST /api/dash/relay/stop`) and confirmed the Pi's own background uploader resumed the same
|
|
file's chunked upload exactly where the phone left off (shared `/status`/`.upload_tmp` state on
|
|
the server made this seamless, no wasted re-transfer).
|
|
|
|
**Next idea (not yet built)**: the user wants a way to browse/preview individual dashcam clips on
|
|
the phone and select specific ones to view or upload (as opposed to bulk-relaying the entire
|
|
backlog) — useful for "just grab this one important clip" rather than a wholesale speed
|
|
optimization. Not scoped/designed yet.
|
|
|
|
**Update — tested the "away from home" hotspot+cellular scenario for real, found a real phone-OS
|
|
limitation (not a code bug)**: the user asked to test the specific scenario the relay feature was
|
|
actually designed for — phone connects to the **Pi's own hotspot** (not home WiFi) to download
|
|
pending files, then re-uploads using the phone's **cellular data**, bypassing the Pi's own (in
|
|
hotspot mode, nonexistent) internet connection entirely. Confirmed via code review this needs zero
|
|
changes: `/api/dash/relay/config` already hands the phone the public `https://yaris.ceaf.dev`
|
|
`SERVER_URL` (not a LAN-only address), so the re-upload leg was always going to be internet-bound
|
|
regardless of which physical network carries it.
|
|
|
|
Live-tested with a real dummy file (dashcam upload toggle paused first so the Pi's own uploader
|
|
couldn't grab it before the phone did): got `TypeError: Failed to fetch` trying to relay. Confirmed
|
|
via a direct test (loading `https://yaris.ceaf.dev/health` as a plain page navigation, not a
|
|
`fetch()`, ruling out CORS) that **the phone's own OS was declining to route internet-bound traffic
|
|
over cellular while still associated with the Pi's no-internet-gateway hotspot WiFi** — it only
|
|
succeeded once the phone auto-fell-back to the home WiFi network (which does have internet)
|
|
instead. This is a phone/OS-level "smart network switching" behavior (common on both iOS and
|
|
Android to avoid using a WiFi network with no internet gateway) that cannot be fixed in this
|
|
project's code — the architecture is sound and was already proven correct in the local end-to-end
|
|
simulation and the byte-for-byte reassembly tests earlier in this phase.
|
|
|
|
**Practical implication**: testing this exact scenario **at home is inherently confounded** — the
|
|
phone always has a known-good home WiFi it can silently fall back to, defeating the test. A true
|
|
validation needs either (a) actually being away from home with no known WiFi in range, forcing the
|
|
phone to use cellular for internet traffic, or (b) manually disabling the phone's WiFi (or its
|
|
"switch to mobile data automatically" setting) while connected to the Pi's hotspot, to force
|
|
cellular use for this one operation.
|
|
|
|
**Update — retested more rigorously, confirmed this is a fundamental architecture-level dead end,
|
|
not just an untested corner case.** Re-ran the same live test multiple times with longer/managed
|
|
"dead-man's-switch" windows (`yaris-netwatch.timer` explicitly stopped during each test run so it
|
|
wouldn't yank the Pi back to home WiFi mid-test — an earlier run's failure was actually caused by
|
|
netwatch reconnecting home WiFi under the phone, see Phase 6 update, not a phone-network issue).
|
|
With netwatch correctly disabled and the Pi confirmed to stay in AP mode for the whole test window,
|
|
still got `Failed to fetch`. Directly tested a plain page navigation (not `fetch()`) to
|
|
`https://yaris.ceaf.dev/health` with WiFi *fully turned off* (pure cellular) — **that also failed**,
|
|
while every other website loaded fine over the same cellular connection with WiFi off. This
|
|
narrows the cause precisely: **the browser/phone genuinely cannot reach the public internet at all
|
|
while `wlan0`/the phone's WiFi radio is associated with the Pi's hotspot, full stop** — this is not
|
|
a settings toggle problem (checked Pixel-specific "switch to mobile data automatically" options,
|
|
not present on this device/Android version) but a fundamental **OS-level "default network" routing
|
|
decision**: a device's default route for *all* IP traffic (including a web page's `fetch()` calls)
|
|
is a single, whole-device choice made by the OS's connectivity manager, not something a browser
|
|
page's JavaScript can override per-request. Real per-request network binding (e.g. Android's
|
|
`ConnectivityManager.bindProcessToNetwork()`) is an API only available to **native
|
|
apps**, not exposed to web pages at all, in any browser, on any phone. Given this, the phone-relay
|
|
feature as designed (a web page doing simultaneous WiFi-to-Pi + cellular-to-server transfers)
|
|
**cannot ever work while the phone is connected to the Pi's no-internet hotspot** — this isn't a
|
|
bug to fix, it's a hard platform ceiling.
|
|
|
|
**Decision: removed the phone-relay feature entirely** (was Phase 11, now reverted - see below).
|
|
Two ways to actually deliver on the original "upload from the road without the Pi's slow WiFi"
|
|
goal were discussed: (1) restructure as two distinct, non-simultaneous phases (fully download to
|
|
the phone's local storage first while on the Pi's hotspot, then have the user manually switch to
|
|
cellular, then upload from local storage) - no native app needed but a more awkward two-step user
|
|
flow, or (2) a real native Android app, which *can* hold both networks open at once via
|
|
`bindProcessToNetwork()` for a true simultaneous relay. Neither has been built - shelved for now
|
|
in favor of simplifying the dashcam retrieval UI (see the new phases below). The Pi's own
|
|
background uploader (plain WiFi/Ethernet, optionally SIM868 cellular fallback for tracking data
|
|
only) remains the only upload path; it's just slower when the Pi's own WiFi is weak, with no
|
|
workaround currently in place for that specific case.
|
|
|
|
**Side-fix from this test session**: found and fixed a real, unrelated bug in `yaris-netwatch.sh`
|
|
during this same test — see the Phase 6 update above.
|
|
|
|
### Phase 12: Remove phone-relay feature — ✅ DONE
|
|
Undo Phase 11 cleanly given the Phase 11 update above (fundamental browser platform limitation,
|
|
not fixable). Removed:
|
|
- `app.py`: `/api/dash/pending`, `/api/dash/file/<path>`, `/api/dash/relay/config`,
|
|
`/api/dash/relay/heartbeat`, `/api/dash/relay/stop`, `/api/dash/relay/mark_uploaded` routes
|
|
(and the now-unused `os`/`send_file`/`abort`/`safe_join` imports they needed).
|
|
- `modules/app_upload.py`: `list_pending_dashcam_files()`, `relay_heartbeat()`/`relay_stop()`/
|
|
`is_relay_active()`, `RELAY_LEASE_SECONDS`/`_relay_lease_until`, and the `is_relay_active()`
|
|
check inside `_upload_dashcam_files()`/`get_status()`.
|
|
- `templates/dash.html`: the "Relay via phone" card.
|
|
- `static/js/dash.js`: all relay JS (`relayOneFile`, `fetchChunkWithRetry`,
|
|
`uploadChunkWithRetry`, heartbeat timers, pending-file list rendering, etc).
|
|
- `server/app.py`: the CORS `after_request` hook, added solely for the phone's cross-origin
|
|
browser calls (the Pi's own Python `requests`-based client was never subject to CORS - that's
|
|
a browser-only enforcement mechanism).
|
|
- Kept untouched, as planned: the chunked upload protocol itself
|
|
(`/api/v1/dashcam/upload/chunk` + `/status`), `dashcam_hash_cache` table + `_hash_dashcam_file()`
|
|
(still used by the Pi's own background uploader to avoid rehashing its 20GB+ backlog every
|
|
loop), `DASHCAM_CHUNK_SIZE`/Cloudflare 100MB workaround - none of that is relay-specific.
|
|
|
|
**Verified**: `python3 -m py_compile` on all touched Python files, `node --check` on `dash.js`,
|
|
grepped all five touched files for any leftover `relay`/`RELAY` reference (none found). Smoke-
|
|
tested by actually importing `app.py` in a scratch venv and hitting it with Flask's test client:
|
|
app imports cleanly (GPS/OBD/SQL/upload startup all log their normal "no hardware" messages, no
|
|
crash), `app.url_map` confirms every relay route is gone while `/`, `/dash`, `/dtc`, and the
|
|
remaining dashcam/upload API routes are still registered, and both `/` and `/dash` render with
|
|
HTTP 200 (`/dash`'s HTML contains zero occurrences of "relay"). Deployed to both the Pi
|
|
(`git pull --ff-only` + `systemctl restart yaris-obd.service`, confirmed `active` and `/dash`
|
|
returns HTTP 200 with no "relay" text over the real network) and the remote server (needed a full
|
|
`docker compose build api && docker compose up -d api` on unraid, not just a restart, since the
|
|
CORS-hook removal is baked into the image at build time - confirmed `/health` still returns
|
|
`{"status":"ok"}` and the response no longer carries any `Access-Control-*` header).
|
|
|
|
**Update 2026-08-09**: Phase 13 (OBD hardening) was completed next; see its section below.
|
|
Phase 15 is the recommended next software phase because Phases 16/17 and the later list-based
|
|
features depend on its shared UI foundation. Phase 14 remains independent but requires a careful
|
|
hardware/filesystem deployment pass rather than ordinary application deployment.
|
|
|
|
### Phase 13: OBD real-hardware readiness (code hardening) — ✅ DONE
|
|
**Reordered from Phase 17 on 2026-08-09** because it was completed directly after Phase 12;
|
|
the phase numbers now follow actual execution order instead of leaving completed work below
|
|
several still-planned phases.
|
|
**Independent of everything else in this plan — only touches `app_obd.py`/`app_events.py`, no UI
|
|
dependency for the backend part, so this can be done anytime/in parallel with the phases above.**
|
|
Nothing has been installed or tested in the actual 2013 Yaris yet - `modules/app_obd.py` has
|
|
only ever been tested standalone on the Pi with nothing plugged into the OBD-II port (correctly
|
|
reports "no connection to car"). Per explicit user decision, this phase is **code hardening
|
|
only** - the user will handle the physical install/bring-up testing themselves. Planned fixes,
|
|
all in `modules/app_obd.py` / `modules/app_events.py`:
|
|
- **Add a genuine background reconnect loop.** Today, `connect_obd()`/`obd_connection()` only
|
|
ever run once per browser tab's socket `connect` event (`app_events.py`'s
|
|
`connect_event()`) - there is no periodic retry. In the real car, this means: if the page
|
|
happens to load before the adapter/ignition is ready, the connection attempt fails once and
|
|
never retries until another browser tab connects. Add a small polling loop (mirroring the
|
|
pattern already proven in `app_gps.py`'s `_poll_loop()`) that periodically calls
|
|
`connect_obd()` while disconnected, independent of any browser being open.
|
|
- **Catch exceptions from `connection.query()`.** Currently a communication hiccup
|
|
(adapter timeout, malformed response, USB blip) would raise inside `obd_speed()`/`obd_rpm()`
|
|
uncaught, silently killing that polling thread forever with no recovery - not just returning
|
|
a null response, an actual crash. Wrap query calls, treat exceptions like a lost connection
|
|
(stop polling, let the reconnect loop above pick it back up) instead of dying silently.
|
|
- **Add an `OBD_PROTOCOL` env override**, mirroring the existing `OBD_PORT` override pattern -
|
|
in case the 2013 Yaris's ECU doesn't cleanly auto-negotiate via ELM327's `ATSP0` (auto) and
|
|
needs an explicit protocol code forced (Toyotas this age are normally ISO 15765-4 CAN 11-bit
|
|
500kbaud, but this has never been confirmed against this specific car).
|
|
- **Improve connection-failure diagnostics.** Distinguish and log clearly: no adapter detected
|
|
at all (USB/port issue) vs adapter detected but ELM327 handshake failed vs handshake OK but no
|
|
response from the car's ECU (car off/ignition not on) - currently all three collapse into the
|
|
same generic "OBD connection failed" message, which will make first-real-car troubleshooting
|
|
much harder than it needs to be.
|
|
- **Add a small connect-time diagnostic log** (ELM327 firmware version + negotiated protocol,
|
|
both available from `python-obd`'s connection object) written once per successful connect -
|
|
useful evidence to sanity-check against Toyota's expected protocol on the very first real test.
|
|
- **Expand the data pulled from the car beyond speed/RPM** (backend only - the dashboard cards
|
|
displaying this land in Phase 22 once the redesign reaches them): coolant temperature, fuel
|
|
level, throttle position, and DTC/MIL status (check-engine light on/off + stored trouble codes,
|
|
reusing/wiring up the already-scaffolded but currently-empty `/dtc` page and `GET_DTC` command).
|
|
**Odometer/mileage is not guaranteed** - the standard OBD-II mode 1 PID for total odometer
|
|
(0xA6) was only added to the SAE J1979 spec in later revisions and is inconsistently supported
|
|
across manufacturers/years; whether the 2013 Yaris's ECU exposes it at all can only be
|
|
confirmed once tested against the real car (falls under "code hardening only" for now - add
|
|
the query with graceful "unsupported" handling rather than assuming it will work).
|
|
|
|
**Implemented 2026-08-09**:
|
|
- Added an idempotent, browser-independent reconnect daemon started from `app.py`; it retries
|
|
every 10 seconds and restarts any dead polling workers after reconnect.
|
|
- Serialized all python-obd queries so speed, RPM, and the expanded sensor worker cannot issue
|
|
overlapping serial exchanges. Query exceptions and silent connection-state changes now close
|
|
and discard the stale handle, emit disconnected state, and hand recovery back to the reconnect
|
|
loop instead of permanently killing a worker.
|
|
- Added `OBD_PROTOCOL` support and distinct logs for no serial adapter, failed ELM327 handshake,
|
|
and ELM327 connected but no ECU response. Successful connections log port, ELM327 firmware,
|
|
negotiated protocol ID, and protocol name once.
|
|
- Added backend polling/emits for coolant temperature, fuel level, throttle position, MIL/DTC
|
|
status, stored DTCs, and SAE PID `0xA6` odometer. Odometer uses a custom python-obd command with
|
|
a null/unsupported result left as `None`; support is not assumed for the 2013 Yaris.
|
|
- Verified against the Pi's installed python-obd 0.7.3 API, plus focused mocked checks covering
|
|
successful connection, forced protocol, ECU-off status, silent disconnect cleanup, query
|
|
exceptions, and odometer decoding. The Pi currently has `/dev/ttyUSB0` present but is not
|
|
connected to the car, so actual PID availability and negotiated Toyota protocol still require
|
|
the user-planned first in-car test.
|
|
- Committed as `8d7909d`, pushed, and deployed to the Pi. After restart the service remained
|
|
`active`, `/` returned HTTP 200, and the log showed four autonomous reconnect attempts with the
|
|
correct "ELM327 handshake succeeded, but the car ECU did not respond" diagnosis and no traceback.
|
|
|
|
### Phase 14: Power-loss resilience & watchdog — 🟡 PARTIAL (watchdog done; filesystem/shutdown deferred)
|
|
**New phase, not code-dependent on anything else — OS/hardware-level, can be done anytime,
|
|
ideally before the Pi lives in the car full-time.** A car cuts power abruptly at every key-off;
|
|
nothing in the project currently protects against that, and repeated unclean power loss is a
|
|
classic cause of SD card corruption on embedded Pi projects. Two complementary angles:
|
|
- **Filesystem hardening**: convert the Pi's root filesystem to read-only with a writable overlay
|
|
(`overlayroot`/`raspi-config`'s built-in overlay FS option), so an abrupt power cut can no longer
|
|
corrupt the OS itself. App data that must persist (`db/sql/`, `dash/videos/`, logs) needs to
|
|
live on an explicitly writable partition/bind-mount carved out of the overlay, since those
|
|
still need real writes.
|
|
- **Clean shutdown before power loss**: either (a) a small delayed-shutdown power module/circuit
|
|
(holds 5V for a few seconds after ignition-off so the Pi can run `shutdown -h now` cleanly), or
|
|
(b) if the SIM868 HAT or another ADC/GPIO source can sense ignition/voltage state, a software
|
|
voltage-drop watcher that triggers a clean shutdown proactively. Exact approach TBD - depends
|
|
on what's easiest to wire into the existing power setup.
|
|
- **Hardware watchdog**: enable the Pi's built-in `bcm2835_wdt` watchdog (via `systemd`'s
|
|
`RuntimeWatchdogSec`/`watchdog-device` support) so a fully hung app or kernel gets a hard reboot
|
|
automatically instead of sitting dead in the car until someone notices.
|
|
|
|
**Implemented 2026-08-09**:
|
|
- Added `deploy/yaris-watchdog.conf` with systemd's hardware watchdog set to 10 seconds and
|
|
`deploy/install_watchdog.sh` to install, activate, and verify it. Deployed to the Pi; confirmed
|
|
`/dev/watchdog0` is the Broadcom BCM2835 watchdog and systemd reports
|
|
`RuntimeWatchdogUSec=10s`.
|
|
- Rebooted the Pi and verified the watchdog setting persisted, systemd returned to `running`,
|
|
`yaris-obd.service` returned to `active`, and `/` returned HTTP 200 after startup.
|
|
- **Explicitly deferred by user decision**: read-only root/filesystem hardening and clean-shutdown
|
|
hardware. The current 59GB root partition fills the SD card and contains 28GB of dashcam data,
|
|
so overlay root needs a separate persistent writable storage decision first; shrinking the live
|
|
mounted root filesystem is not safe. Clean shutdown likewise needs delayed-power or
|
|
ignition-sense hardware before software can implement it.
|
|
|
|
### Phase 15: UI design-system foundation — ✅ DONE
|
|
**New phase, pulled forward from the old "do the whole redesign at the end" plan.** Building the
|
|
Phase 16/17 file browsers (and later the trip-history/Pi-health/settings pages) against the
|
|
current old admin-template style just means rebuilding the same UI twice once the full redesign
|
|
(Phase 22) happens. Instead, do a *small, scoped* slice of the redesign right now: a shared page
|
|
layout/shell plus one reusable list/table component (sortable columns, per-row action buttons,
|
|
delete-with-confirmation pattern, empty-state messaging) that every subsequent list-based feature
|
|
in this plan (dashcam browser, video browser, trip history, settings) can be built on directly,
|
|
instead of writing bespoke list/delete UI code from scratch each time. Not a full redesign of
|
|
every existing page yet - just establishes the pattern once so nothing built after this point
|
|
needs to be redone in Phase 22.
|
|
|
|
**Implemented 2026-08-09**:
|
|
- Added a shared Jinja shell (`templates/base.html`) for navigation, top bar, footer, common
|
|
styles/scripts, page titles, and per-route active navigation. Converted `/`, `/dash`, and
|
|
`/dtc` to extend it, removing their duplicated shell markup while preserving page content and
|
|
page-specific assets.
|
|
- Added a reusable data-table macro (`templates/components.html`) and generic
|
|
`static/js/data-list.js` behavior for dynamic empty states, numeric/text sorting, accessible
|
|
sort direction, and capture-phase confirmation that can stop destructive row handlers before
|
|
they execute. `static/yaris.css` supplies the shared list/action styling.
|
|
- Used the empty `/dtc` page as the first component consumer without pulling its Phase 22 OBD
|
|
wiring forward. Also stopped loading dashboard-only `yaris.js` there and guarded its optional
|
|
`#obd-error` target after browser testing exposed an existing null dereference.
|
|
- Verified Jinja rendering and one active nav item on all three routes, JavaScript syntax, and
|
|
live Pi HTTP 200 responses. Browser-tested all pages at 1280px and 390px with no horizontal
|
|
overflow; injected sample rows confirmed ascending sort, `aria-sort`, empty-state switching,
|
|
and cancellation of a destructive handler.
|
|
|
|
### Phase 16: Dashcam USB file browser (list + delete) — ✅ DONE
|
|
Replace the current all-or-nothing "Import videos" flow's blind spot (no visibility into what's
|
|
actually on the camera) with a real file browser for the mounted USB dashcam device, built on the
|
|
Phase 15 shared list component:
|
|
- New endpoint(s) in `modules/app_dash.py` to list every video file on the mounted dashcam
|
|
(relative path, size, modified time) without copying anything - reuses the existing
|
|
`find_video_files()` walk, pointed at the dashcam's mount point instead of the local
|
|
destination folder.
|
|
- Delete support: individual file delete and a "delete all" bulk action, each requiring the
|
|
mount to be briefly remounted read-write (currently always `ro`) - only for the duration of
|
|
the delete call, then back to read-only (or unmounted) afterward, to minimize risk of
|
|
filesystem corruption from an unclean power loss while writable.
|
|
- UI: a `/dash` card showing total file count + total size on the dashcam, a scrollable file
|
|
list with a delete button per row, a "Delete all" button behind a confirmation dialog (real
|
|
data loss risk - no undo).
|
|
- Explicitly deferred/out of scope: no in-browser preview/playback of files still on the
|
|
dashcam itself (only after import - see Phase 17). No changes to the USB
|
|
select-mode/data-communication toggle behavior - the user replugs power manually to return the
|
|
camera to recording mode, as they already do.
|
|
|
|
**Implemented 2026-08-09**:
|
|
- Added list, single-delete, and delete-all backend operations using the existing USB-only storage
|
|
detection. Each list/import/delete operation shares one lock so mount and file operations cannot
|
|
race; listings include relative path, byte size, modified timestamp, count, and total size.
|
|
- Deletes accept only recognized video extensions, resolve and re-check every target beneath the
|
|
selected mount, and reject absolute paths, traversal, missing files, and symlink escapes.
|
|
- Delete operations mount or remount the dashcam read-write only for the operation, then unmount
|
|
or restore it read-only. If unmount fails, the cleanup path immediately attempts a read-only
|
|
remount and reports a cleanup failure if neither safety action succeeds.
|
|
- Added `/api/dash/files` and `/api/dash/file` routes plus a `/dash` browser built on Phase 15's
|
|
shared component: file/size/time sorting, totals, refresh, per-row confirmed delete, confirmed
|
|
delete-all, empty/no-device states, and error feedback.
|
|
- Verified metadata, traversal rejection, one/all deletion, non-video preservation, import
|
|
regression behavior, and cleanup-failure reporting against temporary filesystem fixtures.
|
|
Deployed to the Pi; the live API correctly returned no storage devices while ignoring the
|
|
attached PL2303 OBD adapter, and the responsive browser UI passed no-device and mocked populated
|
|
list/sort/confirmation/delete tests.
|
|
- **Real-hardware deletion verification (2026-08-09)**: connected the 29.7GB FAT dashcam and listed
|
|
all 87 original videos (29,615,111,758 bytes) through the live API. The first controlled delete
|
|
exposed that FAT files were root-owned despite the read-write remount, so unprivileged
|
|
`os.remove()` failed; added a privileged `sudo rm -- <validated-path>` fallback only after the
|
|
existing extension, realpath, and mount-boundary checks pass. Created and deleted a tiny
|
|
disposable `YARIS_OBD_DELETE_TEST.MP4` twice through the real API. Both cycles returned exactly
|
|
to the original 87-file count and byte total, left every real video untouched, unmounted
|
|
`/mnt/dashcam`, and kept the service active. Bulk delete uses the same validated per-file removal
|
|
path but was not run against the real card because it would intentionally erase all footage.
|
|
|
|
### Phase 17: Local imported-video browser + player — 🚧 PLANNED
|
|
Same file-browser treatment for videos already imported to the Pi's own disk
|
|
(`dash/videos/`, populated by `import_dashcam()`), built on the Phase 15 shared list component:
|
|
- List endpoint (reuses `find_video_files()` against `app_dash.DEFAULT_DESTINATION`) showing
|
|
relative path, size, modified time, and upload status (cross-check against
|
|
`app_sql.dashcam_file_marked_uploaded()` - so it's visible at a glance which imported clips
|
|
have already reached the remote server and which are still pending the Pi's own background
|
|
uploader).
|
|
- Delete support: **local Pi copy only** - remote server copies are intentionally left alone
|
|
as the safe backup/archive (explicit user decision - see 2026-08-09 discussion). No
|
|
"also delete from server" option is being built for now.
|
|
- In-browser video player: since the dashcam records **MP4** (confirmed with the user), this
|
|
is a plain HTML5 `<video>` element - no server-side transcoding needed. Reuses the same
|
|
Range-request-capable file-serving pattern already proven in the (now-removed) relay's
|
|
`/api/dash/file/<path>` endpoint (needed for correct seeking/scrubbing, not just playback).
|
|
- UI: a `/dash` card (or a dedicated sub-page if the list gets long) with a file list, a
|
|
"Play" action that opens the video inline, and a delete button per row with a confirmation
|
|
prompt.
|
|
- **QOL: video thumbnails.** Scrolling a list of `VID_00234.MOV`-style filenames to find a
|
|
specific clip is rough. Grab one low-res frame per file via `ffmpeg` (cheap, done once per file
|
|
and cached, not on every page load) so the browser shows an actual thumbnail per row instead of
|
|
just a filename.
|
|
- **QOL: retention/auto-cleanup policy.** Once a file is confirmed uploaded to the remote server,
|
|
it's safe to free local space. Add an optional automatic cleanup (configurable via the Phase 21
|
|
Settings page - e.g. "delete local copies older than N days once uploaded", off by default)
|
|
so the Pi's limited storage doesn't need to be babysat manually as the dashcam backlog grows.
|
|
|
|
### Phase 18: Trip history & replay — 🚧 PLANNED
|
|
**New phase.** Right now the `tracking` table (Phase 7) is just a raw stream of GPS/speed points
|
|
with no concept of a discrete "trip" - there's no way to look back and see "trip #47, left home
|
|
8:03am, arrived work 8:31am, 22km." Add:
|
|
- Trip boundary detection: group existing tracking rows into discrete trips based on speed>0
|
|
gaps (a simple, dependency-free heuristic - e.g. a gap of N minutes with no motion ends a trip),
|
|
computed either live as rows are logged or in a batch pass over existing data. Does not need to
|
|
wait on Phase 13's OBD hardening - can ship with GPS-only boundary detection first, refine with
|
|
ignition-on/off signals from OBD later if useful.
|
|
- A `trips` table (start/end time, start/end coordinates, distance, duration) in
|
|
`modules/app_sql.py`.
|
|
- A new history page/list (built on the Phase 15 shared list component) showing past trips, and
|
|
a map replay view (reusing the existing offline-tile Leaflet setup from Phase 3/4) that draws
|
|
the recorded route for a selected trip.
|
|
|
|
### Phase 19: Pi health monitoring card — 🚧 PLANNED
|
|
**New phase, small and standalone - no dependencies.** CPU temperature, free disk space, uptime,
|
|
and RAM usage are all cheap to read (`vcgencmd measure_temp`, `psutil`/`/proc` stats) and cheap to
|
|
display, but currently invisible - the only way to notice a problem (thermal throttling in a hot
|
|
car, an SD card filling up) today is to SSH in and check manually. Add a small backend endpoint
|
|
and a dashboard card (built on the Phase 15 component set) surfacing these at a glance. Also
|
|
feeds directly into Phase 20's alerting.
|
|
|
|
### Phase 20: Alerts/notifications — 🚧 PLANNED
|
|
**New phase - depends on Phases 13/17/19 existing first**, since it needs real data sources to
|
|
alert on. Right now, finding out about a problem (check-engine light on, dashcam SD card nearly
|
|
full, upload backlog growing unbounded, Pi overheating, GPS fix lost for a long stretch) requires
|
|
manually opening the web UI. Add a lightweight push-notification integration - no new
|
|
infrastructure needed, just an outbound webhook to something like `ntfy.sh` or a Telegram bot -
|
|
fired on a handful of conditions:
|
|
- DTC/MIL status changes to "on" (from Phase 13's expanded OBD data).
|
|
- Dashcam USB storage or local `dash/videos/` disk usage crossing a configurable threshold.
|
|
- Upload backlog (pending tracking rows or dashcam files) growing past a configurable size,
|
|
suggesting the Pi's been offline/uploading too slowly for a while.
|
|
- Pi CPU temperature crossing a configurable threshold (from Phase 19).
|
|
- GPS fix lost for longer than a configurable duration while the car appears to be moving
|
|
(OBD speed > 0 with no fix - needs Phase 13's data).
|
|
Webhook target/thresholds configurable via the Phase 21 Settings page rather than hardcoded.
|
|
|
|
### Phase 21: Settings page — 🚧 PLANNED
|
|
**New phase - depends on Phase 15's shared components, and benefits from Phases 17/20 existing**
|
|
so there's real configuration to consolidate. Several settings currently only exist as
|
|
SSH-edited files or scattered individual toggles (dashcam upload pause switch on `/dash`,
|
|
cellular APN in `db/upload_creds.env`, hotspot SSID/password via `nmcli` directly) or don't exist
|
|
yet (retention days from Phase 17, alert webhook/thresholds from Phase 20). Add one dedicated
|
|
Settings page consolidating anything reasonably safe to expose over the local-only web UI:
|
|
upload toggle, cellular APN, local video retention policy, alert webhook URL + thresholds. WiFi
|
|
SSID/password changes likely stay out of scope (higher-risk to expose over a UI that could lock
|
|
the Pi out of its own network if fat-fingered) unless it turns out to be easy/safe to add later.
|
|
|
|
### Phase 22: Finish web UI redesign — 🚧 PLANNED
|
|
**Roadmap decision 2026-08-09: keep this as Phase 22 rather than moving it forward.** Phase 15
|
|
already pulled the architecture-level work forward: every page now shares one Jinja shell and
|
|
the later list-based features have a reusable component contract. Phases 16-21 still add most of
|
|
the final screens and dashboard data. Redesigning their visual details before those features
|
|
exist would mean guessing at layouts and polishing the same surfaces twice; keeping the final
|
|
pass here lets it cover the complete product once, while new phases still build on consistent
|
|
foundations in the meantime.
|
|
|
|
The current `templates/`/`static/` UI is an old, mostly-unmodified admin dashboard template
|
|
(SB Admin 2-style Bootstrap theme) with leftover placeholder cruft never replaced with real
|
|
content - e.g. `index.html`'s "Placeholder button", dropdown menus full of "Action" / "Another
|
|
action" / "Something else here" stub links, and a `/dtc` page whose new shared empty-state table
|
|
still has no live DTC data wired into it. Phase 15 already pulled the *foundational* piece of this
|
|
work forward (shared layout/list component, used by every phase since); this phase is the
|
|
remaining cleanup once all the functional phases above have landed:
|
|
- Replace/remove all remaining placeholder buttons, dropdown menus, and stub links that don't do
|
|
anything.
|
|
- Wire up the actually-empty `/dtc` page to show real DTC/MIL data from Phase 13.
|
|
- Add the new dashboard cards for Phase 13's expanded OBD data (coolant temp, fuel level,
|
|
throttle) and Phase 19's Pi health card, using the Phase 15 component patterns.
|
|
- Final visual polish pass across every page now that the full feature set (dashcam/video
|
|
browsers, trip history, settings, alerts) exists, rather than redesigning against a moving
|
|
target.
|
|
- **Frontend stack decision**: keep Bootstrap 5, Jinja, and the Phase 15 shared shell/components;
|
|
clean up or replace the remaining SB Admin theme styling within that stack. A wholesale
|
|
framework rewrite would add deployment weight and offline assets on the Pi without improving
|
|
the tracker features, and would discard the foundation deliberately built before Phases 16-21.
|
|
|
|
### Phase 23: Native Android companion app — 🚧 PLANNED (low priority, do last)
|
|
Revives the phone-relay goal (Phase 11/12) properly this time, using a real native app instead of
|
|
a web page - the only way to actually hold a WiFi connection to the Pi's hotspot and a cellular
|
|
connection to the internet open simultaneously (`ConnectivityManager.bindProcessToNetwork()`,
|
|
unavailable to browser JS - see the Phase 11 update). Explicitly **not urgent** - deprioritized to
|
|
the end of the roadmap, after all web-side work (Phases 12-22) is done. Scope, at a high level
|
|
(not yet designed in detail):
|
|
- **Dashcam file transfer**: browse/select/delete dashcam clips already imported to the Pi (same
|
|
data as the Phase 17 local video browser, just a native client instead of/alongside the web UI),
|
|
download them to the phone, and upload to the remote server over cellular while the phone stays
|
|
on the Pi's hotspot for the download leg - the true simultaneous dual-network relay the web
|
|
version could never deliver.
|
|
- **Trip/location info**: surface live GPS position, speed, and trip history (Phase 18) pulled
|
|
from the Pi while connected to its hotspot (reusing the existing `/api/gps`-style data feeds),
|
|
so the app is useful as a real "check on the car" companion, not just a file-transfer tool.
|
|
- Not yet scoped: distribution (sideload vs Play Store), whether it talks to the Pi directly over
|
|
the hotspot only or also to the remote server directly for trip history when away from the car,
|
|
and how much of the existing REST API surface can be reused as-is vs needs new
|
|
mobile-friendly endpoints. To be designed once Phases 12-22 (web-side cleanup + hardening +
|
|
redesign) are done.
|
|
|
|
## Working Notes / Environment Gotchas
|
|
|
|
- **Pi credentials/SSH**: `ssh yaris@<pi-ip>`. Confirm current IP before connecting (varies by
|
|
network — check locally, not recorded here) — usually reachable via `yaris.local` too (mDNS,
|
|
see Phase 6).
|
|
- **Local dev repo**: work happens on a separate dev machine, not on the Pi. Push to
|
|
`origin/shameless-aislop`, the repo's current default branch (repo is public).
|
|
- **unraid server SSH**: `ssh [email protected]` (alias `unraid`). Real git checkout lives at
|
|
`/mnt/user/app/git/yaris-obd` (not `/mnt/user/appdata/...`, which is only the docker-managed data
|
|
volume for the `db`/`dashcam` containers). Deploy with
|
|
`cd /mnt/user/app/git/yaris-obd && git pull --ff-only && cd server && docker compose up -d --build`.
|
|
Container names: `yaris-obd-db` (Postgres 17), `yaris-obd-api`.
|
|
- **Pi deploy loop**: `ssh yaris@<pi-ip>` (alias `yarisobd`) ->
|
|
`cd ~/app/yaris-obd && git pull --ff-only` -> `sudo systemctl restart yaris-obd.service`
|
|
(passwordless sudo confirmed via `yaris ALL=(ALL) NOPASSWD: ALL`). App stdout/stderr goes to
|
|
`~/app/yaris-obd/yaris-obd.log` (not journald - the systemd unit redirects both streams there),
|
|
so use `tail`/`grep` on that file, not `journalctl`, to see application-level prints (upload
|
|
progress, OBD/GNSS errors, etc.) - `journalctl -u yaris-obd.service` only shows service
|
|
start/stop/crash lifecycle events. **Update 2026-08-09**: plain `print()` calls were silently
|
|
invisible in that log for months because Python fully-buffers stdout when it's not a TTY (a
|
|
file, in this case) — `logging`-based lines (e.g. the `obd` library's own logger, which uses
|
|
stderr) showed up fine, but bare `print()` diagnostics from `app_gps.py` never did. Fixed by
|
|
adding `Environment=PYTHONUNBUFFERED=1` to the systemd unit
|
|
(`deploy/yaris-obd.service`) — **remember to also copy the updated unit file to
|
|
`/etc/systemd/system/yaris-obd.service` and run `sudo systemctl daemon-reload`** on the Pi, a
|
|
plain `git pull` + restart does NOT pick up systemd unit file changes (the live
|
|
`/etc/systemd/system/` copy is independent of the repo's `deploy/` copy).
|
|
- **Pi deploy loop, requirements.txt changes**: after any `requirements.txt` change (e.g. adding
|
|
`RPi.GPIO`), also run `~/app/yaris-obd/venv/bin/pip install -r requirements.txt` on the Pi — a
|
|
`git pull` alone doesn't install new dependencies into the venv.
|
|
- **Pi memory budget**: ~900MB usable RAM, no swap. Any code path that might buffer a whole
|
|
dashcam-sized file (hundreds of MB) in memory is unsafe on this hardware - stream, don't buffer
|
|
(see Phase 10).
|
|
- **Git push auth**: if `git push` fails with a credential/auth error, run `gh auth login`
|
|
(device-code browser flow) then `gh auth setup-git` to wire git to use the `gh` CLI's stored
|
|
credentials — this fixed push auth issues in this environment without needing SSH keys or PATs.
|
|
- **Pi environment gap**: Debian Bookworm blocks system-wide `pip install` (PEP 668,
|
|
"externally-managed-environment"). Use a venv for the real app dependency install (Phase 5)
|
|
rather than `--break-system-packages` everywhere. (Resolved in Phase 5 — venv at
|
|
`/home/yaris/app/yaris-obd/venv`.)
|
|
- **Pi → GitHub auth for `git pull` (private repo)**: the Pi's pre-existing
|
|
`~/.ssh/id_ed25519.pub` is an *inbound* key only (private half is on the dev machine) — can't
|
|
be reused for the Pi's own outbound git auth. Generated a dedicated keypair on the Pi
|
|
(`~/.ssh/yaris_deploy_key`), added it as a GitHub **read-only deploy key** on this repo, and
|
|
added an `~/.ssh/config` entry pinning `github.com` to that key. `[email protected]:...` (SSH
|
|
remote) then works standalone on the Pi, no PAT/interactive login needed.
|
|
- **piwheels**: pip on the Pi automatically uses `piwheels.org` as an extra index (Raspberry Pi
|
|
OS default `pip.conf`) — this is why `gevent` (which normally needs a C compile) installed in
|
|
seconds from a prebuilt ARM wheel instead of taking many minutes to build from source.
|
|
- **python-obd note**: `obd.OBD()` (or with `portstr=`) performs the ELM327 handshake at
|
|
connection time; if there's no car on the other end, it raises rather than silently returning
|
|
a dead handle — `app_obd.py`'s `connect_obd()` already catches this and sets `connection = None`.
|