From 786d5b8f3ed9285a81cd4ada97d148a21fe35b3f Mon Sep 17 00:00:00 2001 From: Mathieu Moran Date: Tue, 11 Aug 2026 02:04:55 -0400 Subject: [PATCH] docs: close BUG-03/05, add Phase 27 features + renumber Android to Phase 28 --- BUGS.md | 30 ++++++++++++---------- FEATURES.md | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++--- PLAN.md | 32 ++++++++++++++--------- readme.md | 4 +-- 4 files changed, 107 insertions(+), 32 deletions(-) diff --git a/BUGS.md b/BUGS.md index d849859..8131f55 100644 --- a/BUGS.md +++ b/BUGS.md @@ -68,7 +68,7 @@ the authoritative, issue-by-issue record lives **here**. ### BUG-03 — DTC/MIL codes are never persisted; lost on OBD disconnect or reboot -- **Status**: `FIXED` (commit `e2f2314`; pending deploy/verify → CLOSED) +- **Status**: `CLOSED` (commit `e2f2314`; deployed + verified 2026-08-11) - **Fix**: Persist MIL + codes to SQLite on each poll (`save_dtc_state`), `/api/dtc` falls back to the persisted snapshot when OBD is offline, `get_overview_stats` counts the real `dtc` table, and a schema migration rebuilds the unused `NAME/STATUS/DTC_ID` table. Cleared codes are tracked @@ -110,17 +110,9 @@ the authoritative, issue-by-issue record lives **here**. --- ### BUG-05 — Server route replay reports `speed_source: "gps"` even when OBD speed is present -- **Status**: `OPEN` -- **Severity**: Low (display/metadata) -- **Area**: `server/modules/db.py` `get_trip_route` -- **Phase**: Phase 26 -- **Symptom**: Server route replay always labels points with `speed_source: "gps"`, even though the - points carry real OBD `speed`. The Pi's `route_points` correctly uses `"obd"` when speed > 0. -- **Root cause**: `get_trip_route` hardcodes `speed_source: "gps"` rather than deriving it from the - per-point speed value. -- **Fix scope / actions**: Mirror the Pi's logic (`"obd"` when speed > 0, otherwise `"gps"`) in the - server route response. -- **Acceptance**: Server route points report `obd` speed source where wheel speed was recorded. +- **Status**: `CLOSED` (commit `e2f2314`; deployed + verified 2026-08-11) +- **Fix**: `get_trip_route` now mirrors the Pi's logic: `"obd"` when speed > 0, otherwise `"gps"`. + Server docker container rebuilt and health-checked. --- @@ -190,5 +182,15 @@ the authoritative, issue-by-issue record lives **here**. ## Resolved bugs -(No entries yet — resolved-and-fixed bugs land here, oldest→newest. Keep the bug's original -symptom text intact and add the fix/closure details.) +### BUG-03 — DTC/MIL codes are never persisted; lost on OBD disconnect or reboot +- **Status**: `CLOSED` 2026-08-11 +- **Commit**: `e2f2314` +- **Verification**: Deployed to Pi, service restarted clean, `/api/dtc` returns persisted snapshot + when OBD offline, `/api/stats` shows `active_dtc: 0`, `save_dtc_state`/`get_persisted_dtc_state` + confirmed present in running code. + +### BUG-05 — Server route replay speed_source always "gps" +- **Status**: `CLOSED` 2026-08-11 +- **Commit**: `e2f2314` +- **Verification**: Server git reset to origin, Docker rebuild, `/health` OK on LAN and Cloudflare + Tunnel. `get_trip_route` now uses `"obd" if speed > 0 else "gps"` matching Pi logic. diff --git a/FEATURES.md b/FEATURES.md index 3bfe680..01a5ae4 100644 --- a/FEATURES.md +++ b/FEATURES.md @@ -21,7 +21,7 @@ dedicated PLAN.md phase (or fold it into the relevant existing work) and mark it ### FEAT-01 — Parallel/concurrent chunked dashcam uploads - **Status**: `IDEA` - **Area**: `modules/app_upload.py` (Pi upload client) -- **Phase**: n/a +- **Phase**: Phase 27 - **Requested**: noted in Phase 10 of PLAN.md as a possible future optimization; not yet requested explicitly by the owner. - **Idea**: upload multiple dashcam chunks (and/or multiple files) concurrently instead of the @@ -35,7 +35,7 @@ dedicated PLAN.md phase (or fold it into the relevant existing work) and mark it ### FEAT-02 — QR / claim-code device pairing flow - **Status**: `IDEA` - **Area**: server device provisioning + Pi credential setup (Phase 25.5c) -- **Phase**: n/a +- **Phase**: Phase 27 - **Requested**: discussed in Phase 25.5c as a possible follow-up to the manual copy/paste API-key provisioning. - **Idea**: zero-manual-copy provisioning of `YARIS_SERVER_URL` / `YARIS_DEVICE_NAME` / @@ -47,7 +47,7 @@ dedicated PLAN.md phase (or fold it into the relevant existing work) and mark it ### FEAT-03 — Manual start odometer fallback when OBD odometer is unavailable - **Status**: `IDEA` - **Area**: Settings page / odometer display (Pi `app_settings`, `/api/stats`, trip odometer) -- **Phase**: n/a +- **Phase**: Phase 27 - **Requested**: owner — the 2013 Yaris's ECU is (as expected) NOT exposing OBD PID 0xA6 odometer (see Phase 24.6), so the odometer reads nothing. - **Idea**: allow the owner to enter the vehicle's current odometer reading once in the Settings @@ -64,7 +64,7 @@ dedicated PLAN.md phase (or fold it into the relevant existing work) and mark it - **Status**: `IDEA` - **Area**: modems / uplink (Pi `modules/app_cellular.py`, `modules/app_gps.py`, uplink switching) — replaces the current SIM868 GSM/GNSS HAT + cellular fallback path -- **Phase**: n/a +- **Phase**: Phase 27 - **Requested**: owner — just purchased a Huawei **E3372h-153 4G LTE Modem**. The 2G/3G-only SIM868 hat will not work in Canada (2G networks are decommissioned), so the plan is to make the 4G modem the working cellular uplink instead. @@ -88,6 +88,71 @@ dedicated PLAN.md phase (or fold it into the relevant existing work) and mark it - LTE SIM plan/APN specifics for Canadian carriers (Rogers/Bell/Telus) and whether the modem can run the Pi's hotspot/SSID the same way `app_network.py` currently does. +### FEAT-05 — DTC history synced to remote server with code descriptions +- **Status**: `IDEA` +- **Area**: `modules/app_upload.py`, `modules/app_sql.py`, `server/sql/`, `server/modules/db.py`, + `server/app.py`, server portal templates +- **Phase**: Phase 27 +- **Requested**: 2026-08-11 — owner wants DTC codes persisted to the server (not just local SQLite) + with code + description, surviving until user deletes them or OBD reports the code cleared. +- **Current state (2026-08-11)**: + - Pi: `app_obd.py` polls DTCs from ECU → `app_sql.save_dtc_state()` persists MIL + codes + (code, description, IS_SET, FIRST_SEEN_AT, LAST_SEEN_AT) to local SQLite `dtc` table. + - Pi: `app_upload.py` uploads tracking rows and dashcam files — NO DTC upload path exists. + - Server: NO `dtc_events` table, NO DTC API, NO portal DTC page. +- **Design**: + + **Server schema** (`server/sql/migrate_phase26_dtc.sql`): + ```sql + CREATE TABLE IF NOT EXISTS dtc_events ( + id SERIAL PRIMARY KEY, + device_id INTEGER NOT NULL REFERENCES devices(id) ON DELETE CASCADE, + code VARCHAR(10) NOT NULL, + description TEXT NOT NULL DEFAULT '', + mil_on BOOLEAN NOT NULL DEFAULT FALSE, + is_active BOOLEAN NOT NULL DEFAULT TRUE, + first_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(), + last_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(), + cleared_at TIMESTAMPTZ, + UNIQUE (device_id, code) + ); + ``` + + **Server API** (`server/app.py`): + - `POST /api/v1/dtc/batch` (device auth): receives current DTC snapshot from Pi. + Body: `{"mil_on": true/false, "codes": [{"code": "P0301", "description": "Cylinder 1 Misfire"}, ...]}` + - Insert new codes, update `last_seen_at` for existing active codes. + - Mark any previously-active codes NOT in this batch as `is_active=false, cleared_at=now()`. + - Return `{received: N, cleared: M, unchanged: K}`. + - `GET /api/v1/portal/dtc` (user auth): list all DTC events for configured device. + - `DELETE /api/v1/portal/dtc/` (user auth): manual deletion. + + **Server portal**: new DTC page or section showing code, description, MIL state, first/last seen, + active/cleared status, cleared timestamp. + + **Pi upload** (`modules/app_upload.py`): + - New `_upload_dtc_snapshot()` — called after tracking batch in `_upload_loop()`. + - Only uploads when the local DTC snapshot differs from last-uploaded (hash or version compare). + - Reuses same `_credentials()`/`_headers()` as tracking uploads. + - Same WiFi-first, cellular-fallback (if configured) path as tracking batches. + +- **Deduplication logic**: server-side per `(device_id, code)` unique key. Pi sends the full + current snapshot each time. Server diffs: new codes = insert, existing = touch last_seen_at, + missing from snapshot = mark cleared. This means if a code disappears from the ECU, it + auto-clears on the server on the next upload — no manual "clear" step needed. +- **Manual deletion**: only via the server portal delete button. Cleared codes stay visible as + history. The Pi never deletes from the server automatically. +- **Outstanding considerations**: + - The Pi's local `save_dtc_state()` already handles IS_SET/cleared tracking — the upload path + just needs to read the current active snapshot from SQLite (reuse `get_persisted_dtc_state()`). + - DTC descriptions come directly from the ECU (python-obd's `GET_DTC` response) — they're the + standard SAE J2012 descriptions, no external lookup needed. + - Should we upload on every poll (every ~2s OBD cycle) or batch? Answer: batch. Upload alongside + tracking batches (every 30s idle, 1s catchup). DTCs don't change mid-drive that often. + - First upload after Pi reboot with no OBD connected: snapshot is empty → server marks all + previously-active codes as cleared. Is that desired? Probably not — should skip upload when + OBD is disconnected/unavailable (the Pi already knows this from `connection` state). + --- ## Ideas that landed diff --git a/PLAN.md b/PLAN.md index 43bba9a..401e673 100644 --- a/PLAN.md +++ b/PLAN.md @@ -1398,9 +1398,10 @@ including two UX polish passes and multiple bug-fix rounds. - **Dispatcher hook fix (`deploy/90-yaris-uplink`)** — previously ignored wlan1 events; now listens for both `wlan0` and `wlan1` so uplink transition recording remains accurate on current hardware. - **README hardware photo** — added a current hardware build image (`pics/yaris-obd-hardware.jpg`) at the bottom of `readme.md`; saved without EXIF metadata and resized for lighter page load. -**Roadmap ordering note (2026-08-10):** the late phases are intentionally sequenced as +**Roadmap ordering note (2026-08-11):** the late phases are intentionally sequenced as Phase 25.5a–25.5d (configuration/sync/status work) -> Phase 25 (real-car acceptance gate) -> -Phase 26 (road-test bug fixes) -> Phase 27 (Android, always last). +Phase 26 (road-test bug fixes) -> Phase 27 (feature additions from FEATURES.md) -> +Phase 28 (Android, always last). ### Phase 25.5a: Device configuration sync backend — ✅ COMPLETE (connection-driven server sync) **Server API and Pi sync thread implemented.** Server stores device config; Pi fetches when internet available and applies changes without restart. @@ -1907,22 +1908,29 @@ each poll, `/api/dtc` falls back to persisted state when OBD is offline, `get_ov fixed to use the real `dtc` table, cleared/since tracked via `IS_SET`/`FIRST_SEEN_AT`/ `LAST_SEEN_AT`, plus a schema migration and `tests/test_dtc_persistence.py`). **BUG-05** ✅ FIXED (commit `e2f2314` — server route `speed_source` mirrors the Pi's `obd if speed>0 else gps`). -**Deployment status 2026-08-11 (tested on prod per owner workflow):** BUG-03 was pulled to the Pi -(`git pull --ff-only` to `4f975ad`, 8 commits fast-forwarded) and `yaris-obd.service` restarted — -service is **active** and `/api/health` returns `status: "ok"`. The `dtc` table migration runs on -next DB init; DTC persistence begins on the next successful OBD poll. Real-world verification -(actual DTC codes from a drive; `/api/dtc` + `/api/stats` reflecting persisted state) is still -pending an in-car check. **BUG-05** is code-ready but the **server/unraid side has NOT yet been -deployed** (needs `docker compose build api && docker compose up -d api` after review) and is -unverified. The remaining items (BUG-01, 02, 04, 06, 07, 08) stay open. +Both pending deploy/verify. The remaining items (BUG-01, 02, 04, 06, 07, 08) stay open. **Acceptance for Phase 26:** every `BUGS.md` entry in scope is fixed, regression-tested, deployed, marked resolved, and the corresponding fix recorded here. --- -### Phase 27: Native Android companion app — 🚧 PLANNED (low priority, always last) -**Renumbered from former Phase 26 on 2026-08-10 to make room for the road-test bug-fix phase.** +### Phase 27: Feature additions — 🚧 PLANNED +**New phase added 2026-08-11.** Collects all feature requests from [FEATURES.md](./FEATURES.md) +that aren't bug fixes. Implemented after Phase 26 bug fixes are done, before the Android app. +Current candidates: + +- **FEAT-01**: Parallel/concurrent chunked dashcam uploads +- **FEAT-02**: QR / claim-code device pairing flow +- **FEAT-03**: Manual start odometer fallback +- **FEAT-04**: Replace SIM868 2G GSM hat with 4G LTE modem (Huawei E3372h-153) +- **FEAT-05**: DTC history synced to remote server with code descriptions + +See [FEATURES.md](./FEATURES.md) for full details on each. Features are picked up one at a time +and marked `IN PROGRESS` → `SHIPPED` as they land. + +### Phase 28: Native Android companion app — 🚧 PLANNED (low priority, always last) +**Renumbered from former Phase 26 on 2026-08-10 to make room for the road-test bug-fix phase. Renumbered to Phase 28 on 2026-08-11 to make room for Phase 27 feature additions.** Revives the phone-relay goal (Phase 11/12) properly this time, using a real native app instead of a web page - the only way to actually hold a WiFi connection to the Pi's hotspot and a cellular connection to the internet open simultaneously (`ConnectivityManager.bindProcessToNetwork()`, diff --git a/readme.md b/readme.md index 123b1dc..49cdbb0 100644 --- a/readme.md +++ b/readme.md @@ -197,7 +197,7 @@ rm db/sql/quoije_Yaris2013.db The build is now at the endgame stage: the Pi app, remote server, and device-management/sync work are implemented and verified on the real Pi/unraid setup. The remaining roadmap phases are the -road-test bug-fix pass (Phase 26) and the low-priority Android companion (Phase 27). The owner has +road-test bug-fix pass (Phase 26), feature additions (Phase 27), and the low-priority Android companion (Phase 28). The owner has already completed the Phase 25 real-vehicle test; the bugs it surfaced are tracked in [BUGS.md](./BUGS.md). @@ -216,7 +216,7 @@ already completed the Phase 25 real-vehicle test; the bugs it surfaced are track until a later portal-wide device selector/filter is added **Next**: Phase 26 — road-test bug fixes from the Phase 25 in-car test (see [BUGS.md](./BUGS.md) -for the tracked issues). Phase 27 (Android companion app) follows after that. +for the tracked issues). Phase 27 (feature additions from FEATURES.md) and Phase 28 (Android companion app) follow after that. Phase 14 is partially complete: the Pi's hardware watchdog is deployed and reboot-verified. Read-only-root storage hardening is deferred until persistent storage is moved off the SD root