Document Phase 8/10 real-hardware validation in PLAN.md and refresh READMEs

- PLAN.md: mark Phase 8 (dashcam retrieval) verified on real hardware, add
  Phase 10 documenting the OOM/multipart and Cloudflare body-size fixes plus
  the chunked/resumable upload redesign, both verified end-to-end against
  real dashcam footage. Update architecture decision #3 (Mongo -> Postgres
  supersession), add unraid/Pi deploy notes and memory-budget gotcha.
- readme.md: refresh feature list (remote upload, chunked dashcam uploads,
  toggle) and project status (all 10 phases done, only cellular fallback
  untested).
- server/README.md: document the new chunked upload endpoints and the
  Cloudflare body-size-limit rationale behind them.
This commit is contained in:
Mathieu Moran
2026-08-08 23:33:01 -04:00
parent 7d7be79377
commit 8c33c669a5
3 changed files with 141 additions and 19 deletions
+96 -8
View File
@@ -58,8 +58,11 @@ pick up the project without needing prior chat history.
driving — it is not part of the upload path. Remote-server uploads happen independently via
home WiFi (when in range) or the SIM868 cellular fallback (decision #5) — the phone connecting
to the hotspot never relays/carries upload traffic.
3. **Local/remote persistence**: Keep SQLite locally (reliable local write path per trip) +
MongoDB Atlas for remote sync when online. Fix existing bugs rather than replace the design.
3. **Local/remote persistence**: Keep SQLite locally (reliable local write path per trip). The
original plan paired this with MongoDB Atlas for remote sync (`modules/app_db.py`, fixed in
Phase 7) — **superseded by decision #4/Phase 9**: the actual remote store is the self-hosted
Postgres server, uploaded via `modules/app_upload.py`. `app_db.py`'s Mongo path is not wired
into `app.py` and is effectively dead code at this point, kept only for reference.
4. **Remote server**: Self-hosted — a custom server/API (e.g. on a home server or VPS) receives
itinerary and dashcam uploads, rather than 3rd-party managed cloud services.
5. **Remote upload uplink**: Use the SIM868's GSM/GPRS cellular connection as a fallback uplink
@@ -255,7 +258,7 @@ Neither file was called from `app.py` or anywhere else - orphaned from a prior i
of crashing, since `db/creds.env` still holds empty placeholder credentials (real Mongo
credentials are a user-provided follow-up, out of scope here).
### Phase 8: Dashcam retrieval (app_dash.py) — DONE (tested without physical hardware)
### Phase 8: Dashcam retrieval (app_dash.py) — ✅ DONE, verified on real hardware
Rewrote `app_dash.py` to be brand/device-agnostic: detection via `lsblk -J`, matching partitions
whose parent disk reports `TRAN == "usb"` (this is what safely distinguishes a plugged-in
dashcam/USB stick from the Pi's own onboard storage, which is always `mmc` transport - no more
@@ -292,10 +295,23 @@ Added `dash/videos/` to `.gitignore` (imported footage is per-device data, not s
detected" (accurate, since none is plugged into the dev machine), Import button correctly
stays disabled until a device is found, no JS console errors, no regression on `/` or `/dtc`.
**Still to verify once the physical dashcam is available**: plug it into the Pi, hit "Scan for
dashcam" on `/dash`, confirm it's detected and its real videos import correctly into
`dash/videos/` on the Pi. No code changes should be needed for that - only the transport (loop
device vs. real USB) differs, and the detection/copy logic is transport-agnostic by design.
**Verified against the real physical dashcam** (Jieli-chipset USB ID `1224:2725`, 29.7GB SD card,
`DCIM/1/VID_###.MOV` layout, 87 files, ~28GB total). This first live hardware test found and fixed
three real bugs the loop-device simulation couldn't have caught:
1. **Superfloppy SD card** - the card has no partition table at all (`lsblk` reports the whole
`/dev/sda` disk directly as `fstype: vfat`, no child partition) - the original detection assumed
a partition would always exist. Fixed to also match a `TRAN == "usb"` disk with its own
`fstype` set directly, not just partitions under it.
2. **Mount permission error** - `mount_device()` called `os.makedirs()` directly, but the service
runs as unprivileged `yaris` and `/mnt` is root-owned. Fixed to `sudo mkdir -p <mount_point>`
before the existing `sudo mount`, mirroring that pattern.
3. **OOM crash during upload** (see Phase 10 below) - not a copy-path bug, but discovered
immediately after the first successful real import.
End-to-end result: full 28GB/87-file import from the SD card completed successfully (resumed
cleanly after being interrupted once by the OOM crash below - `copy_files()`'s skip-if-already-
copied logic meant re-running `/api/dash/import` just picked up the remaining files, no data loss
or re-copy of the first 19 files). SD card unmounted cleanly afterward.
### Phase 9: Remote server for itinerary/dashcam upload — DONE
Per architecture decisions #4 and #5: self-hosted server/API to receive itinerary (from the
@@ -356,6 +372,63 @@ confirmed matching row counts on the server via direct psql query, live upload s
Exposed externally at `https://yaris.ceaf.dev` via the user's Cloudflare Tunnel (public hostname
added in the Cloudflare dashboard, pointing at `192.168.10.10:8098`).
### Phase 10: Dashcam upload memory/size fixes — ✅ DONE, verified on real hardware
Two real-hardware-only bugs surfaced once actual ~300MB+ video files (not small test fixtures) hit
the upload path for the first time:
1. **OOM crash from in-memory multipart uploads.** `modules/app_upload.py` originally uploaded
dashcam files via `requests.post(..., files={"file": f})`. The `requests` library builds the
*entire* multipart/form-data body in memory before sending when using `files=` - a well-known
gotcha, not specific to this codebase. On the Pi 3's ~900MB RAM, uploading one ~320MB video this
way spiked the process past 600MB+ RSS, triggering the kernel OOM killer within minutes
(confirmed via `dmesg`: repeated `Out of memory: Killed process ... anon-rss:823928kB`, and via
`pmap -x` showing a single ~362MB anonymous mapping matching the file size almost exactly).
**Fixed**: added `requests-toolbelt` and rewrote the upload body construction to use
`MultipartEncoder`, which streams the file in chunks as the request is sent instead of
materializing it up front. Confirmed via an isolated 300MB test (628MB RSS old way -> 29MB new
way, byte-identical result) and live on the Pi (RSS stayed flat ~55-59MB during real uploads,
vs. previously crashing every few minutes).
2. **Cloudflare Tunnel's hard ~100MB request-body cap.** Even after the memory fix, real uploads
still failed once routed through `https://yaris.ceaf.dev` - a bare `413 Payload Too Large` from
Cloudflare itself (confirmed: the identical file succeeds direct-LAN, 413s only through the
tunnel). This is a free-tier platform limit, unrelated to `MAX_DASHCAM_UPLOAD_MB` on the Flask
side. **Fixed** via a chunked/resumable upload redesign (well under the 100MB cap):
- Client (`modules/app_upload.py`): `DASHCAM_CHUNK_SIZE = 40MB`; new
`_upload_video_chunked()` checks `/status` first (enables resume across restarts/failures),
then POSTs missing chunks one at a time via `MultipartEncoder`.
- Server (`server/app.py`): new `GET /api/v1/dashcam/upload/status` (reports
`already_uploaded` or `received_chunk_indices`) and `POST /api/v1/dashcam/upload/chunk`
(stages chunks under `STORAGE_DIR/.upload_tmp/<device>/<sha256>/`, reassembles + verifies
SHA256 once all chunks arrive, records via the existing `db.insert_dashcam_file`, cleans up
the staging dir). A `.total_chunks` marker guards against stale/mismatched partial sets if the
chunk size ever changes between client versions. The original single-shot
`/api/v1/dashcam/upload` endpoint is untouched, kept for reference/compatibility.
**Verified end-to-end against the real deployed server through the real Cloudflare tunnel**,
first with a synthetic interrupted-then-resumed 250MB test (status transitions
`not_started` -> partial upload -> `in_progress` w/ correct `received_chunk_indices` -> resume ->
`complete` -> `already_uploaded`, SHA256 match confirmed on disk), then with real dashcam footage:
the Pi copied all 87 real files locally and the background upload thread drained the real backlog
one file at a time, each one verified byte-for-byte (SHA256 match Pi vs. server) and correctly
recorded in Postgres, while the Pi's service ran 50+ minutes straight with RSS staying in a bounded
~55-141MB range (no OOM kills) - a stark contrast to the previous crash-every-few-minutes behavior.
Given home-upload-bandwidth constraints, the full 28GB backlog drains at roughly one ~350MB file
per 4-5 minutes and is expected to take several hours total - this runs unattended in the
background with no user action needed once started.
**Not yet built**: parallel/concurrent chunk uploads (currently strictly sequential, one chunk
then one file at a time) - would speed up the multi-hour full-backlog drain but adds complexity;
not requested by the user yet, flagged here as a possible future optimization if the sequential
pace becomes annoying in practice.
**Known manual step (not yet automated)**: this dashcam model boots into a "select mode" menu
(dashcam / mass-storage / webcam) when it senses USB data lines are live, and needs USB data
communication physically absent to boot straight into recording mode, but present when plugging
into the Pi for import. No code in this repo handles that switch - it's a manual physical step by
the user (e.g. a data-blocking USB cable/adapter, or plugging/unplugging appropriately) each time
before recording vs. before importing. Worth revisiting if it becomes tedious in daily use (e.g. a
Pi-controlled USB data-line switch), but out of scope for now.
### SIM868 cellular fallback for tracking uploads — built, UNTESTED (no SIM card available)
`modules/app_cellular.py` gives the tracking-upload path a fallback over the SIM868's cellular
data connection when WiFi/Ethernet is unreachable. Design notes:
@@ -391,7 +464,22 @@ data connection when WiFi/Ethernet is unreachable. Design notes:
network — check locally, not recorded here) — usually reachable via `yaris.local` too (mDNS,
see Phase 6).
- **Local dev repo**: work happens on a separate dev machine, not on the Pi. Push to
`origin/shameless-aislop` (not yet merged to `main`) when work is ready.
`origin/shameless-aislop`, the repo's current default branch (repo is public).
- **unraid server SSH**: `ssh [email protected]` (alias `unraid`). Real git checkout lives at
`/mnt/user/app/git/yaris-obd` (not `/mnt/user/appdata/...`, which is only the docker-managed data
volume for the `db`/`dashcam` containers). Deploy with
`cd /mnt/user/app/git/yaris-obd && git pull --ff-only && cd server && docker compose up -d --build`.
Container names: `yaris-obd-db` (Postgres 17), `yaris-obd-api`.
- **Pi deploy loop**: `ssh yaris@<pi-ip>` (alias `yarisobd`) ->
`cd ~/app/yaris-obd && git pull --ff-only` -> `sudo systemctl restart yaris-obd.service`
(passwordless sudo confirmed via `yaris ALL=(ALL) NOPASSWD: ALL`). App stdout/stderr goes to
`~/app/yaris-obd/yaris-obd.log` (not journald - the systemd unit redirects both streams there),
so use `tail`/`grep` on that file, not `journalctl`, to see application-level prints (upload
progress, OBD/GNSS errors, etc.) - `journalctl -u yaris-obd.service` only shows service
start/stop/crash lifecycle events.
- **Pi memory budget**: ~900MB usable RAM, no swap. Any code path that might buffer a whole
dashcam-sized file (hundreds of MB) in memory is unsafe on this hardware - stream, don't buffer
(see Phase 10).
- **Git push auth**: if `git push` fails with a credential/auth error, run `gh auth login`
(device-code browser flow) then `gh auth setup-git` to wire git to use the `gh` CLI's stored
credentials — this fixed push auth issues in this environment without needing SSH keys or PATs.