Document Phase 8/10 real-hardware validation in PLAN.md and refresh READMEs
- PLAN.md: mark Phase 8 (dashcam retrieval) verified on real hardware, add Phase 10 documenting the OOM/multipart and Cloudflare body-size fixes plus the chunked/resumable upload redesign, both verified end-to-end against real dashcam footage. Update architecture decision #3 (Mongo -> Postgres supersession), add unraid/Pi deploy notes and memory-budget gotcha. - readme.md: refresh feature list (remote upload, chunked dashcam uploads, toggle) and project status (all 10 phases done, only cellular fallback untested). - server/README.md: document the new chunked upload endpoints and the Cloudflare body-size-limit rationale behind them.
This commit is contained in:
@@ -58,8 +58,11 @@ pick up the project without needing prior chat history.
|
||||
driving — it is not part of the upload path. Remote-server uploads happen independently via
|
||||
home WiFi (when in range) or the SIM868 cellular fallback (decision #5) — the phone connecting
|
||||
to the hotspot never relays/carries upload traffic.
|
||||
3. **Local/remote persistence**: Keep SQLite locally (reliable local write path per trip) +
|
||||
MongoDB Atlas for remote sync when online. Fix existing bugs rather than replace the design.
|
||||
3. **Local/remote persistence**: Keep SQLite locally (reliable local write path per trip). The
|
||||
original plan paired this with MongoDB Atlas for remote sync (`modules/app_db.py`, fixed in
|
||||
Phase 7) — **superseded by decision #4/Phase 9**: the actual remote store is the self-hosted
|
||||
Postgres server, uploaded via `modules/app_upload.py`. `app_db.py`'s Mongo path is not wired
|
||||
into `app.py` and is effectively dead code at this point, kept only for reference.
|
||||
4. **Remote server**: Self-hosted — a custom server/API (e.g. on a home server or VPS) receives
|
||||
itinerary and dashcam uploads, rather than 3rd-party managed cloud services.
|
||||
5. **Remote upload uplink**: Use the SIM868's GSM/GPRS cellular connection as a fallback uplink
|
||||
@@ -255,7 +258,7 @@ Neither file was called from `app.py` or anywhere else - orphaned from a prior i
|
||||
of crashing, since `db/creds.env` still holds empty placeholder credentials (real Mongo
|
||||
credentials are a user-provided follow-up, out of scope here).
|
||||
|
||||
### Phase 8: Dashcam retrieval (app_dash.py) — DONE (tested without physical hardware)
|
||||
### Phase 8: Dashcam retrieval (app_dash.py) — ✅ DONE, verified on real hardware
|
||||
Rewrote `app_dash.py` to be brand/device-agnostic: detection via `lsblk -J`, matching partitions
|
||||
whose parent disk reports `TRAN == "usb"` (this is what safely distinguishes a plugged-in
|
||||
dashcam/USB stick from the Pi's own onboard storage, which is always `mmc` transport - no more
|
||||
@@ -292,10 +295,23 @@ Added `dash/videos/` to `.gitignore` (imported footage is per-device data, not s
|
||||
detected" (accurate, since none is plugged into the dev machine), Import button correctly
|
||||
stays disabled until a device is found, no JS console errors, no regression on `/` or `/dtc`.
|
||||
|
||||
**Still to verify once the physical dashcam is available**: plug it into the Pi, hit "Scan for
|
||||
dashcam" on `/dash`, confirm it's detected and its real videos import correctly into
|
||||
`dash/videos/` on the Pi. No code changes should be needed for that - only the transport (loop
|
||||
device vs. real USB) differs, and the detection/copy logic is transport-agnostic by design.
|
||||
**Verified against the real physical dashcam** (Jieli-chipset USB ID `1224:2725`, 29.7GB SD card,
|
||||
`DCIM/1/VID_###.MOV` layout, 87 files, ~28GB total). This first live hardware test found and fixed
|
||||
three real bugs the loop-device simulation couldn't have caught:
|
||||
1. **Superfloppy SD card** - the card has no partition table at all (`lsblk` reports the whole
|
||||
`/dev/sda` disk directly as `fstype: vfat`, no child partition) - the original detection assumed
|
||||
a partition would always exist. Fixed to also match a `TRAN == "usb"` disk with its own
|
||||
`fstype` set directly, not just partitions under it.
|
||||
2. **Mount permission error** - `mount_device()` called `os.makedirs()` directly, but the service
|
||||
runs as unprivileged `yaris` and `/mnt` is root-owned. Fixed to `sudo mkdir -p <mount_point>`
|
||||
before the existing `sudo mount`, mirroring that pattern.
|
||||
3. **OOM crash during upload** (see Phase 10 below) - not a copy-path bug, but discovered
|
||||
immediately after the first successful real import.
|
||||
|
||||
End-to-end result: full 28GB/87-file import from the SD card completed successfully (resumed
|
||||
cleanly after being interrupted once by the OOM crash below - `copy_files()`'s skip-if-already-
|
||||
copied logic meant re-running `/api/dash/import` just picked up the remaining files, no data loss
|
||||
or re-copy of the first 19 files). SD card unmounted cleanly afterward.
|
||||
|
||||
### Phase 9: Remote server for itinerary/dashcam upload — DONE
|
||||
Per architecture decisions #4 and #5: self-hosted server/API to receive itinerary (from the
|
||||
@@ -356,6 +372,63 @@ confirmed matching row counts on the server via direct psql query, live upload s
|
||||
Exposed externally at `https://yaris.ceaf.dev` via the user's Cloudflare Tunnel (public hostname
|
||||
added in the Cloudflare dashboard, pointing at `192.168.10.10:8098`).
|
||||
|
||||
### Phase 10: Dashcam upload memory/size fixes — ✅ DONE, verified on real hardware
|
||||
Two real-hardware-only bugs surfaced once actual ~300MB+ video files (not small test fixtures) hit
|
||||
the upload path for the first time:
|
||||
|
||||
1. **OOM crash from in-memory multipart uploads.** `modules/app_upload.py` originally uploaded
|
||||
dashcam files via `requests.post(..., files={"file": f})`. The `requests` library builds the
|
||||
*entire* multipart/form-data body in memory before sending when using `files=` - a well-known
|
||||
gotcha, not specific to this codebase. On the Pi 3's ~900MB RAM, uploading one ~320MB video this
|
||||
way spiked the process past 600MB+ RSS, triggering the kernel OOM killer within minutes
|
||||
(confirmed via `dmesg`: repeated `Out of memory: Killed process ... anon-rss:823928kB`, and via
|
||||
`pmap -x` showing a single ~362MB anonymous mapping matching the file size almost exactly).
|
||||
**Fixed**: added `requests-toolbelt` and rewrote the upload body construction to use
|
||||
`MultipartEncoder`, which streams the file in chunks as the request is sent instead of
|
||||
materializing it up front. Confirmed via an isolated 300MB test (628MB RSS old way -> 29MB new
|
||||
way, byte-identical result) and live on the Pi (RSS stayed flat ~55-59MB during real uploads,
|
||||
vs. previously crashing every few minutes).
|
||||
2. **Cloudflare Tunnel's hard ~100MB request-body cap.** Even after the memory fix, real uploads
|
||||
still failed once routed through `https://yaris.ceaf.dev` - a bare `413 Payload Too Large` from
|
||||
Cloudflare itself (confirmed: the identical file succeeds direct-LAN, 413s only through the
|
||||
tunnel). This is a free-tier platform limit, unrelated to `MAX_DASHCAM_UPLOAD_MB` on the Flask
|
||||
side. **Fixed** via a chunked/resumable upload redesign (well under the 100MB cap):
|
||||
- Client (`modules/app_upload.py`): `DASHCAM_CHUNK_SIZE = 40MB`; new
|
||||
`_upload_video_chunked()` checks `/status` first (enables resume across restarts/failures),
|
||||
then POSTs missing chunks one at a time via `MultipartEncoder`.
|
||||
- Server (`server/app.py`): new `GET /api/v1/dashcam/upload/status` (reports
|
||||
`already_uploaded` or `received_chunk_indices`) and `POST /api/v1/dashcam/upload/chunk`
|
||||
(stages chunks under `STORAGE_DIR/.upload_tmp/<device>/<sha256>/`, reassembles + verifies
|
||||
SHA256 once all chunks arrive, records via the existing `db.insert_dashcam_file`, cleans up
|
||||
the staging dir). A `.total_chunks` marker guards against stale/mismatched partial sets if the
|
||||
chunk size ever changes between client versions. The original single-shot
|
||||
`/api/v1/dashcam/upload` endpoint is untouched, kept for reference/compatibility.
|
||||
|
||||
**Verified end-to-end against the real deployed server through the real Cloudflare tunnel**,
|
||||
first with a synthetic interrupted-then-resumed 250MB test (status transitions
|
||||
`not_started` -> partial upload -> `in_progress` w/ correct `received_chunk_indices` -> resume ->
|
||||
`complete` -> `already_uploaded`, SHA256 match confirmed on disk), then with real dashcam footage:
|
||||
the Pi copied all 87 real files locally and the background upload thread drained the real backlog
|
||||
one file at a time, each one verified byte-for-byte (SHA256 match Pi vs. server) and correctly
|
||||
recorded in Postgres, while the Pi's service ran 50+ minutes straight with RSS staying in a bounded
|
||||
~55-141MB range (no OOM kills) - a stark contrast to the previous crash-every-few-minutes behavior.
|
||||
Given home-upload-bandwidth constraints, the full 28GB backlog drains at roughly one ~350MB file
|
||||
per 4-5 minutes and is expected to take several hours total - this runs unattended in the
|
||||
background with no user action needed once started.
|
||||
|
||||
**Not yet built**: parallel/concurrent chunk uploads (currently strictly sequential, one chunk
|
||||
then one file at a time) - would speed up the multi-hour full-backlog drain but adds complexity;
|
||||
not requested by the user yet, flagged here as a possible future optimization if the sequential
|
||||
pace becomes annoying in practice.
|
||||
|
||||
**Known manual step (not yet automated)**: this dashcam model boots into a "select mode" menu
|
||||
(dashcam / mass-storage / webcam) when it senses USB data lines are live, and needs USB data
|
||||
communication physically absent to boot straight into recording mode, but present when plugging
|
||||
into the Pi for import. No code in this repo handles that switch - it's a manual physical step by
|
||||
the user (e.g. a data-blocking USB cable/adapter, or plugging/unplugging appropriately) each time
|
||||
before recording vs. before importing. Worth revisiting if it becomes tedious in daily use (e.g. a
|
||||
Pi-controlled USB data-line switch), but out of scope for now.
|
||||
|
||||
### SIM868 cellular fallback for tracking uploads — built, UNTESTED (no SIM card available)
|
||||
`modules/app_cellular.py` gives the tracking-upload path a fallback over the SIM868's cellular
|
||||
data connection when WiFi/Ethernet is unreachable. Design notes:
|
||||
@@ -391,7 +464,22 @@ data connection when WiFi/Ethernet is unreachable. Design notes:
|
||||
network — check locally, not recorded here) — usually reachable via `yaris.local` too (mDNS,
|
||||
see Phase 6).
|
||||
- **Local dev repo**: work happens on a separate dev machine, not on the Pi. Push to
|
||||
`origin/shameless-aislop` (not yet merged to `main`) when work is ready.
|
||||
`origin/shameless-aislop`, the repo's current default branch (repo is public).
|
||||
- **unraid server SSH**: `ssh [email protected]` (alias `unraid`). Real git checkout lives at
|
||||
`/mnt/user/app/git/yaris-obd` (not `/mnt/user/appdata/...`, which is only the docker-managed data
|
||||
volume for the `db`/`dashcam` containers). Deploy with
|
||||
`cd /mnt/user/app/git/yaris-obd && git pull --ff-only && cd server && docker compose up -d --build`.
|
||||
Container names: `yaris-obd-db` (Postgres 17), `yaris-obd-api`.
|
||||
- **Pi deploy loop**: `ssh yaris@<pi-ip>` (alias `yarisobd`) ->
|
||||
`cd ~/app/yaris-obd && git pull --ff-only` -> `sudo systemctl restart yaris-obd.service`
|
||||
(passwordless sudo confirmed via `yaris ALL=(ALL) NOPASSWD: ALL`). App stdout/stderr goes to
|
||||
`~/app/yaris-obd/yaris-obd.log` (not journald - the systemd unit redirects both streams there),
|
||||
so use `tail`/`grep` on that file, not `journalctl`, to see application-level prints (upload
|
||||
progress, OBD/GNSS errors, etc.) - `journalctl -u yaris-obd.service` only shows service
|
||||
start/stop/crash lifecycle events.
|
||||
- **Pi memory budget**: ~900MB usable RAM, no swap. Any code path that might buffer a whole
|
||||
dashcam-sized file (hundreds of MB) in memory is unsafe on this hardware - stream, don't buffer
|
||||
(see Phase 10).
|
||||
- **Git push auth**: if `git push` fails with a credential/auth error, run `gh auth login`
|
||||
(device-code browser flow) then `gh auth setup-git` to wire git to use the `gh` CLI's stored
|
||||
credentials — this fixed push auth issues in this environment without needing SSH keys or PATs.
|
||||
|
||||
Reference in New Issue
Block a user