Document Phase 8/10 real-hardware validation in PLAN.md and refresh READMEs

- PLAN.md: mark Phase 8 (dashcam retrieval) verified on real hardware, add
  Phase 10 documenting the OOM/multipart and Cloudflare body-size fixes plus
  the chunked/resumable upload redesign, both verified end-to-end against
  real dashcam footage. Update architecture decision #3 (Mongo -> Postgres
  supersession), add unraid/Pi deploy notes and memory-budget gotcha.
- readme.md: refresh feature list (remote upload, chunked dashcam uploads,
  toggle) and project status (all 10 phases done, only cellular fallback
  untested).
- server/README.md: document the new chunked upload endpoints and the
  Cloudflare body-size-limit rationale behind them.

PLAN.md

@@ -58,8 +58,11 @@ pick up the project without needing prior chat history.
    driving — it is not part of the upload path. Remote-server uploads happen independently via
    home WiFi (when in range) or the SIM868 cellular fallback (decision #5) — the phone connecting
    to the hotspot never relays/carries upload traffic.
-3. **Local/remote persistence**: Keep SQLite locally (reliable local write path per trip) +
+3. **Local/remote persistence**: Keep SQLite locally (reliable local write path per trip). The
-   MongoDB Atlas for remote sync when online. Fix existing bugs rather than replace the design.
+   original plan paired this with MongoDB Atlas for remote sync (`modules/app_db.py`, fixed in
+   Phase 7) — **superseded by decision #4/Phase 9**: the actual remote store is the self-hosted
+   Postgres server, uploaded via `modules/app_upload.py`. `app_db.py`'s Mongo path is not wired
+   into `app.py` and is effectively dead code at this point, kept only for reference.
 4. **Remote server**: Self-hosted — a custom server/API (e.g. on a home server or VPS) receives
    itinerary and dashcam uploads, rather than 3rd-party managed cloud services.
 5. **Remote upload uplink**: Use the SIM868's GSM/GPRS cellular connection as a fallback uplink
@@ -255,7 +258,7 @@ Neither file was called from `app.py` or anywhere else - orphaned from a prior i
   of crashing, since `db/creds.env` still holds empty placeholder credentials (real Mongo
   credentials are a user-provided follow-up, out of scope here).
 
-### Phase 8: Dashcam retrieval (app_dash.py) — DONE (tested without physical hardware)
+### Phase 8: Dashcam retrieval (app_dash.py) — ✅ DONE, verified on real hardware
 Rewrote `app_dash.py` to be brand/device-agnostic: detection via `lsblk -J`, matching partitions
 whose parent disk reports `TRAN == "usb"` (this is what safely distinguishes a plugged-in
 dashcam/USB stick from the Pi's own onboard storage, which is always `mmc` transport - no more
@@ -292,10 +295,23 @@ Added `dash/videos/` to `.gitignore` (imported footage is per-device data, not s
    detected" (accurate, since none is plugged into the dev machine), Import button correctly
    stays disabled until a device is found, no JS console errors, no regression on `/` or `/dtc`.
 
-**Still to verify once the physical dashcam is available**: plug it into the Pi, hit "Scan for
+**Verified against the real physical dashcam** (Jieli-chipset USB ID `1224:2725`, 29.7GB SD card,
-dashcam" on `/dash`, confirm it's detected and its real videos import correctly into
+`DCIM/1/VID_###.MOV` layout, 87 files, ~28GB total). This first live hardware test found and fixed
-`dash/videos/` on the Pi. No code changes should be needed for that - only the transport (loop
+three real bugs the loop-device simulation couldn't have caught:
-device vs. real USB) differs, and the detection/copy logic is transport-agnostic by design.
+1. **Superfloppy SD card** - the card has no partition table at all (`lsblk` reports the whole
+   `/dev/sda` disk directly as `fstype: vfat`, no child partition) - the original detection assumed
+   a partition would always exist. Fixed to also match a `TRAN == "usb"` disk with its own
+   `fstype` set directly, not just partitions under it.
+2. **Mount permission error** - `mount_device()` called `os.makedirs()` directly, but the service
+   runs as unprivileged `yaris` and `/mnt` is root-owned. Fixed to `sudo mkdir -p <mount_point>`
+   before the existing `sudo mount`, mirroring that pattern.
+3. **OOM crash during upload** (see Phase 10 below) - not a copy-path bug, but discovered
+   immediately after the first successful real import.
+
+End-to-end result: full 28GB/87-file import from the SD card completed successfully (resumed
+cleanly after being interrupted once by the OOM crash below - `copy_files()`'s skip-if-already-
+copied logic meant re-running `/api/dash/import` just picked up the remaining files, no data loss
+or re-copy of the first 19 files). SD card unmounted cleanly afterward.
 
 ### Phase 9: Remote server for itinerary/dashcam upload — DONE
 Per architecture decisions #4 and #5: self-hosted server/API to receive itinerary (from the
@@ -356,6 +372,63 @@ confirmed matching row counts on the server via direct psql query, live upload s
 Exposed externally at `https://yaris.ceaf.dev` via the user's Cloudflare Tunnel (public hostname
 added in the Cloudflare dashboard, pointing at `192.168.10.10:8098`).
 
+### Phase 10: Dashcam upload memory/size fixes — ✅ DONE, verified on real hardware
+Two real-hardware-only bugs surfaced once actual ~300MB+ video files (not small test fixtures) hit
+the upload path for the first time:
+
+1. **OOM crash from in-memory multipart uploads.** `modules/app_upload.py` originally uploaded
+   dashcam files via `requests.post(..., files={"file": f})`. The `requests` library builds the
+   *entire* multipart/form-data body in memory before sending when using `files=` - a well-known
+   gotcha, not specific to this codebase. On the Pi 3's ~900MB RAM, uploading one ~320MB video this
+   way spiked the process past 600MB+ RSS, triggering the kernel OOM killer within minutes
+   (confirmed via `dmesg`: repeated `Out of memory: Killed process ... anon-rss:823928kB`, and via
+   `pmap -x` showing a single ~362MB anonymous mapping matching the file size almost exactly).
+   **Fixed**: added `requests-toolbelt` and rewrote the upload body construction to use
+   `MultipartEncoder`, which streams the file in chunks as the request is sent instead of
+   materializing it up front. Confirmed via an isolated 300MB test (628MB RSS old way -> 29MB new
+   way, byte-identical result) and live on the Pi (RSS stayed flat ~55-59MB during real uploads,
+   vs. previously crashing every few minutes).
+2. **Cloudflare Tunnel's hard ~100MB request-body cap.** Even after the memory fix, real uploads
+   still failed once routed through `https://yaris.ceaf.dev` - a bare `413 Payload Too Large` from
+   Cloudflare itself (confirmed: the identical file succeeds direct-LAN, 413s only through the
+   tunnel). This is a free-tier platform limit, unrelated to `MAX_DASHCAM_UPLOAD_MB` on the Flask
+   side. **Fixed** via a chunked/resumable upload redesign (well under the 100MB cap):
+   - Client (`modules/app_upload.py`): `DASHCAM_CHUNK_SIZE = 40MB`; new
+     `_upload_video_chunked()` checks `/status` first (enables resume across restarts/failures),
+     then POSTs missing chunks one at a time via `MultipartEncoder`.
+   - Server (`server/app.py`): new `GET /api/v1/dashcam/upload/status` (reports
+     `already_uploaded` or `received_chunk_indices`) and `POST /api/v1/dashcam/upload/chunk`
+     (stages chunks under `STORAGE_DIR/.upload_tmp/<device>/<sha256>/`, reassembles + verifies
+     SHA256 once all chunks arrive, records via the existing `db.insert_dashcam_file`, cleans up
+     the staging dir). A `.total_chunks` marker guards against stale/mismatched partial sets if the
+     chunk size ever changes between client versions. The original single-shot
+     `/api/v1/dashcam/upload` endpoint is untouched, kept for reference/compatibility.
+
+**Verified end-to-end against the real deployed server through the real Cloudflare tunnel**,
+first with a synthetic interrupted-then-resumed 250MB test (status transitions
+`not_started` -> partial upload -> `in_progress` w/ correct `received_chunk_indices` -> resume ->
+`complete` -> `already_uploaded`, SHA256 match confirmed on disk), then with real dashcam footage:
+the Pi copied all 87 real files locally and the background upload thread drained the real backlog
+one file at a time, each one verified byte-for-byte (SHA256 match Pi vs. server) and correctly
+recorded in Postgres, while the Pi's service ran 50+ minutes straight with RSS staying in a bounded
+~55-141MB range (no OOM kills) - a stark contrast to the previous crash-every-few-minutes behavior.
+Given home-upload-bandwidth constraints, the full 28GB backlog drains at roughly one ~350MB file
+per 4-5 minutes and is expected to take several hours total - this runs unattended in the
+background with no user action needed once started.
+
+**Not yet built**: parallel/concurrent chunk uploads (currently strictly sequential, one chunk
+then one file at a time) - would speed up the multi-hour full-backlog drain but adds complexity;
+not requested by the user yet, flagged here as a possible future optimization if the sequential
+pace becomes annoying in practice.
+
+**Known manual step (not yet automated)**: this dashcam model boots into a "select mode" menu
+(dashcam / mass-storage / webcam) when it senses USB data lines are live, and needs USB data
+communication physically absent to boot straight into recording mode, but present when plugging
+into the Pi for import. No code in this repo handles that switch - it's a manual physical step by
+the user (e.g. a data-blocking USB cable/adapter, or plugging/unplugging appropriately) each time
+before recording vs. before importing. Worth revisiting if it becomes tedious in daily use (e.g. a
+Pi-controlled USB data-line switch), but out of scope for now.
+
 ### SIM868 cellular fallback for tracking uploads — built, UNTESTED (no SIM card available)
 `modules/app_cellular.py` gives the tracking-upload path a fallback over the SIM868's cellular
 data connection when WiFi/Ethernet is unreachable. Design notes:
@@ -391,7 +464,22 @@ data connection when WiFi/Ethernet is unreachable. Design notes:
   network — check locally, not recorded here) — usually reachable via `yaris.local` too (mDNS,
   see Phase 6).
 - **Local dev repo**: work happens on a separate dev machine, not on the Pi. Push to
-  `origin/shameless-aislop` (not yet merged to `main`) when work is ready.
+  `origin/shameless-aislop`, the repo's current default branch (repo is public).
+- **unraid server SSH**: `ssh [email protected]` (alias `unraid`). Real git checkout lives at
+  `/mnt/user/app/git/yaris-obd` (not `/mnt/user/appdata/...`, which is only the docker-managed data
+  volume for the `db`/`dashcam` containers). Deploy with
+  `cd /mnt/user/app/git/yaris-obd && git pull --ff-only && cd server && docker compose up -d --build`.
+  Container names: `yaris-obd-db` (Postgres 17), `yaris-obd-api`.
+- **Pi deploy loop**: `ssh yaris@<pi-ip>` (alias `yarisobd`) ->
+  `cd ~/app/yaris-obd && git pull --ff-only` -> `sudo systemctl restart yaris-obd.service`
+  (passwordless sudo confirmed via `yaris ALL=(ALL) NOPASSWD: ALL`). App stdout/stderr goes to
+  `~/app/yaris-obd/yaris-obd.log` (not journald - the systemd unit redirects both streams there),
+  so use `tail`/`grep` on that file, not `journalctl`, to see application-level prints (upload
+  progress, OBD/GNSS errors, etc.) - `journalctl -u yaris-obd.service` only shows service
+  start/stop/crash lifecycle events.
+- **Pi memory budget**: ~900MB usable RAM, no swap. Any code path that might buffer a whole
+  dashcam-sized file (hundreds of MB) in memory is unsafe on this hardware - stream, don't buffer
+  (see Phase 10).
 - **Git push auth**: if `git push` fails with a credential/auth error, run `gh auth login`
   (device-code browser flow) then `gh auth setup-git` to wire git to use the `gh` CLI's stored
   credentials — this fixed push auth issues in this environment without needing SSH keys or PATs.

readme.md

@@ -36,8 +36,13 @@ i have a dream
 - 📶 **Self-hosted WiFi hotspot** so you can load the web UI on your phone from inside the car,
   with automatic fallback to your home WiFi when in range (`http://yaris.local` — no port number)
 - 🎥 **Dashcam file retrieval** — detects a USB dashcam automatically (any brand/folder layout,
-  no hardcoded device path) and copies its footage locally
+  no hardcoded device path), copies its footage locally, verified against real hardware
-- 🗄️ Local SQLite trip logging, with an optional MongoDB remote sync path
+- ☁️ **Remote upload** — a self-hosted server (Flask + Postgres, `server/`) receives trip/GPS
+  logs and dashcam footage over the internet (via Cloudflare Tunnel or a plain port-forward).
+  Dashcam videos upload in resumable 40MB chunks (works around tunnel body-size limits and keeps
+  the Pi's memory footprint flat regardless of file size); a UI toggle can pause dashcam uploads
+  independently while trip logs keep syncing
+- 🗄️ Local SQLite trip logging, retried/synced to the remote server with automatic backoff
 - 🔌 Runs fully headless as a `systemd` service, autostarting on boot
 
 ## Hardware
@@ -66,8 +71,12 @@ The app serves on `http://<host>:5000`. On the Pi it also runs as a `systemd` se
 (see [deploy/](./deploy)) and is reachable with no port number at `http://yaris.local` once the
 mDNS + port-redirect setup described in [PLAN.md](./PLAN.md) is installed.
 
+For the remote server (receives trip logs + dashcam uploads), see [server/README.md](./server/README.md).
+
 ## Project status
 
-This project was revived after a couple of years on the shelf. Progress is tracked phase-by-phase
+This project was revived after a couple of years on the shelf. All 10 planned phases are done and
-in [PLAN.md](./PLAN.md) — currently through **Phase 8** (dashcam retrieval), with the remote
+the full pipeline — OBD/GPS logging, offline web UI, WiFi hotspot, dashcam retrieval, and remote
-server/upload phase still to come.
+upload — has been verified against real hardware end-to-end, including a full real dashcam import
+and upload run. Only the SIM868 cellular fallback (for uploading while out of WiFi range) remains
+untested, pending a SIM card. Full phase-by-phase history in [PLAN.md](./PLAN.md).

server/README.md

@@ -9,13 +9,24 @@ docker-compose stack on an unraid server (or any Docker host).
 
 - `GET /health` — unauthenticated liveness check.
 - `POST /api/v1/tracking/batch` — bulk-insert a batch of GPS/trip points.
-- `POST /api/v1/dashcam/upload` — upload one dashcam video file (multipart,
+- `POST /api/v1/dashcam/upload` — upload one dashcam video file in a single
-  content-addressed by sha256 so retries/duplicates are safe).
+  request (multipart, content-addressed by sha256). Kept for reference/
+  compatibility, but the Pi no longer calls this directly for real dashcam
+  footage — see chunked upload below.
+- `GET /api/v1/dashcam/upload/status` — check upload progress for a
+  `(relative_path, sha256)` pair: `not_started`, `in_progress` (with the list
+  of already-received chunk indices, for resuming), or `already_uploaded`.
+- `POST /api/v1/dashcam/upload/chunk` — upload one chunk of a dashcam video
+  (keyed by `relative_path`/`sha256`/`chunk_index`/`total_chunks`). Chunks are
+  staged on disk and reassembled + sha256-verified once all have arrived, then
+  recorded the same way as the single-shot endpoint. This is what the Pi
+  actually uses for real dashcam files — see "Why chunked uploads" below.
 
-Both upload endpoints require two headers: `X-Device-Name` and `X-Api-Key`
+Both `/api/v1/tracking/batch` and the dashcam upload endpoints require two
-(see [Registering a device](#registering-a-device-and-getting-an-api-key)
+headers: `X-Device-Name` and `X-Api-Key` (see
-below). Postgres is **not** published to the LAN — only the `api` container
+[Registering a device](#registering-a-device-and-getting-an-api-key) below).
-talks to it, over the internal compose network.
+Postgres is **not** published to the LAN — only the `api` container talks to
+it, over the internal compose network.
 
 ## Deploying on unraid
 
@@ -79,6 +90,20 @@ override/rewrite (e.g. in AdGuard Home, same trick used for `yaris.local` on
 the Pi) so the hostname resolves directly to the unraid LAN IP while on the
 home network, bypassing the tunnel entirely for big transfers.
 
+## Why chunked uploads
+
+Cloudflare Tunnel's free tier enforces a hard ~100MB request-body cap,
+returning a bare `413 Payload Too Large` before the request even reaches this
+API — confirmed against real dashcam footage (avg ~320MB/file), which the
+single-shot `/api/v1/dashcam/upload` endpoint could never pass through the
+tunnel. The Pi's upload client instead splits each file into 40MB chunks and
+POSTs them one at a time to `/api/v1/dashcam/upload/chunk`, checking
+`/api/v1/dashcam/upload/status` first so an interrupted upload (Pi
+reboot/crash, network drop) resumes from the last received chunk instead of
+restarting the whole file. Verified end-to-end on real hardware: a full
+28GB/87-file dashcam backlog uploaded successfully this way, each file
+byte-for-byte sha256-verified against the source on the Pi.
+
 ## Toggling dashcam uploads off
 
 The Pi's web UI has a setting to pause dashcam file uploads while keeping