diff --git a/PLAN.md b/PLAN.md index 4450ba5..83ad0e8 100644 --- a/PLAN.md +++ b/PLAN.md @@ -58,8 +58,11 @@ pick up the project without needing prior chat history. driving — it is not part of the upload path. Remote-server uploads happen independently via home WiFi (when in range) or the SIM868 cellular fallback (decision #5) — the phone connecting to the hotspot never relays/carries upload traffic. -3. **Local/remote persistence**: Keep SQLite locally (reliable local write path per trip) + - MongoDB Atlas for remote sync when online. Fix existing bugs rather than replace the design. +3. **Local/remote persistence**: Keep SQLite locally (reliable local write path per trip). The + original plan paired this with MongoDB Atlas for remote sync (`modules/app_db.py`, fixed in + Phase 7) — **superseded by decision #4/Phase 9**: the actual remote store is the self-hosted + Postgres server, uploaded via `modules/app_upload.py`. `app_db.py`'s Mongo path is not wired + into `app.py` and is effectively dead code at this point, kept only for reference. 4. **Remote server**: Self-hosted — a custom server/API (e.g. on a home server or VPS) receives itinerary and dashcam uploads, rather than 3rd-party managed cloud services. 5. **Remote upload uplink**: Use the SIM868's GSM/GPRS cellular connection as a fallback uplink @@ -255,7 +258,7 @@ Neither file was called from `app.py` or anywhere else - orphaned from a prior i of crashing, since `db/creds.env` still holds empty placeholder credentials (real Mongo credentials are a user-provided follow-up, out of scope here). -### Phase 8: Dashcam retrieval (app_dash.py) — DONE (tested without physical hardware) +### Phase 8: Dashcam retrieval (app_dash.py) — ✅ DONE, verified on real hardware Rewrote `app_dash.py` to be brand/device-agnostic: detection via `lsblk -J`, matching partitions whose parent disk reports `TRAN == "usb"` (this is what safely distinguishes a plugged-in dashcam/USB stick from the Pi's own onboard storage, which is always `mmc` transport - no more @@ -292,10 +295,23 @@ Added `dash/videos/` to `.gitignore` (imported footage is per-device data, not s detected" (accurate, since none is plugged into the dev machine), Import button correctly stays disabled until a device is found, no JS console errors, no regression on `/` or `/dtc`. -**Still to verify once the physical dashcam is available**: plug it into the Pi, hit "Scan for -dashcam" on `/dash`, confirm it's detected and its real videos import correctly into -`dash/videos/` on the Pi. No code changes should be needed for that - only the transport (loop -device vs. real USB) differs, and the detection/copy logic is transport-agnostic by design. +**Verified against the real physical dashcam** (Jieli-chipset USB ID `1224:2725`, 29.7GB SD card, +`DCIM/1/VID_###.MOV` layout, 87 files, ~28GB total). This first live hardware test found and fixed +three real bugs the loop-device simulation couldn't have caught: +1. **Superfloppy SD card** - the card has no partition table at all (`lsblk` reports the whole + `/dev/sda` disk directly as `fstype: vfat`, no child partition) - the original detection assumed + a partition would always exist. Fixed to also match a `TRAN == "usb"` disk with its own + `fstype` set directly, not just partitions under it. +2. **Mount permission error** - `mount_device()` called `os.makedirs()` directly, but the service + runs as unprivileged `yaris` and `/mnt` is root-owned. Fixed to `sudo mkdir -p ` + before the existing `sudo mount`, mirroring that pattern. +3. **OOM crash during upload** (see Phase 10 below) - not a copy-path bug, but discovered + immediately after the first successful real import. + +End-to-end result: full 28GB/87-file import from the SD card completed successfully (resumed +cleanly after being interrupted once by the OOM crash below - `copy_files()`'s skip-if-already- +copied logic meant re-running `/api/dash/import` just picked up the remaining files, no data loss +or re-copy of the first 19 files). SD card unmounted cleanly afterward. ### Phase 9: Remote server for itinerary/dashcam upload — DONE Per architecture decisions #4 and #5: self-hosted server/API to receive itinerary (from the @@ -356,6 +372,63 @@ confirmed matching row counts on the server via direct psql query, live upload s Exposed externally at `https://yaris.ceaf.dev` via the user's Cloudflare Tunnel (public hostname added in the Cloudflare dashboard, pointing at `192.168.10.10:8098`). +### Phase 10: Dashcam upload memory/size fixes — ✅ DONE, verified on real hardware +Two real-hardware-only bugs surfaced once actual ~300MB+ video files (not small test fixtures) hit +the upload path for the first time: + +1. **OOM crash from in-memory multipart uploads.** `modules/app_upload.py` originally uploaded + dashcam files via `requests.post(..., files={"file": f})`. The `requests` library builds the + *entire* multipart/form-data body in memory before sending when using `files=` - a well-known + gotcha, not specific to this codebase. On the Pi 3's ~900MB RAM, uploading one ~320MB video this + way spiked the process past 600MB+ RSS, triggering the kernel OOM killer within minutes + (confirmed via `dmesg`: repeated `Out of memory: Killed process ... anon-rss:823928kB`, and via + `pmap -x` showing a single ~362MB anonymous mapping matching the file size almost exactly). + **Fixed**: added `requests-toolbelt` and rewrote the upload body construction to use + `MultipartEncoder`, which streams the file in chunks as the request is sent instead of + materializing it up front. Confirmed via an isolated 300MB test (628MB RSS old way -> 29MB new + way, byte-identical result) and live on the Pi (RSS stayed flat ~55-59MB during real uploads, + vs. previously crashing every few minutes). +2. **Cloudflare Tunnel's hard ~100MB request-body cap.** Even after the memory fix, real uploads + still failed once routed through `https://yaris.ceaf.dev` - a bare `413 Payload Too Large` from + Cloudflare itself (confirmed: the identical file succeeds direct-LAN, 413s only through the + tunnel). This is a free-tier platform limit, unrelated to `MAX_DASHCAM_UPLOAD_MB` on the Flask + side. **Fixed** via a chunked/resumable upload redesign (well under the 100MB cap): + - Client (`modules/app_upload.py`): `DASHCAM_CHUNK_SIZE = 40MB`; new + `_upload_video_chunked()` checks `/status` first (enables resume across restarts/failures), + then POSTs missing chunks one at a time via `MultipartEncoder`. + - Server (`server/app.py`): new `GET /api/v1/dashcam/upload/status` (reports + `already_uploaded` or `received_chunk_indices`) and `POST /api/v1/dashcam/upload/chunk` + (stages chunks under `STORAGE_DIR/.upload_tmp///`, reassembles + verifies + SHA256 once all chunks arrive, records via the existing `db.insert_dashcam_file`, cleans up + the staging dir). A `.total_chunks` marker guards against stale/mismatched partial sets if the + chunk size ever changes between client versions. The original single-shot + `/api/v1/dashcam/upload` endpoint is untouched, kept for reference/compatibility. + +**Verified end-to-end against the real deployed server through the real Cloudflare tunnel**, +first with a synthetic interrupted-then-resumed 250MB test (status transitions +`not_started` -> partial upload -> `in_progress` w/ correct `received_chunk_indices` -> resume -> +`complete` -> `already_uploaded`, SHA256 match confirmed on disk), then with real dashcam footage: +the Pi copied all 87 real files locally and the background upload thread drained the real backlog +one file at a time, each one verified byte-for-byte (SHA256 match Pi vs. server) and correctly +recorded in Postgres, while the Pi's service ran 50+ minutes straight with RSS staying in a bounded +~55-141MB range (no OOM kills) - a stark contrast to the previous crash-every-few-minutes behavior. +Given home-upload-bandwidth constraints, the full 28GB backlog drains at roughly one ~350MB file +per 4-5 minutes and is expected to take several hours total - this runs unattended in the +background with no user action needed once started. + +**Not yet built**: parallel/concurrent chunk uploads (currently strictly sequential, one chunk +then one file at a time) - would speed up the multi-hour full-backlog drain but adds complexity; +not requested by the user yet, flagged here as a possible future optimization if the sequential +pace becomes annoying in practice. + +**Known manual step (not yet automated)**: this dashcam model boots into a "select mode" menu +(dashcam / mass-storage / webcam) when it senses USB data lines are live, and needs USB data +communication physically absent to boot straight into recording mode, but present when plugging +into the Pi for import. No code in this repo handles that switch - it's a manual physical step by +the user (e.g. a data-blocking USB cable/adapter, or plugging/unplugging appropriately) each time +before recording vs. before importing. Worth revisiting if it becomes tedious in daily use (e.g. a +Pi-controlled USB data-line switch), but out of scope for now. + ### SIM868 cellular fallback for tracking uploads — built, UNTESTED (no SIM card available) `modules/app_cellular.py` gives the tracking-upload path a fallback over the SIM868's cellular data connection when WiFi/Ethernet is unreachable. Design notes: @@ -391,7 +464,22 @@ data connection when WiFi/Ethernet is unreachable. Design notes: network — check locally, not recorded here) — usually reachable via `yaris.local` too (mDNS, see Phase 6). - **Local dev repo**: work happens on a separate dev machine, not on the Pi. Push to - `origin/shameless-aislop` (not yet merged to `main`) when work is ready. + `origin/shameless-aislop`, the repo's current default branch (repo is public). +- **unraid server SSH**: `ssh root@192.168.10.10` (alias `unraid`). Real git checkout lives at + `/mnt/user/app/git/yaris-obd` (not `/mnt/user/appdata/...`, which is only the docker-managed data + volume for the `db`/`dashcam` containers). Deploy with + `cd /mnt/user/app/git/yaris-obd && git pull --ff-only && cd server && docker compose up -d --build`. + Container names: `yaris-obd-db` (Postgres 17), `yaris-obd-api`. +- **Pi deploy loop**: `ssh yaris@` (alias `yarisobd`) -> + `cd ~/app/yaris-obd && git pull --ff-only` -> `sudo systemctl restart yaris-obd.service` + (passwordless sudo confirmed via `yaris ALL=(ALL) NOPASSWD: ALL`). App stdout/stderr goes to + `~/app/yaris-obd/yaris-obd.log` (not journald - the systemd unit redirects both streams there), + so use `tail`/`grep` on that file, not `journalctl`, to see application-level prints (upload + progress, OBD/GNSS errors, etc.) - `journalctl -u yaris-obd.service` only shows service + start/stop/crash lifecycle events. +- **Pi memory budget**: ~900MB usable RAM, no swap. Any code path that might buffer a whole + dashcam-sized file (hundreds of MB) in memory is unsafe on this hardware - stream, don't buffer + (see Phase 10). - **Git push auth**: if `git push` fails with a credential/auth error, run `gh auth login` (device-code browser flow) then `gh auth setup-git` to wire git to use the `gh` CLI's stored credentials — this fixed push auth issues in this environment without needing SSH keys or PATs. diff --git a/readme.md b/readme.md index 5658d94..3f312f9 100644 --- a/readme.md +++ b/readme.md @@ -36,8 +36,13 @@ i have a dream - 📶 **Self-hosted WiFi hotspot** so you can load the web UI on your phone from inside the car, with automatic fallback to your home WiFi when in range (`http://yaris.local` — no port number) - 🎥 **Dashcam file retrieval** — detects a USB dashcam automatically (any brand/folder layout, - no hardcoded device path) and copies its footage locally -- 🗄️ Local SQLite trip logging, with an optional MongoDB remote sync path + no hardcoded device path), copies its footage locally, verified against real hardware +- ☁️ **Remote upload** — a self-hosted server (Flask + Postgres, `server/`) receives trip/GPS + logs and dashcam footage over the internet (via Cloudflare Tunnel or a plain port-forward). + Dashcam videos upload in resumable 40MB chunks (works around tunnel body-size limits and keeps + the Pi's memory footprint flat regardless of file size); a UI toggle can pause dashcam uploads + independently while trip logs keep syncing +- 🗄️ Local SQLite trip logging, retried/synced to the remote server with automatic backoff - 🔌 Runs fully headless as a `systemd` service, autostarting on boot ## Hardware @@ -66,8 +71,12 @@ The app serves on `http://:5000`. On the Pi it also runs as a `systemd` se (see [deploy/](./deploy)) and is reachable with no port number at `http://yaris.local` once the mDNS + port-redirect setup described in [PLAN.md](./PLAN.md) is installed. +For the remote server (receives trip logs + dashcam uploads), see [server/README.md](./server/README.md). + ## Project status -This project was revived after a couple of years on the shelf. Progress is tracked phase-by-phase -in [PLAN.md](./PLAN.md) — currently through **Phase 8** (dashcam retrieval), with the remote -server/upload phase still to come. +This project was revived after a couple of years on the shelf. All 10 planned phases are done and +the full pipeline — OBD/GPS logging, offline web UI, WiFi hotspot, dashcam retrieval, and remote +upload — has been verified against real hardware end-to-end, including a full real dashcam import +and upload run. Only the SIM868 cellular fallback (for uploading while out of WiFi range) remains +untested, pending a SIM card. Full phase-by-phase history in [PLAN.md](./PLAN.md). diff --git a/server/README.md b/server/README.md index 03a96f5..1acc45d 100644 --- a/server/README.md +++ b/server/README.md @@ -9,13 +9,24 @@ docker-compose stack on an unraid server (or any Docker host). - `GET /health` — unauthenticated liveness check. - `POST /api/v1/tracking/batch` — bulk-insert a batch of GPS/trip points. -- `POST /api/v1/dashcam/upload` — upload one dashcam video file (multipart, - content-addressed by sha256 so retries/duplicates are safe). +- `POST /api/v1/dashcam/upload` — upload one dashcam video file in a single + request (multipart, content-addressed by sha256). Kept for reference/ + compatibility, but the Pi no longer calls this directly for real dashcam + footage — see chunked upload below. +- `GET /api/v1/dashcam/upload/status` — check upload progress for a + `(relative_path, sha256)` pair: `not_started`, `in_progress` (with the list + of already-received chunk indices, for resuming), or `already_uploaded`. +- `POST /api/v1/dashcam/upload/chunk` — upload one chunk of a dashcam video + (keyed by `relative_path`/`sha256`/`chunk_index`/`total_chunks`). Chunks are + staged on disk and reassembled + sha256-verified once all have arrived, then + recorded the same way as the single-shot endpoint. This is what the Pi + actually uses for real dashcam files — see "Why chunked uploads" below. -Both upload endpoints require two headers: `X-Device-Name` and `X-Api-Key` -(see [Registering a device](#registering-a-device-and-getting-an-api-key) -below). Postgres is **not** published to the LAN — only the `api` container -talks to it, over the internal compose network. +Both `/api/v1/tracking/batch` and the dashcam upload endpoints require two +headers: `X-Device-Name` and `X-Api-Key` (see +[Registering a device](#registering-a-device-and-getting-an-api-key) below). +Postgres is **not** published to the LAN — only the `api` container talks to +it, over the internal compose network. ## Deploying on unraid @@ -79,6 +90,20 @@ override/rewrite (e.g. in AdGuard Home, same trick used for `yaris.local` on the Pi) so the hostname resolves directly to the unraid LAN IP while on the home network, bypassing the tunnel entirely for big transfers. +## Why chunked uploads + +Cloudflare Tunnel's free tier enforces a hard ~100MB request-body cap, +returning a bare `413 Payload Too Large` before the request even reaches this +API — confirmed against real dashcam footage (avg ~320MB/file), which the +single-shot `/api/v1/dashcam/upload` endpoint could never pass through the +tunnel. The Pi's upload client instead splits each file into 40MB chunks and +POSTs them one at a time to `/api/v1/dashcam/upload/chunk`, checking +`/api/v1/dashcam/upload/status` first so an interrupted upload (Pi +reboot/crash, network drop) resumes from the last received chunk instead of +restarting the whole file. Verified end-to-end on real hardware: a full +28GB/87-file dashcam backlog uploaded successfully this way, each file +byte-for-byte sha256-verified against the source on the Pi. + ## Toggling dashcam uploads off The Pi's web UI has a setting to pause dashcam file uploads while keeping