Add phone-relay dashcam upload to bypass the Pi's slow WiFi

Root-caused the slow-upload complaint to the Pi's own congested 2.4GHz
WiFi radio (confirmed via iperf3 - same degraded throughput even on
pure local LAN traffic, ruling out Cloudflare/ISP). Since the Pi 3 B
has no 5GHz radio, add a way to bypass it for the internet-bound leg:
a phone opens /dash and manually relays pending dashcam files straight
to the remote server over its own network, reusing the exact same
chunked upload protocol the Pi's client already uses.

- server/app.py: add CORS headers (after_request) so the phone's
  browser can call the remote server cross-origin from a page served
  by the Pi.
- app.py: new endpoints - GET /api/dash/pending (list not-yet-uploaded
  files), GET /api/dash/file/<path> (Range-request-capable download,
  path-traversal safe), GET /api/dash/relay/config (hands the phone
  the device credentials), POST /api/dash/relay/heartbeat|stop (lease-
  based pause of the Pi's own background uploader, self-heals if the
  phone's tab closes uncleanly), POST /api/dash/relay/mark_uploaded.
- modules/app_upload.py: list_pending_dashcam_files(), relay lease
  helpers, _upload_dashcam_files() now pauses while a relay lease is
  active.
- modules/app_sql.py: new dashcam_hash_cache table so a file's sha256
  is only computed once (keyed by path+size+mtime) instead of being
  recomputed on every loop iteration/pending-list call - benefits the
  existing background uploader too.
- templates/dash.html / static/js/dash.js: new "Relay via phone" card
  - per-file and "relay all" buttons (manual trigger only, per user
  preference), downloads+uploads one 40MB slice at a time so the
  phone never holds a whole file in memory, retries chunks up to 3x.
- PLAN.md/readme.md/server/README.md updated (Phase 11).

Tested via a full local end-to-end simulation driving the real Pi and
server Flask apps side by side (Postgres mocked only at the db module
boundary - all HTTP/chunking/auth/CORS logic is real): verified Range
downloads (206), byte-for-byte reassembly via SHA256, path-traversal
rejection, and the relay lease correctly pausing/resuming the
background uploader. Not yet tested against a real phone/live network.
This commit is contained in:
Mathieu Moran
2026-08-09 00:15:27 -04:00
parent 8c33c669a5
commit 0d67a21ae1
9 changed files with 549 additions and 13 deletions
+72
View File
@@ -458,6 +458,78 @@ data connection when WiFi/Ethernet is unreachable. Design notes:
a plain-HTTP router port-forward path, avoiding the need for the modem's TLS stack to validate
a cert chain).
### Phase 11: Phone-relay dashcam upload — ✅ DONE
Root cause investigation into "why is the transfer from the Pi to the server so slow?" (real
dashcam backlog draining at only ~350MB/4-5min) found the bottleneck is the **Pi's own onboard
BCM43438 WiFi chip** (2.4GHz-only, congested/interfered channel) — confirmed via `iperf3`
raw-TCP tests showing only ~14.7Mbit/s with active retransmits even over a same-subnet direct-LAN
hop that never touches the internet/Cloudflare, ruling out ISP/Cloudflare limits. Since the Pi 3 B
has no 5GHz radio and can't be fixed in software, added a way to bypass its weak WiFi for the
internet-bound leg entirely: a phone opens the same `/dash` page (over the Pi's hotspot/home WiFi)
and manually relays pending dashcam files straight to the remote server using the phone's own
network (WiFi or cellular data), instead of routing everything through the Pi.
**Zero new remote-server upload logic** — the phone's browser JS calls the *exact same*
`/api/v1/dashcam/upload/status` + `/api/v1/dashcam/upload/chunk` endpoints the Pi's own Python
client already uses, in the same 40MB-chunk protocol, just orchestrated by `fetch()`/`FormData`
instead of `requests`/`MultipartEncoder`. The server dedupes purely by SHA256 regardless of which
device sends the bytes, so no server-side changes were needed for the upload path itself — only a
CORS `after_request` hook (`server/app.py`) was added, since the phone's browser calls
`yaris.ceaf.dev` from a page served by a *different* origin (the Pi), which browsers block without
`Access-Control-Allow-*` headers on both the real request and its preflight `OPTIONS`. Wildcard
origin is fine here — auth is header-based (`X-Device-Name`/`X-Api-Key`), not cookie/session-based.
**New Pi-side endpoints** (`app.py`):
- `GET /api/dash/pending` — lists not-yet-uploaded dashcam files (`relative_path`, `sha256`,
`size`) via a new `app_upload.list_pending_dashcam_files()`.
- `GET /api/dash/file/<path:relpath>` — serves a dashcam file's bytes for the phone to download,
via `send_file(..., conditional=True)` for **Range-request support** (206 Partial Content) so
the phone downloads (and immediately re-uploads) one 40MB slice at a time instead of ever
holding a whole 300+MB file in the browser's memory — the exact same memory-safety lesson
learned the hard way in Phase 10, applied symmetrically to the download side. Path-traversal
safe via Werkzeug's `safe_join` against `app_dash.DEFAULT_DESTINATION`.
- `GET /api/dash/relay/config` — hands the phone the same `SERVER_URL`/`DEVICE_NAME`/`API_KEY`
the Pi's own client uses, so its JS can auth directly with the remote server. Same trust
boundary as the rest of this project's unauthenticated local UI (anyone on the Pi's own
hotspot/LAN is already implicitly trusted).
- `POST /api/dash/relay/heartbeat` / `POST /api/dash/relay/stop` — a **lease-based pause**
(`RELAY_LEASE_SECONDS = 45`) for the Pi's own background dashcam uploader, chosen over a plain
on/off toggle specifically to self-heal if the phone's browser tab closes/loses connection
uncleanly (a simple toggle could get stuck "paused" forever). The phone page heartbeats every
15s while relaying; `_upload_dashcam_files()` skips entirely while a lease is active, re-checked
between files so it stops promptly if a relay starts mid-drain. Also released via
`navigator.sendBeacon()` on `beforeunload` as a best-effort immediate release.
- `POST /api/dash/relay/mark_uploaded` — lets the phone tell the Pi "I just uploaded this file"
right after a successful relay, so the local dedupe table (and the pending list / Pi's own
background loop) reflect it immediately instead of waiting for the next full backlog scan.
**Performance side-fix**: `modules/app_upload.py`'s dashcam loop previously recomputed SHA256 for
*every* candidate file on *every* loop iteration, including files long since uploaded — wasteful
CPU/IO against a growing, possibly 20GB+, local backlog. Added a `dashcam_hash_cache` SQLite table
(`modules/app_sql.py`, keyed by `relative_path`+`size`+`mtime`) so a file's hash is only computed
once; both the existing background uploader and the new pending-files endpoint benefit.
UI (`templates/dash.html` / `static/js/dash.js`): a new "Relay via phone" card on `/dash` lists
pending files with a per-file "Relay" button and a "Relay all pending" button (manual trigger only
— chosen deliberately over an automatic one, per explicit user preference), plus a "Stop relay"
button. Chunk downloads/uploads retry up to 3 times with backoff before surfacing an error, and a
resumed relay re-checks `/status` first so a previously-interrupted file picks up from its first
missing chunk instead of restarting.
**Tested**: a full local end-to-end simulation (no physical phone/network needed) drove the real
Pi `app.py` and real `server/app.py` Flask apps side by side (Postgres mocked at the `db` module
boundary only — all HTTP routing, chunking, auth, and CORS logic is real), reproducing the exact
sequence the phone's browser JS performs: list pending -> check remote `/status` -> download each
chunk from the Pi via `Range` requests (verified real `206 Partial Content` responses) -> upload
each chunk to the mock-backed server -> verify the reassembled file matches the original
byte-for-byte via SHA256 -> mark uploaded on the Pi -> confirm it drops out of the pending list.
Also verified: path-traversal attempts on `/api/dash/file/` correctly 404, the relay
heartbeat/stop lease correctly pauses/resumes `_upload_dashcam_files()`, and the sha256 cache
table is populated and reused across repeated `/api/dash/pending` calls. **Not yet tested on a
real phone against the real deployed server** — the wire protocol and every server-visible
behavior is verified, but real mobile-browser quirks (background-tab throttling, flaky cellular
mid-transfer) haven't been exercised against live hardware yet.
## Working Notes / Environment Gotchas
- **Pi credentials/SSH**: `ssh yaris@<pi-ip>`. Confirm current IP before connecting (varies by