Add phone-relay dashcam upload to bypass the Pi's slow WiFi
Root-caused the slow-upload complaint to the Pi's own congested 2.4GHz WiFi radio (confirmed via iperf3 - same degraded throughput even on pure local LAN traffic, ruling out Cloudflare/ISP). Since the Pi 3 B has no 5GHz radio, add a way to bypass it for the internet-bound leg: a phone opens /dash and manually relays pending dashcam files straight to the remote server over its own network, reusing the exact same chunked upload protocol the Pi's client already uses. - server/app.py: add CORS headers (after_request) so the phone's browser can call the remote server cross-origin from a page served by the Pi. - app.py: new endpoints - GET /api/dash/pending (list not-yet-uploaded files), GET /api/dash/file/<path> (Range-request-capable download, path-traversal safe), GET /api/dash/relay/config (hands the phone the device credentials), POST /api/dash/relay/heartbeat|stop (lease- based pause of the Pi's own background uploader, self-heals if the phone's tab closes uncleanly), POST /api/dash/relay/mark_uploaded. - modules/app_upload.py: list_pending_dashcam_files(), relay lease helpers, _upload_dashcam_files() now pauses while a relay lease is active. - modules/app_sql.py: new dashcam_hash_cache table so a file's sha256 is only computed once (keyed by path+size+mtime) instead of being recomputed on every loop iteration/pending-list call - benefits the existing background uploader too. - templates/dash.html / static/js/dash.js: new "Relay via phone" card - per-file and "relay all" buttons (manual trigger only, per user preference), downloads+uploads one 40MB slice at a time so the phone never holds a whole file in memory, retries chunks up to 3x. - PLAN.md/readme.md/server/README.md updated (Phase 11). Tested via a full local end-to-end simulation driving the real Pi and server Flask apps side by side (Postgres mocked only at the db module boundary - all HTTP/chunking/auth/CORS logic is real): verified Range downloads (206), byte-for-byte reassembly via SHA256, path-traversal rejection, and the relay lease correctly pausing/resuming the background uploader. Not yet tested against a real phone/live network.
This commit is contained in:
@@ -458,6 +458,78 @@ data connection when WiFi/Ethernet is unreachable. Design notes:
|
||||
a plain-HTTP router port-forward path, avoiding the need for the modem's TLS stack to validate
|
||||
a cert chain).
|
||||
|
||||
### Phase 11: Phone-relay dashcam upload — ✅ DONE
|
||||
Root cause investigation into "why is the transfer from the Pi to the server so slow?" (real
|
||||
dashcam backlog draining at only ~350MB/4-5min) found the bottleneck is the **Pi's own onboard
|
||||
BCM43438 WiFi chip** (2.4GHz-only, congested/interfered channel) — confirmed via `iperf3`
|
||||
raw-TCP tests showing only ~14.7Mbit/s with active retransmits even over a same-subnet direct-LAN
|
||||
hop that never touches the internet/Cloudflare, ruling out ISP/Cloudflare limits. Since the Pi 3 B
|
||||
has no 5GHz radio and can't be fixed in software, added a way to bypass its weak WiFi for the
|
||||
internet-bound leg entirely: a phone opens the same `/dash` page (over the Pi's hotspot/home WiFi)
|
||||
and manually relays pending dashcam files straight to the remote server using the phone's own
|
||||
network (WiFi or cellular data), instead of routing everything through the Pi.
|
||||
|
||||
**Zero new remote-server upload logic** — the phone's browser JS calls the *exact same*
|
||||
`/api/v1/dashcam/upload/status` + `/api/v1/dashcam/upload/chunk` endpoints the Pi's own Python
|
||||
client already uses, in the same 40MB-chunk protocol, just orchestrated by `fetch()`/`FormData`
|
||||
instead of `requests`/`MultipartEncoder`. The server dedupes purely by SHA256 regardless of which
|
||||
device sends the bytes, so no server-side changes were needed for the upload path itself — only a
|
||||
CORS `after_request` hook (`server/app.py`) was added, since the phone's browser calls
|
||||
`yaris.ceaf.dev` from a page served by a *different* origin (the Pi), which browsers block without
|
||||
`Access-Control-Allow-*` headers on both the real request and its preflight `OPTIONS`. Wildcard
|
||||
origin is fine here — auth is header-based (`X-Device-Name`/`X-Api-Key`), not cookie/session-based.
|
||||
|
||||
**New Pi-side endpoints** (`app.py`):
|
||||
- `GET /api/dash/pending` — lists not-yet-uploaded dashcam files (`relative_path`, `sha256`,
|
||||
`size`) via a new `app_upload.list_pending_dashcam_files()`.
|
||||
- `GET /api/dash/file/<path:relpath>` — serves a dashcam file's bytes for the phone to download,
|
||||
via `send_file(..., conditional=True)` for **Range-request support** (206 Partial Content) so
|
||||
the phone downloads (and immediately re-uploads) one 40MB slice at a time instead of ever
|
||||
holding a whole 300+MB file in the browser's memory — the exact same memory-safety lesson
|
||||
learned the hard way in Phase 10, applied symmetrically to the download side. Path-traversal
|
||||
safe via Werkzeug's `safe_join` against `app_dash.DEFAULT_DESTINATION`.
|
||||
- `GET /api/dash/relay/config` — hands the phone the same `SERVER_URL`/`DEVICE_NAME`/`API_KEY`
|
||||
the Pi's own client uses, so its JS can auth directly with the remote server. Same trust
|
||||
boundary as the rest of this project's unauthenticated local UI (anyone on the Pi's own
|
||||
hotspot/LAN is already implicitly trusted).
|
||||
- `POST /api/dash/relay/heartbeat` / `POST /api/dash/relay/stop` — a **lease-based pause**
|
||||
(`RELAY_LEASE_SECONDS = 45`) for the Pi's own background dashcam uploader, chosen over a plain
|
||||
on/off toggle specifically to self-heal if the phone's browser tab closes/loses connection
|
||||
uncleanly (a simple toggle could get stuck "paused" forever). The phone page heartbeats every
|
||||
15s while relaying; `_upload_dashcam_files()` skips entirely while a lease is active, re-checked
|
||||
between files so it stops promptly if a relay starts mid-drain. Also released via
|
||||
`navigator.sendBeacon()` on `beforeunload` as a best-effort immediate release.
|
||||
- `POST /api/dash/relay/mark_uploaded` — lets the phone tell the Pi "I just uploaded this file"
|
||||
right after a successful relay, so the local dedupe table (and the pending list / Pi's own
|
||||
background loop) reflect it immediately instead of waiting for the next full backlog scan.
|
||||
|
||||
**Performance side-fix**: `modules/app_upload.py`'s dashcam loop previously recomputed SHA256 for
|
||||
*every* candidate file on *every* loop iteration, including files long since uploaded — wasteful
|
||||
CPU/IO against a growing, possibly 20GB+, local backlog. Added a `dashcam_hash_cache` SQLite table
|
||||
(`modules/app_sql.py`, keyed by `relative_path`+`size`+`mtime`) so a file's hash is only computed
|
||||
once; both the existing background uploader and the new pending-files endpoint benefit.
|
||||
|
||||
UI (`templates/dash.html` / `static/js/dash.js`): a new "Relay via phone" card on `/dash` lists
|
||||
pending files with a per-file "Relay" button and a "Relay all pending" button (manual trigger only
|
||||
— chosen deliberately over an automatic one, per explicit user preference), plus a "Stop relay"
|
||||
button. Chunk downloads/uploads retry up to 3 times with backoff before surfacing an error, and a
|
||||
resumed relay re-checks `/status` first so a previously-interrupted file picks up from its first
|
||||
missing chunk instead of restarting.
|
||||
|
||||
**Tested**: a full local end-to-end simulation (no physical phone/network needed) drove the real
|
||||
Pi `app.py` and real `server/app.py` Flask apps side by side (Postgres mocked at the `db` module
|
||||
boundary only — all HTTP routing, chunking, auth, and CORS logic is real), reproducing the exact
|
||||
sequence the phone's browser JS performs: list pending -> check remote `/status` -> download each
|
||||
chunk from the Pi via `Range` requests (verified real `206 Partial Content` responses) -> upload
|
||||
each chunk to the mock-backed server -> verify the reassembled file matches the original
|
||||
byte-for-byte via SHA256 -> mark uploaded on the Pi -> confirm it drops out of the pending list.
|
||||
Also verified: path-traversal attempts on `/api/dash/file/` correctly 404, the relay
|
||||
heartbeat/stop lease correctly pauses/resumes `_upload_dashcam_files()`, and the sha256 cache
|
||||
table is populated and reused across repeated `/api/dash/pending` calls. **Not yet tested on a
|
||||
real phone against the real deployed server** — the wire protocol and every server-visible
|
||||
behavior is verified, but real mobile-browser quirks (background-tab throttling, flaky cellular
|
||||
mid-transfer) haven't been exercised against live hardware yet.
|
||||
|
||||
## Working Notes / Environment Gotchas
|
||||
|
||||
- **Pi credentials/SSH**: `ssh yaris@<pi-ip>`. Confirm current IP before connecting (varies by
|
||||
|
||||
Reference in New Issue
Block a user