Mathieu Moran
0d67a21ae1
Root-caused the slow-upload complaint to the Pi's own congested 2.4GHz WiFi radio (confirmed via iperf3 - same degraded throughput even on pure local LAN traffic, ruling out Cloudflare/ISP). Since the Pi 3 B has no 5GHz radio, add a way to bypass it for the internet-bound leg: a phone opens /dash and manually relays pending dashcam files straight to the remote server over its own network, reusing the exact same chunked upload protocol the Pi's client already uses. - server/app.py: add CORS headers (after_request) so the phone's browser can call the remote server cross-origin from a page served by the Pi. - app.py: new endpoints - GET /api/dash/pending (list not-yet-uploaded files), GET /api/dash/file/<path> (Range-request-capable download, path-traversal safe), GET /api/dash/relay/config (hands the phone the device credentials), POST /api/dash/relay/heartbeat|stop (lease- based pause of the Pi's own background uploader, self-heals if the phone's tab closes uncleanly), POST /api/dash/relay/mark_uploaded. - modules/app_upload.py: list_pending_dashcam_files(), relay lease helpers, _upload_dashcam_files() now pauses while a relay lease is active. - modules/app_sql.py: new dashcam_hash_cache table so a file's sha256 is only computed once (keyed by path+size+mtime) instead of being recomputed on every loop iteration/pending-list call - benefits the existing background uploader too. - templates/dash.html / static/js/dash.js: new "Relay via phone" card - per-file and "relay all" buttons (manual trigger only, per user preference), downloads+uploads one 40MB slice at a time so the phone never holds a whole file in memory, retries chunks up to 3x. - PLAN.md/readme.md/server/README.md updated (Phase 11). Tested via a full local end-to-end simulation driving the real Pi and server Flask apps side by side (Postgres mocked only at the db module boundary - all HTTP/chunking/auth/CORS logic is real): verified Range downloads (206), byte-for-byte reassembly via SHA256, path-traversal rejection, and the relay lease correctly pausing/resuming the background uploader. Not yet tested against a real phone/live network.
129 lines
5.7 KiB
Markdown
129 lines
5.7 KiB
Markdown
# yaris-obd remote server
|
|
|
|
A small Flask + Postgres API that receives trip/GPS log batches and dashcam
|
|
video files uploaded by the Pi (see the repo root [readme.md](../readme.md)
|
|
and [PLAN.md](../PLAN.md) for the overall project). Designed to run as a
|
|
docker-compose stack on an unraid server (or any Docker host).
|
|
|
|
## What it does
|
|
|
|
- `GET /health` — unauthenticated liveness check.
|
|
- `POST /api/v1/tracking/batch` — bulk-insert a batch of GPS/trip points.
|
|
- `POST /api/v1/dashcam/upload` — upload one dashcam video file in a single
|
|
request (multipart, content-addressed by sha256). Kept for reference/
|
|
compatibility, but the Pi no longer calls this directly for real dashcam
|
|
footage — see chunked upload below.
|
|
- `GET /api/v1/dashcam/upload/status` — check upload progress for a
|
|
`(relative_path, sha256)` pair: `not_started`, `in_progress` (with the list
|
|
of already-received chunk indices, for resuming), or `already_uploaded`.
|
|
- `POST /api/v1/dashcam/upload/chunk` — upload one chunk of a dashcam video
|
|
(keyed by `relative_path`/`sha256`/`chunk_index`/`total_chunks`). Chunks are
|
|
staged on disk and reassembled + sha256-verified once all have arrived, then
|
|
recorded the same way as the single-shot endpoint. This is what the Pi
|
|
actually uses for real dashcam files — see "Why chunked uploads" below.
|
|
|
|
Both `/api/v1/tracking/batch` and the dashcam upload endpoints require two
|
|
headers: `X-Device-Name` and `X-Api-Key` (see
|
|
[Registering a device](#registering-a-device-and-getting-an-api-key) below).
|
|
Postgres is **not** published to the LAN — only the `api` container talks to
|
|
it, over the internal compose network.
|
|
|
|
**CORS is enabled on every response** (`Access-Control-Allow-Origin: *`) so a
|
|
device's registered API key can be used directly from browser JS running on a
|
|
*different* origin — specifically, the Pi's phone-relay feature (see
|
|
[PLAN.md](../PLAN.md) Phase 11), where a phone's browser calls this API
|
|
straight from a page served by the Pi itself. This is safe here because
|
|
auth is header-based (`X-Device-Name`/`X-Api-Key`), not cookie/session-based,
|
|
so a permissive origin can't be used to steal a logged-in session.
|
|
|
|
## Deploying on unraid
|
|
|
|
1. Get the code onto the unraid box (same repo as the Pi, just the
|
|
`server/` sub-folder matters here):
|
|
```bash
|
|
ssh root@<unraid-ip>
|
|
mkdir -p /mnt/user/app/git && cd /mnt/user/app/git
|
|
git clone [email protected]:quoije/yaris-obd.git
|
|
cd yaris-obd/server
|
|
```
|
|
2. Copy the env template and fill in a real Postgres password:
|
|
```bash
|
|
cp .env.example .env
|
|
nano .env # set POSTGRES_PASSWORD, and DATA_DIR if you want data on the array
|
|
```
|
|
Recommended `DATA_DIR` for unraid: `/mnt/user/appdata/yaris-obd-server`
|
|
(keeps Postgres data + dashcam files on the array, backed up like your
|
|
other appdata).
|
|
3. Build and start the stack:
|
|
```bash
|
|
docker compose up -d --build
|
|
```
|
|
This is also importable into unraid's **Compose Manager** plugin if you'd
|
|
rather manage start/stop/updates from the Unraid UI.
|
|
4. Confirm it's healthy:
|
|
```bash
|
|
docker compose ps
|
|
curl http://localhost:8098/health
|
|
```
|
|
|
|
## Registering a device and getting an API key
|
|
|
|
Run this once per device (e.g. once for the Pi) after the stack is up:
|
|
|
|
```bash
|
|
docker compose exec api python3 scripts/create_device.py yaris-2013
|
|
```
|
|
|
|
This prints a one-time API key — copy the `YARIS_UPLOAD_DEVICE_NAME` /
|
|
`YARIS_UPLOAD_API_KEY` values into the Pi's upload-client config. Only the
|
|
key's sha256 hash is stored server-side; the raw key is never shown again
|
|
(re-run the script with the same device name to rotate/replace the key).
|
|
|
|
## Exposing it outside the LAN
|
|
|
|
Two options, either works with this API unchanged:
|
|
|
|
- **Cloudflare Tunnel** (already running on this unraid box): add a public
|
|
hostname rule in the Cloudflare dashboard pointing at
|
|
`http://<unraid-lan-ip>:8098` (or `http://api:8098` if you point the
|
|
tunnel at the `api` container directly on the compose network). Good for
|
|
small tracking-data batches; large dashcam uploads may hit the tunnel's
|
|
per-request size limit.
|
|
- **Router port-forward** (fallback if needed): forward an external port to
|
|
`<unraid-lan-ip>:8098` and let your router/firewall (e.g. Unifi) handle
|
|
the security side. No changes needed here either way.
|
|
|
|
For large dashcam uploads from home, consider adding a local DNS
|
|
override/rewrite (e.g. in AdGuard Home, same trick used for `yaris.local` on
|
|
the Pi) so the hostname resolves directly to the unraid LAN IP while on the
|
|
home network, bypassing the tunnel entirely for big transfers.
|
|
|
|
## Why chunked uploads
|
|
|
|
Cloudflare Tunnel's free tier enforces a hard ~100MB request-body cap,
|
|
returning a bare `413 Payload Too Large` before the request even reaches this
|
|
API — confirmed against real dashcam footage (avg ~320MB/file), which the
|
|
single-shot `/api/v1/dashcam/upload` endpoint could never pass through the
|
|
tunnel. The Pi's upload client instead splits each file into 40MB chunks and
|
|
POSTs them one at a time to `/api/v1/dashcam/upload/chunk`, checking
|
|
`/api/v1/dashcam/upload/status` first so an interrupted upload (Pi
|
|
reboot/crash, network drop) resumes from the last received chunk instead of
|
|
restarting the whole file. Verified end-to-end on real hardware: a full
|
|
28GB/87-file dashcam backlog uploaded successfully this way, each file
|
|
byte-for-byte sha256-verified against the source on the Pi.
|
|
|
|
## Toggling dashcam uploads off
|
|
|
|
The Pi's web UI has a setting to pause dashcam file uploads while keeping
|
|
GPS/trip log uploads running — no server-side change needed, the Pi simply
|
|
stops calling `/api/v1/dashcam/upload` while the toggle is off.
|
|
|
|
## Local development / testing without unraid
|
|
|
|
```bash
|
|
cd server
|
|
cp .env.example .env # edit POSTGRES_PASSWORD
|
|
docker compose up --build
|
|
curl http://localhost:8098/health
|
|
```
|