Mathieu Moran
8c33c669a5
- PLAN.md: mark Phase 8 (dashcam retrieval) verified on real hardware, add Phase 10 documenting the OOM/multipart and Cloudflare body-size fixes plus the chunked/resumable upload redesign, both verified end-to-end against real dashcam footage. Update architecture decision #3 (Mongo -> Postgres supersession), add unraid/Pi deploy notes and memory-budget gotcha. - readme.md: refresh feature list (remote upload, chunked dashcam uploads, toggle) and project status (all 10 phases done, only cellular fallback untested). - server/README.md: document the new chunked upload endpoints and the Cloudflare body-size-limit rationale behind them.
121 lines
5.2 KiB
Markdown
121 lines
5.2 KiB
Markdown
# yaris-obd remote server
|
|
|
|
A small Flask + Postgres API that receives trip/GPS log batches and dashcam
|
|
video files uploaded by the Pi (see the repo root [readme.md](../readme.md)
|
|
and [PLAN.md](../PLAN.md) for the overall project). Designed to run as a
|
|
docker-compose stack on an unraid server (or any Docker host).
|
|
|
|
## What it does
|
|
|
|
- `GET /health` — unauthenticated liveness check.
|
|
- `POST /api/v1/tracking/batch` — bulk-insert a batch of GPS/trip points.
|
|
- `POST /api/v1/dashcam/upload` — upload one dashcam video file in a single
|
|
request (multipart, content-addressed by sha256). Kept for reference/
|
|
compatibility, but the Pi no longer calls this directly for real dashcam
|
|
footage — see chunked upload below.
|
|
- `GET /api/v1/dashcam/upload/status` — check upload progress for a
|
|
`(relative_path, sha256)` pair: `not_started`, `in_progress` (with the list
|
|
of already-received chunk indices, for resuming), or `already_uploaded`.
|
|
- `POST /api/v1/dashcam/upload/chunk` — upload one chunk of a dashcam video
|
|
(keyed by `relative_path`/`sha256`/`chunk_index`/`total_chunks`). Chunks are
|
|
staged on disk and reassembled + sha256-verified once all have arrived, then
|
|
recorded the same way as the single-shot endpoint. This is what the Pi
|
|
actually uses for real dashcam files — see "Why chunked uploads" below.
|
|
|
|
Both `/api/v1/tracking/batch` and the dashcam upload endpoints require two
|
|
headers: `X-Device-Name` and `X-Api-Key` (see
|
|
[Registering a device](#registering-a-device-and-getting-an-api-key) below).
|
|
Postgres is **not** published to the LAN — only the `api` container talks to
|
|
it, over the internal compose network.
|
|
|
|
## Deploying on unraid
|
|
|
|
1. Get the code onto the unraid box (same repo as the Pi, just the
|
|
`server/` sub-folder matters here):
|
|
```bash
|
|
ssh root@<unraid-ip>
|
|
mkdir -p /mnt/user/app/git && cd /mnt/user/app/git
|
|
git clone [email protected]:quoije/yaris-obd.git
|
|
cd yaris-obd/server
|
|
```
|
|
2. Copy the env template and fill in a real Postgres password:
|
|
```bash
|
|
cp .env.example .env
|
|
nano .env # set POSTGRES_PASSWORD, and DATA_DIR if you want data on the array
|
|
```
|
|
Recommended `DATA_DIR` for unraid: `/mnt/user/appdata/yaris-obd-server`
|
|
(keeps Postgres data + dashcam files on the array, backed up like your
|
|
other appdata).
|
|
3. Build and start the stack:
|
|
```bash
|
|
docker compose up -d --build
|
|
```
|
|
This is also importable into unraid's **Compose Manager** plugin if you'd
|
|
rather manage start/stop/updates from the Unraid UI.
|
|
4. Confirm it's healthy:
|
|
```bash
|
|
docker compose ps
|
|
curl http://localhost:8098/health
|
|
```
|
|
|
|
## Registering a device and getting an API key
|
|
|
|
Run this once per device (e.g. once for the Pi) after the stack is up:
|
|
|
|
```bash
|
|
docker compose exec api python3 scripts/create_device.py yaris-2013
|
|
```
|
|
|
|
This prints a one-time API key — copy the `YARIS_UPLOAD_DEVICE_NAME` /
|
|
`YARIS_UPLOAD_API_KEY` values into the Pi's upload-client config. Only the
|
|
key's sha256 hash is stored server-side; the raw key is never shown again
|
|
(re-run the script with the same device name to rotate/replace the key).
|
|
|
|
## Exposing it outside the LAN
|
|
|
|
Two options, either works with this API unchanged:
|
|
|
|
- **Cloudflare Tunnel** (already running on this unraid box): add a public
|
|
hostname rule in the Cloudflare dashboard pointing at
|
|
`http://<unraid-lan-ip>:8098` (or `http://api:8098` if you point the
|
|
tunnel at the `api` container directly on the compose network). Good for
|
|
small tracking-data batches; large dashcam uploads may hit the tunnel's
|
|
per-request size limit.
|
|
- **Router port-forward** (fallback if needed): forward an external port to
|
|
`<unraid-lan-ip>:8098` and let your router/firewall (e.g. Unifi) handle
|
|
the security side. No changes needed here either way.
|
|
|
|
For large dashcam uploads from home, consider adding a local DNS
|
|
override/rewrite (e.g. in AdGuard Home, same trick used for `yaris.local` on
|
|
the Pi) so the hostname resolves directly to the unraid LAN IP while on the
|
|
home network, bypassing the tunnel entirely for big transfers.
|
|
|
|
## Why chunked uploads
|
|
|
|
Cloudflare Tunnel's free tier enforces a hard ~100MB request-body cap,
|
|
returning a bare `413 Payload Too Large` before the request even reaches this
|
|
API — confirmed against real dashcam footage (avg ~320MB/file), which the
|
|
single-shot `/api/v1/dashcam/upload` endpoint could never pass through the
|
|
tunnel. The Pi's upload client instead splits each file into 40MB chunks and
|
|
POSTs them one at a time to `/api/v1/dashcam/upload/chunk`, checking
|
|
`/api/v1/dashcam/upload/status` first so an interrupted upload (Pi
|
|
reboot/crash, network drop) resumes from the last received chunk instead of
|
|
restarting the whole file. Verified end-to-end on real hardware: a full
|
|
28GB/87-file dashcam backlog uploaded successfully this way, each file
|
|
byte-for-byte sha256-verified against the source on the Pi.
|
|
|
|
## Toggling dashcam uploads off
|
|
|
|
The Pi's web UI has a setting to pause dashcam file uploads while keeping
|
|
GPS/trip log uploads running — no server-side change needed, the Pi simply
|
|
stops calling `/api/v1/dashcam/upload` while the toggle is off.
|
|
|
|
## Local development / testing without unraid
|
|
|
|
```bash
|
|
cd server
|
|
cp .env.example .env # edit POSTGRES_PASSWORD
|
|
docker compose up --build
|
|
curl http://localhost:8098/health
|
|
```
|