quoije/yaris-obd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Mark Phase 9 (remote server) as DONE in PLAN.md

Browse Source 
Reflects the completed deployment: server running on unraid, exposed
at https://yaris.ceaf.dev via Cloudflare Tunnel, Pi-side upload client
live and backlog-drain tuned.

Co-authored-by: Copilot <[email protected]>
This commit is contained in:
Mathieu Moran
2026-08-08 20:21:22 -04:00
co-authored by Copilot
parent 975526667e
commit cea62d8c10
1 changed files with 41 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PLAN.md
+41 -24
View File
@@ -297,23 +297,23 @@ dashcam" on `/dash`, confirm it's detected and its real videos import correctly
`dash/videos/` on the Pi. No code changes should be needed for that - only the transport (loop
device vs. real USB) differs, and the detection/copy logic is transport-agnostic by design.
### Phase 9: Remote server for itinerary/dashcam upload — 🚧 IN PROGRESS
Per architecture decisions #4 and #5: build a self-hosted server/API to receive itinerary
(from the SQLite/Mongo pipeline) and dashcam file uploads, with the Pi using its home WiFi when
available and falling back to the SIM868's GSM/GPRS cellular connection otherwise.
### Phase 9: Remote server for itinerary/dashcam upload — DONE
Per architecture decisions #4 and #5: self-hosted server/API to receive itinerary (from the
SQLite pipeline) and dashcam file uploads.
Decided: Flask API + dedicated Postgres, deployed as a docker-compose stack on the user's
Decided/built: Flask API + dedicated Postgres, deployed as a docker-compose stack on the user's
unraid server (separate from any existing containers/DBs there), living in `server/` in this
same repo (not a separate repo). Auth: simple `X-Device-Name`/`X-Api-Key` header pair per
device, key hash stored server-side, real key never committed (raw key printed once by
`server/scripts/create_device.py`, saved locally on the Pi only). External access: Cloudflare
Tunnel already running on the unraid box (add a public-hostname rule pointing at the API's
published port) — router port-forward + the user's Unifi firewall is a documented fallback if
ever needed instead. Large dashcam uploads should prefer a local-network path (e.g. an AdGuard
DNS override to bypass the tunnel while on home WiFi) since tunnels typically cap per-request
upload size.
same repo. Auth: simple `X-Device-Name`/`X-Api-Key` header pair per device, key hash stored
server-side, real key never committed (raw key printed once by `server/scripts/create_device.py`,
saved locally on the Pi only, in gitignored `db/upload_creds.env`). External access: exposed at
`https://yaris.ceaf.dev` via the user's existing Cloudflare Tunnel (public-hostname rule added in
the Cloudflare dashboard, pointing at `192.168.10.10:8098`) — router port-forward + the user's
Unifi firewall is the documented fallback if ever needed instead. Large dashcam uploads can
still prefer a local-network path (e.g. an AdGuard DNS override to bypass the tunnel while on
home WiFi) since tunnels typically cap per-request upload size — noted in `server/README.md`,
not yet needed/built.
Built so far (`server/` folder):
Built (`server/` folder):
- `server/sql/init.sql` — Postgres schema: `devices` (name + sha256 api-key-hash), `tracking_points`
(device_id FK, GPS/speed/odometer + `recorded_at`, unique per device+timestamp so retried
batches can't duplicate rows), `dashcam_files` (device_id FK, relative_path + sha256, unique per
@@ -323,23 +323,40 @@ Built so far (`server/` folder):
- `server/modules/auth.py` — `require_device_auth` decorator checking the two headers against
the stored key hash.
- `server/app.py` — Flask app: `GET /health` (unauthenticated), `POST /api/v1/tracking/batch`
(bulk JSON insert), `POST /api/v1/dashcam/upload` (multipart, streamed to disk with sha256
verification + path-traversal guard on `relative_path`).
(bulk JSON insert, capped at 5000 points/request as a sanity guard), `POST /api/v1/dashcam/upload`
(multipart, streamed to disk with sha256 verification + path-traversal guard on `relative_path`).
- `server/Dockerfile` + `server/docker-compose.yml` — `api` (gunicorn, published on host port
8098 by default) + `db` (Postgres 17, internal-only, not published to the LAN) services, data
volume defaults to `server/data/` but recommended to point `DATA_DIR` at
`/mnt/user/appdata/yaris-obd-server` on unraid.
volume defaults to `server/data/` but pointed at `/mnt/user/appdata/yaris-obd-server` on unraid.
- `server/.env.example` — template for `POSTGRES_PASSWORD`/`API_PORT`/`DATA_DIR`/upload size cap.
- `server/scripts/create_device.py` — run once per device inside the `api` container to register
it and print its one-time API key.
it and print its one-time API key. Ran once for device `yaris-2013`.
- `server/README.md` — full unraid deployment + Cloudflare Tunnel exposure + device-registration
instructions.
Still TODO: deploy the stack on the actual unraid box and verify end-to-end (containers healthy,
`/health` reachable from the LAN), register the Pi as a device and get its real API key, then
build the Pi-side upload client (`modules/app_upload.py`) that drains the local SQLite queue and
dashcam files to this API with retry/backoff — including a web UI toggle to pause dashcam
uploads specifically while always continuing to upload GPS/trip logs.
Pi-side upload client (`modules/app_upload.py`):
- Background thread, uploads unsent `tracking` rows (batches of 500) and not-yet-uploaded
dashcam files (sha256-verified) via the API above, with exponential retry/backoff on failure.
- Independent dashcam-upload-only pause toggle (persisted `settings` table + a switch on the
`/dash` web UI) — trip/GPS logs always keep uploading regardless of the toggle, per explicit
user request (useful on a slow/metered connection).
- **Backlog-drain tuning**: since tracking rows log every 5s locally, an offline/stale device can
queue up a large backlog. The loop drains back-to-back (short 1s pause between batches) while a
batch comes back full, only settling into the slow 30s idle cadence once actually caught up —
a simulated 1250-row backlog now drains in 3 batches (~0.35s) instead of 13 batches over ~6.5
minutes with the original fixed 100-row/30s-sleep design.
- `modules/app_sql.py` extended: `UPLOADED` column + migration on `tracking`, `settings` table,
`dashcam_uploads` dedupe table, and helper functions backing all of the above.
Tested end-to-end against the real deployed server (tracking batch + dedupe, dashcam upload +
dedupe + path-traversal rejection, bad-API-key rejection, toggle-off correctly pausing only
dashcam uploads) and on the real Pi (its actual ~1800-row pre-feature backlog drained and
confirmed matching row counts on the server via direct psql query, live upload status visible on
`/dash`). Committed+pushed (`4ff99f0`, `c4db1ab`, `8dc304e`), deployed on both unraid and the Pi.
Not attempted (future, no action needed unless requested): SIM868 GSM/GPRS cellular fallback
uplink for uploads when no network path to the server is available at all (today uploads need
the Pi to reach `yaris.ceaf.dev`, e.g. via home WiFi) — remains a possible future enhancement.
## Working Notes / Environment Gotchas