From cea62d8c10b33e6cb46d961c74a26913565ab26d Mon Sep 17 00:00:00 2001 From: Mathieu Moran Date: Sat, 8 Aug 2026 20:21:22 -0400 Subject: [PATCH] Mark Phase 9 (remote server) as DONE in PLAN.md Reflects the completed deployment: server running on unraid, exposed at https://yaris.ceaf.dev via Cloudflare Tunnel, Pi-side upload client live and backlog-drain tuned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- PLAN.md | 65 ++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 41 insertions(+), 24 deletions(-) diff --git a/PLAN.md b/PLAN.md index 77afdff..9452f94 100644 --- a/PLAN.md +++ b/PLAN.md @@ -297,23 +297,23 @@ dashcam" on `/dash`, confirm it's detected and its real videos import correctly `dash/videos/` on the Pi. No code changes should be needed for that - only the transport (loop device vs. real USB) differs, and the detection/copy logic is transport-agnostic by design. -### Phase 9: Remote server for itinerary/dashcam upload — 🚧 IN PROGRESS -Per architecture decisions #4 and #5: build a self-hosted server/API to receive itinerary -(from the SQLite/Mongo pipeline) and dashcam file uploads, with the Pi using its home WiFi when -available and falling back to the SIM868's GSM/GPRS cellular connection otherwise. +### Phase 9: Remote server for itinerary/dashcam upload — DONE +Per architecture decisions #4 and #5: self-hosted server/API to receive itinerary (from the +SQLite pipeline) and dashcam file uploads. -Decided: Flask API + dedicated Postgres, deployed as a docker-compose stack on the user's +Decided/built: Flask API + dedicated Postgres, deployed as a docker-compose stack on the user's unraid server (separate from any existing containers/DBs there), living in `server/` in this -same repo (not a separate repo). Auth: simple `X-Device-Name`/`X-Api-Key` header pair per -device, key hash stored server-side, real key never committed (raw key printed once by -`server/scripts/create_device.py`, saved locally on the Pi only). External access: Cloudflare -Tunnel already running on the unraid box (add a public-hostname rule pointing at the API's -published port) — router port-forward + the user's Unifi firewall is a documented fallback if -ever needed instead. Large dashcam uploads should prefer a local-network path (e.g. an AdGuard -DNS override to bypass the tunnel while on home WiFi) since tunnels typically cap per-request -upload size. +same repo. Auth: simple `X-Device-Name`/`X-Api-Key` header pair per device, key hash stored +server-side, real key never committed (raw key printed once by `server/scripts/create_device.py`, +saved locally on the Pi only, in gitignored `db/upload_creds.env`). External access: exposed at +`https://yaris.ceaf.dev` via the user's existing Cloudflare Tunnel (public-hostname rule added in +the Cloudflare dashboard, pointing at `192.168.10.10:8098`) — router port-forward + the user's +Unifi firewall is the documented fallback if ever needed instead. Large dashcam uploads can +still prefer a local-network path (e.g. an AdGuard DNS override to bypass the tunnel while on +home WiFi) since tunnels typically cap per-request upload size — noted in `server/README.md`, +not yet needed/built. -Built so far (`server/` folder): +Built (`server/` folder): - `server/sql/init.sql` — Postgres schema: `devices` (name + sha256 api-key-hash), `tracking_points` (device_id FK, GPS/speed/odometer + `recorded_at`, unique per device+timestamp so retried batches can't duplicate rows), `dashcam_files` (device_id FK, relative_path + sha256, unique per @@ -323,23 +323,40 @@ Built so far (`server/` folder): - `server/modules/auth.py` — `require_device_auth` decorator checking the two headers against the stored key hash. - `server/app.py` — Flask app: `GET /health` (unauthenticated), `POST /api/v1/tracking/batch` - (bulk JSON insert), `POST /api/v1/dashcam/upload` (multipart, streamed to disk with sha256 - verification + path-traversal guard on `relative_path`). + (bulk JSON insert, capped at 5000 points/request as a sanity guard), `POST /api/v1/dashcam/upload` + (multipart, streamed to disk with sha256 verification + path-traversal guard on `relative_path`). - `server/Dockerfile` + `server/docker-compose.yml` — `api` (gunicorn, published on host port 8098 by default) + `db` (Postgres 17, internal-only, not published to the LAN) services, data - volume defaults to `server/data/` but recommended to point `DATA_DIR` at - `/mnt/user/appdata/yaris-obd-server` on unraid. + volume defaults to `server/data/` but pointed at `/mnt/user/appdata/yaris-obd-server` on unraid. - `server/.env.example` — template for `POSTGRES_PASSWORD`/`API_PORT`/`DATA_DIR`/upload size cap. - `server/scripts/create_device.py` — run once per device inside the `api` container to register - it and print its one-time API key. + it and print its one-time API key. Ran once for device `yaris-2013`. - `server/README.md` — full unraid deployment + Cloudflare Tunnel exposure + device-registration instructions. -Still TODO: deploy the stack on the actual unraid box and verify end-to-end (containers healthy, -`/health` reachable from the LAN), register the Pi as a device and get its real API key, then -build the Pi-side upload client (`modules/app_upload.py`) that drains the local SQLite queue and -dashcam files to this API with retry/backoff — including a web UI toggle to pause dashcam -uploads specifically while always continuing to upload GPS/trip logs. +Pi-side upload client (`modules/app_upload.py`): +- Background thread, uploads unsent `tracking` rows (batches of 500) and not-yet-uploaded + dashcam files (sha256-verified) via the API above, with exponential retry/backoff on failure. +- Independent dashcam-upload-only pause toggle (persisted `settings` table + a switch on the + `/dash` web UI) — trip/GPS logs always keep uploading regardless of the toggle, per explicit + user request (useful on a slow/metered connection). +- **Backlog-drain tuning**: since tracking rows log every 5s locally, an offline/stale device can + queue up a large backlog. The loop drains back-to-back (short 1s pause between batches) while a + batch comes back full, only settling into the slow 30s idle cadence once actually caught up — + a simulated 1250-row backlog now drains in 3 batches (~0.35s) instead of 13 batches over ~6.5 + minutes with the original fixed 100-row/30s-sleep design. +- `modules/app_sql.py` extended: `UPLOADED` column + migration on `tracking`, `settings` table, + `dashcam_uploads` dedupe table, and helper functions backing all of the above. + +Tested end-to-end against the real deployed server (tracking batch + dedupe, dashcam upload + +dedupe + path-traversal rejection, bad-API-key rejection, toggle-off correctly pausing only +dashcam uploads) and on the real Pi (its actual ~1800-row pre-feature backlog drained and +confirmed matching row counts on the server via direct psql query, live upload status visible on +`/dash`). Committed+pushed (`4ff99f0`, `c4db1ab`, `8dc304e`), deployed on both unraid and the Pi. + +Not attempted (future, no action needed unless requested): SIM868 GSM/GPRS cellular fallback +uplink for uploads when no network path to the server is available at all (today uploads need +the Pi to reach `yaris.ceaf.dev`, e.g. via home WiFi) — remains a possible future enhancement. ## Working Notes / Environment Gotchas