Phase 9: scaffold remote server (Flask + Postgres, docker-compose for unraid)
- server/sql/init.sql: devices/tracking_points/dashcam_files schema - server/modules/db.py: psycopg3 pool + query helpers - server/modules/auth.py: X-Device-Name/X-Api-Key header auth - server/app.py: /health, /api/v1/tracking/batch, /api/v1/dashcam/upload - server/Dockerfile + docker-compose.yml: api (gunicorn) + postgres services - server/scripts/create_device.py: one-time device/API-key registration - server/README.md: unraid deployment + Cloudflare Tunnel exposure docs - PLAN.md: Phase 9 marked in progress with full build notes Co-authored-by: Copilot <[email protected]>
This commit is contained in:
@@ -25,3 +25,8 @@ db/sql/
|
|||||||
|
|
||||||
# Imported dashcam footage (large, per-device data, not source)
|
# Imported dashcam footage (large, per-device data, not source)
|
||||||
dash/videos/
|
dash/videos/
|
||||||
|
|
||||||
|
# Remote server: local secrets and runtime data (Postgres volume, uploaded
|
||||||
|
# dashcam files) - never commit real credentials or per-device data
|
||||||
|
server/.env
|
||||||
|
server/data/
|
||||||
|
|||||||
@@ -297,11 +297,50 @@ dashcam" on `/dash`, confirm it's detected and its real videos import correctly
|
|||||||
`dash/videos/` on the Pi. No code changes should be needed for that - only the transport (loop
|
`dash/videos/` on the Pi. No code changes should be needed for that - only the transport (loop
|
||||||
device vs. real USB) differs, and the detection/copy logic is transport-agnostic by design.
|
device vs. real USB) differs, and the detection/copy logic is transport-agnostic by design.
|
||||||
|
|
||||||
### Phase 9: Remote server for itinerary/dashcam upload — ⏳ TODO
|
### Phase 9: Remote server for itinerary/dashcam upload — 🚧 IN PROGRESS
|
||||||
Per architecture decisions #4 and #5: build a self-hosted server/API to receive itinerary
|
Per architecture decisions #4 and #5: build a self-hosted server/API to receive itinerary
|
||||||
(from the SQLite/Mongo pipeline) and dashcam file uploads, with the Pi using its home WiFi when
|
(from the SQLite/Mongo pipeline) and dashcam file uploads, with the Pi using its home WiFi when
|
||||||
available and falling back to the SIM868's GSM/GPRS cellular connection otherwise.
|
available and falling back to the SIM868's GSM/GPRS cellular connection otherwise.
|
||||||
|
|
||||||
|
Decided: Flask API + dedicated Postgres, deployed as a docker-compose stack on the user's
|
||||||
|
unraid server (separate from any existing containers/DBs there), living in `server/` in this
|
||||||
|
same repo (not a separate repo). Auth: simple `X-Device-Name`/`X-Api-Key` header pair per
|
||||||
|
device, key hash stored server-side, real key never committed (raw key printed once by
|
||||||
|
`server/scripts/create_device.py`, saved locally on the Pi only). External access: Cloudflare
|
||||||
|
Tunnel already running on the unraid box (add a public-hostname rule pointing at the API's
|
||||||
|
published port) — router port-forward + the user's Unifi firewall is a documented fallback if
|
||||||
|
ever needed instead. Large dashcam uploads should prefer a local-network path (e.g. an AdGuard
|
||||||
|
DNS override to bypass the tunnel while on home WiFi) since tunnels typically cap per-request
|
||||||
|
upload size.
|
||||||
|
|
||||||
|
Built so far (`server/` folder):
|
||||||
|
- `server/sql/init.sql` — Postgres schema: `devices` (name + sha256 api-key-hash), `tracking_points`
|
||||||
|
(device_id FK, GPS/speed/odometer + `recorded_at`, unique per device+timestamp so retried
|
||||||
|
batches can't duplicate rows), `dashcam_files` (device_id FK, relative_path + sha256, unique per
|
||||||
|
device+path+hash so re-uploads are no-ops).
|
||||||
|
- `server/modules/db.py` — psycopg3 connection pool + query helpers (bulk tracking insert with
|
||||||
|
`ON CONFLICT DO NOTHING`, dashcam dedupe-check-before-write).
|
||||||
|
- `server/modules/auth.py` — `require_device_auth` decorator checking the two headers against
|
||||||
|
the stored key hash.
|
||||||
|
- `server/app.py` — Flask app: `GET /health` (unauthenticated), `POST /api/v1/tracking/batch`
|
||||||
|
(bulk JSON insert), `POST /api/v1/dashcam/upload` (multipart, streamed to disk with sha256
|
||||||
|
verification + path-traversal guard on `relative_path`).
|
||||||
|
- `server/Dockerfile` + `server/docker-compose.yml` — `api` (gunicorn, published on host port
|
||||||
|
8098 by default) + `db` (Postgres 17, internal-only, not published to the LAN) services, data
|
||||||
|
volume defaults to `server/data/` but recommended to point `DATA_DIR` at
|
||||||
|
`/mnt/user/appdata/yaris-obd-server` on unraid.
|
||||||
|
- `server/.env.example` — template for `POSTGRES_PASSWORD`/`API_PORT`/`DATA_DIR`/upload size cap.
|
||||||
|
- `server/scripts/create_device.py` — run once per device inside the `api` container to register
|
||||||
|
it and print its one-time API key.
|
||||||
|
- `server/README.md` — full unraid deployment + Cloudflare Tunnel exposure + device-registration
|
||||||
|
instructions.
|
||||||
|
|
||||||
|
Still TODO: deploy the stack on the actual unraid box and verify end-to-end (containers healthy,
|
||||||
|
`/health` reachable from the LAN), register the Pi as a device and get its real API key, then
|
||||||
|
build the Pi-side upload client (`modules/app_upload.py`) that drains the local SQLite queue and
|
||||||
|
dashcam files to this API with retry/backoff — including a web UI toggle to pause dashcam
|
||||||
|
uploads specifically while always continuing to upload GPS/trip logs.
|
||||||
|
|
||||||
## Working Notes / Environment Gotchas
|
## Working Notes / Environment Gotchas
|
||||||
|
|
||||||
- **Pi credentials/SSH**: `ssh yaris@<pi-ip>`. Confirm current IP before connecting (varies by
|
- **Pi credentials/SSH**: `ssh yaris@<pi-ip>`. Confirm current IP before connecting (varies by
|
||||||
|
|||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# Copy this file to .env and fill in real values before running
|
||||||
|
# `docker compose up -d`. .env itself is gitignored - never commit real
|
||||||
|
# credentials.
|
||||||
|
|
||||||
|
# Postgres credentials (internal to the compose network only, never exposed
|
||||||
|
# to the LAN/internet - pick any strong password).
|
||||||
|
POSTGRES_DB=yarisobd
|
||||||
|
POSTGRES_USER=yarisobd
|
||||||
|
POSTGRES_PASSWORD=changeme
|
||||||
|
|
||||||
|
# Host port the Flask API is published on (this is what you point the
|
||||||
|
# Cloudflare Tunnel / router port-forward at).
|
||||||
|
API_PORT=8098
|
||||||
|
|
||||||
|
# Where Postgres data + uploaded dashcam files are stored on the unraid host.
|
||||||
|
# Recommended: a path on the array, e.g. /mnt/user/appdata/yaris-obd-server
|
||||||
|
DATA_DIR=./data
|
||||||
|
|
||||||
|
# Reject dashcam uploads larger than this (MB). Bump if your dashcam
|
||||||
|
# produces very large individual clips.
|
||||||
|
MAX_DASHCAM_UPLOAD_MB=2048
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
FROM python:3.12-slim
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
COPY requirements.txt .
|
||||||
|
RUN pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
|
COPY app.py .
|
||||||
|
COPY modules/ ./modules/
|
||||||
|
COPY scripts/ ./scripts/
|
||||||
|
|
||||||
|
EXPOSE 8098
|
||||||
|
|
||||||
|
# gunicorn instead of the Flask dev server for production use; single worker
|
||||||
|
# is enough for this low-traffic single-device use case, threaded to allow
|
||||||
|
# concurrent tracking + dashcam-upload requests.
|
||||||
|
CMD ["gunicorn", "--bind", "0.0.0.0:8098", "--workers", "1", "--threads", "4", \
|
||||||
|
"--timeout", "300", "app:app"]
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# yaris-obd remote server
|
||||||
|
|
||||||
|
A small Flask + Postgres API that receives trip/GPS log batches and dashcam
|
||||||
|
video files uploaded by the Pi (see the repo root [readme.md](../readme.md)
|
||||||
|
and [PLAN.md](../PLAN.md) for the overall project). Designed to run as a
|
||||||
|
docker-compose stack on an unraid server (or any Docker host).
|
||||||
|
|
||||||
|
## What it does
|
||||||
|
|
||||||
|
- `GET /health` — unauthenticated liveness check.
|
||||||
|
- `POST /api/v1/tracking/batch` — bulk-insert a batch of GPS/trip points.
|
||||||
|
- `POST /api/v1/dashcam/upload` — upload one dashcam video file (multipart,
|
||||||
|
content-addressed by sha256 so retries/duplicates are safe).
|
||||||
|
|
||||||
|
Both upload endpoints require two headers: `X-Device-Name` and `X-Api-Key`
|
||||||
|
(see [Registering a device](#registering-a-device-and-getting-an-api-key)
|
||||||
|
below). Postgres is **not** published to the LAN — only the `api` container
|
||||||
|
talks to it, over the internal compose network.
|
||||||
|
|
||||||
|
## Deploying on unraid
|
||||||
|
|
||||||
|
1. Get the code onto the unraid box (same repo as the Pi, just the
|
||||||
|
`server/` sub-folder matters here):
|
||||||
|
```bash
|
||||||
|
ssh root@<unraid-ip>
|
||||||
|
mkdir -p /mnt/user/app/git && cd /mnt/user/app/git
|
||||||
|
git clone [email protected]:quoije/yaris-obd.git
|
||||||
|
cd yaris-obd/server
|
||||||
|
```
|
||||||
|
2. Copy the env template and fill in a real Postgres password:
|
||||||
|
```bash
|
||||||
|
cp .env.example .env
|
||||||
|
nano .env # set POSTGRES_PASSWORD, and DATA_DIR if you want data on the array
|
||||||
|
```
|
||||||
|
Recommended `DATA_DIR` for unraid: `/mnt/user/appdata/yaris-obd-server`
|
||||||
|
(keeps Postgres data + dashcam files on the array, backed up like your
|
||||||
|
other appdata).
|
||||||
|
3. Build and start the stack:
|
||||||
|
```bash
|
||||||
|
docker compose up -d --build
|
||||||
|
```
|
||||||
|
This is also importable into unraid's **Compose Manager** plugin if you'd
|
||||||
|
rather manage start/stop/updates from the Unraid UI.
|
||||||
|
4. Confirm it's healthy:
|
||||||
|
```bash
|
||||||
|
docker compose ps
|
||||||
|
curl http://localhost:8098/health
|
||||||
|
```
|
||||||
|
|
||||||
|
## Registering a device and getting an API key
|
||||||
|
|
||||||
|
Run this once per device (e.g. once for the Pi) after the stack is up:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose exec api python3 scripts/create_device.py yaris-2013
|
||||||
|
```
|
||||||
|
|
||||||
|
This prints a one-time API key — copy the `YARIS_UPLOAD_DEVICE_NAME` /
|
||||||
|
`YARIS_UPLOAD_API_KEY` values into the Pi's upload-client config. Only the
|
||||||
|
key's sha256 hash is stored server-side; the raw key is never shown again
|
||||||
|
(re-run the script with the same device name to rotate/replace the key).
|
||||||
|
|
||||||
|
## Exposing it outside the LAN
|
||||||
|
|
||||||
|
Two options, either works with this API unchanged:
|
||||||
|
|
||||||
|
- **Cloudflare Tunnel** (already running on this unraid box): add a public
|
||||||
|
hostname rule in the Cloudflare dashboard pointing at
|
||||||
|
`http://<unraid-lan-ip>:8098` (or `http://api:8098` if you point the
|
||||||
|
tunnel at the `api` container directly on the compose network). Good for
|
||||||
|
small tracking-data batches; large dashcam uploads may hit the tunnel's
|
||||||
|
per-request size limit.
|
||||||
|
- **Router port-forward** (fallback if needed): forward an external port to
|
||||||
|
`<unraid-lan-ip>:8098` and let your router/firewall (e.g. Unifi) handle
|
||||||
|
the security side. No changes needed here either way.
|
||||||
|
|
||||||
|
For large dashcam uploads from home, consider adding a local DNS
|
||||||
|
override/rewrite (e.g. in AdGuard Home, same trick used for `yaris.local` on
|
||||||
|
the Pi) so the hostname resolves directly to the unraid LAN IP while on the
|
||||||
|
home network, bypassing the tunnel entirely for big transfers.
|
||||||
|
|
||||||
|
## Toggling dashcam uploads off
|
||||||
|
|
||||||
|
The Pi's web UI has a setting to pause dashcam file uploads while keeping
|
||||||
|
GPS/trip log uploads running — no server-side change needed, the Pi simply
|
||||||
|
stops calling `/api/v1/dashcam/upload` while the toggle is off.
|
||||||
|
|
||||||
|
## Local development / testing without unraid
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd server
|
||||||
|
cp .env.example .env # edit POSTGRES_PASSWORD
|
||||||
|
docker compose up --build
|
||||||
|
curl http://localhost:8098/health
|
||||||
|
```
|
||||||
+112
@@ -0,0 +1,112 @@
|
|||||||
|
"""yaris-obd remote server: receives trip/GPS log batches and dashcam files
|
||||||
|
uploaded by the Pi, storing them in Postgres + on-disk file storage.
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
import os
|
||||||
|
|
||||||
|
from flask import Flask, request, jsonify, g
|
||||||
|
|
||||||
|
from modules import db
|
||||||
|
from modules.auth import require_device_auth
|
||||||
|
|
||||||
|
app = Flask(__name__)
|
||||||
|
|
||||||
|
# Where uploaded dashcam files get written on disk (mounted to the unraid
|
||||||
|
# array via docker-compose - see DASHCAM_STORAGE_DIR in .env).
|
||||||
|
STORAGE_DIR = os.environ.get("DASHCAM_STORAGE_DIR", "/data/dashcam")
|
||||||
|
MAX_DASHCAM_UPLOAD_MB = int(os.environ.get("MAX_DASHCAM_UPLOAD_MB", "2048"))
|
||||||
|
app.config["MAX_CONTENT_LENGTH"] = MAX_DASHCAM_UPLOAD_MB * 1024 * 1024
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/health")
|
||||||
|
def health():
|
||||||
|
"""Unauthenticated liveness check - used by docker-compose healthcheck
|
||||||
|
and to sanity-test connectivity from the Pi before attempting uploads."""
|
||||||
|
return jsonify(status="ok"), 200
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/api/v1/tracking/batch")
|
||||||
|
@require_device_auth
|
||||||
|
def upload_tracking_batch():
|
||||||
|
"""Bulk-insert a batch of GPS/trip tracking points for the authenticated device.
|
||||||
|
|
||||||
|
Expected JSON body: {"points": [{"recorded_at": "...", "gps_lat": ..., ...}, ...]}
|
||||||
|
recorded_at must be an ISO-8601 timestamp string. Duplicate (device,
|
||||||
|
recorded_at) pairs are silently ignored, so the Pi can safely re-send a
|
||||||
|
batch after a network failure without creating duplicate rows.
|
||||||
|
"""
|
||||||
|
body = request.get_json(silent=True) or {}
|
||||||
|
points = body.get("points")
|
||||||
|
if not isinstance(points, list) or not points:
|
||||||
|
return jsonify(error="expected a non-empty 'points' list"), 400
|
||||||
|
|
||||||
|
for p in points:
|
||||||
|
if "recorded_at" not in p:
|
||||||
|
return jsonify(error="every point requires 'recorded_at'"), 400
|
||||||
|
|
||||||
|
inserted = db.insert_tracking_points(g.device_id, points)
|
||||||
|
return jsonify(status="ok", received=inserted), 201
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/api/v1/dashcam/upload")
|
||||||
|
@require_device_auth
|
||||||
|
def upload_dashcam_file():
|
||||||
|
"""Upload a single dashcam video file.
|
||||||
|
|
||||||
|
Multipart form fields:
|
||||||
|
file: the video file itself
|
||||||
|
relative_path: original sub-folder path on the dashcam SD card (form field)
|
||||||
|
sha256: hex digest of the file, computed by the Pi before sending
|
||||||
|
|
||||||
|
Skips (returns 200, not re-saved) if a file with the same
|
||||||
|
(device, relative_path, sha256) was already uploaded - safe to retry.
|
||||||
|
"""
|
||||||
|
if "file" not in request.files:
|
||||||
|
return jsonify(error="missing 'file' in multipart form"), 400
|
||||||
|
|
||||||
|
relative_path = request.form.get("relative_path", "").strip()
|
||||||
|
expected_sha256 = request.form.get("sha256", "").strip().lower()
|
||||||
|
if not relative_path or not expected_sha256:
|
||||||
|
return jsonify(error="missing 'relative_path' or 'sha256' form field"), 400
|
||||||
|
|
||||||
|
device_id = g.device_id
|
||||||
|
if db.dashcam_file_exists(device_id, relative_path, expected_sha256):
|
||||||
|
return jsonify(status="already_uploaded"), 200
|
||||||
|
|
||||||
|
uploaded = request.files["file"]
|
||||||
|
device_dir = os.path.join(STORAGE_DIR, g.device_name)
|
||||||
|
dest_path = os.path.normpath(os.path.join(device_dir, relative_path))
|
||||||
|
# Guard against a malicious/buggy relative_path (e.g. "../../etc") escaping
|
||||||
|
# the per-device storage directory.
|
||||||
|
if os.path.commonpath([device_dir, dest_path]) != os.path.normpath(device_dir):
|
||||||
|
return jsonify(error="invalid relative_path"), 400
|
||||||
|
os.makedirs(os.path.dirname(dest_path), exist_ok=True)
|
||||||
|
|
||||||
|
hasher = hashlib.sha256()
|
||||||
|
tmp_path = dest_path + ".part"
|
||||||
|
size_bytes = 0
|
||||||
|
with open(tmp_path, "wb") as f:
|
||||||
|
while True:
|
||||||
|
chunk = uploaded.stream.read(1024 * 1024)
|
||||||
|
if not chunk:
|
||||||
|
break
|
||||||
|
hasher.update(chunk)
|
||||||
|
size_bytes += len(chunk)
|
||||||
|
f.write(chunk)
|
||||||
|
|
||||||
|
actual_sha256 = hasher.hexdigest()
|
||||||
|
if actual_sha256 != expected_sha256:
|
||||||
|
os.remove(tmp_path)
|
||||||
|
return jsonify(error="sha256 mismatch, upload corrupted in transit"), 422
|
||||||
|
|
||||||
|
os.replace(tmp_path, dest_path)
|
||||||
|
db.insert_dashcam_file(device_id, relative_path, size_bytes, actual_sha256, dest_path)
|
||||||
|
return jsonify(status="ok", size_bytes=size_bytes), 201
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
app.run(host="0.0.0.0", port=8098)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
services:
|
||||||
|
db:
|
||||||
|
image: postgres:17
|
||||||
|
container_name: yaris-obd-db
|
||||||
|
restart: unless-stopped
|
||||||
|
environment:
|
||||||
|
POSTGRES_DB: ${POSTGRES_DB:-yarisobd}
|
||||||
|
POSTGRES_USER: ${POSTGRES_USER:-yarisobd}
|
||||||
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set a POSTGRES_PASSWORD in .env}
|
||||||
|
volumes:
|
||||||
|
- ${DATA_DIR:-./data}/postgres:/var/lib/postgresql/data
|
||||||
|
- ./sql/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-yarisobd}"]
|
||||||
|
interval: 10s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
# Not published to the host/LAN on purpose - only reachable from the
|
||||||
|
# `api` container over the internal compose network.
|
||||||
|
|
||||||
|
api:
|
||||||
|
build: .
|
||||||
|
container_name: yaris-obd-api
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
db:
|
||||||
|
condition: service_healthy
|
||||||
|
environment:
|
||||||
|
POSTGRES_HOST: db
|
||||||
|
POSTGRES_PORT: 5432
|
||||||
|
POSTGRES_DB: ${POSTGRES_DB:-yarisobd}
|
||||||
|
POSTGRES_USER: ${POSTGRES_USER:-yarisobd}
|
||||||
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set a POSTGRES_PASSWORD in .env}
|
||||||
|
DASHCAM_STORAGE_DIR: /data/dashcam
|
||||||
|
MAX_DASHCAM_UPLOAD_MB: ${MAX_DASHCAM_UPLOAD_MB:-2048}
|
||||||
|
volumes:
|
||||||
|
- ${DATA_DIR:-./data}/dashcam:/data/dashcam
|
||||||
|
ports:
|
||||||
|
- "${API_PORT:-8098}:8098"
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8098/health', timeout=3)"]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 3
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
"""API-key authentication for device uploads.
|
||||||
|
|
||||||
|
Devices authenticate with two headers:
|
||||||
|
X-Device-Name: <device name row in the devices table, e.g. "yaris-2013">
|
||||||
|
X-Api-Key: <raw API key, only ever compared as a sha256 hash>
|
||||||
|
|
||||||
|
The raw key is never stored - only its sha256 hex digest is kept in the
|
||||||
|
devices.api_key_hash column, generated once via scripts/create_device.py.
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
from functools import wraps
|
||||||
|
|
||||||
|
from flask import request, jsonify, g
|
||||||
|
|
||||||
|
from modules import db
|
||||||
|
|
||||||
|
|
||||||
|
def _hash_key(raw_key):
|
||||||
|
return hashlib.sha256(raw_key.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def require_device_auth(view):
|
||||||
|
@wraps(view)
|
||||||
|
def wrapped(*args, **kwargs):
|
||||||
|
device_name = request.headers.get("X-Device-Name", "")
|
||||||
|
api_key = request.headers.get("X-Api-Key", "")
|
||||||
|
|
||||||
|
if not device_name or not api_key:
|
||||||
|
return jsonify(error="missing X-Device-Name or X-Api-Key header"), 401
|
||||||
|
|
||||||
|
device = db.get_device_by_name(device_name)
|
||||||
|
if device is None or device["api_key_hash"] != _hash_key(api_key):
|
||||||
|
return jsonify(error="invalid device name or api key"), 401
|
||||||
|
|
||||||
|
g.device_id = device["id"]
|
||||||
|
g.device_name = device["name"]
|
||||||
|
db.touch_device_last_seen(device["id"])
|
||||||
|
return view(*args, **kwargs)
|
||||||
|
|
||||||
|
return wrapped
|
||||||
@@ -0,0 +1,108 @@
|
|||||||
|
"""Postgres connection pool + small query helpers for the yaris-obd remote server."""
|
||||||
|
import os
|
||||||
|
from contextlib import contextmanager
|
||||||
|
|
||||||
|
from psycopg_pool import ConnectionPool
|
||||||
|
|
||||||
|
_POOL = None
|
||||||
|
|
||||||
|
|
||||||
|
def _dsn():
|
||||||
|
return (
|
||||||
|
f"host={os.environ.get('POSTGRES_HOST', 'db')} "
|
||||||
|
f"port={os.environ.get('POSTGRES_PORT', '5432')} "
|
||||||
|
f"dbname={os.environ.get('POSTGRES_DB', 'yarisobd')} "
|
||||||
|
f"user={os.environ.get('POSTGRES_USER', 'yarisobd')} "
|
||||||
|
f"password={os.environ.get('POSTGRES_PASSWORD', '')}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def get_pool():
|
||||||
|
global _POOL
|
||||||
|
if _POOL is None:
|
||||||
|
_POOL = ConnectionPool(conninfo=_dsn(), min_size=1, max_size=5, open=True)
|
||||||
|
return _POOL
|
||||||
|
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def get_cursor(commit=False):
|
||||||
|
"""Yield a cursor from the pool, optionally committing on clean exit."""
|
||||||
|
pool = get_pool()
|
||||||
|
with pool.connection() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
yield cur
|
||||||
|
if commit:
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def get_device_by_name(name):
|
||||||
|
with get_cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
"SELECT id, name, api_key_hash FROM devices WHERE name = %s",
|
||||||
|
(name,),
|
||||||
|
)
|
||||||
|
row = cur.fetchone()
|
||||||
|
if row is None:
|
||||||
|
return None
|
||||||
|
return {"id": row[0], "name": row[1], "api_key_hash": row[2]}
|
||||||
|
|
||||||
|
|
||||||
|
def touch_device_last_seen(device_id):
|
||||||
|
with get_cursor(commit=True) as cur:
|
||||||
|
cur.execute(
|
||||||
|
"UPDATE devices SET last_seen_at = now() WHERE id = %s",
|
||||||
|
(device_id,),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def insert_tracking_points(device_id, points):
|
||||||
|
"""Bulk-insert tracking points, skipping any that already exist
|
||||||
|
(device_id, recorded_at) so a client can safely re-send a batch after a
|
||||||
|
partial-failure retry without creating duplicates."""
|
||||||
|
if not points:
|
||||||
|
return 0
|
||||||
|
with get_cursor(commit=True) as cur:
|
||||||
|
cur.executemany(
|
||||||
|
"""
|
||||||
|
INSERT INTO tracking_points
|
||||||
|
(device_id, vehicle_name, driver_name, recorded_at, gps_lat, gps_lon, speed, odometer)
|
||||||
|
VALUES (%(device_id)s, %(vehicle_name)s, %(driver_name)s, %(recorded_at)s,
|
||||||
|
%(gps_lat)s, %(gps_lon)s, %(speed)s, %(odometer)s)
|
||||||
|
ON CONFLICT (device_id, recorded_at) DO NOTHING
|
||||||
|
""",
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"device_id": device_id,
|
||||||
|
"vehicle_name": p.get("vehicle_name"),
|
||||||
|
"driver_name": p.get("driver_name"),
|
||||||
|
"recorded_at": p["recorded_at"],
|
||||||
|
"gps_lat": p.get("gps_lat"),
|
||||||
|
"gps_lon": p.get("gps_lon"),
|
||||||
|
"speed": p.get("speed"),
|
||||||
|
"odometer": p.get("odometer"),
|
||||||
|
}
|
||||||
|
for p in points
|
||||||
|
],
|
||||||
|
)
|
||||||
|
return len(points)
|
||||||
|
|
||||||
|
|
||||||
|
def dashcam_file_exists(device_id, relative_path, sha256):
|
||||||
|
with get_cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
"SELECT 1 FROM dashcam_files WHERE device_id = %s AND relative_path = %s AND sha256 = %s",
|
||||||
|
(device_id, relative_path, sha256),
|
||||||
|
)
|
||||||
|
return cur.fetchone() is not None
|
||||||
|
|
||||||
|
|
||||||
|
def insert_dashcam_file(device_id, relative_path, size_bytes, sha256, storage_path):
|
||||||
|
with get_cursor(commit=True) as cur:
|
||||||
|
cur.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO dashcam_files (device_id, relative_path, size_bytes, sha256, storage_path)
|
||||||
|
VALUES (%s, %s, %s, %s, %s)
|
||||||
|
ON CONFLICT (device_id, relative_path, sha256) DO NOTHING
|
||||||
|
""",
|
||||||
|
(device_id, relative_path, size_bytes, sha256, storage_path),
|
||||||
|
)
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
Flask
|
||||||
|
gunicorn
|
||||||
|
psycopg[binary]
|
||||||
|
psycopg-pool
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Register a new device (e.g. the Pi) and print its API key.
|
||||||
|
|
||||||
|
Run this INSIDE the api container so it shares the same DB connection env
|
||||||
|
vars already configured by docker-compose, e.g.:
|
||||||
|
|
||||||
|
docker compose exec api python3 scripts/create_device.py yaris-2013
|
||||||
|
|
||||||
|
The raw API key is only ever shown once here - copy it into the Pi's
|
||||||
|
upload-client config immediately. Only its sha256 hash is stored server-side.
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
import secrets
|
||||||
|
import sys
|
||||||
|
|
||||||
|
sys.path.insert(0, "/app")
|
||||||
|
|
||||||
|
from modules import db # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
if len(sys.argv) != 2:
|
||||||
|
print(f"usage: {sys.argv[0]} <device-name>", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
device_name = sys.argv[1]
|
||||||
|
raw_key = secrets.token_urlsafe(32)
|
||||||
|
key_hash = hashlib.sha256(raw_key.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
with db.get_cursor(commit=True) as cur:
|
||||||
|
cur.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO devices (name, api_key_hash) VALUES (%s, %s)
|
||||||
|
ON CONFLICT (name) DO UPDATE SET api_key_hash = EXCLUDED.api_key_hash
|
||||||
|
RETURNING id
|
||||||
|
""",
|
||||||
|
(device_name, key_hash),
|
||||||
|
)
|
||||||
|
device_id = cur.fetchone()[0]
|
||||||
|
|
||||||
|
print(f"Device '{device_name}' registered (id={device_id}).")
|
||||||
|
print()
|
||||||
|
print("Save these on the Pi (e.g. in server/.env-style local config, gitignored):")
|
||||||
|
print(f" YARIS_UPLOAD_DEVICE_NAME={device_name}")
|
||||||
|
print(f" YARIS_UPLOAD_API_KEY={raw_key}")
|
||||||
|
print()
|
||||||
|
print("This key will not be shown again - only its hash is stored server-side.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
-- yaris-obd remote server schema
|
||||||
|
-- Applied automatically on first Postgres container start (mounted into
|
||||||
|
-- /docker-entrypoint-initdb.d/), so it only ever runs once against a fresh
|
||||||
|
-- empty database volume.
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS devices (
|
||||||
|
id SERIAL PRIMARY KEY,
|
||||||
|
name TEXT NOT NULL UNIQUE, -- e.g. "yaris-2013"
|
||||||
|
api_key_hash TEXT NOT NULL, -- sha256 hex digest of the device's API key, never the raw key
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
last_seen_at TIMESTAMPTZ
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS tracking_points (
|
||||||
|
id BIGSERIAL PRIMARY KEY,
|
||||||
|
device_id INTEGER NOT NULL REFERENCES devices(id) ON DELETE CASCADE,
|
||||||
|
vehicle_name TEXT,
|
||||||
|
driver_name TEXT,
|
||||||
|
recorded_at TIMESTAMPTZ NOT NULL, -- timestamp taken on the Pi when the point was captured
|
||||||
|
gps_lat DOUBLE PRECISION,
|
||||||
|
gps_lon DOUBLE PRECISION,
|
||||||
|
speed DOUBLE PRECISION,
|
||||||
|
odometer DOUBLE PRECISION,
|
||||||
|
received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
-- Prevent the same locally-queued row from being inserted twice on retry
|
||||||
|
UNIQUE (device_id, recorded_at)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_tracking_points_device_time
|
||||||
|
ON tracking_points (device_id, recorded_at DESC);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS dashcam_files (
|
||||||
|
id BIGSERIAL PRIMARY KEY,
|
||||||
|
device_id INTEGER NOT NULL REFERENCES devices(id) ON DELETE CASCADE,
|
||||||
|
relative_path TEXT NOT NULL, -- original sub-folder path on the dashcam SD card
|
||||||
|
size_bytes BIGINT NOT NULL,
|
||||||
|
sha256 TEXT NOT NULL,
|
||||||
|
storage_path TEXT NOT NULL, -- where the file landed on the server's disk
|
||||||
|
uploaded_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
UNIQUE (device_id, relative_path, sha256)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_dashcam_files_device
|
||||||
|
ON dashcam_files (device_id, uploaded_at DESC);
|
||||||
Reference in New Issue
Block a user