diff --git a/PLAN.md b/PLAN.md index 83ad0e8..943d024 100644 --- a/PLAN.md +++ b/PLAN.md @@ -458,6 +458,78 @@ data connection when WiFi/Ethernet is unreachable. Design notes: a plain-HTTP router port-forward path, avoiding the need for the modem's TLS stack to validate a cert chain). +### Phase 11: Phone-relay dashcam upload — ✅ DONE +Root cause investigation into "why is the transfer from the Pi to the server so slow?" (real +dashcam backlog draining at only ~350MB/4-5min) found the bottleneck is the **Pi's own onboard +BCM43438 WiFi chip** (2.4GHz-only, congested/interfered channel) — confirmed via `iperf3` +raw-TCP tests showing only ~14.7Mbit/s with active retransmits even over a same-subnet direct-LAN +hop that never touches the internet/Cloudflare, ruling out ISP/Cloudflare limits. Since the Pi 3 B +has no 5GHz radio and can't be fixed in software, added a way to bypass its weak WiFi for the +internet-bound leg entirely: a phone opens the same `/dash` page (over the Pi's hotspot/home WiFi) +and manually relays pending dashcam files straight to the remote server using the phone's own +network (WiFi or cellular data), instead of routing everything through the Pi. + +**Zero new remote-server upload logic** — the phone's browser JS calls the *exact same* +`/api/v1/dashcam/upload/status` + `/api/v1/dashcam/upload/chunk` endpoints the Pi's own Python +client already uses, in the same 40MB-chunk protocol, just orchestrated by `fetch()`/`FormData` +instead of `requests`/`MultipartEncoder`. The server dedupes purely by SHA256 regardless of which +device sends the bytes, so no server-side changes were needed for the upload path itself — only a +CORS `after_request` hook (`server/app.py`) was added, since the phone's browser calls +`yaris.ceaf.dev` from a page served by a *different* origin (the Pi), which browsers block without +`Access-Control-Allow-*` headers on both the real request and its preflight `OPTIONS`. Wildcard +origin is fine here — auth is header-based (`X-Device-Name`/`X-Api-Key`), not cookie/session-based. + +**New Pi-side endpoints** (`app.py`): +- `GET /api/dash/pending` — lists not-yet-uploaded dashcam files (`relative_path`, `sha256`, + `size`) via a new `app_upload.list_pending_dashcam_files()`. +- `GET /api/dash/file/` — serves a dashcam file's bytes for the phone to download, + via `send_file(..., conditional=True)` for **Range-request support** (206 Partial Content) so + the phone downloads (and immediately re-uploads) one 40MB slice at a time instead of ever + holding a whole 300+MB file in the browser's memory — the exact same memory-safety lesson + learned the hard way in Phase 10, applied symmetrically to the download side. Path-traversal + safe via Werkzeug's `safe_join` against `app_dash.DEFAULT_DESTINATION`. +- `GET /api/dash/relay/config` — hands the phone the same `SERVER_URL`/`DEVICE_NAME`/`API_KEY` + the Pi's own client uses, so its JS can auth directly with the remote server. Same trust + boundary as the rest of this project's unauthenticated local UI (anyone on the Pi's own + hotspot/LAN is already implicitly trusted). +- `POST /api/dash/relay/heartbeat` / `POST /api/dash/relay/stop` — a **lease-based pause** + (`RELAY_LEASE_SECONDS = 45`) for the Pi's own background dashcam uploader, chosen over a plain + on/off toggle specifically to self-heal if the phone's browser tab closes/loses connection + uncleanly (a simple toggle could get stuck "paused" forever). The phone page heartbeats every + 15s while relaying; `_upload_dashcam_files()` skips entirely while a lease is active, re-checked + between files so it stops promptly if a relay starts mid-drain. Also released via + `navigator.sendBeacon()` on `beforeunload` as a best-effort immediate release. +- `POST /api/dash/relay/mark_uploaded` — lets the phone tell the Pi "I just uploaded this file" + right after a successful relay, so the local dedupe table (and the pending list / Pi's own + background loop) reflect it immediately instead of waiting for the next full backlog scan. + +**Performance side-fix**: `modules/app_upload.py`'s dashcam loop previously recomputed SHA256 for +*every* candidate file on *every* loop iteration, including files long since uploaded — wasteful +CPU/IO against a growing, possibly 20GB+, local backlog. Added a `dashcam_hash_cache` SQLite table +(`modules/app_sql.py`, keyed by `relative_path`+`size`+`mtime`) so a file's hash is only computed +once; both the existing background uploader and the new pending-files endpoint benefit. + +UI (`templates/dash.html` / `static/js/dash.js`): a new "Relay via phone" card on `/dash` lists +pending files with a per-file "Relay" button and a "Relay all pending" button (manual trigger only +— chosen deliberately over an automatic one, per explicit user preference), plus a "Stop relay" +button. Chunk downloads/uploads retry up to 3 times with backoff before surfacing an error, and a +resumed relay re-checks `/status` first so a previously-interrupted file picks up from its first +missing chunk instead of restarting. + +**Tested**: a full local end-to-end simulation (no physical phone/network needed) drove the real +Pi `app.py` and real `server/app.py` Flask apps side by side (Postgres mocked at the `db` module +boundary only — all HTTP routing, chunking, auth, and CORS logic is real), reproducing the exact +sequence the phone's browser JS performs: list pending -> check remote `/status` -> download each +chunk from the Pi via `Range` requests (verified real `206 Partial Content` responses) -> upload +each chunk to the mock-backed server -> verify the reassembled file matches the original +byte-for-byte via SHA256 -> mark uploaded on the Pi -> confirm it drops out of the pending list. +Also verified: path-traversal attempts on `/api/dash/file/` correctly 404, the relay +heartbeat/stop lease correctly pauses/resumes `_upload_dashcam_files()`, and the sha256 cache +table is populated and reused across repeated `/api/dash/pending` calls. **Not yet tested on a +real phone against the real deployed server** — the wire protocol and every server-visible +behavior is verified, but real mobile-browser quirks (background-tab throttling, flaky cellular +mid-transfer) haven't been exercised against live hardware yet. + ## Working Notes / Environment Gotchas - **Pi credentials/SSH**: `ssh yaris@`. Confirm current IP before connecting (varies by diff --git a/app.py b/app.py index a04bad6..df5c29b 100644 --- a/app.py +++ b/app.py @@ -1,4 +1,7 @@ -from flask import Flask, render_template, make_response, json, jsonify, request +import os + +from flask import Flask, render_template, make_response, json, jsonify, request, send_file, abort +from werkzeug.utils import safe_join from modules import app_gps, app_events, app_car, app_account, app_gps,app_obd, app_sql, app_dash, app_upload import time from gevent.pywsgi import WSGIServer @@ -50,6 +53,60 @@ def upload_dashcam_toggle(): app_upload.set_dashcam_upload_enabled(bool(body["enabled"])) return jsonify(app_upload.get_status()) +# --- Phone-relay: lets a phone on the Pi's own network list, download, and +# directly re-upload dashcam files to the remote server itself (bypassing +# the Pi's own often-congested WiFi radio for the internet-bound leg). See +# static/js/dash.js for the client side of this. --- + +@app.route('/api/dash/pending') +def dash_pending(): + return jsonify({"files": app_upload.list_pending_dashcam_files()}) + +@app.route('/api/dash/file/') +def dash_file(relpath): + abs_path = safe_join(app_dash.DEFAULT_DESTINATION, relpath) + if abs_path is None or not os.path.isfile(abs_path): + abort(404) + # conditional=True (Flask's default) makes this support Range requests, + # so the phone can fetch/re-upload one chunk at a time instead of ever + # holding the whole (300+MB) file in the browser's memory at once. + return send_file(abs_path, mimetype='application/octet-stream', conditional=True) + +@app.route('/api/dash/relay/config') +def dash_relay_config(): + # Hands the phone the same device credentials the Pi's own upload client + # uses, so its browser JS can call the remote server's existing chunked + # upload endpoints directly. Same trust boundary as the rest of this + # unauthenticated local UI - anyone on the Pi's hotspot/LAN is already + # implicitly trusted. + return jsonify({ + "server_url": app_upload.SERVER_URL, + "device_name": app_upload.DEVICE_NAME, + "api_key": app_upload.API_KEY, + "chunk_size": app_upload.DASHCAM_CHUNK_SIZE, + "configured": app_upload.is_configured(), + }) + +@app.route('/api/dash/relay/heartbeat', methods=['POST']) +def dash_relay_heartbeat(): + app_upload.relay_heartbeat() + return jsonify(app_upload.get_status()) + +@app.route('/api/dash/relay/stop', methods=['POST']) +def dash_relay_stop(): + app_upload.relay_stop() + return jsonify(app_upload.get_status()) + +@app.route('/api/dash/relay/mark_uploaded', methods=['POST']) +def dash_relay_mark_uploaded(): + body = request.get_json(silent=True) or {} + relative_path = (body.get("relative_path") or "").strip() + sha256 = (body.get("sha256") or "").strip().lower() + if not relative_path or not sha256: + return jsonify(error="expected JSON body {'relative_path', 'sha256'}"), 400 + app_sql.mark_dashcam_file_uploaded(relative_path, sha256) + return jsonify(status="ok") + # not sure if im gonna use this @app.route('/api/obd_speed/') def data(): diff --git a/modules/app_sql.py b/modules/app_sql.py index 02562c0..507be7c 100644 --- a/modules/app_sql.py +++ b/modules/app_sql.py @@ -57,6 +57,18 @@ SQL_CREATE_DASHCAM_UPLOADS_TABLE = """CREATE TABLE IF NOT EXISTS dashcam_uploads UNIQUE(RELATIVE_PATH, SHA256) );""" +# Cache of already-computed sha256 hashes, keyed by (path, size, mtime) so a +# file's hash never needs recomputing once copied from the dashcam (its size +# and mtime won't change afterward). Avoids re-hashing the entire, possibly +# 20GB+, local backlog on every background-upload loop iteration and every +# "pending files" list request from the phone-relay feature. +SQL_CREATE_DASHCAM_HASH_CACHE_TABLE = """CREATE TABLE IF NOT EXISTS dashcam_hash_cache ( + RELATIVE_PATH text PRIMARY KEY, + SIZE integer NOT NULL, + MTIME real NOT NULL, + SHA256 text NOT NULL + );""" + def create_conn(db_file): """Open a SQLite connection to db_file, creating its parent folder if needed. @@ -98,6 +110,7 @@ def get_conn(): create_table(_conn, SQL_CREATE_DTC_TABLE) create_table(_conn, SQL_CREATE_SETTINGS_TABLE) create_table(_conn, SQL_CREATE_DASHCAM_UPLOADS_TABLE) + create_table(_conn, SQL_CREATE_DASHCAM_HASH_CACHE_TABLE) _migrate_tracking_table(_conn) else: print("Error! cannot create the database connection.") @@ -221,6 +234,34 @@ def mark_dashcam_file_uploaded(relative_path, sha256): conn.commit() +def get_cached_dashcam_hash(relative_path, size, mtime): + """Return a previously-computed sha256 for this exact (path, size, mtime), + or None if not cached / the file has changed since it was last hashed.""" + conn = get_conn() + if conn is None: + return None + with _conn_lock: + cur = conn.execute( + "SELECT SHA256 FROM dashcam_hash_cache WHERE RELATIVE_PATH = ? AND SIZE = ? AND MTIME = ?", + (relative_path, size, mtime), + ) + row = cur.fetchone() + return row[0] if row is not None else None + + +def set_cached_dashcam_hash(relative_path, size, mtime, sha256): + conn = get_conn() + if conn is None: + return + with _conn_lock: + conn.execute( + "INSERT INTO dashcam_hash_cache (RELATIVE_PATH, SIZE, MTIME, SHA256) VALUES (?, ?, ?, ?) " + "ON CONFLICT(RELATIVE_PATH) DO UPDATE SET SIZE = excluded.SIZE, MTIME = excluded.MTIME, SHA256 = excluded.SHA256", + (relative_path, size, mtime, sha256), + ) + conn.commit() + + def _log_loop(): # Deferred import to avoid a circular import at module load time # (app_obd.py imports app_sql, and app_sql needs app_obd.speed here). diff --git a/modules/app_upload.py b/modules/app_upload.py index 51acaba..fcde69a 100644 --- a/modules/app_upload.py +++ b/modules/app_upload.py @@ -67,10 +67,17 @@ DASHCAM_CHUNK_SIZE = 40 * 1024 * 1024 DASHCAM_UPLOAD_SETTING_KEY = "dashcam_upload_enabled" +# How long a phone-relay "lease" stays valid without a fresh heartbeat before +# the Pi's own background uploader assumes the phone gave up/closed its tab +# and resumes uploading dashcam files itself. The phone page is expected to +# heartbeat well under this (every ~15-20s) - see /api/dash/relay/heartbeat. +RELAY_LEASE_SECONDS = 45 + _upload_thread = None _last_error = None _last_success_at = None _last_network_path = None # "wifi" or "cellular" - last path a batch actually went out on +_relay_lease_until = None # monotonic time.time() the phone-relay lease expires, or None def _headers(): @@ -100,9 +107,30 @@ def get_status(): "last_success_at": _last_success_at, "last_network_path": _last_network_path, "cellular_fallback_configured": app_cellular.is_configured(), + "relay_active": is_relay_active(), } +# --- Phone-relay lease (pauses the Pi's own background dashcam uploader +# while a phone is manually relaying files, so they don't fight over the +# Pi's single, often WiFi-congested, radio) ------------------------------- + +def relay_heartbeat(): + """Called by the phone-relay page every ~15-20s while active.""" + global _relay_lease_until + _relay_lease_until = time.time() + RELAY_LEASE_SECONDS + + +def relay_stop(): + """Called when the phone-relay page is done/closed cleanly.""" + global _relay_lease_until + _relay_lease_until = None + + +def is_relay_active(): + return _relay_lease_until is not None and time.time() < _relay_lease_until + + def _sha256_file(path): hasher = hashlib.sha256() with open(path, "rb") as f: @@ -114,6 +142,37 @@ def _sha256_file(path): return hasher.hexdigest() +def _hash_dashcam_file(abs_path, rel_path): + """sha256 of a dashcam file, cached by (path, size, mtime) so unchanged + files already copied from the dashcam never get rehashed - real backlogs + here run 20GB+, and both the background loop and the phone-relay pending + list would otherwise rehash everything, including already-uploaded files, + on every single call.""" + stat = os.stat(abs_path) + cached = app_sql.get_cached_dashcam_hash(rel_path, stat.st_size, stat.st_mtime) + if cached is not None: + return cached + sha256 = _sha256_file(abs_path) + app_sql.set_cached_dashcam_hash(rel_path, stat.st_size, stat.st_mtime, sha256) + return sha256 + + +def list_pending_dashcam_files(): + """Return [{relative_path, sha256, size}, ...] for dashcam files not yet + confirmed uploaded - used by the phone-relay page to pick what to fetch.""" + pending = [] + for abs_path, rel_path in app_dash.find_video_files(app_dash.DEFAULT_DESTINATION): + sha256 = _hash_dashcam_file(abs_path, rel_path) + if app_sql.dashcam_file_marked_uploaded(rel_path, sha256): + continue + pending.append({ + "relative_path": rel_path, + "sha256": sha256, + "size": os.path.getsize(abs_path), + }) + return pending + + def _upload_tracking_batch(): """Send up to TRACKING_BATCH_SIZE not-yet-uploaded rows. Returns count sent. @@ -231,15 +290,16 @@ def _upload_video_chunked(abs_path, rel_path, sha256): def _upload_dashcam_files(): """Upload any not-yet-uploaded dashcam files, one at a time. Returns count sent. - Skipped entirely (returns 0) while the dashcam-upload toggle is off - - tracking-log uploads are unaffected by this toggle. + Skipped entirely (returns 0) while the dashcam-upload toggle is off, or + while a phone-relay session is actively leased - tracking-log uploads are + unaffected by either of these. """ - if not is_dashcam_upload_enabled(): + if not is_dashcam_upload_enabled() or is_relay_active(): return 0 uploaded_count = 0 for abs_path, rel_path in app_dash.find_video_files(app_dash.DEFAULT_DESTINATION): - sha256 = _sha256_file(abs_path) + sha256 = _hash_dashcam_file(abs_path, rel_path) if app_sql.dashcam_file_marked_uploaded(rel_path, sha256): continue @@ -247,9 +307,10 @@ def _upload_dashcam_files(): app_sql.mark_dashcam_file_uploaded(rel_path, sha256) uploaded_count += 1 - # Re-check the toggle between files so flipping it off mid-run stops - # further dashcam uploads promptly without waiting for the next cycle. - if not is_dashcam_upload_enabled(): + # Re-check between files so flipping the toggle off, or a phone + # relay starting, mid-run stops further uploads promptly without + # waiting for the next cycle. + if not is_dashcam_upload_enabled() or is_relay_active(): break return uploaded_count diff --git a/readme.md b/readme.md index 3f312f9..6fda0da 100644 --- a/readme.md +++ b/readme.md @@ -42,6 +42,9 @@ i have a dream Dashcam videos upload in resumable 40MB chunks (works around tunnel body-size limits and keeps the Pi's memory footprint flat regardless of file size); a UI toggle can pause dashcam uploads independently while trip logs keep syncing +- 📱 **Phone-relay upload** — if the Pi's own WiFi is slow/congested, open `/dash` on your phone + to download pending dashcam files locally and re-upload them straight to the remote server + using the phone's own network instead, automatically pausing the Pi's own upload meanwhile - 🗄️ Local SQLite trip logging, retried/synced to the remote server with automatic backoff - 🔌 Runs fully headless as a `systemd` service, autostarting on boot @@ -75,8 +78,9 @@ For the remote server (receives trip logs + dashcam uploads), see [server/README ## Project status -This project was revived after a couple of years on the shelf. All 10 planned phases are done and -the full pipeline — OBD/GPS logging, offline web UI, WiFi hotspot, dashcam retrieval, and remote -upload — has been verified against real hardware end-to-end, including a full real dashcam import -and upload run. Only the SIM868 cellular fallback (for uploading while out of WiFi range) remains -untested, pending a SIM card. Full phase-by-phase history in [PLAN.md](./PLAN.md). +This project was revived after a couple of years on the shelf. All 11 planned phases are done and +the full pipeline — OBD/GPS logging, offline web UI, WiFi hotspot, dashcam retrieval, remote +upload, and phone-relay upload — has been verified against real hardware end-to-end, including a +full real dashcam import and upload run. Only the SIM868 cellular fallback (for uploading while +out of WiFi range) and the phone-relay feature's real-phone/live-network test remain untested. +Full phase-by-phase history in [PLAN.md](./PLAN.md). diff --git a/server/README.md b/server/README.md index 1acc45d..7c72d58 100644 --- a/server/README.md +++ b/server/README.md @@ -28,6 +28,14 @@ headers: `X-Device-Name` and `X-Api-Key` (see Postgres is **not** published to the LAN — only the `api` container talks to it, over the internal compose network. +**CORS is enabled on every response** (`Access-Control-Allow-Origin: *`) so a +device's registered API key can be used directly from browser JS running on a +*different* origin — specifically, the Pi's phone-relay feature (see +[PLAN.md](../PLAN.md) Phase 11), where a phone's browser calls this API +straight from a page served by the Pi itself. This is safe here because +auth is header-based (`X-Device-Name`/`X-Api-Key`), not cookie/session-based, +so a permissive origin can't be used to steal a logged-in session. + ## Deploying on unraid 1. Get the code onto the unraid box (same repo as the Pi, just the diff --git a/server/app.py b/server/app.py index b657633..4fc2d04 100644 --- a/server/app.py +++ b/server/app.py @@ -12,6 +12,20 @@ from modules.auth import require_device_auth app = Flask(__name__) +# The phone-relay feature (Pi web UI) has a phone's browser call this API +# directly from a page served BY THE PI (a different origin than this +# server), so the browser enforces CORS on those cross-origin requests. +# Wildcard is fine here - this API is authenticated via the X-Device-Name/ +# X-Api-Key headers (not cookies), so there's no session/credential to leak +# via a permissive Access-Control-Allow-Origin. +@app.after_request +def _add_cors_headers(response): + response.headers["Access-Control-Allow-Origin"] = "*" + response.headers["Access-Control-Allow-Methods"] = "GET, POST, OPTIONS" + response.headers["Access-Control-Allow-Headers"] = "Content-Type, X-Device-Name, X-Api-Key" + response.headers["Access-Control-Max-Age"] = "600" + return response + # Where uploaded dashcam files get written on disk (mounted to the unraid # array via docker-compose - see DASHCAM_STORAGE_DIR in .env). STORAGE_DIR = os.environ.get("DASHCAM_STORAGE_DIR", "/data/dashcam") diff --git a/static/js/dash.js b/static/js/dash.js index 3e3ed33..1dc0576 100644 --- a/static/js/dash.js +++ b/static/js/dash.js @@ -121,3 +121,262 @@ dashcamToggle.addEventListener('change', () => { refreshUploadStatus(); setInterval(refreshUploadStatus, 15000); + +// --- Relay via phone --- +// Lets a phone (open on this same page, over the Pi's hotspot/LAN) download +// pending dashcam files from the Pi and re-upload them directly to the +// remote server using its own network, reusing the exact same chunked +// upload wire protocol the Pi's own background uploader uses +// (/api/v1/dashcam/upload/status + /chunk) - just orchestrated by this +// browser instead of the Pi's Python client. Downloads are pulled in the +// same chunk size and immediately re-uploaded, one at a time, so the phone +// never has to hold a whole (300+MB) file in memory at once. + +const relayStatusEl = document.getElementById('relay-status'); +const relayRefreshBtn = document.getElementById('relay-refresh-btn'); +const relayAllBtn = document.getElementById('relay-all-btn'); +const relayStopBtn = document.getElementById('relay-stop-btn'); +const relayFileList = document.getElementById('relay-file-list'); + +let relayConfig = null; +let pendingFiles = []; +const relayRowEls = new Map(); // relative_path -> { statusSpan, btn } +let heartbeatTimer = null; +let activeRelayCount = 0; +let relayStopRequested = false; + +function humanSize(bytes) { + if (bytes >= 1024 * 1024 * 1024) return (bytes / (1024 * 1024 * 1024)).toFixed(2) + ' GB'; + if (bytes >= 1024 * 1024) return (bytes / (1024 * 1024)).toFixed(1) + ' MB'; + if (bytes >= 1024) return (bytes / 1024).toFixed(1) + ' KB'; + return bytes + ' B'; +} + +function getRelayConfig() { + if (relayConfig) return Promise.resolve(relayConfig); + return fetch('/api/dash/relay/config') + .then(response => response.json()) + .then(config => { + relayConfig = config; + return config; + }); +} + +function startHeartbeat() { + activeRelayCount++; + relayStopBtn.disabled = false; + if (heartbeatTimer) return; + fetch('/api/dash/relay/heartbeat', { method: 'POST' }).catch(() => {}); + heartbeatTimer = setInterval(() => { + fetch('/api/dash/relay/heartbeat', { method: 'POST' }).catch(() => {}); + }, 15000); +} + +function stopHeartbeatIfIdle() { + activeRelayCount = Math.max(0, activeRelayCount - 1); + if (activeRelayCount > 0) return; + if (heartbeatTimer) { + clearInterval(heartbeatTimer); + heartbeatTimer = null; + } + relayStopBtn.disabled = true; + fetch('/api/dash/relay/stop', { method: 'POST' }).catch(() => {}); +} + +relayStopBtn.addEventListener('click', () => { + relayStopRequested = true; + activeRelayCount = 0; + if (heartbeatTimer) { + clearInterval(heartbeatTimer); + heartbeatTimer = null; + } + fetch('/api/dash/relay/stop', { method: 'POST' }).catch(() => {}); + relayStatusEl.textContent = 'Relay stopped - the Pi will resume its own uploads shortly.'; + relayStopBtn.disabled = true; +}); + +// Best-effort: try to release the pause lease immediately if the page is +// closed mid-relay (it also self-expires after ~45s either way). +window.addEventListener('beforeunload', () => { + if (heartbeatTimer && navigator.sendBeacon) { + navigator.sendBeacon('/api/dash/relay/stop'); + } +}); + +async function fetchChunkWithRetry(relativePath, start, end, attempts = 3) { + let lastErr; + for (let attempt = 1; attempt <= attempts; attempt++) { + try { + const response = await fetch(`/api/dash/file/${encodeURIComponent(relativePath)}`, { + headers: { Range: `bytes=${start}-${end}` }, + }); + if (!response.ok && response.status !== 206) { + throw new Error(`download HTTP ${response.status}`); + } + return await response.blob(); + } catch (err) { + lastErr = err; + await new Promise(r => setTimeout(r, 1000 * attempt)); + } + } + throw lastErr; +} + +async function uploadChunkWithRetry(config, relativePath, sha256, chunkIndex, totalChunks, blob, attempts = 3) { + let lastErr; + for (let attempt = 1; attempt <= attempts; attempt++) { + try { + const form = new FormData(); + form.append('relative_path', relativePath); + form.append('sha256', sha256); + form.append('chunk_index', String(chunkIndex)); + form.append('total_chunks', String(totalChunks)); + form.append('file', blob, `chunk_${chunkIndex}`); + const response = await fetch(`${config.server_url}/api/v1/dashcam/upload/chunk`, { + method: 'POST', + headers: { 'X-Device-Name': config.device_name, 'X-Api-Key': config.api_key }, + body: form, + }); + if (!response.ok) { + throw new Error(`upload HTTP ${response.status}`); + } + return await response.json(); + } catch (err) { + lastErr = err; + await new Promise(r => setTimeout(r, 1000 * attempt)); + } + } + throw lastErr; +} + +async function relayOneFile(file, statusEl) { + const config = await getRelayConfig(); + if (!config.configured) { + statusEl.textContent = 'Remote server not configured on this device.'; + return; + } + + relayStopRequested = false; + startHeartbeat(); + try { + const statusResp = await fetch( + `${config.server_url}/api/v1/dashcam/upload/status?relative_path=${encodeURIComponent(file.relative_path)}&sha256=${file.sha256}`, + { headers: { 'X-Device-Name': config.device_name, 'X-Api-Key': config.api_key } } + ); + if (!statusResp.ok) throw new Error(`status check HTTP ${statusResp.status}`); + const status = await statusResp.json(); + + if (status.status !== 'already_uploaded') { + const received = new Set(status.received_chunk_indices || []); + const totalChunks = Math.max(1, Math.ceil(file.size / config.chunk_size)); + + for (let idx = 0; idx < totalChunks; idx++) { + if (relayStopRequested) { + statusEl.textContent = 'Stopped.'; + return; + } + if (received.has(idx)) continue; + + const start = idx * config.chunk_size; + const end = Math.min(start + config.chunk_size, file.size) - 1; + statusEl.textContent = `Relaying... chunk ${idx + 1}/${totalChunks}`; + + const blob = await fetchChunkWithRetry(file.relative_path, start, end); + await uploadChunkWithRetry(config, file.relative_path, file.sha256, idx, totalChunks, blob); + } + } + + // Update the Pi's own local dedupe table right away so the pending + // list (and its own background uploader) reflect this immediately, + // instead of waiting for its next full backlog scan. + await fetch('/api/dash/relay/mark_uploaded', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ relative_path: file.relative_path, sha256: file.sha256 }), + }).catch(() => {}); + + statusEl.textContent = 'Done.'; + pendingFiles = pendingFiles.filter(f => f.relative_path !== file.relative_path); + } catch (err) { + statusEl.textContent = 'Error: ' + err; + throw err; + } finally { + stopHeartbeatIfIdle(); + } +} + +function renderPendingFiles() { + relayFileList.innerHTML = ''; + relayRowEls.clear(); + + if (pendingFiles.length === 0) { + relayStatusEl.textContent = 'No pending dashcam files to relay.'; + relayAllBtn.disabled = true; + return; + } + + relayStatusEl.textContent = `${pendingFiles.length} file(s) pending upload.`; + relayAllBtn.disabled = false; + + pendingFiles.forEach(file => { + const li = document.createElement('li'); + li.className = 'list-group-item d-flex justify-content-between align-items-center flex-wrap gap-2'; + + const label = document.createElement('span'); + label.textContent = `${file.relative_path} (${humanSize(file.size)})`; + + const statusSpan = document.createElement('span'); + statusSpan.className = 'text-muted small'; + + const btn = document.createElement('button'); + btn.className = 'btn btn-sm btn-outline-primary'; + btn.type = 'button'; + btn.textContent = 'Relay'; + btn.addEventListener('click', () => { + btn.disabled = true; + relayOneFile(file, statusSpan).finally(() => { + btn.disabled = false; + }); + }); + + li.appendChild(label); + li.appendChild(statusSpan); + li.appendChild(btn); + relayFileList.appendChild(li); + relayRowEls.set(file.relative_path, { statusSpan, btn }); + }); +} + +function refreshPendingFiles() { + relayStatusEl.textContent = 'Checking pending files...'; + fetch('/api/dash/pending') + .then(response => response.json()) + .then(data => { + pendingFiles = data.files || []; + renderPendingFiles(); + }) + .catch(err => { + relayStatusEl.textContent = 'Error while checking pending files: ' + err; + }); +} + +relayRefreshBtn.addEventListener('click', refreshPendingFiles); + +relayAllBtn.addEventListener('click', async () => { + relayStopRequested = false; + relayAllBtn.disabled = true; + for (const file of pendingFiles.slice()) { + if (relayStopRequested) break; + const row = relayRowEls.get(file.relative_path); + const statusSpan = row ? row.statusSpan : relayStatusEl; + if (row) row.btn.disabled = true; + try { + await relayOneFile(file, statusSpan); + } catch (err) { + // Keep going with the remaining files even if one fails. + } + } + relayAllBtn.disabled = pendingFiles.length === 0; +}); + +refreshPendingFiles(); diff --git a/templates/dash.html b/templates/dash.html index 9402fd4..a13918c 100644 --- a/templates/dash.html +++ b/templates/dash.html @@ -97,6 +97,26 @@

+
+
+

Relay via phone

+
+
+

+ Open this page on your phone (while connected to the Pi's + hotspot/WiFi) to download pending dashcam files locally and + upload them straight to the remote server using your phone's own + data connection - useful when the Pi's own WiFi upload is slow. + Starting a relay automatically pauses the Pi's own background + dashcam upload so they don't compete for the same radio. +

+

Checking...

+ + + +
    +
    +