Add phone-relay dashcam upload to bypass the Pi's slow WiFi
Root-caused the slow-upload complaint to the Pi's own congested 2.4GHz WiFi radio (confirmed via iperf3 - same degraded throughput even on pure local LAN traffic, ruling out Cloudflare/ISP). Since the Pi 3 B has no 5GHz radio, add a way to bypass it for the internet-bound leg: a phone opens /dash and manually relays pending dashcam files straight to the remote server over its own network, reusing the exact same chunked upload protocol the Pi's client already uses. - server/app.py: add CORS headers (after_request) so the phone's browser can call the remote server cross-origin from a page served by the Pi. - app.py: new endpoints - GET /api/dash/pending (list not-yet-uploaded files), GET /api/dash/file/<path> (Range-request-capable download, path-traversal safe), GET /api/dash/relay/config (hands the phone the device credentials), POST /api/dash/relay/heartbeat|stop (lease- based pause of the Pi's own background uploader, self-heals if the phone's tab closes uncleanly), POST /api/dash/relay/mark_uploaded. - modules/app_upload.py: list_pending_dashcam_files(), relay lease helpers, _upload_dashcam_files() now pauses while a relay lease is active. - modules/app_sql.py: new dashcam_hash_cache table so a file's sha256 is only computed once (keyed by path+size+mtime) instead of being recomputed on every loop iteration/pending-list call - benefits the existing background uploader too. - templates/dash.html / static/js/dash.js: new "Relay via phone" card - per-file and "relay all" buttons (manual trigger only, per user preference), downloads+uploads one 40MB slice at a time so the phone never holds a whole file in memory, retries chunks up to 3x. - PLAN.md/readme.md/server/README.md updated (Phase 11). Tested via a full local end-to-end simulation driving the real Pi and server Flask apps side by side (Postgres mocked only at the db module boundary - all HTTP/chunking/auth/CORS logic is real): verified Range downloads (206), byte-for-byte reassembly via SHA256, path-traversal rejection, and the relay lease correctly pausing/resuming the background uploader. Not yet tested against a real phone/live network.
This commit is contained in:
@@ -458,6 +458,78 @@ data connection when WiFi/Ethernet is unreachable. Design notes:
|
|||||||
a plain-HTTP router port-forward path, avoiding the need for the modem's TLS stack to validate
|
a plain-HTTP router port-forward path, avoiding the need for the modem's TLS stack to validate
|
||||||
a cert chain).
|
a cert chain).
|
||||||
|
|
||||||
|
### Phase 11: Phone-relay dashcam upload — ✅ DONE
|
||||||
|
Root cause investigation into "why is the transfer from the Pi to the server so slow?" (real
|
||||||
|
dashcam backlog draining at only ~350MB/4-5min) found the bottleneck is the **Pi's own onboard
|
||||||
|
BCM43438 WiFi chip** (2.4GHz-only, congested/interfered channel) — confirmed via `iperf3`
|
||||||
|
raw-TCP tests showing only ~14.7Mbit/s with active retransmits even over a same-subnet direct-LAN
|
||||||
|
hop that never touches the internet/Cloudflare, ruling out ISP/Cloudflare limits. Since the Pi 3 B
|
||||||
|
has no 5GHz radio and can't be fixed in software, added a way to bypass its weak WiFi for the
|
||||||
|
internet-bound leg entirely: a phone opens the same `/dash` page (over the Pi's hotspot/home WiFi)
|
||||||
|
and manually relays pending dashcam files straight to the remote server using the phone's own
|
||||||
|
network (WiFi or cellular data), instead of routing everything through the Pi.
|
||||||
|
|
||||||
|
**Zero new remote-server upload logic** — the phone's browser JS calls the *exact same*
|
||||||
|
`/api/v1/dashcam/upload/status` + `/api/v1/dashcam/upload/chunk` endpoints the Pi's own Python
|
||||||
|
client already uses, in the same 40MB-chunk protocol, just orchestrated by `fetch()`/`FormData`
|
||||||
|
instead of `requests`/`MultipartEncoder`. The server dedupes purely by SHA256 regardless of which
|
||||||
|
device sends the bytes, so no server-side changes were needed for the upload path itself — only a
|
||||||
|
CORS `after_request` hook (`server/app.py`) was added, since the phone's browser calls
|
||||||
|
`yaris.ceaf.dev` from a page served by a *different* origin (the Pi), which browsers block without
|
||||||
|
`Access-Control-Allow-*` headers on both the real request and its preflight `OPTIONS`. Wildcard
|
||||||
|
origin is fine here — auth is header-based (`X-Device-Name`/`X-Api-Key`), not cookie/session-based.
|
||||||
|
|
||||||
|
**New Pi-side endpoints** (`app.py`):
|
||||||
|
- `GET /api/dash/pending` — lists not-yet-uploaded dashcam files (`relative_path`, `sha256`,
|
||||||
|
`size`) via a new `app_upload.list_pending_dashcam_files()`.
|
||||||
|
- `GET /api/dash/file/<path:relpath>` — serves a dashcam file's bytes for the phone to download,
|
||||||
|
via `send_file(..., conditional=True)` for **Range-request support** (206 Partial Content) so
|
||||||
|
the phone downloads (and immediately re-uploads) one 40MB slice at a time instead of ever
|
||||||
|
holding a whole 300+MB file in the browser's memory — the exact same memory-safety lesson
|
||||||
|
learned the hard way in Phase 10, applied symmetrically to the download side. Path-traversal
|
||||||
|
safe via Werkzeug's `safe_join` against `app_dash.DEFAULT_DESTINATION`.
|
||||||
|
- `GET /api/dash/relay/config` — hands the phone the same `SERVER_URL`/`DEVICE_NAME`/`API_KEY`
|
||||||
|
the Pi's own client uses, so its JS can auth directly with the remote server. Same trust
|
||||||
|
boundary as the rest of this project's unauthenticated local UI (anyone on the Pi's own
|
||||||
|
hotspot/LAN is already implicitly trusted).
|
||||||
|
- `POST /api/dash/relay/heartbeat` / `POST /api/dash/relay/stop` — a **lease-based pause**
|
||||||
|
(`RELAY_LEASE_SECONDS = 45`) for the Pi's own background dashcam uploader, chosen over a plain
|
||||||
|
on/off toggle specifically to self-heal if the phone's browser tab closes/loses connection
|
||||||
|
uncleanly (a simple toggle could get stuck "paused" forever). The phone page heartbeats every
|
||||||
|
15s while relaying; `_upload_dashcam_files()` skips entirely while a lease is active, re-checked
|
||||||
|
between files so it stops promptly if a relay starts mid-drain. Also released via
|
||||||
|
`navigator.sendBeacon()` on `beforeunload` as a best-effort immediate release.
|
||||||
|
- `POST /api/dash/relay/mark_uploaded` — lets the phone tell the Pi "I just uploaded this file"
|
||||||
|
right after a successful relay, so the local dedupe table (and the pending list / Pi's own
|
||||||
|
background loop) reflect it immediately instead of waiting for the next full backlog scan.
|
||||||
|
|
||||||
|
**Performance side-fix**: `modules/app_upload.py`'s dashcam loop previously recomputed SHA256 for
|
||||||
|
*every* candidate file on *every* loop iteration, including files long since uploaded — wasteful
|
||||||
|
CPU/IO against a growing, possibly 20GB+, local backlog. Added a `dashcam_hash_cache` SQLite table
|
||||||
|
(`modules/app_sql.py`, keyed by `relative_path`+`size`+`mtime`) so a file's hash is only computed
|
||||||
|
once; both the existing background uploader and the new pending-files endpoint benefit.
|
||||||
|
|
||||||
|
UI (`templates/dash.html` / `static/js/dash.js`): a new "Relay via phone" card on `/dash` lists
|
||||||
|
pending files with a per-file "Relay" button and a "Relay all pending" button (manual trigger only
|
||||||
|
— chosen deliberately over an automatic one, per explicit user preference), plus a "Stop relay"
|
||||||
|
button. Chunk downloads/uploads retry up to 3 times with backoff before surfacing an error, and a
|
||||||
|
resumed relay re-checks `/status` first so a previously-interrupted file picks up from its first
|
||||||
|
missing chunk instead of restarting.
|
||||||
|
|
||||||
|
**Tested**: a full local end-to-end simulation (no physical phone/network needed) drove the real
|
||||||
|
Pi `app.py` and real `server/app.py` Flask apps side by side (Postgres mocked at the `db` module
|
||||||
|
boundary only — all HTTP routing, chunking, auth, and CORS logic is real), reproducing the exact
|
||||||
|
sequence the phone's browser JS performs: list pending -> check remote `/status` -> download each
|
||||||
|
chunk from the Pi via `Range` requests (verified real `206 Partial Content` responses) -> upload
|
||||||
|
each chunk to the mock-backed server -> verify the reassembled file matches the original
|
||||||
|
byte-for-byte via SHA256 -> mark uploaded on the Pi -> confirm it drops out of the pending list.
|
||||||
|
Also verified: path-traversal attempts on `/api/dash/file/` correctly 404, the relay
|
||||||
|
heartbeat/stop lease correctly pauses/resumes `_upload_dashcam_files()`, and the sha256 cache
|
||||||
|
table is populated and reused across repeated `/api/dash/pending` calls. **Not yet tested on a
|
||||||
|
real phone against the real deployed server** — the wire protocol and every server-visible
|
||||||
|
behavior is verified, but real mobile-browser quirks (background-tab throttling, flaky cellular
|
||||||
|
mid-transfer) haven't been exercised against live hardware yet.
|
||||||
|
|
||||||
## Working Notes / Environment Gotchas
|
## Working Notes / Environment Gotchas
|
||||||
|
|
||||||
- **Pi credentials/SSH**: `ssh yaris@<pi-ip>`. Confirm current IP before connecting (varies by
|
- **Pi credentials/SSH**: `ssh yaris@<pi-ip>`. Confirm current IP before connecting (varies by
|
||||||
|
|||||||
@@ -1,4 +1,7 @@
|
|||||||
from flask import Flask, render_template, make_response, json, jsonify, request
|
import os
|
||||||
|
|
||||||
|
from flask import Flask, render_template, make_response, json, jsonify, request, send_file, abort
|
||||||
|
from werkzeug.utils import safe_join
|
||||||
from modules import app_gps, app_events, app_car, app_account, app_gps,app_obd, app_sql, app_dash, app_upload
|
from modules import app_gps, app_events, app_car, app_account, app_gps,app_obd, app_sql, app_dash, app_upload
|
||||||
import time
|
import time
|
||||||
from gevent.pywsgi import WSGIServer
|
from gevent.pywsgi import WSGIServer
|
||||||
@@ -50,6 +53,60 @@ def upload_dashcam_toggle():
|
|||||||
app_upload.set_dashcam_upload_enabled(bool(body["enabled"]))
|
app_upload.set_dashcam_upload_enabled(bool(body["enabled"]))
|
||||||
return jsonify(app_upload.get_status())
|
return jsonify(app_upload.get_status())
|
||||||
|
|
||||||
|
# --- Phone-relay: lets a phone on the Pi's own network list, download, and
|
||||||
|
# directly re-upload dashcam files to the remote server itself (bypassing
|
||||||
|
# the Pi's own often-congested WiFi radio for the internet-bound leg). See
|
||||||
|
# static/js/dash.js for the client side of this. ---
|
||||||
|
|
||||||
|
@app.route('/api/dash/pending')
|
||||||
|
def dash_pending():
|
||||||
|
return jsonify({"files": app_upload.list_pending_dashcam_files()})
|
||||||
|
|
||||||
|
@app.route('/api/dash/file/<path:relpath>')
|
||||||
|
def dash_file(relpath):
|
||||||
|
abs_path = safe_join(app_dash.DEFAULT_DESTINATION, relpath)
|
||||||
|
if abs_path is None or not os.path.isfile(abs_path):
|
||||||
|
abort(404)
|
||||||
|
# conditional=True (Flask's default) makes this support Range requests,
|
||||||
|
# so the phone can fetch/re-upload one chunk at a time instead of ever
|
||||||
|
# holding the whole (300+MB) file in the browser's memory at once.
|
||||||
|
return send_file(abs_path, mimetype='application/octet-stream', conditional=True)
|
||||||
|
|
||||||
|
@app.route('/api/dash/relay/config')
|
||||||
|
def dash_relay_config():
|
||||||
|
# Hands the phone the same device credentials the Pi's own upload client
|
||||||
|
# uses, so its browser JS can call the remote server's existing chunked
|
||||||
|
# upload endpoints directly. Same trust boundary as the rest of this
|
||||||
|
# unauthenticated local UI - anyone on the Pi's hotspot/LAN is already
|
||||||
|
# implicitly trusted.
|
||||||
|
return jsonify({
|
||||||
|
"server_url": app_upload.SERVER_URL,
|
||||||
|
"device_name": app_upload.DEVICE_NAME,
|
||||||
|
"api_key": app_upload.API_KEY,
|
||||||
|
"chunk_size": app_upload.DASHCAM_CHUNK_SIZE,
|
||||||
|
"configured": app_upload.is_configured(),
|
||||||
|
})
|
||||||
|
|
||||||
|
@app.route('/api/dash/relay/heartbeat', methods=['POST'])
|
||||||
|
def dash_relay_heartbeat():
|
||||||
|
app_upload.relay_heartbeat()
|
||||||
|
return jsonify(app_upload.get_status())
|
||||||
|
|
||||||
|
@app.route('/api/dash/relay/stop', methods=['POST'])
|
||||||
|
def dash_relay_stop():
|
||||||
|
app_upload.relay_stop()
|
||||||
|
return jsonify(app_upload.get_status())
|
||||||
|
|
||||||
|
@app.route('/api/dash/relay/mark_uploaded', methods=['POST'])
|
||||||
|
def dash_relay_mark_uploaded():
|
||||||
|
body = request.get_json(silent=True) or {}
|
||||||
|
relative_path = (body.get("relative_path") or "").strip()
|
||||||
|
sha256 = (body.get("sha256") or "").strip().lower()
|
||||||
|
if not relative_path or not sha256:
|
||||||
|
return jsonify(error="expected JSON body {'relative_path', 'sha256'}"), 400
|
||||||
|
app_sql.mark_dashcam_file_uploaded(relative_path, sha256)
|
||||||
|
return jsonify(status="ok")
|
||||||
|
|
||||||
# not sure if im gonna use this
|
# not sure if im gonna use this
|
||||||
@app.route('/api/obd_speed/')
|
@app.route('/api/obd_speed/')
|
||||||
def data():
|
def data():
|
||||||
|
|||||||
@@ -57,6 +57,18 @@ SQL_CREATE_DASHCAM_UPLOADS_TABLE = """CREATE TABLE IF NOT EXISTS dashcam_uploads
|
|||||||
UNIQUE(RELATIVE_PATH, SHA256)
|
UNIQUE(RELATIVE_PATH, SHA256)
|
||||||
);"""
|
);"""
|
||||||
|
|
||||||
|
# Cache of already-computed sha256 hashes, keyed by (path, size, mtime) so a
|
||||||
|
# file's hash never needs recomputing once copied from the dashcam (its size
|
||||||
|
# and mtime won't change afterward). Avoids re-hashing the entire, possibly
|
||||||
|
# 20GB+, local backlog on every background-upload loop iteration and every
|
||||||
|
# "pending files" list request from the phone-relay feature.
|
||||||
|
SQL_CREATE_DASHCAM_HASH_CACHE_TABLE = """CREATE TABLE IF NOT EXISTS dashcam_hash_cache (
|
||||||
|
RELATIVE_PATH text PRIMARY KEY,
|
||||||
|
SIZE integer NOT NULL,
|
||||||
|
MTIME real NOT NULL,
|
||||||
|
SHA256 text NOT NULL
|
||||||
|
);"""
|
||||||
|
|
||||||
|
|
||||||
def create_conn(db_file):
|
def create_conn(db_file):
|
||||||
"""Open a SQLite connection to db_file, creating its parent folder if needed.
|
"""Open a SQLite connection to db_file, creating its parent folder if needed.
|
||||||
@@ -98,6 +110,7 @@ def get_conn():
|
|||||||
create_table(_conn, SQL_CREATE_DTC_TABLE)
|
create_table(_conn, SQL_CREATE_DTC_TABLE)
|
||||||
create_table(_conn, SQL_CREATE_SETTINGS_TABLE)
|
create_table(_conn, SQL_CREATE_SETTINGS_TABLE)
|
||||||
create_table(_conn, SQL_CREATE_DASHCAM_UPLOADS_TABLE)
|
create_table(_conn, SQL_CREATE_DASHCAM_UPLOADS_TABLE)
|
||||||
|
create_table(_conn, SQL_CREATE_DASHCAM_HASH_CACHE_TABLE)
|
||||||
_migrate_tracking_table(_conn)
|
_migrate_tracking_table(_conn)
|
||||||
else:
|
else:
|
||||||
print("Error! cannot create the database connection.")
|
print("Error! cannot create the database connection.")
|
||||||
@@ -221,6 +234,34 @@ def mark_dashcam_file_uploaded(relative_path, sha256):
|
|||||||
conn.commit()
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def get_cached_dashcam_hash(relative_path, size, mtime):
|
||||||
|
"""Return a previously-computed sha256 for this exact (path, size, mtime),
|
||||||
|
or None if not cached / the file has changed since it was last hashed."""
|
||||||
|
conn = get_conn()
|
||||||
|
if conn is None:
|
||||||
|
return None
|
||||||
|
with _conn_lock:
|
||||||
|
cur = conn.execute(
|
||||||
|
"SELECT SHA256 FROM dashcam_hash_cache WHERE RELATIVE_PATH = ? AND SIZE = ? AND MTIME = ?",
|
||||||
|
(relative_path, size, mtime),
|
||||||
|
)
|
||||||
|
row = cur.fetchone()
|
||||||
|
return row[0] if row is not None else None
|
||||||
|
|
||||||
|
|
||||||
|
def set_cached_dashcam_hash(relative_path, size, mtime, sha256):
|
||||||
|
conn = get_conn()
|
||||||
|
if conn is None:
|
||||||
|
return
|
||||||
|
with _conn_lock:
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO dashcam_hash_cache (RELATIVE_PATH, SIZE, MTIME, SHA256) VALUES (?, ?, ?, ?) "
|
||||||
|
"ON CONFLICT(RELATIVE_PATH) DO UPDATE SET SIZE = excluded.SIZE, MTIME = excluded.MTIME, SHA256 = excluded.SHA256",
|
||||||
|
(relative_path, size, mtime, sha256),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
def _log_loop():
|
def _log_loop():
|
||||||
# Deferred import to avoid a circular import at module load time
|
# Deferred import to avoid a circular import at module load time
|
||||||
# (app_obd.py imports app_sql, and app_sql needs app_obd.speed here).
|
# (app_obd.py imports app_sql, and app_sql needs app_obd.speed here).
|
||||||
|
|||||||
+68
-7
@@ -67,10 +67,17 @@ DASHCAM_CHUNK_SIZE = 40 * 1024 * 1024
|
|||||||
|
|
||||||
DASHCAM_UPLOAD_SETTING_KEY = "dashcam_upload_enabled"
|
DASHCAM_UPLOAD_SETTING_KEY = "dashcam_upload_enabled"
|
||||||
|
|
||||||
|
# How long a phone-relay "lease" stays valid without a fresh heartbeat before
|
||||||
|
# the Pi's own background uploader assumes the phone gave up/closed its tab
|
||||||
|
# and resumes uploading dashcam files itself. The phone page is expected to
|
||||||
|
# heartbeat well under this (every ~15-20s) - see /api/dash/relay/heartbeat.
|
||||||
|
RELAY_LEASE_SECONDS = 45
|
||||||
|
|
||||||
_upload_thread = None
|
_upload_thread = None
|
||||||
_last_error = None
|
_last_error = None
|
||||||
_last_success_at = None
|
_last_success_at = None
|
||||||
_last_network_path = None # "wifi" or "cellular" - last path a batch actually went out on
|
_last_network_path = None # "wifi" or "cellular" - last path a batch actually went out on
|
||||||
|
_relay_lease_until = None # monotonic time.time() the phone-relay lease expires, or None
|
||||||
|
|
||||||
|
|
||||||
def _headers():
|
def _headers():
|
||||||
@@ -100,9 +107,30 @@ def get_status():
|
|||||||
"last_success_at": _last_success_at,
|
"last_success_at": _last_success_at,
|
||||||
"last_network_path": _last_network_path,
|
"last_network_path": _last_network_path,
|
||||||
"cellular_fallback_configured": app_cellular.is_configured(),
|
"cellular_fallback_configured": app_cellular.is_configured(),
|
||||||
|
"relay_active": is_relay_active(),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# --- Phone-relay lease (pauses the Pi's own background dashcam uploader
|
||||||
|
# while a phone is manually relaying files, so they don't fight over the
|
||||||
|
# Pi's single, often WiFi-congested, radio) -------------------------------
|
||||||
|
|
||||||
|
def relay_heartbeat():
|
||||||
|
"""Called by the phone-relay page every ~15-20s while active."""
|
||||||
|
global _relay_lease_until
|
||||||
|
_relay_lease_until = time.time() + RELAY_LEASE_SECONDS
|
||||||
|
|
||||||
|
|
||||||
|
def relay_stop():
|
||||||
|
"""Called when the phone-relay page is done/closed cleanly."""
|
||||||
|
global _relay_lease_until
|
||||||
|
_relay_lease_until = None
|
||||||
|
|
||||||
|
|
||||||
|
def is_relay_active():
|
||||||
|
return _relay_lease_until is not None and time.time() < _relay_lease_until
|
||||||
|
|
||||||
|
|
||||||
def _sha256_file(path):
|
def _sha256_file(path):
|
||||||
hasher = hashlib.sha256()
|
hasher = hashlib.sha256()
|
||||||
with open(path, "rb") as f:
|
with open(path, "rb") as f:
|
||||||
@@ -114,6 +142,37 @@ def _sha256_file(path):
|
|||||||
return hasher.hexdigest()
|
return hasher.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _hash_dashcam_file(abs_path, rel_path):
|
||||||
|
"""sha256 of a dashcam file, cached by (path, size, mtime) so unchanged
|
||||||
|
files already copied from the dashcam never get rehashed - real backlogs
|
||||||
|
here run 20GB+, and both the background loop and the phone-relay pending
|
||||||
|
list would otherwise rehash everything, including already-uploaded files,
|
||||||
|
on every single call."""
|
||||||
|
stat = os.stat(abs_path)
|
||||||
|
cached = app_sql.get_cached_dashcam_hash(rel_path, stat.st_size, stat.st_mtime)
|
||||||
|
if cached is not None:
|
||||||
|
return cached
|
||||||
|
sha256 = _sha256_file(abs_path)
|
||||||
|
app_sql.set_cached_dashcam_hash(rel_path, stat.st_size, stat.st_mtime, sha256)
|
||||||
|
return sha256
|
||||||
|
|
||||||
|
|
||||||
|
def list_pending_dashcam_files():
|
||||||
|
"""Return [{relative_path, sha256, size}, ...] for dashcam files not yet
|
||||||
|
confirmed uploaded - used by the phone-relay page to pick what to fetch."""
|
||||||
|
pending = []
|
||||||
|
for abs_path, rel_path in app_dash.find_video_files(app_dash.DEFAULT_DESTINATION):
|
||||||
|
sha256 = _hash_dashcam_file(abs_path, rel_path)
|
||||||
|
if app_sql.dashcam_file_marked_uploaded(rel_path, sha256):
|
||||||
|
continue
|
||||||
|
pending.append({
|
||||||
|
"relative_path": rel_path,
|
||||||
|
"sha256": sha256,
|
||||||
|
"size": os.path.getsize(abs_path),
|
||||||
|
})
|
||||||
|
return pending
|
||||||
|
|
||||||
|
|
||||||
def _upload_tracking_batch():
|
def _upload_tracking_batch():
|
||||||
"""Send up to TRACKING_BATCH_SIZE not-yet-uploaded rows. Returns count sent.
|
"""Send up to TRACKING_BATCH_SIZE not-yet-uploaded rows. Returns count sent.
|
||||||
|
|
||||||
@@ -231,15 +290,16 @@ def _upload_video_chunked(abs_path, rel_path, sha256):
|
|||||||
def _upload_dashcam_files():
|
def _upload_dashcam_files():
|
||||||
"""Upload any not-yet-uploaded dashcam files, one at a time. Returns count sent.
|
"""Upload any not-yet-uploaded dashcam files, one at a time. Returns count sent.
|
||||||
|
|
||||||
Skipped entirely (returns 0) while the dashcam-upload toggle is off -
|
Skipped entirely (returns 0) while the dashcam-upload toggle is off, or
|
||||||
tracking-log uploads are unaffected by this toggle.
|
while a phone-relay session is actively leased - tracking-log uploads are
|
||||||
|
unaffected by either of these.
|
||||||
"""
|
"""
|
||||||
if not is_dashcam_upload_enabled():
|
if not is_dashcam_upload_enabled() or is_relay_active():
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
uploaded_count = 0
|
uploaded_count = 0
|
||||||
for abs_path, rel_path in app_dash.find_video_files(app_dash.DEFAULT_DESTINATION):
|
for abs_path, rel_path in app_dash.find_video_files(app_dash.DEFAULT_DESTINATION):
|
||||||
sha256 = _sha256_file(abs_path)
|
sha256 = _hash_dashcam_file(abs_path, rel_path)
|
||||||
if app_sql.dashcam_file_marked_uploaded(rel_path, sha256):
|
if app_sql.dashcam_file_marked_uploaded(rel_path, sha256):
|
||||||
continue
|
continue
|
||||||
|
|
||||||
@@ -247,9 +307,10 @@ def _upload_dashcam_files():
|
|||||||
app_sql.mark_dashcam_file_uploaded(rel_path, sha256)
|
app_sql.mark_dashcam_file_uploaded(rel_path, sha256)
|
||||||
uploaded_count += 1
|
uploaded_count += 1
|
||||||
|
|
||||||
# Re-check the toggle between files so flipping it off mid-run stops
|
# Re-check between files so flipping the toggle off, or a phone
|
||||||
# further dashcam uploads promptly without waiting for the next cycle.
|
# relay starting, mid-run stops further uploads promptly without
|
||||||
if not is_dashcam_upload_enabled():
|
# waiting for the next cycle.
|
||||||
|
if not is_dashcam_upload_enabled() or is_relay_active():
|
||||||
break
|
break
|
||||||
|
|
||||||
return uploaded_count
|
return uploaded_count
|
||||||
|
|||||||
@@ -42,6 +42,9 @@ i have a dream
|
|||||||
Dashcam videos upload in resumable 40MB chunks (works around tunnel body-size limits and keeps
|
Dashcam videos upload in resumable 40MB chunks (works around tunnel body-size limits and keeps
|
||||||
the Pi's memory footprint flat regardless of file size); a UI toggle can pause dashcam uploads
|
the Pi's memory footprint flat regardless of file size); a UI toggle can pause dashcam uploads
|
||||||
independently while trip logs keep syncing
|
independently while trip logs keep syncing
|
||||||
|
- 📱 **Phone-relay upload** — if the Pi's own WiFi is slow/congested, open `/dash` on your phone
|
||||||
|
to download pending dashcam files locally and re-upload them straight to the remote server
|
||||||
|
using the phone's own network instead, automatically pausing the Pi's own upload meanwhile
|
||||||
- 🗄️ Local SQLite trip logging, retried/synced to the remote server with automatic backoff
|
- 🗄️ Local SQLite trip logging, retried/synced to the remote server with automatic backoff
|
||||||
- 🔌 Runs fully headless as a `systemd` service, autostarting on boot
|
- 🔌 Runs fully headless as a `systemd` service, autostarting on boot
|
||||||
|
|
||||||
@@ -75,8 +78,9 @@ For the remote server (receives trip logs + dashcam uploads), see [server/README
|
|||||||
|
|
||||||
## Project status
|
## Project status
|
||||||
|
|
||||||
This project was revived after a couple of years on the shelf. All 10 planned phases are done and
|
This project was revived after a couple of years on the shelf. All 11 planned phases are done and
|
||||||
the full pipeline — OBD/GPS logging, offline web UI, WiFi hotspot, dashcam retrieval, and remote
|
the full pipeline — OBD/GPS logging, offline web UI, WiFi hotspot, dashcam retrieval, remote
|
||||||
upload — has been verified against real hardware end-to-end, including a full real dashcam import
|
upload, and phone-relay upload — has been verified against real hardware end-to-end, including a
|
||||||
and upload run. Only the SIM868 cellular fallback (for uploading while out of WiFi range) remains
|
full real dashcam import and upload run. Only the SIM868 cellular fallback (for uploading while
|
||||||
untested, pending a SIM card. Full phase-by-phase history in [PLAN.md](./PLAN.md).
|
out of WiFi range) and the phone-relay feature's real-phone/live-network test remain untested.
|
||||||
|
Full phase-by-phase history in [PLAN.md](./PLAN.md).
|
||||||
|
|||||||
@@ -28,6 +28,14 @@ headers: `X-Device-Name` and `X-Api-Key` (see
|
|||||||
Postgres is **not** published to the LAN — only the `api` container talks to
|
Postgres is **not** published to the LAN — only the `api` container talks to
|
||||||
it, over the internal compose network.
|
it, over the internal compose network.
|
||||||
|
|
||||||
|
**CORS is enabled on every response** (`Access-Control-Allow-Origin: *`) so a
|
||||||
|
device's registered API key can be used directly from browser JS running on a
|
||||||
|
*different* origin — specifically, the Pi's phone-relay feature (see
|
||||||
|
[PLAN.md](../PLAN.md) Phase 11), where a phone's browser calls this API
|
||||||
|
straight from a page served by the Pi itself. This is safe here because
|
||||||
|
auth is header-based (`X-Device-Name`/`X-Api-Key`), not cookie/session-based,
|
||||||
|
so a permissive origin can't be used to steal a logged-in session.
|
||||||
|
|
||||||
## Deploying on unraid
|
## Deploying on unraid
|
||||||
|
|
||||||
1. Get the code onto the unraid box (same repo as the Pi, just the
|
1. Get the code onto the unraid box (same repo as the Pi, just the
|
||||||
|
|||||||
@@ -12,6 +12,20 @@ from modules.auth import require_device_auth
|
|||||||
|
|
||||||
app = Flask(__name__)
|
app = Flask(__name__)
|
||||||
|
|
||||||
|
# The phone-relay feature (Pi web UI) has a phone's browser call this API
|
||||||
|
# directly from a page served BY THE PI (a different origin than this
|
||||||
|
# server), so the browser enforces CORS on those cross-origin requests.
|
||||||
|
# Wildcard is fine here - this API is authenticated via the X-Device-Name/
|
||||||
|
# X-Api-Key headers (not cookies), so there's no session/credential to leak
|
||||||
|
# via a permissive Access-Control-Allow-Origin.
|
||||||
|
@app.after_request
|
||||||
|
def _add_cors_headers(response):
|
||||||
|
response.headers["Access-Control-Allow-Origin"] = "*"
|
||||||
|
response.headers["Access-Control-Allow-Methods"] = "GET, POST, OPTIONS"
|
||||||
|
response.headers["Access-Control-Allow-Headers"] = "Content-Type, X-Device-Name, X-Api-Key"
|
||||||
|
response.headers["Access-Control-Max-Age"] = "600"
|
||||||
|
return response
|
||||||
|
|
||||||
# Where uploaded dashcam files get written on disk (mounted to the unraid
|
# Where uploaded dashcam files get written on disk (mounted to the unraid
|
||||||
# array via docker-compose - see DASHCAM_STORAGE_DIR in .env).
|
# array via docker-compose - see DASHCAM_STORAGE_DIR in .env).
|
||||||
STORAGE_DIR = os.environ.get("DASHCAM_STORAGE_DIR", "/data/dashcam")
|
STORAGE_DIR = os.environ.get("DASHCAM_STORAGE_DIR", "/data/dashcam")
|
||||||
|
|||||||
@@ -121,3 +121,262 @@ dashcamToggle.addEventListener('change', () => {
|
|||||||
|
|
||||||
refreshUploadStatus();
|
refreshUploadStatus();
|
||||||
setInterval(refreshUploadStatus, 15000);
|
setInterval(refreshUploadStatus, 15000);
|
||||||
|
|
||||||
|
// --- Relay via phone ---
|
||||||
|
// Lets a phone (open on this same page, over the Pi's hotspot/LAN) download
|
||||||
|
// pending dashcam files from the Pi and re-upload them directly to the
|
||||||
|
// remote server using its own network, reusing the exact same chunked
|
||||||
|
// upload wire protocol the Pi's own background uploader uses
|
||||||
|
// (/api/v1/dashcam/upload/status + /chunk) - just orchestrated by this
|
||||||
|
// browser instead of the Pi's Python client. Downloads are pulled in the
|
||||||
|
// same chunk size and immediately re-uploaded, one at a time, so the phone
|
||||||
|
// never has to hold a whole (300+MB) file in memory at once.
|
||||||
|
|
||||||
|
const relayStatusEl = document.getElementById('relay-status');
|
||||||
|
const relayRefreshBtn = document.getElementById('relay-refresh-btn');
|
||||||
|
const relayAllBtn = document.getElementById('relay-all-btn');
|
||||||
|
const relayStopBtn = document.getElementById('relay-stop-btn');
|
||||||
|
const relayFileList = document.getElementById('relay-file-list');
|
||||||
|
|
||||||
|
let relayConfig = null;
|
||||||
|
let pendingFiles = [];
|
||||||
|
const relayRowEls = new Map(); // relative_path -> { statusSpan, btn }
|
||||||
|
let heartbeatTimer = null;
|
||||||
|
let activeRelayCount = 0;
|
||||||
|
let relayStopRequested = false;
|
||||||
|
|
||||||
|
function humanSize(bytes) {
|
||||||
|
if (bytes >= 1024 * 1024 * 1024) return (bytes / (1024 * 1024 * 1024)).toFixed(2) + ' GB';
|
||||||
|
if (bytes >= 1024 * 1024) return (bytes / (1024 * 1024)).toFixed(1) + ' MB';
|
||||||
|
if (bytes >= 1024) return (bytes / 1024).toFixed(1) + ' KB';
|
||||||
|
return bytes + ' B';
|
||||||
|
}
|
||||||
|
|
||||||
|
function getRelayConfig() {
|
||||||
|
if (relayConfig) return Promise.resolve(relayConfig);
|
||||||
|
return fetch('/api/dash/relay/config')
|
||||||
|
.then(response => response.json())
|
||||||
|
.then(config => {
|
||||||
|
relayConfig = config;
|
||||||
|
return config;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function startHeartbeat() {
|
||||||
|
activeRelayCount++;
|
||||||
|
relayStopBtn.disabled = false;
|
||||||
|
if (heartbeatTimer) return;
|
||||||
|
fetch('/api/dash/relay/heartbeat', { method: 'POST' }).catch(() => {});
|
||||||
|
heartbeatTimer = setInterval(() => {
|
||||||
|
fetch('/api/dash/relay/heartbeat', { method: 'POST' }).catch(() => {});
|
||||||
|
}, 15000);
|
||||||
|
}
|
||||||
|
|
||||||
|
function stopHeartbeatIfIdle() {
|
||||||
|
activeRelayCount = Math.max(0, activeRelayCount - 1);
|
||||||
|
if (activeRelayCount > 0) return;
|
||||||
|
if (heartbeatTimer) {
|
||||||
|
clearInterval(heartbeatTimer);
|
||||||
|
heartbeatTimer = null;
|
||||||
|
}
|
||||||
|
relayStopBtn.disabled = true;
|
||||||
|
fetch('/api/dash/relay/stop', { method: 'POST' }).catch(() => {});
|
||||||
|
}
|
||||||
|
|
||||||
|
relayStopBtn.addEventListener('click', () => {
|
||||||
|
relayStopRequested = true;
|
||||||
|
activeRelayCount = 0;
|
||||||
|
if (heartbeatTimer) {
|
||||||
|
clearInterval(heartbeatTimer);
|
||||||
|
heartbeatTimer = null;
|
||||||
|
}
|
||||||
|
fetch('/api/dash/relay/stop', { method: 'POST' }).catch(() => {});
|
||||||
|
relayStatusEl.textContent = 'Relay stopped - the Pi will resume its own uploads shortly.';
|
||||||
|
relayStopBtn.disabled = true;
|
||||||
|
});
|
||||||
|
|
||||||
|
// Best-effort: try to release the pause lease immediately if the page is
|
||||||
|
// closed mid-relay (it also self-expires after ~45s either way).
|
||||||
|
window.addEventListener('beforeunload', () => {
|
||||||
|
if (heartbeatTimer && navigator.sendBeacon) {
|
||||||
|
navigator.sendBeacon('/api/dash/relay/stop');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
async function fetchChunkWithRetry(relativePath, start, end, attempts = 3) {
|
||||||
|
let lastErr;
|
||||||
|
for (let attempt = 1; attempt <= attempts; attempt++) {
|
||||||
|
try {
|
||||||
|
const response = await fetch(`/api/dash/file/${encodeURIComponent(relativePath)}`, {
|
||||||
|
headers: { Range: `bytes=${start}-${end}` },
|
||||||
|
});
|
||||||
|
if (!response.ok && response.status !== 206) {
|
||||||
|
throw new Error(`download HTTP ${response.status}`);
|
||||||
|
}
|
||||||
|
return await response.blob();
|
||||||
|
} catch (err) {
|
||||||
|
lastErr = err;
|
||||||
|
await new Promise(r => setTimeout(r, 1000 * attempt));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
throw lastErr;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function uploadChunkWithRetry(config, relativePath, sha256, chunkIndex, totalChunks, blob, attempts = 3) {
|
||||||
|
let lastErr;
|
||||||
|
for (let attempt = 1; attempt <= attempts; attempt++) {
|
||||||
|
try {
|
||||||
|
const form = new FormData();
|
||||||
|
form.append('relative_path', relativePath);
|
||||||
|
form.append('sha256', sha256);
|
||||||
|
form.append('chunk_index', String(chunkIndex));
|
||||||
|
form.append('total_chunks', String(totalChunks));
|
||||||
|
form.append('file', blob, `chunk_${chunkIndex}`);
|
||||||
|
const response = await fetch(`${config.server_url}/api/v1/dashcam/upload/chunk`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'X-Device-Name': config.device_name, 'X-Api-Key': config.api_key },
|
||||||
|
body: form,
|
||||||
|
});
|
||||||
|
if (!response.ok) {
|
||||||
|
throw new Error(`upload HTTP ${response.status}`);
|
||||||
|
}
|
||||||
|
return await response.json();
|
||||||
|
} catch (err) {
|
||||||
|
lastErr = err;
|
||||||
|
await new Promise(r => setTimeout(r, 1000 * attempt));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
throw lastErr;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function relayOneFile(file, statusEl) {
|
||||||
|
const config = await getRelayConfig();
|
||||||
|
if (!config.configured) {
|
||||||
|
statusEl.textContent = 'Remote server not configured on this device.';
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
relayStopRequested = false;
|
||||||
|
startHeartbeat();
|
||||||
|
try {
|
||||||
|
const statusResp = await fetch(
|
||||||
|
`${config.server_url}/api/v1/dashcam/upload/status?relative_path=${encodeURIComponent(file.relative_path)}&sha256=${file.sha256}`,
|
||||||
|
{ headers: { 'X-Device-Name': config.device_name, 'X-Api-Key': config.api_key } }
|
||||||
|
);
|
||||||
|
if (!statusResp.ok) throw new Error(`status check HTTP ${statusResp.status}`);
|
||||||
|
const status = await statusResp.json();
|
||||||
|
|
||||||
|
if (status.status !== 'already_uploaded') {
|
||||||
|
const received = new Set(status.received_chunk_indices || []);
|
||||||
|
const totalChunks = Math.max(1, Math.ceil(file.size / config.chunk_size));
|
||||||
|
|
||||||
|
for (let idx = 0; idx < totalChunks; idx++) {
|
||||||
|
if (relayStopRequested) {
|
||||||
|
statusEl.textContent = 'Stopped.';
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (received.has(idx)) continue;
|
||||||
|
|
||||||
|
const start = idx * config.chunk_size;
|
||||||
|
const end = Math.min(start + config.chunk_size, file.size) - 1;
|
||||||
|
statusEl.textContent = `Relaying... chunk ${idx + 1}/${totalChunks}`;
|
||||||
|
|
||||||
|
const blob = await fetchChunkWithRetry(file.relative_path, start, end);
|
||||||
|
await uploadChunkWithRetry(config, file.relative_path, file.sha256, idx, totalChunks, blob);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Update the Pi's own local dedupe table right away so the pending
|
||||||
|
// list (and its own background uploader) reflect this immediately,
|
||||||
|
// instead of waiting for its next full backlog scan.
|
||||||
|
await fetch('/api/dash/relay/mark_uploaded', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ relative_path: file.relative_path, sha256: file.sha256 }),
|
||||||
|
}).catch(() => {});
|
||||||
|
|
||||||
|
statusEl.textContent = 'Done.';
|
||||||
|
pendingFiles = pendingFiles.filter(f => f.relative_path !== file.relative_path);
|
||||||
|
} catch (err) {
|
||||||
|
statusEl.textContent = 'Error: ' + err;
|
||||||
|
throw err;
|
||||||
|
} finally {
|
||||||
|
stopHeartbeatIfIdle();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderPendingFiles() {
|
||||||
|
relayFileList.innerHTML = '';
|
||||||
|
relayRowEls.clear();
|
||||||
|
|
||||||
|
if (pendingFiles.length === 0) {
|
||||||
|
relayStatusEl.textContent = 'No pending dashcam files to relay.';
|
||||||
|
relayAllBtn.disabled = true;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
relayStatusEl.textContent = `${pendingFiles.length} file(s) pending upload.`;
|
||||||
|
relayAllBtn.disabled = false;
|
||||||
|
|
||||||
|
pendingFiles.forEach(file => {
|
||||||
|
const li = document.createElement('li');
|
||||||
|
li.className = 'list-group-item d-flex justify-content-between align-items-center flex-wrap gap-2';
|
||||||
|
|
||||||
|
const label = document.createElement('span');
|
||||||
|
label.textContent = `${file.relative_path} (${humanSize(file.size)})`;
|
||||||
|
|
||||||
|
const statusSpan = document.createElement('span');
|
||||||
|
statusSpan.className = 'text-muted small';
|
||||||
|
|
||||||
|
const btn = document.createElement('button');
|
||||||
|
btn.className = 'btn btn-sm btn-outline-primary';
|
||||||
|
btn.type = 'button';
|
||||||
|
btn.textContent = 'Relay';
|
||||||
|
btn.addEventListener('click', () => {
|
||||||
|
btn.disabled = true;
|
||||||
|
relayOneFile(file, statusSpan).finally(() => {
|
||||||
|
btn.disabled = false;
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
li.appendChild(label);
|
||||||
|
li.appendChild(statusSpan);
|
||||||
|
li.appendChild(btn);
|
||||||
|
relayFileList.appendChild(li);
|
||||||
|
relayRowEls.set(file.relative_path, { statusSpan, btn });
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function refreshPendingFiles() {
|
||||||
|
relayStatusEl.textContent = 'Checking pending files...';
|
||||||
|
fetch('/api/dash/pending')
|
||||||
|
.then(response => response.json())
|
||||||
|
.then(data => {
|
||||||
|
pendingFiles = data.files || [];
|
||||||
|
renderPendingFiles();
|
||||||
|
})
|
||||||
|
.catch(err => {
|
||||||
|
relayStatusEl.textContent = 'Error while checking pending files: ' + err;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
relayRefreshBtn.addEventListener('click', refreshPendingFiles);
|
||||||
|
|
||||||
|
relayAllBtn.addEventListener('click', async () => {
|
||||||
|
relayStopRequested = false;
|
||||||
|
relayAllBtn.disabled = true;
|
||||||
|
for (const file of pendingFiles.slice()) {
|
||||||
|
if (relayStopRequested) break;
|
||||||
|
const row = relayRowEls.get(file.relative_path);
|
||||||
|
const statusSpan = row ? row.statusSpan : relayStatusEl;
|
||||||
|
if (row) row.btn.disabled = true;
|
||||||
|
try {
|
||||||
|
await relayOneFile(file, statusSpan);
|
||||||
|
} catch (err) {
|
||||||
|
// Keep going with the remaining files even if one fails.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
relayAllBtn.disabled = pendingFiles.length === 0;
|
||||||
|
});
|
||||||
|
|
||||||
|
refreshPendingFiles();
|
||||||
|
|||||||
@@ -97,6 +97,26 @@
|
|||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
<div class="card shadow mt-4">
|
||||||
|
<div class="card-header py-3">
|
||||||
|
<p class="text-primary m-0 fw-bold">Relay via phone</p>
|
||||||
|
</div>
|
||||||
|
<div class="card-body">
|
||||||
|
<p class="text-muted small">
|
||||||
|
Open this page on your phone (while connected to the Pi's
|
||||||
|
hotspot/WiFi) to download pending dashcam files locally and
|
||||||
|
upload them straight to the remote server using your phone's own
|
||||||
|
data connection - useful when the Pi's own WiFi upload is slow.
|
||||||
|
Starting a relay automatically pauses the Pi's own background
|
||||||
|
dashcam upload so they don't compete for the same radio.
|
||||||
|
</p>
|
||||||
|
<p id="relay-status" class="mb-2">Checking...</p>
|
||||||
|
<button id="relay-refresh-btn" class="btn btn-secondary btn-sm" type="button">Refresh pending files</button>
|
||||||
|
<button id="relay-all-btn" class="btn btn-primary btn-sm" type="button" disabled>Relay all pending</button>
|
||||||
|
<button id="relay-stop-btn" class="btn btn-outline-danger btn-sm" type="button" disabled>Stop relay</button>
|
||||||
|
<ul id="relay-file-list" class="list-group mt-3"></ul>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<footer class="bg-white sticky-footer">
|
<footer class="bg-white sticky-footer">
|
||||||
|
|||||||
Reference in New Issue
Block a user