Confirmed against the real deployed server: Cloudflare Tunnel's free
tier caps request bodies at ~100MB and returns a plain 413 - a direct
LAN upload of the same 250MB test file succeeds, the identical request
through yaris.ceaf.dev does not. Real dashcam clips average ~320MB, so
no dashcam video could ever complete an upload through the tunnel as
it stood, regardless of the earlier memory-streaming fix.
Server: new /api/v1/dashcam/upload/chunk (accepts one chunk at a time,
keyed by relative_path+sha256+chunk_index/total_chunks, reassembles and
verifies the sha256 once every chunk has arrived) and
/api/v1/dashcam/upload/status (reports which chunk indices a given
upload already has, or "already_uploaded" if it's done) endpoints.
Chunks are staged under STORAGE_DIR/.upload_tmp/<device>/<sha256>/ and
cleaned up after reassembly; a stale partial set left by a differently
-chunked earlier attempt is detected via a small marker file and wiped
before restarting. The original single-shot /upload endpoint is left
in place unchanged.
Client (modules/app_upload.py): _upload_dashcam_files() now calls
_upload_video_chunked(), which checks /status first (so a prior
partial upload resumes instead of restarting from byte 0) and then
POSTs any missing 40MB chunks (DASHCAM_CHUNK_SIZE, comfortably under
Cloudflare's cap) via the same streaming MultipartEncoder approach as
before.
Co-authored-by: Copilot <[email protected]>
First real-hardware test of dashcam retrieval kept crashing the whole
yaris-obd service (repeated SIGKILL/OOM on the Pi 3's ~900MB RAM) a few
minutes into every run, aborting in-flight file copies. Root cause:
_upload_dashcam_files() used requests.post(files={"file": f}), which
builds the entire multipart/form-data body in memory before sending -
fine for small payloads, but this dashcam's real clips average ~320MB
each (up to a couple hundred more with overhead), so every dashcam
upload attempt alone could push the process well past 300-600MB, and
the Pi's OOM killer stepped in immediately.
Reproduced in isolation: uploading a 300MB file the old way spiked the
client process from 28MB to 628MB RSS; switching to requests-toolbelt's
MultipartEncoder (which streams the file in chunks as the request body
is sent, instead of materializing it up front) kept it flat at ~29MB
for the same upload, with an identical byte-for-byte result on the
receiving end. No server-side changes needed - the wire format is
unchanged, still ordinary multipart/form-data.
Added requests-toolbelt to requirements.txt.
Co-authored-by: Copilot <[email protected]>
modules/app_cellular.py: uses the SIM868's built-in AT-command HTTP(S)
client (AT+HTTPINIT/HTTPPARA/HTTPACTION/HTTPREAD) instead of a real
ppp0 interface, since the module's GNSS (polled continuously by
app_gps.py) and its GSM/GPRS radio share the same physical UART
(/dev/serial0) - a pppd/chat dial-up would take over that port
entirely and starve GNSS polling. Staying in AT-command mode lets
cellular HTTP calls interleave safely through the same serial lock.
Only carries small JSON tracking batches, never dashcam video -
cellular is far too slow/limited for that, and the modem's HTTP AT
interface has too small an internal buffer anyway. Dashcam uploads
remain WiFi/Ethernet-only, unchanged.
modules/app_upload.py: tracking-batch upload always tries WiFi/
Ethernet first; only on a connection-level failure (not an HTTP error
status, which means the server was reached) does it retry over
cellular. Exposes last_network_path/cellular_fallback_configured in
/api/upload/status; UI shows which path the last batch used.
Feature-flagged off by default via new YARIS_CELLULAR_ENABLED/
YARIS_CELLULAR_APN/_APN_USER/_APN_PASS keys in db/upload_creds.env
(all blank by default) - has zero effect on the existing WiFi upload
path unless explicitly configured. Regression-tested against the real
deployed server that normal WiFi uploads are unaffected while
cellular stays unconfigured.
*** UNTESTED against real cellular hardware - no SIM card was
*** available. See docstrings in app_cellular.py and PLAN.md for
*** exactly what needs validation once a SIM card is available.
Co-authored-by: Copilot <[email protected]>
The upload loop previously always slept the full 30s UPLOAD_INTERVAL
between batches, even when a full batch was just sent (meaning more
rows were likely still queued). With GPS/OBD rows logged every 5s
locally, an offline/stale device could build up a large backlog that
took a very long time to drain at 100 rows/30s.
- Bump TRACKING_BATCH_SIZE 100 -> 500 (cuts round-trips per backlog).
- Only sleep the full UPLOAD_INTERVAL once actually caught up (a
short batch came back); while a full batch keeps coming back, drain
back-to-back with a short CATCHUP_INTERVAL (1s) instead.
- server/app.py: add a sane MAX_TRACKING_BATCH_POINTS (5000) cap on
the tracking/batch endpoint as a defensive guard against a runaway
client, independent of the Pi's own batch size.
Verified against the real deployed server: a simulated 1250-row
backlog now drains in 3 batches (~0.35s server round-trip time) instead
of 13 batches spread across ~6.5 minutes of idle sleeping.
Co-authored-by: Copilot <[email protected]>
- app_sql.py: add UPLOADED column + migration on tracking table, add
settings and dashcam_uploads tables, add helpers for unsent-row
queries, marking rows uploaded, and dashcam upload dedupe tracking.
- app_upload.py: new background upload client. Periodically batches
unsent tracking rows to POST /api/v1/tracking/batch and uploads new
dashcam files (sha256-verified) to POST /api/v1/dashcam/upload, with
exponential backoff on failure. Exposes get_status() and a
dashcam-upload enable/disable toggle backed by the settings table.
- app.py: start the upload client thread at startup, add
GET /api/upload/status and POST /api/upload/dashcam_toggle routes.
- dash.html/dash.js: add a "Remote server upload" card with upload
status and a toggle switch to pause dashcam video uploads while
trip/GPS log uploads keep running (per user request - useful on a
slow/metered connection).
- db/upload_creds.env: new gitignore-exempted template (mirrors
db/creds.env) for YARIS_SERVER_URL/YARIS_DEVICE_NAME/YARIS_API_KEY.
- requirements.txt: add requests.
Tested against the real deployed unraid server: tracking batch upload
+ dedupe, dashcam upload + dedupe, and toggle-off correctly pausing
dashcam uploads while tracking uploads continue. Also verified the
full Flask app imports/runs cleanly and the new API routes + dash.html
toggle render correctly in a headless browser.
Co-authored-by: Copilot <[email protected]>
