Phase 16: complete web UI redesign - the current template is full of
never-replaced placeholder cruft (stub buttons, empty /dtc page); do
one real design pass once the functional phases land rather than
keep bolting onto the old admin-template style.
Phase 17: native Android companion app - the real fix for the
phone-relay goal that a web page could never deliver (true
simultaneous WiFi-to-Pi + cellular-to-server via
bindProcessToNetwork), plus a trip/location info view. Explicitly
deprioritized to the end of the roadmap per user - not urgent.
Also expanded Phase 15 (OBD hardening) scope to pull richer dashboard
data (coolant temp, fuel level, throttle, DTC/MIL status) alongside
speed/RPM, noting odometer/mileage support is not guaranteed on this
vehicle/year and needs real-car confirmation.
Co-authored-by: Copilot <[email protected]>
Retested the hotspot+cellular phone-relay scenario more rigorously
(netwatch explicitly disabled during tests so it wouldn't yank the Pi
back to home WiFi mid-test - an earlier failed run turned out to be
caused by that, not a phone issue). Confirmed with WiFi fully off
(pure cellular) that requests to the public server still failed while
associated with the Pi's hotspot, and that this is a fundamental
OS-level 'single default network for the whole device' limitation -
not something a web page's JS can work around, only a native app could
(bindProcessToNetwork). Documented the decision to remove the feature.
Added Phase 12 (remove phone-relay), Phase 13 (dashcam USB file
browser: list+delete), Phase 14 (local imported-video browser +
HTML5 player), Phase 15 (OBD code hardening ahead of first real
car test - background reconnect loop, exception handling, protocol
override, better diagnostics).
Co-authored-by: Copilot <[email protected]>
Live-tested the away-from-home hotspot+cellular relay scenario. Found the
netwatch reconnect bug (documented in Phase 6 update) during this test.
Confirmed the relay architecture itself needs no code changes - the
TypeError: Failed to fetch was a phone-OS network-switching behavior
(won't route internet traffic over cellular while connected to a
no-internet WiFi), not a CORS/code issue - ruled out via a plain page
navigation test. Testing this fully requires being genuinely away from
home or manually forcing cellular use on the phone.
Co-authored-by: Copilot <[email protected]>
Real-hardware finding: netwatch repeatedly reported 'home WiFi not found'
4 times over 13min while sitting right next to the router. Root cause:
a plain 'nmcli con up home-wifi' straight out of AP mode relies on NM's
scan cache for wlan0, which is stale/empty coming out of AP mode (the
radio can't scan while acting as an AP). A single con-up attempt didn't
wait for a fresh scan to land before giving up.
Fix: explicitly drop the AP connection first to free the radio, force a
fresh scan with a settle delay, and retry the connect attempt (longer
30s timeout) up to 2 times before conceding and restoring the hotspot.
Bumped the systemd unit's TimeoutStartSec to 180s to give the retry loop
enough headroom (was defaulting to 90s).
Co-authored-by: Copilot <[email protected]>
Root-caused a "GPS not working" report to the SIM868 module chip being
fully powered off (the HAT's 5V rail can be live while the chip itself
stays off until PWRKEY is triggered) - confirmed by getting zero AT
response at any baud rate while the app's GPS thread had the serial
port open normally.
Discovered this specific HAT revision has the "P4" GPIO power-on mod
wire installed (Waveshare FAQ / GSM_PWRKEY.rar script), letting PWRKEY
be triggered via physical pin 7 (BCM GPIO4) instead of a manual button
press. Verified live: pulsing it low for 4s then high brought the
module from zero AT response to responding immediately.
modules/app_gps.py: added _module_responds()/_pulse_pwrkey()/
_ensure_module_powered(), wired into _poll_loop() before the existing
GNSS power-on check. Checks AT first and only pulses PWRKEY if the
module isn't already responding, since PWRKEY is a toggle (pulsing an
already-on module would turn it back off).
Also fixed a related logging gap found while debugging: print()
diagnostics from app_gps.py were invisible in yaris-obd.log for
months because Python fully-buffers stdout when it isn't a TTY.
deploy/yaris-obd.service now sets PYTHONUNBUFFERED=1.
requirements.txt: added RPi.GPIO.
PLAN.md: documented both findings and the Pi deploy gotchas (systemd
unit file changes need a manual copy + daemon-reload; requirements.txt
changes need a manual venv pip install - neither is picked up by a
plain git pull + restart).
Co-authored-by: Copilot <[email protected]>
Retook pics/ui.png to show the current dashboard, including the
Dashcam nav item added in later phases. Also switched the readme's
image link from a hardcoded old-commit GitHub URL to a relative path
so it won't go stale again.
Co-authored-by: Copilot <[email protected]>
Deployed and tested the phone-relay feature end-to-end for real:
found and fixed a deploy gap (CORS fix wasn't rebuilt into the actual
remote server's Docker image, only pulled on the Pi - surfaced as a
generic "TypeError: Failed to fetch" in the browser). After fixing,
measured real throughput from server-side chunk timestamps: ~1.11MB/s
via phone relay (home WiFi) vs ~9-10Mbit/s for the Pi's own direct
upload - statistically the same. Confirms the relay provides no speed
benefit when the phone is on the same home WiFi as the Pi, since the
download-to-phone leg pays the same congested-radio cost either way.
Documented this finding plus a follow-up idea (per-clip browse/select
upload from the phone, not yet designed).
Root-caused the slow-upload complaint to the Pi's own congested 2.4GHz
WiFi radio (confirmed via iperf3 - same degraded throughput even on
pure local LAN traffic, ruling out Cloudflare/ISP). Since the Pi 3 B
has no 5GHz radio, add a way to bypass it for the internet-bound leg:
a phone opens /dash and manually relays pending dashcam files straight
to the remote server over its own network, reusing the exact same
chunked upload protocol the Pi's client already uses.
- server/app.py: add CORS headers (after_request) so the phone's
browser can call the remote server cross-origin from a page served
by the Pi.
- app.py: new endpoints - GET /api/dash/pending (list not-yet-uploaded
files), GET /api/dash/file/<path> (Range-request-capable download,
path-traversal safe), GET /api/dash/relay/config (hands the phone
the device credentials), POST /api/dash/relay/heartbeat|stop (lease-
based pause of the Pi's own background uploader, self-heals if the
phone's tab closes uncleanly), POST /api/dash/relay/mark_uploaded.
- modules/app_upload.py: list_pending_dashcam_files(), relay lease
helpers, _upload_dashcam_files() now pauses while a relay lease is
active.
- modules/app_sql.py: new dashcam_hash_cache table so a file's sha256
is only computed once (keyed by path+size+mtime) instead of being
recomputed on every loop iteration/pending-list call - benefits the
existing background uploader too.
- templates/dash.html / static/js/dash.js: new "Relay via phone" card
- per-file and "relay all" buttons (manual trigger only, per user
preference), downloads+uploads one 40MB slice at a time so the
phone never holds a whole file in memory, retries chunks up to 3x.
- PLAN.md/readme.md/server/README.md updated (Phase 11).
Tested via a full local end-to-end simulation driving the real Pi and
server Flask apps side by side (Postgres mocked only at the db module
boundary - all HTTP/chunking/auth/CORS logic is real): verified Range
downloads (206), byte-for-byte reassembly via SHA256, path-traversal
rejection, and the relay lease correctly pausing/resuming the
background uploader. Not yet tested against a real phone/live network.
- PLAN.md: mark Phase 8 (dashcam retrieval) verified on real hardware, add
Phase 10 documenting the OOM/multipart and Cloudflare body-size fixes plus
the chunked/resumable upload redesign, both verified end-to-end against
real dashcam footage. Update architecture decision #3 (Mongo -> Postgres
supersession), add unraid/Pi deploy notes and memory-budget gotcha.
- readme.md: refresh feature list (remote upload, chunked dashcam uploads,
toggle) and project status (all 10 phases done, only cellular fallback
untested).
- server/README.md: document the new chunked upload endpoints and the
Cloudflare body-size-limit rationale behind them.
Confirmed against the real deployed server: Cloudflare Tunnel's free
tier caps request bodies at ~100MB and returns a plain 413 - a direct
LAN upload of the same 250MB test file succeeds, the identical request
through yaris.ceaf.dev does not. Real dashcam clips average ~320MB, so
no dashcam video could ever complete an upload through the tunnel as
it stood, regardless of the earlier memory-streaming fix.
Server: new /api/v1/dashcam/upload/chunk (accepts one chunk at a time,
keyed by relative_path+sha256+chunk_index/total_chunks, reassembles and
verifies the sha256 once every chunk has arrived) and
/api/v1/dashcam/upload/status (reports which chunk indices a given
upload already has, or "already_uploaded" if it's done) endpoints.
Chunks are staged under STORAGE_DIR/.upload_tmp/<device>/<sha256>/ and
cleaned up after reassembly; a stale partial set left by a differently
-chunked earlier attempt is detected via a small marker file and wiped
before restarting. The original single-shot /upload endpoint is left
in place unchanged.
Client (modules/app_upload.py): _upload_dashcam_files() now calls
_upload_video_chunked(), which checks /status first (so a prior
partial upload resumes instead of restarting from byte 0) and then
POSTs any missing 40MB chunks (DASHCAM_CHUNK_SIZE, comfortably under
Cloudflare's cap) via the same streaming MultipartEncoder approach as
before.
Co-authored-by: Copilot <[email protected]>
First real-hardware test of dashcam retrieval kept crashing the whole
yaris-obd service (repeated SIGKILL/OOM on the Pi 3's ~900MB RAM) a few
minutes into every run, aborting in-flight file copies. Root cause:
_upload_dashcam_files() used requests.post(files={"file": f}), which
builds the entire multipart/form-data body in memory before sending -
fine for small payloads, but this dashcam's real clips average ~320MB
each (up to a couple hundred more with overhead), so every dashcam
upload attempt alone could push the process well past 300-600MB, and
the Pi's OOM killer stepped in immediately.
Reproduced in isolation: uploading a 300MB file the old way spiked the
client process from 28MB to 628MB RSS; switching to requests-toolbelt's
MultipartEncoder (which streams the file in chunks as the request body
is sent, instead of materializing it up front) kept it flat at ~29MB
for the same upload, with an identical byte-for-byte result on the
receiving end. No server-side changes needed - the wire format is
unchanged, still ordinary multipart/form-data.
Added requests-toolbelt to requirements.txt.
Co-authored-by: Copilot <[email protected]>
mount_device() used a bare os.makedirs() to create the mount point,
but the app runs as an unprivileged user while /mnt is root-owned -
surfaced as a real PermissionError during the first actual hardware
test. Create the mount point via sudo mkdir -p instead, consistent
with the existing sudo mount/umount calls in this module.
Co-authored-by: Copilot <[email protected]>
First real-hardware test revealed the connected dashcam's SD card ships
factory-formatted with no partition table - lsblk reports it as a bare
"disk" with an fstype set directly, never as a child "part". The
previous detection logic only matched type=="part", so it silently
missed this real device entirely.
list_usb_storage_partitions() now also matches whole-disk USB devices
that have no children and an fstype set directly on them, in addition
to the existing partition-table case. Verified against a fixture built
from this dashcam's actual `lsblk -J` output.
Co-authored-by: Copilot <[email protected]>
modules/app_cellular.py: uses the SIM868's built-in AT-command HTTP(S)
client (AT+HTTPINIT/HTTPPARA/HTTPACTION/HTTPREAD) instead of a real
ppp0 interface, since the module's GNSS (polled continuously by
app_gps.py) and its GSM/GPRS radio share the same physical UART
(/dev/serial0) - a pppd/chat dial-up would take over that port
entirely and starve GNSS polling. Staying in AT-command mode lets
cellular HTTP calls interleave safely through the same serial lock.
Only carries small JSON tracking batches, never dashcam video -
cellular is far too slow/limited for that, and the modem's HTTP AT
interface has too small an internal buffer anyway. Dashcam uploads
remain WiFi/Ethernet-only, unchanged.
modules/app_upload.py: tracking-batch upload always tries WiFi/
Ethernet first; only on a connection-level failure (not an HTTP error
status, which means the server was reached) does it retry over
cellular. Exposes last_network_path/cellular_fallback_configured in
/api/upload/status; UI shows which path the last batch used.
Feature-flagged off by default via new YARIS_CELLULAR_ENABLED/
YARIS_CELLULAR_APN/_APN_USER/_APN_PASS keys in db/upload_creds.env
(all blank by default) - has zero effect on the existing WiFi upload
path unless explicitly configured. Regression-tested against the real
deployed server that normal WiFi uploads are unaffected while
cellular stays unconfigured.
*** UNTESTED against real cellular hardware - no SIM card was
*** available. See docstrings in app_cellular.py and PLAN.md for
*** exactly what needs validation once a SIM card is available.
Co-authored-by: Copilot <[email protected]>
Reflects the completed deployment: server running on unraid, exposed
at https://yaris.ceaf.dev via Cloudflare Tunnel, Pi-side upload client
live and backlog-drain tuned.
Co-authored-by: Copilot <[email protected]>
The upload loop previously always slept the full 30s UPLOAD_INTERVAL
between batches, even when a full batch was just sent (meaning more
rows were likely still queued). With GPS/OBD rows logged every 5s
locally, an offline/stale device could build up a large backlog that
took a very long time to drain at 100 rows/30s.
- Bump TRACKING_BATCH_SIZE 100 -> 500 (cuts round-trips per backlog).
- Only sleep the full UPLOAD_INTERVAL once actually caught up (a
short batch came back); while a full batch keeps coming back, drain
back-to-back with a short CATCHUP_INTERVAL (1s) instead.
- server/app.py: add a sane MAX_TRACKING_BATCH_POINTS (5000) cap on
the tracking/batch endpoint as a defensive guard against a runaway
client, independent of the Pi's own batch size.
Verified against the real deployed server: a simulated 1250-row
backlog now drains in 3 batches (~0.35s server round-trip time) instead
of 13 batches spread across ~6.5 minutes of idle sleeping.
Co-authored-by: Copilot <[email protected]>
- app_sql.py: add UPLOADED column + migration on tracking table, add
settings and dashcam_uploads tables, add helpers for unsent-row
queries, marking rows uploaded, and dashcam upload dedupe tracking.
- app_upload.py: new background upload client. Periodically batches
unsent tracking rows to POST /api/v1/tracking/batch and uploads new
dashcam files (sha256-verified) to POST /api/v1/dashcam/upload, with
exponential backoff on failure. Exposes get_status() and a
dashcam-upload enable/disable toggle backed by the settings table.
- app.py: start the upload client thread at startup, add
GET /api/upload/status and POST /api/upload/dashcam_toggle routes.
- dash.html/dash.js: add a "Remote server upload" card with upload
status and a toggle switch to pause dashcam video uploads while
trip/GPS log uploads keep running (per user request - useful on a
slow/metered connection).
- db/upload_creds.env: new gitignore-exempted template (mirrors
db/creds.env) for YARIS_SERVER_URL/YARIS_DEVICE_NAME/YARIS_API_KEY.
- requirements.txt: add requests.
Tested against the real deployed unraid server: tracking batch upload
+ dedupe, dashcam upload + dedupe, and toggle-off correctly pausing
dashcam uploads while tracking uploads continue. Also verified the
full Flask app imports/runs cleanly and the new API routes + dash.html
toggle render correctly in a headless browser.
Co-authored-by: Copilot <[email protected]>
Squashed rewrite of the project revival work (replaces prior history on this
branch to remove personal data that had been accidentally committed - real
LAN IP, GPS coordinates, home WiFi SSID, and phone model). See PLAN.md for
full phase-by-phase details. Summary:
- Phase 2: adapt app_obd.py for a USB OBD-II adapter (was Bluetooth-only),
auto-detect /dev/ttyUSB*, fix several pre-existing crash bugs.
- Phase 3/4: make the web UI work fully offline (vendored Socket.IO/Leaflet/
Chart.js/FontAwesome, offline map tile downloader script, live GPS marker
tracking fix).
- Phase 5: systemd deployment/autostart on the Pi.
- Phase 6: WiFi hotspot mode via NetworkManager (auto-switch between home
WiFi and a local AP for in-car web UI access), mDNS hostname + port 80
redirect so the UI is reachable at a plain hostname with no port number.
- Phase 7: fix app_db.py/app_sql.py bugs, wire up local SQLite trip logging.
- Phase 8: brand-agnostic USB dashcam file retrieval (lsblk transport-based
detection, works with any dashcam brand/folder layout).
- Rewritten readme.md with a proper project overview.
Co-authored-by: Copilot <[email protected]>
