quoije/yaris-obd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add SIM868 cellular fallback for tracking uploads (untested, no SIM card)

Browse Source 
modules/app_cellular.py: uses the SIM868's built-in AT-command HTTP(S)
client (AT+HTTPINIT/HTTPPARA/HTTPACTION/HTTPREAD) instead of a real
ppp0 interface, since the module's GNSS (polled continuously by
app_gps.py) and its GSM/GPRS radio share the same physical UART
(/dev/serial0) - a pppd/chat dial-up would take over that port
entirely and starve GNSS polling. Staying in AT-command mode lets
cellular HTTP calls interleave safely through the same serial lock.

Only carries small JSON tracking batches, never dashcam video -
cellular is far too slow/limited for that, and the modem's HTTP AT
interface has too small an internal buffer anyway. Dashcam uploads
remain WiFi/Ethernet-only, unchanged.

modules/app_upload.py: tracking-batch upload always tries WiFi/
Ethernet first; only on a connection-level failure (not an HTTP error
status, which means the server was reached) does it retry over
cellular. Exposes last_network_path/cellular_fallback_configured in
/api/upload/status; UI shows which path the last batch used.

Feature-flagged off by default via new YARIS_CELLULAR_ENABLED/
YARIS_CELLULAR_APN/_APN_USER/_APN_PASS keys in db/upload_creds.env
(all blank by default) - has zero effect on the existing WiFi upload
path unless explicitly configured. Regression-tested against the real
deployed server that normal WiFi uploads are unaffected while
cellular stays unconfigured.

*** UNTESTED against real cellular hardware - no SIM card was
*** available. See docstrings in app_cellular.py and PLAN.md for
*** exactly what needs validation once a SIM card is available.

Co-authored-by: Copilot <[email protected]>
This commit is contained in:
Mathieu Moran
2026-08-08 20:34:53 -04:00
co-authored by Copilot
parent cea62d8c10
commit 6d61b70a06
6 changed files with 311 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PLAN.md
+30 -3
View File
@@ -353,10 +353,37 @@ dedupe + path-traversal rejection, bad-API-key rejection, toggle-off correctly p
dashcam uploads) and on the real Pi (its actual ~1800-row pre-feature backlog drained and
confirmed matching row counts on the server via direct psql query, live upload status visible on
`/dash`). Committed+pushed (`4ff99f0`, `c4db1ab`, `8dc304e`), deployed on both unraid and the Pi.
Exposed externally at `https://yaris.ceaf.dev` via the user's Cloudflare Tunnel (public hostname
added in the Cloudflare dashboard, pointing at `192.168.10.10:8098`).
Not attempted (future, no action needed unless requested): SIM868 GSM/GPRS cellular fallback
uplink for uploads when no network path to the server is available at all (today uploads need
the Pi to reach `yaris.ceaf.dev`, e.g. via home WiFi) — remains a possible future enhancement.
### SIM868 cellular fallback for tracking uploads — built, UNTESTED (no SIM card available)
`modules/app_cellular.py` gives the tracking-upload path a fallback over the SIM868's cellular
data connection when WiFi/Ethernet is unreachable. Design notes:
- Uses the SIM868's built-in AT-command HTTP(S) client (`AT+HTTPINIT`/`HTTPPARA`/`HTTPACTION`/
`HTTPREAD`) rather than a real `ppp0` network interface. Reason: the SIM868's GNSS (polled
continuously by `modules/app_gps.py`) and its GSM/GPRS radio share the *same* physical UART
(`/dev/serial0`) — a classic `pppd`/`chat` dial-up would put that port into raw PPP framing for
as long as the link is up, starving the GNSS polling thread. Staying in AT-command mode lets
cellular HTTP calls interleave safely through the same serial lock
(`modules.app_gps._serial_lock`) — each exchange just briefly delays the next GPS poll.
- Only carries small JSON tracking batches, **never** dashcam video — cellular data is far too
slow/limited for multi-MB video files, and the modem's HTTP AT interface has a small internal
buffer unsuited to them anyway. Dashcam uploads remain WiFi/Ethernet-only, unchanged.
- `modules/app_upload.py`'s tracking-batch upload always tries WiFi/Ethernet (`requests`) first;
only on a connection-level failure (no route/DNS/timeout — as opposed to an HTTP error status,
which means the server WAS reached) does it retry the same batch over cellular.
- Feature-flagged off by default: `YARIS_CELLULAR_ENABLED`/`YARIS_CELLULAR_APN`/
`YARIS_CELLULAR_APN_USER`/`YARIS_CELLULAR_APN_PASS` added to `db/upload_creds.env`, all blank
by default so this has zero effect on the already-verified WiFi upload path until the user
fills in an APN once they get a SIM card. Regression-tested (against the real server) that
normal WiFi uploads are completely unaffected while cellular is unconfigured.
- `/dash` upload-status card now shows which path (`wifi`/`cellular`) the last successful batch
used, and whether cellular fallback is configured.
- **Needs validation once a SIM card is available** — exact AT command dialect/timing (SIMCOM
firmware revisions vary slightly), and whether `AT+HTTPSSL` works for the HTTPS
`yaris.ceaf.dev` endpoint on this specific module's firmware (if not, a documented fallback is
a plain-HTTP router port-forward path, avoiding the need for the modem's TLS stack to validate
a cert chain).
## Working Notes / Environment Gotchas
db/upload_creds.env
+14
View File
@@ -10,3 +10,17 @@
YARIS_SERVER_URL = ""
YARIS_DEVICE_NAME = ""
YARIS_API_KEY = ""
##
## Optional SIM868 cellular fallback for trip/GPS-log uploads only (NOT
## dashcam video - too slow/limited over GPRS/3G). Only takes effect when a
## WiFi/Ethernet upload attempt fails outright (no route/DNS/timeout).
## UNTESTED - no SIM card was available during development, see
## modules/app_cellular.py. Leave YARIS_CELLULAR_ENABLED blank to keep this
## fully disabled (default) with zero effect on normal WiFi uploads.
##
YARIS_CELLULAR_ENABLED = ""
YARIS_CELLULAR_APN = ""
YARIS_CELLULAR_APN_USER = ""
YARIS_CELLULAR_APN_PASS = ""
modules/app_cellular.py
+194
View File
@@ -0,0 +1,194 @@
"""SIM868 GSM/GPRS cellular fallback for uploading trip/GPS tracking batches
when there is no WiFi/Ethernet path to the remote server.
*** UNTESTED - no SIM card was available to validate this against a real
*** cellular network. Written defensively (every AT exchange is wrapped so a
*** failure just raises CellularError, never crashes the app) but the exact
*** AT command dialect/timing may need small tweaks once a SIM is inserted
*** and this can actually be exercised. See "Verifying on real hardware"
*** below and the Phase 9 notes in PLAN.md.
Why AT-command HTTP instead of a real ppp0 network interface: the SIM868's
GNSS (used by modules/app_gps.py) and its GSM/GPRS radio are both driven over
the *same* physical UART (/dev/serial0). A classic pppd/chat dial-up would put
that UART into raw PPP framing for as long as the link is up, which would
starve/conflict with the GNSS AT+CGNSINF polling already running continuously
in app_gps.py. SIMCOM modules instead expose a built-in "HTTP(S) Application"
AT command set (AT+HTTPINIT / AT+HTTPPARA / AT+HTTPACTION / AT+HTTPREAD) that
stays in normal AT command mode the whole time, so it can safely interleave
with GNSS polling through the same serial lock (modules.app_gps._serial_lock) -
each exchange just briefly delays the next GPS poll, it never takes over the
port.
This only carries small JSON tracking-batch payloads - dashcam video uploads
remain WiFi/Ethernet-only (see modules/app_upload.py), both because cellular
data is far too slow/limited for video files and because the modem's HTTP
AT interface has a small internal buffer unsuited to multi-MB bodies.
"""
import os
import time
from modules import app_gps
APN = os.getenv("YARIS_CELLULAR_APN", "")
APN_USER = os.getenv("YARIS_CELLULAR_APN_USER", "")
APN_PASS = os.getenv("YARIS_CELLULAR_APN_PASS", "")
ENABLED = os.getenv("YARIS_CELLULAR_ENABLED", "") == "1"
GPRS_BRINGUP_TIMEOUT = 30 # seconds to wait for AT+CIICR / AT+CIFSR
HTTP_ACTION_TIMEOUT = 60 # seconds to wait for the +HTTPACTION URC (2G/3G can be slow)
_gprs_context_up = False
class CellularError(Exception):
"""Raised for any cellular-modem/AT-command failure - always caught by
the caller (modules/app_upload.py), never allowed to crash the app."""
def is_configured():
"""True once the user has filled in an APN and flipped the enabled flag
in db/upload_creds.env. Off by default - nothing cellular-related runs
unless explicitly opted into."""
return ENABLED and bool(APN)
def _read_until(ser, terminators, timeout):
"""Read from the modem until one of the given substrings shows up in the
accumulated buffer, or timeout. Returns the buffer (decoded, best-effort)."""
deadline = time.time() + timeout
buf = ""
while time.time() < deadline:
chunk = ser.read(1024)
if chunk:
buf += chunk.decode(errors="replace")
if any(t in buf for t in terminators):
return buf
else:
time.sleep(0.1)
return buf
def _at(ser, cmd, terminators=("OK", "ERROR"), timeout=5):
"""Send one AT command and wait for a terminator. Raises CellularError
if the modem responds with ERROR or nothing recognizable within timeout."""
ser.reset_input_buffer()
ser.write((cmd + "\r\n").encode())
resp = _read_until(ser, terminators, timeout)
if "ERROR" in resp and "OK" not in resp:
raise CellularError(f"{cmd!r} -> {resp!r}")
if not any(t in resp for t in terminators):
raise CellularError(f"{cmd!r} timed out, got {resp!r}")
return resp
def _ensure_gprs_context():
"""Bring up the GPRS data context (APN attach) if not already up.
Reuses app_gps's serial connection/lock - see module docstring for why.
"""
global _gprs_context_up
with app_gps._serial_lock:
ser = app_gps._get_serial()
if _gprs_context_up:
# Cheap check that the context is still alive before trusting it.
try:
resp = _at(ser, "AT+CIFSR", terminators=("\r\n",), timeout=5)
if "." in resp and "ERROR" not in resp:
return
except CellularError:
pass
_gprs_context_up = False
_at(ser, "AT+CGATT=1", timeout=10)
apn_cmd = f'AT+CSTT="{APN}","{APN_USER}","{APN_PASS}"'
_at(ser, apn_cmd, timeout=10)
_at(ser, "AT+CIICR", timeout=GPRS_BRINGUP_TIMEOUT)
resp = _at(ser, "AT+CIFSR", terminators=("\r\n",), timeout=10)
if "ERROR" in resp or "." not in resp:
raise CellularError(f"AT+CIFSR did not return an IP: {resp!r}")
_gprs_context_up = True
def http_post_json(url, body_bytes, extra_headers, timeout=HTTP_ACTION_TIMEOUT):
"""POST a small JSON body to `url` via the modem's built-in HTTP client.
Returns (status_code, response_body_bytes). Raises CellularError on any
failure (modem not responding, no network attach, HTTP-layer failure,
etc.) - callers should catch this broadly and treat it the same as a
network-unreachable condition.
"""
if not is_configured():
raise CellularError("cellular fallback not configured (APN/enabled flag not set)")
_ensure_gprs_context()
with app_gps._serial_lock:
ser = app_gps._get_serial()
try:
_at(ser, "AT+HTTPTERM", timeout=3)
except CellularError:
pass # fine if nothing was initialized yet
_at(ser, "AT+HTTPINIT", timeout=5)
try:
_at(ser, 'AT+HTTPPARA="CID",1', timeout=5)
_at(ser, f'AT+HTTPPARA="URL","{url}"', timeout=5)
if url.lower().startswith("https"):
try:
_at(ser, "AT+HTTPSSL=1", timeout=5)
except CellularError:
# Older SIMCOM firmware/certificate stores sometimes
# don't support this - surfacing as a clear error is more
# useful than silently falling back to plain HTTP against
# an HTTPS-only endpoint.
raise CellularError(
"modem does not support AT+HTTPSSL - HTTPS upload via "
"cellular is unavailable on this firmware. Consider a "
"plain-HTTP router port-forward path for cellular use."
)
_at(ser, 'AT+HTTPPARA="CONTENT","application/json"', timeout=5)
header_lines = "".join(f"{k}: {v}\r\n" for k, v in extra_headers.items())
_at(ser, f'AT+HTTPPARA="USERDATA","{header_lines}"', timeout=5)
_at(
ser,
f"AT+HTTPDATA={len(body_bytes)},{int(timeout * 1000)}",
terminators=("DOWNLOAD", "ERROR"),
timeout=5,
)
ser.write(body_bytes)
_read_until(ser, ("OK", "ERROR"), timeout=10)
action_resp = _at(
ser,
"AT+HTTPACTION=1",
terminators=("+HTTPACTION:",),
timeout=timeout,
)
status_code = 0
for line in action_resp.splitlines():
if "+HTTPACTION:" in line:
# +HTTPACTION: <method>,<status_code>,<data_len>
parts = line.split(":", 1)[1].strip().split(",")
if len(parts) >= 2:
status_code = int(parts[1])
break
read_resp = _at(ser, "AT+HTTPREAD", terminators=("OK", "ERROR"), timeout=15)
body = read_resp.encode(errors="replace")
return status_code, body
finally:
try:
_at(ser, "AT+HTTPTERM", timeout=5)
except CellularError:
pass
modules/app_upload.py
+62 -11
View File
@@ -6,6 +6,20 @@ paused independently via a persisted toggle (settings table in the same
SQLite db used for tracking) so a user can stop shipping large video files
over a slow/metered connection while trip logs keep flowing.
Network path: normal WiFi/Ethernet (via `requests`) is always tried first for
everything. If a tracking-batch upload fails with a connection-level error
(no route/DNS/timeout - not an HTTP error status, which means the server WAS
reached) and the SIM868 cellular fallback is configured, the same small JSON
batch is retried over the modem's AT-command HTTP client (see
modules/app_cellular.py). Dashcam uploads never use cellular - only
WiFi/Ethernet - since video files are far too large/slow for a GPRS/3G link;
they simply wait for connectivity to return, same as before.
*** The cellular fallback path is UNTESTED (no SIM card available) - see
*** modules/app_cellular.py's docstring. It's opt-in/feature-flagged off by
*** default (YARIS_CELLULAR_ENABLED unset) so this has zero effect on the
*** already-verified WiFi upload path unless explicitly configured.
Credentials/server URL are loaded from db/upload_creds.env (gitignored,
never commit real values - mirrors the db/creds.env pattern already used by
app_db.py for MongoDB).
@@ -18,7 +32,7 @@ from threading import Thread
import requests
from dotenv import load_dotenv
from modules import app_sql, app_dash
from modules import app_sql, app_dash, app_cellular
_BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
load_dotenv(os.path.join(_BASE_DIR, "db", "upload_creds.env"))
@@ -38,6 +52,7 @@ DASHCAM_UPLOAD_SETTING_KEY = "dashcam_upload_enabled"
_upload_thread = None
_last_error = None
_last_success_at = None
_last_network_path = None # "wifi" or "cellular" - last path a batch actually went out on
def _headers():
@@ -65,6 +80,8 @@ def get_status():
"pending_tracking_rows": app_sql.count_unsent_tracking_rows(),
"last_error": _last_error,
"last_success_at": _last_success_at,
"last_network_path": _last_network_path,
"cellular_fallback_configured": app_cellular.is_configured(),
}
@@ -80,7 +97,16 @@ def _sha256_file(path):
def _upload_tracking_batch():
"""Send up to TRACKING_BATCH_SIZE not-yet-uploaded rows. Returns count sent."""
"""Send up to TRACKING_BATCH_SIZE not-yet-uploaded rows. Returns count sent.
Tries WiFi/Ethernet (`requests`) first. If that fails at the connection
level (no route, DNS failure, timeout - as opposed to the server
responding with an HTTP error, which means it WAS reached) and the
SIM868 cellular fallback is configured, retries the same batch over the
modem's AT-command HTTP client before giving up.
"""
global _last_network_path
rows = app_sql.get_unsent_tracking_rows(limit=TRACKING_BATCH_SIZE)
if not rows:
return 0
@@ -97,15 +123,40 @@ def _upload_tracking_batch():
}
for row in rows
]
row_ids = [row["id"] for row in rows]
resp = requests.post(
f"{SERVER_URL}/api/v1/tracking/batch",
json={"points": points},
headers=_headers(),
timeout=15,
)
resp.raise_for_status()
app_sql.mark_tracking_rows_uploaded([row["id"] for row in rows])
try:
resp = requests.post(
f"{SERVER_URL}/api/v1/tracking/batch",
json={"points": points},
headers=_headers(),
timeout=15,
)
resp.raise_for_status()
_last_network_path = "wifi"
except requests.exceptions.ConnectionError as wifi_err:
if not app_cellular.is_configured():
raise
print(f"WiFi/Ethernet upload unreachable ({wifi_err}), trying cellular fallback...")
import json as _json
body_bytes = _json.dumps({"points": points}).encode()
try:
status_code, _body = app_cellular.http_post_json(
f"{SERVER_URL}/api/v1/tracking/batch", body_bytes, _headers()
)
except app_cellular.CellularError as cell_err:
# Neither path worked - raise the original WiFi error so the
# normal backoff/retry logic in _upload_loop handles it.
print(f"Cellular fallback also failed: {cell_err}")
raise wifi_err
if status_code >= 400:
raise app_cellular.CellularError(
f"cellular upload got HTTP {status_code}"
)
_last_network_path = "cellular"
app_sql.mark_tracking_rows_uploaded(row_ids)
return len(rows)
@@ -172,7 +223,7 @@ def _upload_loop():
time.sleep(CATCHUP_INTERVAL)
else:
time.sleep(UPLOAD_INTERVAL)
except requests.RequestException as e:
except (requests.RequestException, app_cellular.CellularError) as e:
_last_error = str(e)
print(f"Upload failed ({e}), retrying in {backoff}s")
time.sleep(backoff)
static/js/dash.js
+8 -1
View File
@@ -75,7 +75,14 @@ function describeUploadStatus(status) {
}
let text = `Pending trip-log rows to upload: ${status.pending_tracking_rows}.`;
if (status.last_success_at) {
text += ` Last successful upload: ${status.last_success_at}.`;
text += ` Last successful upload: ${status.last_success_at}`;
if (status.last_network_path) {
text += ` (via ${status.last_network_path === 'cellular' ? 'cellular fallback' : 'WiFi/Ethernet'})`;
}
text += '.';
}
if (status.cellular_fallback_configured) {
text += ' Cellular (SIM868) fallback is enabled for trip logs if WiFi/Ethernet is unreachable.';
}
if (status.last_error) {
text += ` Last error: ${status.last_error}.`;
templates/dash.html
+3 -1
View File
@@ -91,7 +91,9 @@
<p class="text-muted small mt-2 mb-0">
Trip/GPS log uploads always stay on. Turning this off only pauses
the (larger) dashcam video uploads - useful on a slow or metered
connection.
connection. If configured, small trip-log batches can also fall
back to the SIM868's cellular data when WiFi/Ethernet is
unreachable - dashcam video never uses cellular.
</p>
</div>
</div>