Fix OOM crash during dashcam upload: stream multipart body instead of buffering in memory

First real-hardware test of dashcam retrieval kept crashing the whole
yaris-obd service (repeated SIGKILL/OOM on the Pi 3's ~900MB RAM) a few
minutes into every run, aborting in-flight file copies. Root cause:
_upload_dashcam_files() used requests.post(files={"file": f}), which
builds the entire multipart/form-data body in memory before sending -
fine for small payloads, but this dashcam's real clips average ~320MB
each (up to a couple hundred more with overhead), so every dashcam
upload attempt alone could push the process well past 300-600MB, and
the Pi's OOM killer stepped in immediately.

Reproduced in isolation: uploading a 300MB file the old way spiked the
client process from 28MB to 628MB RSS; switching to requests-toolbelt's
MultipartEncoder (which streams the file in chunks as the request body
is sent, instead of materializing it up front) kept it flat at ~29MB
for the same upload, with an identical byte-for-byte result on the
receiving end. No server-side changes needed - the wire format is
unchanged, still ordinary multipart/form-data.

Added requests-toolbelt to requirements.txt.

Co-authored-by: Copilot <[email protected]>
This commit is contained in:
Mathieu Moran
2026-08-08 21:56:22 -04:00
co-authored by Copilot
parent 833f008671
commit f88b57d368
2 changed files with 20 additions and 4 deletions
+18 -3
View File
@@ -31,6 +31,7 @@ from threading import Thread
import requests
from dotenv import load_dotenv
from requests_toolbelt import MultipartEncoder
from modules import app_sql, app_dash, app_cellular
@@ -175,12 +176,26 @@ def _upload_dashcam_files():
if app_sql.dashcam_file_marked_uploaded(rel_path, sha256):
continue
# Stream the multipart body via MultipartEncoder instead of requests'
# plain files=/data= kwargs. requests builds a non-streamed files=
# upload by reading the *entire* file into memory up front to
# assemble the multipart body - fine for small files, but real
# dashcam clips run 300+MB each, and on a ~1GB-RAM Pi 3 that alone is
# enough to trigger the OOM killer (confirmed against real hardware:
# RSS spiked in lockstep with dashcam file size). MultipartEncoder
# reads the file in small chunks as the request is sent instead.
with open(abs_path, "rb") as f:
encoder = MultipartEncoder(
fields={
"relative_path": rel_path,
"sha256": sha256,
"file": (os.path.basename(abs_path), f, "application/octet-stream"),
}
)
resp = requests.post(
f"{SERVER_URL}/api/v1/dashcam/upload",
data={"relative_path": rel_path, "sha256": sha256},
files={"file": f},
headers=_headers(),
data=encoder,
headers={**_headers(), "Content-Type": encoder.content_type},
timeout=300,
)
resp.raise_for_status()
+1
View File
@@ -6,3 +6,4 @@ python-dotenv
pyserial
obd
requests
requests-toolbelt