Fix OOM crash during dashcam upload: stream multipart body instead of buffering in memory
First real-hardware test of dashcam retrieval kept crashing the whole
yaris-obd service (repeated SIGKILL/OOM on the Pi 3's ~900MB RAM) a few
minutes into every run, aborting in-flight file copies. Root cause:
_upload_dashcam_files() used requests.post(files={"file": f}), which
builds the entire multipart/form-data body in memory before sending -
fine for small payloads, but this dashcam's real clips average ~320MB
each (up to a couple hundred more with overhead), so every dashcam
upload attempt alone could push the process well past 300-600MB, and
the Pi's OOM killer stepped in immediately.
Reproduced in isolation: uploading a 300MB file the old way spiked the
client process from 28MB to 628MB RSS; switching to requests-toolbelt's
MultipartEncoder (which streams the file in chunks as the request body
is sent, instead of materializing it up front) kept it flat at ~29MB
for the same upload, with an identical byte-for-byte result on the
receiving end. No server-side changes needed - the wire format is
unchanged, still ordinary multipart/form-data.
Added requests-toolbelt to requirements.txt.
Co-authored-by: Copilot <[email protected]>
This commit is contained in:
+18
-3
@@ -31,6 +31,7 @@ from threading import Thread
|
||||
|
||||
import requests
|
||||
from dotenv import load_dotenv
|
||||
from requests_toolbelt import MultipartEncoder
|
||||
|
||||
from modules import app_sql, app_dash, app_cellular
|
||||
|
||||
@@ -175,12 +176,26 @@ def _upload_dashcam_files():
|
||||
if app_sql.dashcam_file_marked_uploaded(rel_path, sha256):
|
||||
continue
|
||||
|
||||
# Stream the multipart body via MultipartEncoder instead of requests'
|
||||
# plain files=/data= kwargs. requests builds a non-streamed files=
|
||||
# upload by reading the *entire* file into memory up front to
|
||||
# assemble the multipart body - fine for small files, but real
|
||||
# dashcam clips run 300+MB each, and on a ~1GB-RAM Pi 3 that alone is
|
||||
# enough to trigger the OOM killer (confirmed against real hardware:
|
||||
# RSS spiked in lockstep with dashcam file size). MultipartEncoder
|
||||
# reads the file in small chunks as the request is sent instead.
|
||||
with open(abs_path, "rb") as f:
|
||||
encoder = MultipartEncoder(
|
||||
fields={
|
||||
"relative_path": rel_path,
|
||||
"sha256": sha256,
|
||||
"file": (os.path.basename(abs_path), f, "application/octet-stream"),
|
||||
}
|
||||
)
|
||||
resp = requests.post(
|
||||
f"{SERVER_URL}/api/v1/dashcam/upload",
|
||||
data={"relative_path": rel_path, "sha256": sha256},
|
||||
files={"file": f},
|
||||
headers=_headers(),
|
||||
data=encoder,
|
||||
headers={**_headers(), "Content-Type": encoder.content_type},
|
||||
timeout=300,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
+2
-1
@@ -5,4 +5,5 @@ pymongo
|
||||
python-dotenv
|
||||
pyserial
|
||||
obd
|
||||
requests
|
||||
requests
|
||||
requests-toolbelt
|
||||
Reference in New Issue
Block a user