fix: BUG-06 — convert naive Pi timestamps to TZ-aware before upload so Postgres TIMESTAMPTZ stores correct instant

This commit is contained in:
Mathieu Moran
2026-08-11 14:47:37 -04:00
parent 9733036510
commit 7ce2a1e5f5
3 changed files with 39 additions and 9 deletions
+11 -6
View File
@@ -28,15 +28,20 @@ When CLOSED, move the entire entry to Resolved — don't duplicate.
--- ---
### BUG-06 — Server timestamps are shifted (naive local time stored as UTC) ### BUG-06 — Server timestamps are shifted (naive local time stored as UTC)
- **Status**: `OPEN` - **Status**: `IN PROGRESS` (fix deployed 2026-08-11, needs new-data verification)
- **Severity**: High-ish - **Severity**: High-ish
- **Area**: `modules/app_upload.py`, `server/sql`, `server/modules/db.py`, portal - **Area**: `modules/app_upload.py`, `server/sql`, `server/modules/db.py`, portal
- **Phase**: Phase 26 - **Phase**: Phase 26
- **Symptom**: Server trip times shifted by EDT offset (~4-5h). - **Symptom**: Server trip times shifted by EDT offset (~4-5h). Pi stores naive
- **Root cause**: Pi sends naive local time; Postgres `TIMESTAMPTZ` interprets as UTC. local time; Postgres `TIMESTAMPTZ` interprets it as UTC.
- **Fix scope**: Pi sends timezone-aware timestamps. Server stores/reads consistently. - **Fix**: New `_to_tz_aware()` helper in `app_upload.py` converts naive SQLite
One-time migration for existing rows — **get owner sign-off**. timestamps to timezone-aware ISO strings before upload (e.g.,
- **Acceptance**: New data at correct local time; existing rows corrected or accepted. `2026-08-10 19:27:30` → `2026-08-10T19:27:30-04:00`). Already-aware timestamps
pass through unchanged. **Existing rows on the server are NOT migrated** — only
new uploads benefit from this fix. Old trips will still show the wrong time
until a one-time server-side `UPDATE` migration is run (needs owner sign-off).
- **Acceptance**: New trips uploaded after this fix display correct local time
on both portal and Pi.
--- ---
+8 -2
View File
@@ -1860,7 +1860,7 @@ phase roadmap.
--- ---
### Phase 26: Road-test bug fixes — 🚧 IN PROGRESS (7 closed, 1 open, 1 hardware) ### Phase 26: Road-test bug fixes — 🚧 IN PROGRESS (7 closed, 1 deployed awaiting test, 1 open, 1 hardware)
**Last updated 2026-08-11.** Targeted bug-fix pass for defects surfaced by the Phase 25 in-car **Last updated 2026-08-11.** Targeted bug-fix pass for defects surfaced by the Phase 25 in-car
road test. Each item tracked in [BUGS.md](./BUGS.md). road test. Each item tracked in [BUGS.md](./BUGS.md).
@@ -1890,11 +1890,17 @@ rather than incremental patching:
- **Same JS file works on both Pi and server**: tile URL/attrib/zoom defaults read from - **Same JS file works on both Pi and server**: tile URL/attrib/zoom defaults read from
`data-*` attributes on `#history-map`, with Pi-local fallbacks when absent. `data-*` attributes on `#history-map`, with Pi-local fallbacks when absent.
**Deployed, awaiting new-data verification:**
- BUG-06: `_to_tz_aware()` helper in `app_upload.py` converts naive SQLite datetimes
to TZ-aware ISO strings (e.g., `2026-08-10T19:27:30-04:00`) before upload. Postgres
`TIMESTAMPTZ` now stores the correct instant. Existing server rows NOT migrated —
only new uploads benefit until a one-time UPDATE is run.
**Deployed and verified on both platforms (2026-08-11):** **Deployed and verified on both platforms (2026-08-11):**
- BUG-01: Dual-radio setup — wlan0 is permanent AP at 10.42.0.1, wlan1 handles internet uplink only. `yaris_uplink.py` no longer touches the AP. In-car access card on Settings page shows hotspot URL + cell-data warning. Netwatch fixed to not tear down AP when clients connected (backup, not needed with dual-radio). - BUG-01: Dual-radio setup — wlan0 is permanent AP at 10.42.0.1, wlan1 handles internet uplink only. `yaris_uplink.py` no longer touches the AP. In-car access card on Settings page shows hotspot URL + cell-data warning. Netwatch fixed to not tear down AP when clients connected (backup, not needed with dual-radio).
- BUG-07 history.js rewrite (see above) — rapid trip-switching no longer freezes, loading spinner + active row highlight provide clear feedback. Verified on Pi and server. - BUG-07 history.js rewrite (see above) — rapid trip-switching no longer freezes, loading spinner + active row highlight provide clear feedback. Verified on Pi and server.
**Still open:** BUG-06 (timezone shift). **Still open:** None.
BUG-02 is hardware (Pi micro-USB port damage) — owner's responsibility. BUG-02 is hardware (Pi micro-USB port damage) — owner's responsibility.
**Acceptance:** every `BUGS.md` entry in scope is fixed, regression-tested, deployed, and marked resolved. **Acceptance:** every `BUGS.md` entry in scope is fixed, regression-tested, deployed, and marked resolved.
+20 -1
View File
@@ -38,6 +38,7 @@ import hashlib
import json import json
import os import os
import time import time
from datetime import datetime
from threading import Thread from threading import Thread
import requests import requests
@@ -139,6 +140,24 @@ def _hash_dashcam_file(abs_path, rel_path):
return sha256 return sha256
def _to_tz_aware(dt_str):
"""Convert a naive datetime string (from SQLite) to a timezone-aware ISO
string the server's TIMESTAMPTZ column will interpret correctly.
SQLite stores naive local-time strings like '2026-08-10 19:27:30.123456'.
Without a timezone, Postgres interprets these as UTC, shifting displayed
times by the local offset. This helper attaches the Pi's local IANA
offset so the server stores and displays the correct local time.
"""
try:
dt = datetime.fromisoformat(str(dt_str))
except (TypeError, ValueError):
return dt_str # unparseable — send as-is and let the server reject it
if dt.tzinfo is not None:
return dt_str # already TZ-aware — nothing to do
return dt.astimezone().isoformat()
def _upload_tracking_batch(): def _upload_tracking_batch():
"""Send up to TRACKING_BATCH_SIZE not-yet-uploaded rows. Returns count sent. """Send up to TRACKING_BATCH_SIZE not-yet-uploaded rows. Returns count sent.
@@ -159,7 +178,7 @@ def _upload_tracking_batch():
points = [ points = [
{ {
"recorded_at": row["datetime"], "recorded_at": _to_tz_aware(row["datetime"]),
"gps_lat": row["gpsx"], "gps_lat": row["gpsx"],
"gps_lon": row["gpsy"], "gps_lon": row["gpsy"],
"speed": row["speed"], "speed": row["speed"],