android: byte-level upload progress + debug logging; document 28.3

- UploadWorker: ProgressRequestBody (okio ForwardingSink) tracks real bytes sent including already-received chunks, instead of chunk count - Robust chunk reads (loop skip/read) + Log.d/Log.e debug logging - DashcamViewModel: upload state/progress debug logging - android/plan.md + PLAN.md: 28.3 marked in progress with built items
2026-08-12 19:00:00 -04:00
parent 6398baaa6b
commit 2e8e42a165
4 changed files with 113 additions and 27 deletions
@@ -1994,7 +1994,7 @@ tracked in FEATURES.md.
 2. **FEAT-02** — QR / claim-code device pairing flow (medium)
 3. **FEAT-04** — Replace SIM868 2G GSM with 4G LTE modem (large)

-### Phase 28: Native Android companion app — 🚧 IN PROGRESS (28.1 ✅, 28.2 ✅, 28.3–28.7 pending)
+### Phase 28: Native Android companion app — 🚧 IN PROGRESS (28.1 ✅, 28.2 ✅, 28.3 🚧, 28.4–28.7 pending)
 **Renumbered from former Phase 26 on 2026-08-10 to make room for the road-test bug-fix phase.
 Renumbered to Phase 28 on 2026-08-11 to make room for Phase 27 feature additions. Planning
 completed 2026-08-11. Detailed implementation plan in [android/plan.md](android/plan.md).**
@@ -2019,6 +2019,13 @@ completed 2026-08-11. Detailed implementation plan in [android/plan.md](android/
 - Bottom nav: Dashcam tab + Test tab.
 - **Verified:** video list loads from real Pi over "yaris 911" hotspot on the owner's Pixel.

+**28.3 — Cellular upload 🚧 IN PROGRESS (2026-08-12)**
+- `ServerApi.kt`: Retrofit chunked-upload endpoints (`/api/v1/dashcam/upload/status` + `.../chunk`).
+- `UploadWorker.kt`: WorkManager foreground worker — SHA256 → `/status` check → 40MB chunks,
+  skipping already-received chunks. Byte-level progress via okio `ForwardingSink` (real bytes
+  sent, not chunk count), debug logging, robust chunk reads, re-upload guard.
+- `DownloadsScreen`: per-file upload status + inline progress bar; in-app test file generator.
+
 #### Why native (not web, not PWA)
 Revives the phone-relay goal (Phase 11/12) properly this time. The Phase 11 update proved
 conclusively that a browser page **cannot** hold a WiFi connection to the Pi's no-internet
@@ -308,8 +308,10 @@ class DashcamViewModel(application: Application) : AndroidViewModel(application)
    fun uploadFile(file: DownloadedFile) {
        val app = getApplication<YarisApp>()
        val ctx = app.applicationContext
+        android.util.Log.d("UploadWorker", "uploadFile called: ${file.name}")

        if (!CredentialStore.isConfigured(ctx)) {
+            android.util.Log.e("UploadWorker", "Credentials not configured")
            viewModelScope.launch { _downloadEvents.emit("✗ Server credentials not configured") }
            return
        }
@@ -337,6 +339,7 @@ class DashcamViewModel(application: Application) : AndroidViewModel(application)

        wm.getWorkInfoByIdLiveData(uuid).observeForever { info ->
            if (info == null) return@observeForever
+            android.util.Log.d("UploadWorker", "State: ${info.state} pct=${info.progress.getInt("pct", -1)}")
            val progress = info.progress.getInt("pct", 0)
            val status = info.progress.getString("status")
            val done = info.state == WorkInfo.State.SUCCEEDED || info.state == WorkInfo.State.FAILED
@@ -2,6 +2,7 @@ package com.yarisobd.app

 import android.content.Context
 import android.os.Environment
+import android.util.Log
 import androidx.work.CoroutineWorker
 import androidx.work.ForegroundInfo
 import androidx.work.WorkerParameters
@@ -11,8 +12,12 @@ import kotlinx.coroutines.withContext
 import okhttp3.MediaType.Companion.toMediaType
 import okhttp3.OkHttpClient
 import okhttp3.Request
+import okhttp3.RequestBody
 import okhttp3.RequestBody.Companion.asRequestBody
 import okhttp3.RequestBody.Companion.toRequestBody
+import okio.Buffer
+import okio.BufferedSink
+import okio.ForwardingSink
 import okio.buffer
 import okio.source
 import org.json.JSONObject
@@ -27,6 +32,7 @@ class UploadWorker(
 ) : CoroutineWorker(context, params) {

    companion object {
+        const val TAG = "UploadWorker"
        const val KEY_FILENAME = "filename"
        const val KEY_SERVER_URL = "server_url"
        const val KEY_DEVICE_NAME = "device_name"
@@ -50,7 +56,11 @@ class UploadWorker(
            Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_MOVIES),
            "yaris-obd/$filename"
        )
-        if (!file.exists()) return@withContext Result.failure(workDataOf("error" to "File not found"))
+        Log.d(TAG, "doWork start: $filename size=${file.length()} server=$serverUrl device=$deviceName")
+        if (!file.exists()) {
+            Log.e(TAG, "File not found: ${file.absolutePath}")
+            return@withContext Result.failure(workDataOf("error" to "File not found"))
+        }

        try {
            // Promote to foreground immediately so large files aren't killed during hashing
@@ -58,21 +68,26 @@ class UploadWorker(
            setProgress(workDataOf("pct" to 0, "status" to "Hashing…"))

            // Compute SHA256
+            Log.d(TAG, "Hashing $filename ...")
            val sha256 = computeSha256(file)
+            Log.d(TAG, "SHA256 done: $sha256")
            val relativePath = filename  // Using filename as relative_path since flat dir

            // Check status
            setProgress(workDataOf("pct" to 0, "status" to "Checking server…"))
            val statusUrl = "$serverUrl/api/v1/dashcam/upload/status?relative_path=$relativePath&sha256=$sha256"
+            Log.d(TAG, "Status check: $statusUrl")
            val statusReq = Request.Builder().url(statusUrl)
                .addHeader("X-Device-Name", deviceName)
                .addHeader("X-Api-Key", apiKey)
                .get().build()
            val statusRes = okHttp.newCall(statusReq).execute()
            val statusBody = statusRes.body?.string() ?: ""
+            Log.d(TAG, "Status response: HTTP ${statusRes.code} body=$statusBody")
            val statusJson = JSONObject(statusBody)

            if (statusJson.optString("status") == "already_uploaded") {
+                Log.d(TAG, "Already uploaded, skipping: $filename")
                return@withContext Result.success(workDataOf("filename" to filename))
            }

@@ -81,19 +96,51 @@ class UploadWorker(
                ?.toSet() ?: emptySet()

            val totalChunks = ((file.length() + CHUNK_SIZE - 1) / CHUNK_SIZE).toInt()
+            Log.d(TAG, "Uploading: totalChunks=$totalChunks received=$receivedIndices")
            setForeground(DownloadWorker.createForegroundNotification(applicationContext, "Uploading $filename", 0))
            setProgress(workDataOf("pct" to 0, "status" to "Uploading…"))

-            // Upload missing chunks
+            // Upload missing chunks with byte-level progress
+            val fileSize = file.length()
+            val chunkSizes = LongArray(totalChunks) { i -> minOf(CHUNK_SIZE, fileSize - i * CHUNK_SIZE) }
+            var baseBytes = receivedIndices.sumOf { chunkSizes[it] }
+            var lastProgressMs = 0L
+            var lastNotifyMs = 0L
+
+            fun reportProgress(bytesWritten: Long) {
+                val now = System.currentTimeMillis()
+                val totalBytes = baseBytes + bytesWritten
+                val pct = ((totalBytes * 100) / fileSize).toInt().coerceIn(0, 100)
+                if (now - lastProgressMs >= 250) {
+                    lastProgressMs = now
+                    Log.d(TAG, "Progress: $pct% ($totalBytes/$fileSize bytes)")
+                    setProgressAsync(workDataOf("pct" to pct, "status" to "Uploading…"))
+                }
+                if (now - lastNotifyMs >= 1000) {
+                    lastNotifyMs = now
+                    setForegroundAsync(DownloadWorker.createForegroundNotification(applicationContext, "Uploading $filename", pct))
+                }
+            }
+
            for (i in 0 until totalChunks) {
                if (i in receivedIndices) continue

                val offset = i * CHUNK_SIZE
-                val chunkSize = minOf(CHUNK_SIZE, file.length() - offset)
-                val chunkBytes = ByteArray(chunkSize.toInt())
+                val chunkSize = chunkSizes[i].toInt()
+                val chunkBytes = ByteArray(chunkSize)
                FileInputStream(file).use { fis ->
-                    fis.skip(offset)
-                    fis.read(chunkBytes)
+                    var skipped = 0L
+                    while (skipped < offset) {
+                        val s = fis.skip(offset - skipped)
+                        if (s <= 0) break
+                        skipped += s
+                    }
+                    var read = 0
+                    while (read < chunkSize) {
+                        val n = fis.read(chunkBytes, read, chunkSize - read)
+                        if (n < 0) break
+                        read += n
+                    }
                }

                val chunkBody = chunkBytes.toRequestBody("application/octet-stream".toMediaType())
@@ -106,25 +153,32 @@ class UploadWorker(
                    .addFormDataPart("total_chunks", totalChunks.toString())
                    .build()

+                val body = ProgressRequestBody(multipart) { written -> reportProgress(written) }
+
                val uploadReq = Request.Builder()
                    .url("$serverUrl/api/v1/dashcam/upload/chunk")
                    .addHeader("X-Device-Name", deviceName)
                    .addHeader("X-Api-Key", apiKey)
-                    .post(multipart)
+                    .post(body)
                    .build()

+                Log.d(TAG, "Uploading chunk $i/$totalChunks ($chunkSize bytes)...")
                val uploadRes = okHttp.newCall(uploadReq).execute()
+                Log.d(TAG, "Chunk $i response: HTTP ${uploadRes.code}")
                if (!uploadRes.isSuccessful) {
+                    Log.e(TAG, "Chunk $i failed: HTTP ${uploadRes.code} body=${uploadRes.body?.string()}")
                    return@withContext Result.failure(workDataOf("error" to "Chunk $i: HTTP ${uploadRes.code}"))
                }

-                val pct = ((i + 1) * 100) / totalChunks
+                baseBytes += chunkSizes[i]
+                val pct = ((baseBytes * 100) / fileSize).toInt().coerceIn(0, 100)
                setForeground(DownloadWorker.createForegroundNotification(applicationContext, "Uploading $filename", pct))
                setProgress(workDataOf("pct" to pct, "status" to "Uploading…"))
            }

            return@withContext Result.success(workDataOf("filename" to filename))
        } catch (e: Exception) {
+            Log.e(TAG, "Upload exception", e)
            return@withContext Result.failure(workDataOf("error" to (e.message ?: "Unknown error")))
        }
    }
@@ -140,4 +194,25 @@ class UploadWorker(
        }
        return digest.digest().joinToString("") { "%02x".format(it) }
    }
+
+    private class ProgressRequestBody(
+        private val delegate: RequestBody,
+        private val onBytesWritten: (Long) -> Unit,
+    ) : RequestBody() {
+        override fun contentType() = delegate.contentType()
+        override fun contentLength() = delegate.contentLength()
+        override fun writeTo(sink: BufferedSink) {
+            var written = 0L
+            val countingSink = object : ForwardingSink(sink) {
+                override fun write(source: Buffer, byteCount: Long) {
+                    super.write(source, byteCount)
+                    written += byteCount
+                    onBytesWritten(written)
+                }
+            }
+            val buffered = countingSink.buffer()
+            delegate.writeTo(buffered)
+            buffered.flush()
+        }
+    }
 }
@@ -124,33 +124,34 @@ android/
 - Phase 28.2 — dashcam video browser + WiFi download
 - fix: downloads use WorkManager foreground service + inline progress

-### 28.3 — Cellular upload ⬜ NOT STARTED
+### 28.3 — Cellular upload 🚧 IN PROGRESS (2026-08-12)
 **Depends on: 28.2 (files on phone local storage).**

-#### 28.3.1 — Server upload data layer
- [ ] Retrofit interface for server chunked upload:
+#### 28.3.1 — Server upload data layer ✅ DONE
+- [x] Retrofit interface for server chunked upload (`data/remote/ServerApi.kt`):
  - `GET /api/v1/dashcam/upload/status?sha256={hash}` → `UploadStatus`
  - `POST /api/v1/dashcam/upload/chunk` → multipart (file chunk, sha256, chunk_index, total_chunks)
- [ ] Data classes: `UploadStatus` (already_uploaded, received_chunk_indices, total_chunks)
- [ ] Commit: `feat: server chunked upload API`
+- [x] Data classes: `UploadStatus` (already_uploaded, received_chunk_indices, total_chunks)

-#### 28.3.2 — Upload worker
- [ ] WorkManager `CoroutineWorker`:
-  - Input: file local URI, sha256, relative_path
-  - Step 1: check `/status` for already-received chunks
+#### 28.3.2 — Upload worker ✅ DONE
+- [x] WorkManager `CoroutineWorker` (`UploadWorker.kt`):
+  - Input: filename, server URL, device name, API key
+  - Step 1: compute SHA256, check `/status` for already-received chunks
  - Step 2: read file in 40MB chunks (matching `DASHCAM_CHUNK_SIZE`)
-  - Step 3: POST each missing chunk via cellular-bound OkHttp client
+  - Step 3: POST each missing chunk (skips already-received chunks)
  - Step 4: verify server reports `already_uploaded` after last chunk
-  - Step 5: delete local file copy on success (optional, configurable)
- [ ] Foreground service notification: "Uploading VID_00234.MOV — 3/8 chunks (37%)"
- [ ] Retry on failure: WorkManager backoff policy (exponential, max 3 attempts)
- [ ] Commit: `feat: WorkManager cellular upload worker`
+- [x] Foreground service notification with live % ("Uploading VID_00234.MOV — 62%")
+- [x] Byte-level upload progress via `ProgressRequestBody` (okio `ForwardingSink`) — progress
+  reflects actual bytes sent including already-received chunks, not just chunk count
+- [x] Robust chunk reads (loop `skip()`/`read()` instead of single calls)
+- [x] Debug logging (`Log.d`/`Log.e`, tag `UploadWorker`) throughout
+- [x] Re-upload guard: skips files already on server (`already_uploaded`)

-#### 28.3.3 — Upload queue UI
- [ ] Screen: list of downloaded files waiting to upload
- [ ] Per-file: status (queued / uploading / done / failed), progress bar, retry button
+#### 28.3.3 — Upload queue UI 🚧 IN PROGRESS
+- [x] Downloads screen shows per-file upload status + inline progress bar
+- [x] In-app test file generator for upload testing
 - [ ] "Upload all" button → enqueue all pending files
- [ ] Observation: watch WorkManager `WorkInfo` via `LiveData`/`Flow` for real-time status
+- [ ] Retry button per failed file
 - [ ] Commit: `feat: upload queue screen`

 ### 28.4 — Live dashboard ⬜ NOT STARTED