Compare commits

...
7 Commits
Author SHA1 Message Date
quoije b378214e9b Run transfer queue jobs concurrently (max 3) so downloads don't block each other
build / Linux x86_64 (push) Successful in 3m15s
build / Windows x64 (Wine) (push) Successful in 3m41s
2026-09-14 12:50:05 -04:00
quoije 0110b50a07 Download cover bundle through the transfer queue (fix QProgressDialog NameError) 2026-09-14 12:47:13 -04:00
quoije 76eb6a6ef4 Smooth store source transitions: cancel stale batches and batch grid updates 2026-09-14 00:49:50 -04:00
quoije 58d348859a Fix store FD exhaustion: run cover/date loaders one at a time 2026-09-14 00:38:37 -04:00
quoije 3359ecf131 Fix store lag: decode cached covers off the GUI thread
build / Linux x86_64 (push) Successful in 2m48s
build / Windows x64 (Wine) (push) Successful in 3m17s
2026-09-14 00:24:48 -04:00
quoije 38a3859a21 Unbundle covers: fetch psppt-covers.zip from repo on demand with user prompt
build / Linux x86_64 (push) Successful in 2m28s
build / Windows x64 (Wine) (push) Successful in 3m8s
2026-09-14 00:03:09 -04:00
quoije 96e876b8ec Harden shutdown: no segfault closing mid-load (thread cleanup + graceful Ctrl+C) 2026-09-13 23:49:17 -04:00
11 changed files with 372 additions and 108 deletions
+4
View File
@@ -24,3 +24,7 @@ data/nps/*.tsv
# PyInstaller output # PyInstaller output
build/ build/
dist/ dist/
# Offline cover bundles (large; fetched from the project release on first run)
data/covers/*.zip
/psppt-covers.zip
+25 -10
View File
@@ -79,8 +79,10 @@ pyinstaller --clean --noconfirm psppt.spec
Notes: Notes:
- The build bundles the DATs, PS1 loader, and the offline cover bundles. The - The build bundles the DATs, PS1 loader, cover templates, and index catalogs
cover zips are large (~850 MB), so builds are slow and the output is big. (~80 MB total). The large offline cover zips (~850 MB) are **not** bundled —
the app fetches them on first run (see
[Offline bundles](#offline-bundles--auto-download)).
- `7z` (p7zip-full on Linux, 7-Zip on Windows) is a **runtime** dependency for - `7z` (p7zip-full on Linux, 7-Zip on Windows) is a **runtime** dependency for
reading store archives — it is not bundled. reading store archives — it is not bundled.
- User data (credentials, manual covers, PS1 templates, NPS cache, library DB, - User data (credentials, manual covers, PS1 templates, NPS cache, library DB,
@@ -170,9 +172,10 @@ Discs are matched by hash first (most precise), then serial, then filename.
- `data/credentials.json` — cover/metadata API keys (IGDB, TheMovieDB, - `data/credentials.json` — cover/metadata API keys (IGDB, TheMovieDB,
ScreenScraper, Google Custom Search). Environment variables take precedence. ScreenScraper, Google Custom Search). Environment variables take precedence.
- `data/manual_covers/games|videos/` — your own cover images. - `data/manual_covers/games|videos/` — your own cover images.
- `data/covers/` — checked-in assets: the UMD video/game cover templates, - `data/covers/` — checked-in assets: the UMD video/game cover templates and
`libretro-index.json` + `openvgdb-index.json` catalogs, `libretro-covers.zip` + the `libretro-index.json` + `openvgdb-index.json` catalogs. The large
`openvgdb-covers.zip` offline cover bundles, and the legacy cover cache. `libretro-covers.zip` + `openvgdb-covers.zip` bundles are git-ignored and
distributed as `psppt-covers.zip` (see above).
- `data/base.PBP` — a PSN PS1 classic EBOOT (any game) supplying `STARTDAT` - `data/base.PBP` — a PSN PS1 classic EBOOT (any game) supplying `STARTDAT`
+ a base `PARAM.SFO`/icon for PS1 conversion (git-ignored, copyrighted). + a base `PARAM.SFO`/icon for PS1 conversion (git-ignored, copyrighted).
- `data/loader/DATA.PSP` + `data/loader/KEYS.BIN` — bundled plain-ELF POPS - `data/loader/DATA.PSP` + `data/loader/KEYS.BIN` — bundled plain-ELF POPS
@@ -223,6 +226,7 @@ QSettings under `psppt`) taking precedence where noted.
| `PSP_CREDENTIALS` | `$PSP_USER_DATA_DIR/credentials.json` | credentials file | | `PSP_CREDENTIALS` | `$PSP_USER_DATA_DIR/credentials.json` | credentials file |
| `PSP_LIBRETRO_COVERS_URL` | *(empty)* | direct URL for `libretro-covers.zip` (else derived from store server) | | `PSP_LIBRETRO_COVERS_URL` | *(empty)* | direct URL for `libretro-covers.zip` (else derived from store server) |
| `PSP_OVGDB_COVERS_URL` | *(empty)* | direct URL for `openvgdb-covers.zip` (else derived from store server) | | `PSP_OVGDB_COVERS_URL` | *(empty)* | direct URL for `openvgdb-covers.zip` (else derived from store server) |
| `PSP_COVERS_BUNDLE_URL` | Gitea `covers` release asset | combined `psppt-covers.zip` download URL |
Cover/metadata keys can also be set as env vars: `IGDB_CLIENT_ID`, Cover/metadata keys can also be set as env vars: `IGDB_CLIENT_ID`,
`IGDB_CLIENT_SECRET`, `TMDB_API_KEY`, `GOOGLE_API_KEY`, `GOOGLE_CX`, `IGDB_CLIENT_SECRET`, `TMDB_API_KEY`, `GOOGLE_API_KEY`, `GOOGLE_CX`,
@@ -255,12 +259,23 @@ Manual covers in `data/manual_covers/videos/` always win.
### Offline bundles & auto-download ### Offline bundles & auto-download
The two bundles (`libretro-covers.zip`, `openvgdb-covers.zip`) are shipped in The two bundles (`libretro-covers.zip`, `openvgdb-covers.zip`) are packaged
`data/covers/`, so a fresh install works with **no internet and no API keys**. into one `psppt-covers.zip` (~850 MB) and attached to the repo's `covers`
If either is missing at startup, the app downloads it from the store server's release — **not** bundled into the app binary.
`/api/covers/<name>.zip` endpoint (see [Server setup](#server-setup)).
Rebuild them yourself: On first launch, if the bundle isn't installed, the app asks whether to
download it (with a progress bar). Covers keep working online via libretro /
OpenVGDB / ScreenScraper / IGDB without it — the bundle just makes them instant
and offline. "Don't ask again" is remembered, and you can re-download from
**Settings → Cover sources → Download offline cover bundle…**.
Build the bundle yourself:
```bash
python3 scripts/make-covers-bundle.py # -> psppt-covers.zip
```
Rebuild the inner zips:
```bash ```bash
python3 -m app.libretro # libretro-thumbnails -> libretro-covers.zip python3 -m app.libretro # libretro-thumbnails -> libretro-covers.zip
+7
View File
@@ -76,6 +76,13 @@ STORE_TOKEN = os.environ.get("PSP_STORE_TOKEN", "")
LIBRETRO_COVERS_URL = os.environ.get("PSP_LIBRETRO_COVERS_URL", "") LIBRETRO_COVERS_URL = os.environ.get("PSP_LIBRETRO_COVERS_URL", "")
OVGDB_COVERS_URL = os.environ.get("PSP_OVGDB_COVERS_URL", "") OVGDB_COVERS_URL = os.environ.get("PSP_OVGDB_COVERS_URL", "")
# Combined offline cover bundle (both inner zips) fetched on first run when the
# user opts in. Defaults to the project's Gitea release asset.
COVERS_BUNDLE_URL = os.environ.get(
"PSP_COVERS_BUNDLE_URL",
"https://git.ceaf.dev/quoije/psp-preservation-tool/releases/download/covers/psppt-covers.zip",
)
# archive.org item hosting the same Redump 7z set (identifier or full URL). # archive.org item hosting the same Redump 7z set (identifier or full URL).
ARCHIVE_ITEM = os.environ.get("PSP_ARCHIVE_ITEM", "psp-redump-roms321.com") ARCHIVE_ITEM = os.environ.get("PSP_ARCHIVE_ITEM", "psp-redump-roms321.com")
+73 -33
View File
@@ -53,8 +53,20 @@ _TMDB_SEARCH = "https://api.themoviedb.org/3/search/movie"
_TMDB_IMAGE = "https://image.tmdb.org/t/p/" _TMDB_IMAGE = "https://image.tmdb.org/t/p/"
_GOOGLE_CSE_API = "https://www.googleapis.com/customsearch/v1" _GOOGLE_CSE_API = "https://www.googleapis.com/customsearch/v1"
_MANUAL_DIR = config.USER_DATA_DIR / "manual_covers" _MANUAL_DIR = config.USER_DATA_DIR / "manual_covers"
_OVGDB_ZIP = config.DATA_DIR / "covers" / "openvgdb-covers.zip" # Offline cover bundles live in the writable cover dir (downloaded on demand),
_LIBRETRO_ZIP = config.DATA_DIR / "covers" / "libretro-covers.zip" # not the read-only app bundle. A source checkout keeps them in data/covers/.
_OVGDB_ZIP = config.COVERS_DIR / "openvgdb-covers.zip"
_LIBRETRO_ZIP = config.COVERS_DIR / "libretro-covers.zip"
_BUNDLE_ZIP = config.COVERS_DIR / "psppt-covers.zip"
def _bundle_zip_path(name):
"""Resolve a cover bundle zip: writable dir first, legacy checkout second."""
for base in (config.COVERS_DIR, config.LEGACY_COVERS_DIR):
p = base / name
if p.exists():
return p
return config.COVERS_DIR / name
_WIKI_SEARCH = "https://en.wikipedia.org/w/api.php" _WIKI_SEARCH = "https://en.wikipedia.org/w/api.php"
_WIKI_SUMMARY = "https://en.wikipedia.org/api/rest_v1/page/summary/" _WIKI_SUMMARY = "https://en.wikipedia.org/api/rest_v1/page/summary/"
@@ -158,48 +170,74 @@ def clear_cover_cache():
pass pass
def _download_file(url, dest, timeout=600): def _download_file(url, dest, timeout=600, progress_cb=None, cancel_event=None):
"""Stream `url` to `dest` (atomic); returns True on success.""" """Stream `url` to `dest` (atomic); returns True on success.
``progress_cb(done, total)`` is called with bytes so far (total is 0 when
the server doesn't report a content length). ``cancel_event`` aborts the
download when set.
"""
req = urllib.request.Request(url, headers=_UA) req = urllib.request.Request(url, headers=_UA)
with urllib.request.urlopen(req, timeout=timeout) as resp: with urllib.request.urlopen(req, timeout=timeout) as resp:
if resp.status != 200: if resp.status != 200:
return False return False
total = int(resp.headers.get("Content-Length") or 0)
tmp = dest.with_name(dest.name + ".tmp") tmp = dest.with_name(dest.name + ".tmp")
done = 0
with open(tmp, "wb") as f: with open(tmp, "wb") as f:
while True: while True:
if cancel_event is not None and cancel_event.is_set():
raise InterruptedError("download cancelled")
chunk = resp.read(1 << 20) chunk = resp.read(1 << 20)
if not chunk: if not chunk:
break break
f.write(chunk) f.write(chunk)
done += len(chunk)
if progress_cb is not None:
progress_cb(done, total)
os.replace(tmp, dest) os.replace(tmp, dest)
return True return True
def ensure_cover_bundles(server_url=""): def covers_bundle_present():
"""Download missing offline cover bundle zips from the store server. """True when the offline cover bundle is installed (both inner zips)."""
return (_bundle_zip_path("libretro-covers.zip").exists()
and _bundle_zip_path("openvgdb-covers.zip").exists())
``server_url`` is the base URL (e.g. ``http://192.168.10.100:8000``); the
zips are fetched from ``<base>/api/covers/<name>.zip`` unless a direct URL def download_covers_bundle(url=None, progress_cb=None, cancel_event=None):
is set in config. Returns the names of bundles downloaded. """Download and extract the combined cover bundle into the cover dir.
``url`` defaults to ``config.COVERS_BUNDLE_URL``. Raises on failure; the
caller decides whether to surface the error.
""" """
base = (server_url or config.STORE_SERVER_URL or "").rstrip("/") url = (url or config.COVERS_BUNDLE_URL or "").strip()
downloaded = [] if not url:
for name, path, direct in ( raise ValueError("no cover bundle URL configured")
("libretro-covers.zip", _LIBRETRO_ZIP, config.LIBRETRO_COVERS_URL), config.COVERS_DIR.mkdir(parents=True, exist_ok=True)
("openvgdb-covers.zip", _OVGDB_ZIP, config.OVGDB_COVERS_URL), _download_file(url, _BUNDLE_ZIP, timeout=3600,
): progress_cb=progress_cb, cancel_event=cancel_event)
if path.exists(): inner = ("libretro-covers.zip", "openvgdb-covers.zip")
continue with zipfile.ZipFile(_BUNDLE_ZIP) as z:
url = direct or (f"{base}/api/covers/{name}" if base else "") missing = [n for n in inner if n not in z.namelist()]
if not url: if missing:
continue raise ValueError(f"cover bundle is missing: {', '.join(missing)}")
try: with zipfile.ZipFile(_BUNDLE_ZIP) as z:
if _download_file(url, path): for name in inner:
downloaded.append(name) out = config.COVERS_DIR / name
log.info("downloaded offline cover bundle %s", name) tmp = out.with_name(out.name + ".tmp")
except Exception as exc: # noqa: BLE001 with z.open(name) as src, open(tmp, "wb") as dst:
log.warning("could not download cover bundle %s: %s", name, exc) while True:
return downloaded chunk = src.read(1 << 20)
if not chunk:
break
dst.write(chunk)
os.replace(tmp, out)
# Invalidate cached namelists so lookups pick up the freshly extracted zips.
global _libretro_names, _ovgdb_names
_libretro_names = None
_ovgdb_names = None
return config.COVERS_DIR
def cover_url(name, kind="Named_Boxarts", system="games"): def cover_url(name, kind="Named_Boxarts", system="games"):
@@ -482,9 +520,10 @@ def _ovgdb_name_list():
global _ovgdb_names global _ovgdb_names
if _ovgdb_names is None: if _ovgdb_names is None:
_ovgdb_names = set() _ovgdb_names = set()
if _OVGDB_ZIP.exists(): path = _bundle_zip_path("openvgdb-covers.zip")
if path.exists():
try: try:
with zipfile.ZipFile(_OVGDB_ZIP) as z: with zipfile.ZipFile(path) as z:
_ovgdb_names = {n[:-4] for n in z.namelist() if n.endswith(".jpg")} _ovgdb_names = {n[:-4] for n in z.namelist() if n.endswith(".jpg")}
except Exception: # noqa: BLE001 except Exception: # noqa: BLE001
pass pass
@@ -535,7 +574,7 @@ def fetch_openvgdb_local(name):
if entry is None: if entry is None:
return None return None
try: try:
with zipfile.ZipFile(_OVGDB_ZIP) as z: with zipfile.ZipFile(_bundle_zip_path("openvgdb-covers.zip")) as z:
data = z.read(entry) data = z.read(entry)
return data if _is_image(data) else None return data if _is_image(data) else None
except Exception: # noqa: BLE001 except Exception: # noqa: BLE001
@@ -559,7 +598,7 @@ def fetch_libretro_local(name):
if entry is None: if entry is None:
return None return None
try: try:
with zipfile.ZipFile(_LIBRETRO_ZIP) as z: with zipfile.ZipFile(_bundle_zip_path("libretro-covers.zip")) as z:
data = z.read(entry) data = z.read(entry)
return data if _is_image(data) else None return data if _is_image(data) else None
except Exception: # noqa: BLE001 except Exception: # noqa: BLE001
@@ -570,9 +609,10 @@ def _libretro_name_list():
global _libretro_names global _libretro_names
if _libretro_names is None: if _libretro_names is None:
_libretro_names = set() _libretro_names = set()
if _LIBRETRO_ZIP.exists(): path = _bundle_zip_path("libretro-covers.zip")
if path.exists():
try: try:
with zipfile.ZipFile(_LIBRETRO_ZIP) as z: with zipfile.ZipFile(path) as z:
_libretro_names = {n[:-4] for n in z.namelist() if n.endswith(".png")} _libretro_names = {n[:-4] for n in z.namelist() if n.endswith(".png")}
except Exception: # noqa: BLE001 except Exception: # noqa: BLE001
pass pass
+158 -24
View File
@@ -7,6 +7,7 @@ Run with::
import datetime import datetime
import json import json
import logging import logging
import signal
import sys import sys
import threading import threading
from pathlib import Path from pathlib import Path
@@ -158,7 +159,9 @@ class CoverLoader(QObject):
else: else:
self.failed.emit(self.gen, i, self.items[i]["name"]) self.failed.emit(self.gen, i, self.items[i]["name"])
finally: finally:
pool.shutdown(wait=True, cancel_futures=True) # Return promptly on cancel/exit instead of blocking the GUI thread
# on in-flight network timeouts during shutdown.
pool.shutdown(wait=not self._cancel.is_set(), cancel_futures=True)
self.finished.emit() self.finished.emit()
@@ -480,7 +483,7 @@ class CoverShowcase(QWidget):
return QIcon(pm) return QIcon(pm)
def _on_loaded(self, gen, index, image): def _on_loaded(self, gen, index, image):
if gen != self._generation: if self._closing or gen != self._generation:
return return
item = self.list.item(index) item = self.list.item(index)
if item is not None: if item is not None:
@@ -502,7 +505,7 @@ class CoverShowcase(QWidget):
self.play_requested.emit(data) self.play_requested.emit(data)
def _on_failed(self, gen, index, name): def _on_failed(self, gen, index, name):
if gen != self._generation: if self._closing or gen != self._generation:
return return
if index not in self.missing: if index not in self.missing:
self.missing.append(index) self.missing.append(index)
@@ -510,6 +513,8 @@ class CoverShowcase(QWidget):
self._update_status() self._update_status()
def _on_loading_done(self): def _on_loading_done(self):
if self._closing:
return
sender = self.sender() sender = self.sender()
if sender is not None and getattr(sender, "gen", None) != self._generation: if sender is not None and getattr(sender, "gen", None) != self._generation:
return return
@@ -517,6 +522,8 @@ class CoverShowcase(QWidget):
self._update_status() self._update_status()
def _update_status(self): def _update_status(self):
if self._closing:
return
if self._loading: if self._loading:
self.status.setText("Loading covers…") self.status.setText("Loading covers…")
self.missing_changed.emit(0) self.missing_changed.emit(0)
@@ -1136,6 +1143,7 @@ class ScrapeDialog(QDialog):
class SettingsPage(QWidget): class SettingsPage(QWidget):
saved = Signal() saved = Signal()
download_covers = Signal()
def __init__(self, games_dir, videos_dir, store_dir, ppsspp_dir, cloud_dir, ppsspp_exe, def __init__(self, games_dir, videos_dir, store_dir, ppsspp_dir, cloud_dir, ppsspp_exe,
store_server_url, store_token, store_archive_id, store_archive_sig, store_archive_user, store_server_url, store_token, store_archive_id, store_archive_sig, store_archive_user,
@@ -1303,6 +1311,9 @@ class SettingsPage(QWidget):
self.clear_covers_btn = QPushButton("Clear cover cache (re-fetch all covers)") self.clear_covers_btn = QPushButton("Clear cover cache (re-fetch all covers)")
self.clear_covers_btn.clicked.connect(self._clear_cover_cache) self.clear_covers_btn.clicked.connect(self._clear_cover_cache)
form.addRow(self.clear_covers_btn) form.addRow(self.clear_covers_btn)
self.download_bundle_btn = QPushButton("Download offline cover bundle…")
self.download_bundle_btn.clicked.connect(self.download_covers.emit)
form.addRow(self.download_bundle_btn)
col.addWidget(box) col.addWidget(box)
# -- Storage ---------------------------------------------------------- # # -- Storage ---------------------------------------------------------- #
@@ -1541,7 +1552,9 @@ class StoreCoverLoader(QObject):
if not img.isNull(): if not img.isNull():
self.loaded.emit(self.gen, row, _fit_image(img, self.icon_w, self.icon_h)) self.loaded.emit(self.gen, row, _fit_image(img, self.icon_w, self.icon_h))
finally: finally:
pool.shutdown(wait=True, cancel_futures=True) # Return promptly on cancel/exit instead of blocking the GUI thread
# on in-flight network timeouts during shutdown.
pool.shutdown(wait=not self._cancel.is_set(), cancel_futures=True)
self.finished.emit() self.finished.emit()
@@ -1818,6 +1831,8 @@ class StorePage(QWidget):
self._lib_cover = {} # norm -> (release_id, system) self._lib_cover = {} # norm -> (release_id, system)
self._psp_norm = set() # norm of library items installed on the PSP self._psp_norm = set() # norm of library items installed on the PSP
self._cover_attempted = set() self._cover_attempted = set()
self._cover_busy = False
self._date_busy = False
self._generation = 0 self._generation = 0
self._scan_gen = 0 self._scan_gen = 0
self._meta_gen = 0 self._meta_gen = 0
@@ -2004,9 +2019,27 @@ class StorePage(QWidget):
if self._items: if self._items:
self._apply_filter() self._apply_filter()
def _begin_scan(self):
"""Clear the stale view and invalidate in-flight cover/date batches."""
self._generation += 1
self._cover_busy = False
self._date_busy = False
self._cover_queue = []
self._cover_next = 0
self._date_queue = []
self._date_next = 0
self._cover_attempted.clear()
for loader in self._loaders:
loader.cancel()
self._loaders = []
self.grid.clear()
self.table.setRowCount(0)
self.status.setText("Loading…")
def refresh(self, force=False): def refresh(self, force=False):
if self._closing: if self._closing:
return return
self._begin_scan()
self._scan_gen += 1 self._scan_gen += 1
gen = self._scan_gen gen = self._scan_gen
if self.source_combo.currentIndex() == 0: # Server if self.source_combo.currentIndex() == 0: # Server
@@ -2067,6 +2100,8 @@ class StorePage(QWidget):
self._items = [] self._items = []
self._filtered = [] self._filtered = []
self._dirty = False self._dirty = False
self.grid.clear()
self.table.setRowCount(0)
log.warning("store unavailable: %s", message) log.warning("store unavailable: %s", message)
self.status.setText(f"Store unavailable: {message}") self.status.setText(f"Store unavailable: {message}")
@@ -2167,6 +2202,8 @@ class StorePage(QWidget):
self._cover_next = 0 self._cover_next = 0
self._date_queue = [] self._date_queue = []
self._date_next = 0 self._date_next = 0
self._cover_busy = False
self._date_busy = False
self._cover_attempted.clear() self._cover_attempted.clear()
self._placeholder = QIcon(_placeholder_pixmap(self.ICON_W, self.ICON_H)) self._placeholder = QIcon(_placeholder_pixmap(self.ICON_W, self.ICON_H))
@@ -2176,6 +2213,7 @@ class StorePage(QWidget):
self._loaders = [] self._loaders = []
self.grid.clear() self.grid.clear()
self.grid.setUpdatesEnabled(False)
self.table.setUpdatesEnabled(False) self.table.setUpdatesEnabled(False)
self.table.setSortingEnabled(False) self.table.setSortingEnabled(False)
self.table.setRowCount(show_count) self.table.setRowCount(show_count)
@@ -2218,10 +2256,10 @@ class StorePage(QWidget):
rid, system = lc rid, system = lc
else: else:
rid, system = covers.stable_key(it["name"]), it.get("system") or "games" rid, system = covers.stable_key(it["name"]), it.get("system") or "games"
p = covers.cover_path(rid, system) # Covers are decoded + scaled off the GUI thread (StoreCoverLoader).
if p.exists() and covers.file_is_image(p): # Never load a QPixmap here — doing so for cached covers made the store
item.setIcon(QIcon(_fit_pixmap(QPixmap(str(p)), self.ICON_W, self.ICON_H))) # unresponsive while filling/scrolling.
elif system not in ("theme", "dlc") and rid not in self._cover_attempted: if system not in ("theme", "dlc") and rid not in self._cover_attempted:
# Themes and DLC have no meaningful box art in libretro/IGDB, and # Themes and DLC have no meaningful box art in libretro/IGDB, and
# bulk-scraping hundreds of them on every switch to the DLC list # bulk-scraping hundreds of them on every switch to the DLC list
# saturates the network/CPU and makes the store unresponsive. # saturates the network/CPU and makes the store unresponsive.
@@ -2280,6 +2318,7 @@ class StorePage(QWidget):
def _finish_fill(self): def _finish_fill(self):
if self._fill_timer is not None: if self._fill_timer is not None:
self._fill_timer.stop() self._fill_timer.stop()
self.grid.setUpdatesEnabled(True)
self.table.setUpdatesEnabled(True) self.table.setUpdatesEnabled(True)
self.table.setSortingEnabled(True) self.table.setSortingEnabled(True)
total = len(self._filtered) total = len(self._filtered)
@@ -2292,13 +2331,14 @@ class StorePage(QWidget):
self._load_date_batch(self.DATE_BATCH) self._load_date_batch(self.DATE_BATCH)
def _load_cover_batch(self, n): def _load_cover_batch(self, n):
if self._closing or not self._cover_queue: if self._closing or not self._cover_queue or self._cover_busy:
return return
end = min(self._cover_next + n, len(self._cover_queue)) end = min(self._cover_next + n, len(self._cover_queue))
if self._cover_next >= end: if self._cover_next >= end:
return return
tasks = self._cover_queue[self._cover_next:end] tasks = self._cover_queue[self._cover_next:end]
self._cover_next = end self._cover_next = end
self._cover_busy = True
self._start_cover_loader(self._generation, tasks) self._start_cover_loader(self._generation, tasks)
def _active_scrollbar(self): def _active_scrollbar(self):
@@ -2329,13 +2369,14 @@ class StorePage(QWidget):
self._fill_timer.start() self._fill_timer.start()
def _load_date_batch(self, n): def _load_date_batch(self, n):
if self._closing or not self._date_queue: if self._closing or not self._date_queue or self._date_busy:
return return
end = min(self._date_next + n, len(self._date_queue)) end = min(self._date_next + n, len(self._date_queue))
if self._date_next >= end: if self._date_next >= end:
return return
tasks = self._date_queue[self._date_next:end] tasks = self._date_queue[self._date_next:end]
self._date_next = end self._date_next = end
self._date_busy = True
self._start_date_loader(self._generation, tasks) self._start_date_loader(self._generation, tasks)
def _start_date_loader(self, gen, tasks): def _start_date_loader(self, gen, tasks):
@@ -2346,14 +2387,24 @@ class StorePage(QWidget):
loader.moveToThread(thread) loader.moveToThread(thread)
thread.started.connect(loader.run) thread.started.connect(loader.run)
loader.loaded.connect(self._on_date_loaded) loader.loaded.connect(self._on_date_loaded)
loader.finished.connect(self._on_date_batch_done)
loader.finished.connect(thread.quit) loader.finished.connect(thread.quit)
thread.finished.connect(self._on_thread_finished) thread.finished.connect(self._on_thread_finished)
self._threads.append(thread) self._threads.append(thread)
self._loaders.append(loader) self._loaders.append(loader)
thread.start() thread.start()
def _on_date_batch_done(self):
if self._closing:
return
loader = self.sender()
if getattr(loader, "gen", None) != self._generation:
return
self._date_busy = False
self._load_date_batch(self.DATE_BATCH)
def _on_date_loaded(self, gen, row, date): def _on_date_loaded(self, gen, row, date):
if gen != self._generation or not date: if self._closing or gen != self._generation or not date:
return return
if 0 <= row < len(self._filtered): if 0 <= row < len(self._filtered):
self._filtered[row]["release_date"] = date self._filtered[row]["release_date"] = date
@@ -2369,14 +2420,24 @@ class StorePage(QWidget):
loader.moveToThread(thread) loader.moveToThread(thread)
thread.started.connect(loader.run) thread.started.connect(loader.run)
loader.loaded.connect(self._on_cover_loaded) loader.loaded.connect(self._on_cover_loaded)
loader.finished.connect(self._on_cover_batch_done)
loader.finished.connect(thread.quit) loader.finished.connect(thread.quit)
thread.finished.connect(self._on_thread_finished) thread.finished.connect(self._on_thread_finished)
self._threads.append(thread) self._threads.append(thread)
self._loaders.append(loader) self._loaders.append(loader)
thread.start() thread.start()
def _on_cover_batch_done(self):
if self._closing:
return
loader = self.sender()
if getattr(loader, "gen", None) != self._generation:
return
self._cover_busy = False
self._load_cover_batch(self.COVER_SCROLL_BATCH)
def _on_cover_loaded(self, gen, row, image): def _on_cover_loaded(self, gen, row, image):
if gen != self._generation: if self._closing or gen != self._generation:
return return
item = self.grid.item(row) item = self.grid.item(row)
if item is not None: if item is not None:
@@ -2609,6 +2670,7 @@ class QueuePage(QWidget):
"umd-dump": "UMD dump", "umd-dump": "UMD dump",
"save-sync": "Save sync", "save-sync": "Save sync",
"nps-download": "NoPayStation", "nps-download": "NoPayStation",
"covers": "Covers bundle",
} }
def __init__(self, parent=None): def __init__(self, parent=None):
@@ -4130,6 +4192,7 @@ class Ps1Page(QWidget):
self._preview_gen = 0 self._preview_gen = 0
self._preview_threads = [] self._preview_threads = []
self._preview_workers = [] self._preview_workers = []
self._closing = False
layout = QVBoxLayout(self) layout = QVBoxLayout(self)
bar = QHBoxLayout() bar = QHBoxLayout()
@@ -4228,7 +4291,7 @@ class Ps1Page(QWidget):
self._preview_workers.append(worker) self._preview_workers.append(worker)
def _on_preview_result(self, gen, game_id, icon0, pic0, pic1): def _on_preview_result(self, gen, game_id, icon0, pic0, pic1):
if gen != self._preview_gen: if self._closing or gen != self._preview_gen:
return return
self.preview_icon0.setText("") self.preview_icon0.setText("")
self.preview_pic0.setText("") self.preview_pic0.setText("")
@@ -4276,6 +4339,8 @@ class Ps1Page(QWidget):
self._thread.start() self._thread.start()
def _on_results(self, entries): def _on_results(self, entries):
if self._closing:
return
# Only show discs that still need converting; converted games already # Only show discs that still need converting; converted games already
# appear in the Library tab. # appear in the Library tab.
entries = [e for e in entries if e["kind"] == "disc" and e.get("status") != "converted"] entries = [e for e in entries if e["kind"] == "disc" and e.get("status") != "converted"]
@@ -4291,7 +4356,22 @@ class Ps1Page(QWidget):
self.table.selectRow(0) self.table.selectRow(0)
def _on_failed(self, message): def _on_failed(self, message):
self.status.setText(f"PS1 scan failed: {message}") if not self._closing:
self.status.setText(f"PS1 scan failed: {message}")
def shutdown(self):
self._closing = True
if self._thread is not None and _thread_running(self._thread):
self._thread.quit()
self._thread.wait()
for t in self._preview_threads:
if _thread_running(t):
t.quit()
t.wait()
self._preview_threads = []
self._preview_workers = []
self._thread = None
self._worker = None
def _convert_selected(self): def _convert_selected(self):
row = self.table.currentRow() row = self.table.currentRow()
@@ -4588,6 +4668,7 @@ class MainWindow(QMainWindow):
self._ps1_dir, self._ps1_output_dir, self._ps1_base_pbp, self._ps1_dir, self._ps1_output_dir, self._ps1_base_pbp,
self._nps_dir) self._nps_dir)
self.settings_page.saved.connect(self._on_settings_saved) self.settings_page.saved.connect(self._on_settings_saved)
self.settings_page.download_covers.connect(self._download_covers_bundle)
return self.settings_page return self.settings_page
def _on_settings_saved(self): def _on_settings_saved(self):
@@ -5325,6 +5406,9 @@ class MainWindow(QMainWindow):
QTimer.singleShot(0, self._refresh_psp) QTimer.singleShot(0, self._refresh_psp)
if kind == "save-sync": if kind == "save-sync":
QTimer.singleShot(0, self._refresh_saves) QTimer.singleShot(0, self._refresh_saves)
if kind == "covers":
covers.clear_miss_cache()
QTimer.singleShot(0, self.load_library)
if kind == "ps1-convert": if kind == "ps1-convert":
QTimer.singleShot(0, self.ps1_page.refresh) QTimer.singleShot(0, self.ps1_page.refresh)
QTimer.singleShot(0, self.load_library) QTimer.singleShot(0, self.load_library)
@@ -5660,6 +5744,48 @@ class MainWindow(QMainWindow):
QMessageBox.information(self, "Done", f"Renamed {renamed} file(s).") QMessageBox.information(self, "Done", f"Renamed {renamed} file(s).")
self.load_library() self.load_library()
def maybe_offer_covers_bundle(self):
"""Offer to download the offline cover bundle once, if it's missing."""
if self._closing or covers.covers_bundle_present():
return
s = QSettings("psppt", "psppt")
if str(s.value("covers_bundle_skip", "false")).lower() == "true":
return
box = QMessageBox(self)
box.setWindowTitle("Offline covers")
box.setIcon(QMessageBox.Question)
box.setText(
"The offline cover bundle (~843 MB) isn't installed.\n\n"
"Download it now from the project repo? Covers keep working online "
"without it — the bundle just makes them instant and offline."
)
download = box.addButton("Download", QMessageBox.AcceptRole)
box.addButton("Not now", QMessageBox.RejectRole)
never = box.addButton("Don't ask again", QMessageBox.DestructiveRole)
box.setDefaultButton(download)
box.exec()
clicked = box.clickedButton()
if clicked is never:
s.setValue("covers_bundle_skip", "true")
elif clicked is download:
self._download_covers_bundle()
def _download_covers_bundle(self):
"""Queue the offline cover bundle download in the transfer queue."""
if self._closing:
return
for j in self.queue.jobs:
if j.kind == "covers" and j.status in ("queued", "running", "cancelling"):
return # already queued or running
self.queue.enqueue(
transfers.CoversBundleSource(),
str(config.COVERS_DIR),
None,
"Offline cover bundle",
kind="covers",
)
self.tabs.setCurrentWidget(self.queue_page)
def closeEvent(self, event): def closeEvent(self, event):
self._closing = True self._closing = True
self.settings.setValue("window_geometry", self.saveGeometry()) self.settings.setValue("window_geometry", self.saveGeometry())
@@ -5685,18 +5811,11 @@ class MainWindow(QMainWindow):
self.psp_page.shutdown() self.psp_page.shutdown()
if hasattr(self, "saves_page"): if hasattr(self, "saves_page"):
self.saves_page.shutdown() self.saves_page.shutdown()
if hasattr(self, "ps1_page"):
self.ps1_page.shutdown()
event.accept() event.accept()
def _start_bundle_check(window):
"""Download any missing offline cover bundles in the background."""
server = getattr(window, "_store_server_url", "") or ""
try:
covers.ensure_cover_bundles(server)
except Exception: # noqa: BLE001 - background best-effort
pass
def main(): def main():
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)-7s %(name)-14s %(message)s") logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)-7s %(name)-14s %(message)s")
init_db() # migrate legacy on-share DB/covers to local storage, then schema init_db() # migrate legacy on-share DB/covers to local storage, then schema
@@ -5717,7 +5836,22 @@ def main():
window.setWindowIcon(app.windowIcon()) window.setWindowIcon(app.windowIcon())
window.show() window.show()
log.info("main window ready") log.info("main window ready")
threading.Thread(target=_start_bundle_check, args=(window,), daemon=True).start() QTimer.singleShot(1200, window.maybe_offer_covers_bundle)
# Ctrl+C should close the window gracefully (running closeEvent's thread
# cleanup) rather than tearing Qt down mid-thread, which segfaults.
interrupt = {"pending": False}
def _sigint_handler(signum, frame):
interrupt["pending"] = True
signal.signal(signal.SIGINT, _sigint_handler)
sigint_timer = QTimer()
sigint_timer.setInterval(100)
sigint_timer.timeout.connect(
lambda: window.close() if interrupt["pending"] else None)
sigint_timer.start()
return app.exec() return app.exec()
+39 -36
View File
@@ -79,13 +79,13 @@ class TransferQueue(QObject):
job_finished = Signal(dict) job_finished = Signal(dict)
job_removed = Signal(int) job_removed = Signal(int)
def __init__(self, parent=None): def __init__(self, parent=None, max_concurrent=3):
super().__init__(parent) super().__init__(parent)
self.max_concurrent = max(1, int(max_concurrent))
self.jobs = [] self.jobs = []
self._pending = [] self._pending = []
self._current = None # job_id -> (QThread, JobWorker) for jobs currently running in parallel.
self._thread = None self._running = {}
self._worker = None
# Keep strong references to every worker/thread for their full lifetime. # Keep strong references to every worker/thread for their full lifetime.
# Dropping a Python ref to a QObject that was moved to another thread # Dropping a Python ref to a QObject that was moved to another thread
# deletes it directly and crashes (bus error). # deletes it directly and crashes (bus error).
@@ -115,8 +115,10 @@ class TransferQueue(QObject):
self.job_updated.emit(job.to_dict()) self.job_updated.emit(job.to_dict())
self.job_finished.emit(job.to_dict()) self.job_finished.emit(job.to_dict())
return return
if job.status == "running" and self._worker is not None: if job.status == "running":
self._worker.cancel() pair = self._running.get(job_id)
if pair is not None:
pair[1].cancel()
job.status = "cancelling" job.status = "cancelling"
self.job_updated.emit(job.to_dict()) self.job_updated.emit(job.to_dict())
@@ -140,8 +142,8 @@ class TransferQueue(QObject):
self.remove(jid) self.remove(jid)
def shutdown(self): def shutdown(self):
if self._worker is not None: for _t, worker in self._running.values():
self._worker.cancel() worker.cancel()
for t in self._threads: for t in self._threads:
try: try:
if t.isRunning(): if t.isRunning():
@@ -157,30 +159,29 @@ class TransferQueue(QObject):
return None return None
def _start_next(self): def _start_next(self):
if self._current is not None or not self._pending: while len(self._running) < self.max_concurrent and self._pending:
return job = self._pending.pop(0)
job = self._pending.pop(0) job.status = "running"
self._current = job log.info("starting job #%d: %s (%s)", job.id, job.title, job.kind)
job.status = "running" self.job_updated.emit(job.to_dict())
log.info("starting job #%d: %s (%s)", job.id, job.title, job.kind)
self.job_updated.emit(job.to_dict())
self._thread = QThread(self) thread = QThread(self)
self._worker = JobWorker(job.source, job.dest_dir, job) worker = JobWorker(job.source, job.dest_dir, job)
self._worker.moveToThread(self._thread) worker.moveToThread(thread)
self._thread.started.connect(self._worker.run) thread.started.connect(worker.run)
self._worker.progress.connect(self._on_progress) worker.progress.connect(self._on_progress)
self._worker.finished.connect(self._on_finished) worker.finished.connect(self._on_finished)
self._worker.skipped.connect(self._on_skipped) worker.skipped.connect(self._on_skipped)
self._worker.cancelled.connect(self._on_cancelled) worker.cancelled.connect(self._on_cancelled)
self._worker.failed.connect(self._on_failed) worker.failed.connect(self._on_failed)
for sig in (self._worker.finished, self._worker.skipped, for sig in (worker.finished, worker.skipped,
self._worker.cancelled, self._worker.failed): worker.cancelled, worker.failed):
sig.connect(self._thread.quit) sig.connect(thread.quit)
self._thread.finished.connect(self._on_thread_finished) thread.finished.connect(self._on_thread_finished)
self._threads.append(self._thread) self._running[job.id] = (thread, worker)
self._workers.append(self._worker) self._threads.append(thread)
self._thread.start() self._workers.append(worker)
thread.start()
def _on_progress(self, job_id, stage, fraction, detail): def _on_progress(self, job_id, stage, fraction, detail):
job = self._find(job_id) job = self._find(job_id)
@@ -218,10 +219,12 @@ class TransferQueue(QObject):
self.job_finished.emit(job.to_dict()) self.job_finished.emit(job.to_dict())
def _on_thread_finished(self): def _on_thread_finished(self):
# Queued to the main thread once the worker thread has fully stopped, # Queued to the main thread once a worker thread has fully stopped, so
# so it is now safe to drop the "current" pointers and advance. # it is now safe to drop its pointers and advance the queue.
self._current = None thread = self.sender()
self._thread = None for jid, pair in list(self._running.items()):
self._worker = None if pair[0] is thread:
del self._running[jid]
break
self._start_next() self._start_next()
+27
View File
@@ -175,6 +175,33 @@ class Source:
raise NotImplementedError raise NotImplementedError
class CoversBundleSource:
"""Download the combined offline cover bundle (runs as a queue job)."""
label = "psppt-covers.zip"
def __init__(self, url=""):
self.url = url
def download(self, item, dest_dir, progress_cb=None, cancel_event=None):
from . import covers
def on_progress(done, total):
if progress_cb is None:
return
frac = (done / total) if total else 0.0
detail = (f"{done / (1 << 20):.0f} / {total / (1 << 20):.0f} MB"
if total else "")
progress_cb("Downloading", frac, detail)
try:
result = covers.download_covers_bundle(self.url,
progress_cb=on_progress,
cancel_event=cancel_event)
except InterruptedError:
raise TransferCancelled() from None
return str(result)
class FileCopySource(Source): class FileCopySource(Source):
"""Copy a single local file (library -> PSP or PSP -> library). """Copy a single local file (library -> PSP or PSP -> library).
Binary file not shown.
Binary file not shown.
+2 -4
View File
@@ -1,8 +1,8 @@
# -*- mode: python ; coding: utf-8 -*- # -*- mode: python ; coding: utf-8 -*-
"""PyInstaller spec: build psppt (onedir). """PyInstaller spec: build psppt (onedir).
Bundles the read-only resources (DATs, PS1 loader, offline cover bundles, Bundles the read-only resources (DATs, PS1 loader, cover templates, index
icons) under ``lib/`` next to the executable. User-writable data catalogs, icons) under ``lib/`` next to the executable. User-writable data
(credentials, manual covers, PS1 templates, NPS cache, library DB, cover (credentials, manual covers, PS1 templates, NPS cache, library DB, cover
cache) lives in the OS state dir at runtime — see ``app/config.py``. cache) lives in the OS state dir at runtime — see ``app/config.py``.
@@ -15,8 +15,6 @@ datas = [
("data/dats", "data/dats"), ("data/dats", "data/dats"),
("data/loader", "data/loader"), ("data/loader", "data/loader"),
("data/supplement.json", "data"), ("data/supplement.json", "data"),
("data/covers/libretro-covers.zip", "data/covers"),
("data/covers/openvgdb-covers.zip", "data/covers"),
("data/covers/libretro-index.json", "data/covers"), ("data/covers/libretro-index.json", "data/covers"),
("data/covers/openvgdb-index.json", "data/covers"), ("data/covers/openvgdb-index.json", "data/covers"),
("data/covers/UMD_COVER_TEMPLATE.png", "data/covers"), ("data/covers/UMD_COVER_TEMPLATE.png", "data/covers"),
+36
View File
@@ -0,0 +1,36 @@
#!/usr/bin/env python3
"""Package the offline cover zips into a single distributable bundle.
Outputs ``psppt-covers.zip`` containing ``libretro-covers.zip`` and
``openvgdb-covers.zip``. Attach it to the Gitea ``covers`` release; the app
downloads it on first run when the user opts in.
Usage: python3 scripts/make-covers-bundle.py [out-path]
"""
import sys
import zipfile
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
COVERS = ROOT / "data" / "covers"
INNER = ("libretro-covers.zip", "openvgdb-covers.zip")
def main(argv):
out = Path(argv[1]) if len(argv) > 1 else ROOT / "psppt-covers.zip"
missing = [n for n in INNER if not (COVERS / n).exists()]
if missing:
print(f"missing: {', '.join(missing)} in {COVERS}", file=sys.stderr)
return 2
# Inner zips are already compressed; store them to avoid recompressing.
with zipfile.ZipFile(out, "w", zipfile.ZIP_STORED) as z:
for n in INNER:
print(f"adding {n} …")
z.write(COVERS / n, arcname=n)
size = out.stat().st_size / (1 << 20)
print(f"wrote {out} ({size:.0f} MB)")
return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv))