Compare commits

...
78 Commits
Author SHA1 Message Date
quoije c61d8e9d3f Revert contents_directory (collides with exe name)
build / Linux x86_64 (push) Failing after 3m8s
build / Windows x64 (Wine) (push) Canceled after 0s
2026-09-13 21:43:28 -04:00
quoije 78f6d44c73 Fix frozen-build SSL certs, taskbar icon, and rename _internal to psp-preserve
build / Linux x86_64 (push) Failing after 7m17s
build / Windows x64 (Wine) (push) Canceled after 0s
- config: point SSL at certifi bundle when frozen (fixes store/NPS/TMDB SSL errors)
- gui: set app name + window icon for the taskbar
- spec: contents_directory=psp-preserve; bundle a .desktop file for Wayland launchers
2026-09-13 21:19:56 -04:00
quoije a7f90cd5a7 Runner image: use WineHQ wine-staging + embeddable Python (Wine 8.0 too old for Py3.12) 2026-09-13 21:13:30 -04:00
quoije c9575817fe Runner image: use full Python installer (bundles pip + OpenSSL) 2026-09-13 21:11:45 -04:00
quoije b576ff5747 Settings: add Storage section showing state dir, DB, cover cache, and user data paths 2026-09-13 21:07:23 -04:00
quoije 6510229b07 Add cover-grid sorting to Library and Store
Library: Title / Release date / Size / Region / Type / Serial
Store: Newest first / Title / Region / Size / In library first
2026-09-13 21:03:49 -04:00
quoije e61d91418e Runner image: add wine32 (winetricks regedit needs 32-bit Wine) 2026-09-13 20:52:42 -04:00
quoije a1c461f6aa Runner image: use embeddable Python (avoid 32-bit installer bootstrap) 2026-09-13 20:50:57 -04:00
quoije 985c8df2cc Runner image: add xauth (xvfb-run requirement) 2026-09-13 20:38:48 -04:00
quoije 6eab2eca9c Use upload-artifact v3 (Gitea lacks v4 artifact protocol)
build / Linux x86_64 (push) Successful in 13m34s
build / Windows x64 (Wine) (push) Canceled after 0s
2026-09-13 20:28:57 -04:00
quoije eac29c428d Add Wine-based runner image for Windows builds
- runner/Dockerfile: Debian + Wine + Windows CPython 3.12 + PyInstaller + act_runner
- runner/entrypoint.sh: self-register with the windows:host label
- workflow windows job now builds via winpython under Wine
2026-09-13 20:18:06 -04:00
quoije ce00e3f518 Add PyInstaller spec and Gitea Actions CI for Linux/Windows binaries
build / Linux x86_64 (push) Failing after 13m46s
build / Windows x64 (push) Canceled after 0s
- run.py: top-level entry point for PyInstaller (relative imports need it)
- psp-preserve.spec: onedir build bundling DATs, loader, cover bundles, icons
- requirements-build.txt: pinned build deps (PySide6 6.8.3 for Windows ICU fix)
- .gitea/workflows/build.yml: builds linux + windows artifacts on tags/manual
- gitignore build/ dist/
2026-09-13 20:09:55 -04:00
quoije c19717402c Polish settings, add matching defaults, drop pkg2zip, externalize infra config
- Rework Settings tab into grouped sections
- Add preferred-region matching and full-hash-by-default setting
- Replace combo-box square arrow with a proper chevron
- Fix UMD Refresh button (clicked bool was clearing the view)
- Remove pkg2zip.exe fallback (pure-Python decryptor is the primary path)
- Move LAN IP/share paths to .env; use example defaults in config
2026-09-13 20:03:44 -04:00
quoije 1ec2e81e15 Scan tab: remove duplicate Scan button, add redo-match-choice context menu 2026-09-13 19:31:28 -04:00
quoije bd9cf47ca9 Run saves scan in background so app startup doesn't freeze 2026-09-09 23:15:18 -04:00
quoije 6e06d1389c Background scrape/eject/refresh, PS1 PIC0 art polish, top-right logo, and updated screenshot 2026-09-09 21:25:34 -04:00
quoije 8c5f9b6ea0 Dark theme, scan auto-ingest, packaging prep, and README refresh 2026-09-09 18:17:23 -04:00
quoije 74164b33ec Update screenshot (5-column grid) 2026-09-09 12:28:18 -04:00
quoije 75db7a3faa Fill cover grid width in library and store (no dead gap) 2026-09-09 12:27:46 -04:00
quoije ed3cc9bc8a Update README screenshot 2026-09-08 23:59:34 -04:00
quoije e01013ce6a Fix DLC switch freeze: skip bulk cover scraping for themes/DLC 2026-09-08 23:55:51 -04:00
quoije 318a920873 Fix store scan signals running on worker thread (bound methods, not lambdas) 2026-09-08 23:27:29 -04:00
quoije ec9f5e6d8e Store: fix NPS refresh, scan race, thread cleanup, and matching perf 2026-09-08 23:18:23 -04:00
quoije be54ff00aa Re-enable cover/metadata scraping for themes and DLC 2026-09-08 23:04:24 -04:00
quoije 115cdd84b0 Store: show cached theme previews in grid and details panel 2026-09-08 22:56:58 -04:00
quoije 62ce4b1913 PSP Sync: show friendly theme/DLC names instead of raw filenames 2026-09-08 22:51:36 -04:00
quoije 46c675ce79 PSP Sync: list themes and DLC in the device inventory 2026-09-08 22:45:36 -04:00
quoije 42fac24cf1 Skip ISO.BIN.EDAT wrapper so DLC shows as a single library item 2026-09-08 22:15:36 -04:00
quoije 7cc5debfa6 Skip NoPayStation items with MISSING (delisted) pkg URLs 2026-09-08 22:00:56 -04:00
quoije c03f4619e6 Show theme/DLC names and embedded theme previews in the library 2026-09-08 17:32:11 -04:00
quoije 1be33be4ab Fix NPS store content-type dropdown fetching wrong list 2026-09-07 23:58:54 -04:00
quoije 2fed95da05 Skip cover/metadata scraping for NoPayStation themes and DLC 2026-09-07 23:53:55 -04:00
quoije c94bcdc993 Add NoPayStation themes, DLC and minis support 2026-09-07 23:34:56 -04:00
quoije 398c9f5dad Add PSP game Category Lite setting for NoPayStation installs 2026-09-07 23:00:04 -04:00
quoije 2ba24028a0 Port PSN pkg decryption to pure Python (drop Wine requirement) 2026-09-07 22:45:00 -04:00
quoije d458a9859c nps: bundle pkg2zip binary and run it via Wine
pkg2zip's prebuilt releases are Windows-only, so ship pkg2zip.exe in
data/bin/ (MIT, mmozeiko) and invoke it through Wine on Linux. find_pkg2zip
prefers the bundled binary, then a configured path, then a native pkg2zip on
PATH. Adds data/bin/ATTRIBUTION.md and README credit.
2026-09-07 22:14:14 -04:00
quoije 6837481f2b gui: narrow saves checkbox column + app window icon
- Saves checkbox column fixed at 28px.
- Add a PlayStation-style app icon (img/app-icon.png) and set it as the
  window icon on launch.
2026-09-07 22:04:27 -04:00
quoije 0ef7d05462 saves: bigger icons, per-row checkboxes, delete + export
- Save icon bumped to 56x40.
- Added a checkbox column for multi-select and a 'Delete selected' button.
- Right-click context menu on a row offers Export (copy the save folder to a
  chosen dir) and Delete (remove from all locations, with confirmation).
2026-09-07 21:48:41 -04:00
quoije de931f422a saves: show the save's ICON0.PNG in the table
The Save column now displays a small (40x32, letterboxed) icon from the
save folder's ICON0.PNG when present, using the first location (PSP,
PPSSPP, cloud) that has one.
2026-09-07 21:45:10 -04:00
quoije 4e0218a562 gui: stop scrape-all dialog from reappearing after cancel
Cancelling the store scrape now closes the dialog and drops its reference
immediately, so queued progress signals can't re-show it (QProgressDialog
re-shows on setValue while a value is pending). _scraping stays true until
the worker finishes to prevent a second concurrent scrape.
2026-09-07 21:41:53 -04:00
quoije 3032f04b72 modify .gitignore 2026-09-07 21:36:40 -04:00
quoije d02152ca06 fix NoPayStation PSP parsing + detail-panel UnboundLocalError
- PSP_GAMES.tsv and PSX_GAMES.tsv use different headers (PSP adds Type/RAP
  columns, PSX adds Original Name). Parse the header row and map columns by
  name instead of hardcoding positions — PSP names were showing as 'Minis'/
  'PC Engine' (the Type column) with broken download URLs.
- GameDetailsPanel.set_item referenced is_ps1 before it was assigned, raising
  UnboundLocalError and leaving the detail panel stale.
2026-09-07 21:32:50 -04:00
quoije efc8522117 nps: cache TSV lists in data/nps/ instead of the state dir
TSVs live with the other data assets; users can also drop PSP_GAMES.tsv /
PSX_GAMES.tsv there manually to browse offline.
2026-09-07 21:22:43 -04:00
quoije 53f671d09a store: add NoPayStation source (PSP + PS1 Classics .pkg)
New 'NoPayStation' option in the Store tab lists PSP_GAMES.tsv and
PSX_GAMES.tsv, downloads each item's encrypted .pkg and decrypts/extracts
it with pkg2zip into a configurable output dir (<dir>/PSP/<TITLE_ID>/ and
<dir>/PSX/<TITLE_ID>/). Downloaded PSP games and PS1 Classics appear in the
Library (games/eboot and ps1/eboot respectively) and copy to the PSP as
PSP/GAME folders.

- app/nopaystation.py: TSV fetch/parse (cached) + pkg2zip discovery/extract.
- app/transfers.py: NoPayStationSource (scan + download).
- Settings: NPS output dir + pkg2zip path (QSettings nps_dir/pkg2zip_path).
- StorePage: NPS source + content-type combo (PSP Games / PS1 Classics),
  per-item cover system, TSV release dates, nps-download job kind.
- Library: _nps_library_items ingest, on-PSP detection, folder copy/delete,
  PPSSPP play disabled for encrypted PSP EBOOTs.
- README: document the source, pkg2zip + npdrm_free requirements.
2026-09-07 21:18:55 -04:00
quoije 4b6e407b3f modify readme.md 2026-09-07 20:27:25 -04:00
quoije 196e3ac82f gui: show the source filename in Library and Store details
Add a muted, selectable filename line under the title in both details
panels, with the full path as its tooltip (archive name in the Store,
disc/ISO filename in the Library).
2026-09-07 20:05:31 -04:00
quoije 2c6b07e286 psp: detect installed PS1 games (incl. Category Lite folders)
PS1 EBOOTs install as PSP/GAME/{category}/{GAME_ID}/EBOOT.PBP, which the
PSP index ignored (it only indexed ISO games/videos), so the app never
marked them 'on PSP'. Index homebrew EBOOTs by folder id + embedded title,
match PS1 library items against them, and make 'Delete from PSP' remove the
whole game folder.
2026-09-07 19:59:30 -04:00
quoije 0ebd173af7 ps1: IGDB-first only for EBOOT icon, restore usual library order
Library covers go back to libretro -> OpenVGDB -> ScreenScraper ->
Wikipedia -> IGDB (last resort) for all systems, including PS1. The EBOOT
ICON0 (ps1_art) alone now tries IGDB first for clean banner-less box art,
falling back to the normal pipeline.
2026-09-07 19:46:50 -04:00
quoije c0c0f841d6 gui: label PS1 games as 'PS1' in the PSP sync table
The sync table reused a games/videos check, so PS1 entries showed as
'Video'. Use the same system->label map as the library table.
2026-09-07 19:39:14 -04:00
quoije 78b57af79e ps1: fit whole cover in ICON0 frame, show full PIC0 title
- ICON0: scale the cover to fit the case slot (no cropping), centred with
  the frame's dark background behind the letterbox bars.
- PIC0: shrink the title font until the full name fits instead of cutting
  it with an ellipsis (falls back to eliding only below 10px).
2026-09-07 19:33:21 -04:00
quoije 43263aa644 ps1: IGDB-first covers, elided PIC0 text, drop PS3 badge
- Cover art: try IGDB first for PS1 so the cover is clean box art without
  the PlayStation banner that libretro/ScreenScraper thumbnails bake in
  (redundant with the ICON0 case frame).
- PIC0: elide long titles with an ellipsis and word-wrap the description to
  3 lines with a trailing ellipsis instead of clipping/overflowing.
- PIC0: remove the PS3 badge; PSP only for now (PS3/PS4 packaging is future
  work).
2026-09-07 19:29:19 -04:00
quoije f6dccbec24 ps1: official-style ICON0/PIC0/PIC1 artwork with PSN templates
Replaces the letterboxed 144x80 icon + cropped PIC1 with official-looking
PS1 Classics art:
- ICON0 (80x80): cover art composited into the bundled case frame (PS logo
  + 'PlayStation.' wordmark) from data/ps1_templates/ICON0.PNG.
- PIC0 (310x180): a PSN-style info board drawn in code — title, IGDB
  description, ESRB badge (from IGDB age_ratings) and PS3/PSP badges.
- PIC1 (480x272): the bundled launch background template.

The converter now embeds PIC0, the PS1 tab previews all three images, and
ps1_art pulls IGDB description/ESRB as best effort. Templates are git-ignored
(user-supplied, like base.PBP) with a drawn-art fallback when missing.
2026-09-07 19:23:38 -04:00
quoije 2433240628 ps1: show only pending discs in the PS1 tab
Converted games already appear in the Library tab, so the PS1 tab now
lists only raw discs that still need converting. Auto-selects the first
pending disc so the preview loads immediately.
2026-09-07 18:45:55 -04:00
quoije c1fcf6176d ps1: preview EBOOT icon/background in the PS1 tab before converting
Selecting a pending disc in the PS1 tab now shows a live preview of the
ICON0 (144x80 XMB icon) and PIC1 (480x272 launch background) that the
converter would embed, fetched asynchronously so the UI stays responsive.
Converted discs/EBOOTs show an 'already converted' notice, and ps1_art now
falls back to a name-derived cache key when a disc has no serial.
2026-09-07 18:42:46 -04:00
quoije fb2a98dd21 ps1: embed real game art (ICON0 + PIC1) in converted EBOOTs
Fetch the game's cover art (libretro / ScreenScraper / IGDB / Wikipedia,
reusing the local cover cache) and bake it into the EBOOT's ICON0 (144x80
XMB icon) and PIC1 (480x272 launch background) instead of the default PS1
icon. Best-effort: falls back to the base icon/empty background when no art
is found. convert()/convert_to_folder() gain optional icon0/pic1 params.
2026-09-07 18:39:29 -04:00
quoije f9d9d639ed queue: add job Type column (Download/Copy to PSP/PS1 convert/etc.)
Add a 'Type' column to the Queue table mapping each job kind to a friendly
label, so downloads, copies, PS1 conversions, UMD dumps and save syncs are
distinguishable at a glance.
2026-09-07 18:27:43 -04:00
quoije abd201396c matcher: token pre-filter for fuzzy name matching (6x faster startup)
The fuzzy fallback ran difflib against every catalog release (~11k) for each
unmatched disc, dominating library load (2.7s) and startup scans. Pre-filter
candidates to those sharing a significant (>=3 char) word with the disc
filename or PARAM.SFO title before the expensive comparison, and cache the
pre-computed word sets. Library load: 2.7s -> 0.44s; scan: ~0.5s.
2026-09-07 18:23:28 -04:00
quoije e179bf2f2e library: refresh scraped cover in grid even when items were rebuilt
refresh_item matched the grid row by object identity, which silently fails
when the library list was rebuilt (background scan/store refresh) while the
scrape dialog was open. Match by release_id + system as a fallback, like
update_item_label already does, so the cover tile updates without a Reload.
2026-09-07 18:16:20 -04:00
quoije 02ed99c9c7 library: only show converted PS1 EBOOTs
Pending PS1 source discs stay in the PS1 tab; the Library now lists only
converted EBOOT entries.
2026-09-07 18:07:08 -04:00
quoije bcc2f4f73c gui: switch to Queue tab on PS1 convert; PS1 Category Lite folder
- _on_ps1_convert now opens the Queue tab so the user sees the conversion
  job start when converting from the Library or PS1 tab.
- Add a 'PS1 category (Category Lite)' setting (ps1_category) alongside the
  existing ISO category; PS1 EBOOTs copy to /PSP/GAME/<category>/<ID>/
  when set, matching the ISO category behaviour.
2026-09-07 18:05:53 -04:00
quoije f61a77a31f ps1: bundle plain-ELF POPS loader + KEYS.BIN (iPoPS, MIT)
Fix 80020148 'game could not be started' on ARK-5: the base PBP's encrypted
~PSP loader fails to link on modern CFW. Use the bundled plain-ELF loader
(data/loader/DATA.PSP) as the DATA.PSP section instead, and write the 16-byte
NPDRM version key (data/loader/KEYS.BIN) alongside the EBOOT so CFW can
decrypt the STARTDAT PGD. Both assets are from iPoPS (MIT, attributed).

- load_data_psp: prefer the bundled ELF, fall back to the base ~PSP loader.
- load_keys_bin: return the bundled 16-byte key when present.
- convert: write KEYS.BIN next to the output EBOOT.
- README + data/loader/ATTRIBUTION.md: document the loader and its license.
2026-09-07 17:44:54 -04:00
quoije ea08f52d44 ps1: general STARTDAT lookup + warn on old base.pbp loader
- extract_startdat: locate STARTDAT at PSAR + p1 (the u32 at PSAR+0x0C)
  instead of popstation's p1 + 0x50000 absolute offset, which only worked
  for bases whose PSAR was aligned to 0x50000. Any PS1 classic can now be
  used as a base.
- _warn_old_base: print a warning when the base PBP is a 3.xx-era classic,
  since its POPS loader often fails to link on 6.60/6.61 CFW (error
  80020148 'game could not be started').
- README: document the 6.60/6.61 base preference and DEFLATE compression.
2026-09-07 17:36:16 -04:00
quoije e50cd02f2a ps1: compress disc data (raw DEFLATE) for real-hardware boot
POPS on real PSP hardware refuses to boot uncompressed (level-0) PSARs
(noted in pop-fe: 'Uncompressed EBOOTs don't work'), which caused 'the game
cannot be started'. Compress each 0x9300 disc block with raw DEFLATE and
switch the index table to the pop-fe layout (offset u32, length u16, flags
u8, SHA-1[16]); patch p1/p2 after the compressed size is known.

- _write_disc_block: deflate each block; store raw bytes when a block does
  not shrink.
- _write_psar: reserve + patch the index table and p1/p2.
- convert/convert_to_folder/CLI: complevel parameter (default 1).
- Reuse the base PBP's ICON0 and drop generated PIC1 to match popstation.
- Self-test: add a DEFLATE round-trip with index/SHA-1 checks.
2026-09-07 17:05:31 -04:00
quoije 2f1fc760a1 ps1: byte-exact popstation output; fix TOC, DATA.PSP, SFO, pregap
Fix the EBOOT.PBP 'data is corrupted' boot failure on CFW by making the
converter match popstation byte-for-byte where it counts:

- build_toc: A0 entry byte 8 is the raw Red Book disc-type bitfield 0x20
  (CD-ROM XA), not BCD(0x20)=0x32; A-point entries use control 0x41.
- extract_data_psp: copy only prx_size (u32 at DATA.PSP+0x2C), not the
  whole zero-padded section.
- PSAR_ALIGNMENT: 0x50000 (matches popstation and the base PBP).
- SFO: patch the base PBP's PARAM.SFO in place (title + DISC_ID) instead
  of rebuilding, preserving integer field types (0x0404) and values.
- Cue pregap: INDEX 00 pregap is already inside the bin file, so do not
  re-emit zero sectors; only a PREGAP command adds external pregap.
  Track starts/lead-out now match PSXPackager's merged-cue output.
2026-09-07 16:50:52 -04:00
quoije 02b4adb1c1 PS1: match popstation PSAR header byte-for-byte; drop fake KEYS.BIN/DOCUMENT.DAT 2026-09-07 16:20:39 -04:00
quoije a9c371c221 PS1: output PSN-style game folders (EBOOT.PBP + KEYS.BIN + DOCUMENT.DAT) 2026-09-07 16:20:39 -04:00
quoije 4d5c48cb94 Add PS1 games to the Library with covers, metadata, and EBOOT actions 2026-09-07 16:20:39 -04:00
quoije 777d316c0f Docs: document PS1 support, cover sources, and new env vars 2026-09-07 16:20:39 -04:00
quoije 625e2f8718 UMD videos: add Wikipedia + Google image cover sources to scrape 2026-09-07 16:20:39 -04:00
quoije 7d8b449846 Scrape: restore full-size movie poster URL for UMD video covers 2026-09-07 16:20:39 -04:00
quoije 27e482d344 Library: refresh details cover box after a scrape 2026-09-07 16:20:39 -04:00
quoije 5d619e5f59 Scrape: include libretro + OpenVGDB covers alongside IGDB 2026-09-07 16:20:39 -04:00
quoije 743d5e8e90 Phase 2: PS1 library scan, settings, conversion queue, and PS1 tab 2026-09-07 16:20:39 -04:00
quoije bb1b8dea66 Use Redump + No-Intro DATs together for games; dedupe fuzzy matches 2026-09-07 16:20:39 -04:00
quoije 2d599141f9 Add Logiqx XML DAT parsing (Redump) and clear stale file matches on re-ingest 2026-09-07 16:20:39 -04:00
quoije 922c01d17c Target Redump DATs/catalog instead of No-Intro 2026-09-07 16:20:39 -04:00
quoije 128b62e5b8 Generate PS1 TOC in EBOOT converter 2026-09-07 16:20:39 -04:00
quoije da3429d918 Add pure-Python PS1 to EBOOT.PBP converter (phase 1) 2026-09-07 16:20:39 -04:00
43 changed files with 23553 additions and 438 deletions
+12
View File
@@ -0,0 +1,12 @@
# Copy this file to `.env` and fill in your values. `.env` is git-ignored.
# Static LAN IP for the API container on Unraid's macvlan bridge (br0).
PSP_LAN_IP=192.168.10.100
# Host paths mounted into the container (Unraid shares).
PSP_STORE_HOST=/mnt/user/downloads/Sony - PlayStation Portable
PSP_UMD_GAMES_HOST=/mnt/user/psp/umd_games
PSP_UMD_VIDEOS_HOST=/mnt/user/psp/umd_videos
# Secret token the desktop app sends as `Authorization: Bearer <token>`.
PSP_API_TOKEN=
+56
View File
@@ -0,0 +1,56 @@
name: build
on:
push:
tags: ["v*"]
workflow_dispatch:
jobs:
linux:
name: Linux x86_64
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install build dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements-build.txt
- name: Build
run: pyinstaller --clean --noconfirm psp-preserve.spec
- name: Package tarball
run: tar -czf psp-preserve-linux-x86_64.tar.gz -C dist psp-preserve
- uses: actions/upload-artifact@v3
with:
name: psp-preserve-linux-x86_64
path: psp-preserve-linux-x86_64.tar.gz
windows:
name: Windows x64 (Wine)
runs-on: windows
steps:
- uses: actions/checkout@v4
- name: Install build dependencies
run: |
winpython -m pip install --upgrade pip
winpython -m pip install -r requirements-build.txt
- name: Build
run: winpython -m PyInstaller --clean --noconfirm psp-preserve.spec
- name: Package zip
run: |
cd dist && zip -rq ../psp-preserve-windows-x64.zip psp-preserve
- uses: actions/upload-artifact@v3
with:
name: psp-preserve-windows-x64
path: psp-preserve-windows-x64.zip
+18
View File
@@ -6,3 +6,21 @@ data/*.sqlite
data/*.sqlite-*
data/credentials.json
session-transcript-2026-09-01.md
# Copyrighted PSN EBOOTs (base.pbp) used only for local PS1 conversion testing
data/*.PBP
data/*.pbp
# User-supplied PS1 Classics art templates (extracted from copyrighted PSN
# EBOOTs) — kept local only; the app falls back to drawn art when absent.
data/ps1_templates/
# nps tsv files
data/nps/*.tsv
# Local environment overrides (LAN paths, secrets). See .env.example
.env
# PyInstaller output
build/
dist/
+235 -39
View File
@@ -1,40 +1,50 @@
# PSP Preservation Tool
the name is terrible, i know.
![AI Generated](https://raw.githubusercontent.com/dotnetdave/ai-usage-badges/refs/heads/main/badges/png/1x/ai-generated.png)
Native Linux GUI (PySide6) for preserving PSP UMD **games** and **videos** as
`.iso` files. It scans your discs, verifies them against a full catalog
(No-Intro DATs), reports what's missing, and retags/renames discs to a canonical
scheme. It also fetches cover art and metadata, syncs discs and save-data with a
real PSP, and can pull missing discs from your own Redump 7z archive collection.
A cross-platform (Linux + Windows) desktop GUI — PySide6 — for preserving PSP
UMD **games** and **videos** as `.iso` files. It scans your discs, verifies them
against the Redump/No-Intro catalogs, reports what's missing, and retags/renames
discs to a canonical scheme. It fetches cover art and metadata, syncs discs and
save-data with a real PSP, can pull missing discs from a Redump 7z archive
collection, downloads PSN content from NoPayStation, and converts PlayStation 1
ISO/CUE/BIN rips into PSP `EBOOT.PBP` homebrew.
![PSP Preservation Tool](img/qt-gui.png)
* Use default app theme for Qt.
* Dark theme with a blue accent — the Library tab in cover view. *
## Features
- **Library** — scan folders of `.iso` discs, verify against the DAT catalog,
browse with cover art, search/filter, see per-game details (developer,
publisher, genre, release date, synopsis, trailer), and launch discs in PPSSPP.
- **Store** — browse and download discs from three sources: a local 7z folder,
your own HTTP store server, or an archive.org item (see
[Store](#store--archiveorg-and-local-folder)).
- **Store** — browse and download discs from four sources: a local 7z folder,
your own HTTP store server, an archive.org item, or NoPayStation (PSP/PS1
PSN `.pkg` content) (see [Store](#store--archiveorg-and-local-folder)).
- **Queue** — background transfers with progress, cancel, per-row delete, etc.
- **PSP Sync** — detect a connected PSP (USB) or a mounted UMD, inventory what's
installed (games / UMD videos / homebrew / PKG), copy discs to the PSP, delete
from the PSP, and dump a UMD to the library.
- **Saves** — two-way sync of save-data between the PSP, PPSSPP, and a cloud
folder.
- **PS1** — scan a folder of PlayStation 1 discs (ISO/BIN/CUE), read each disc's
serial, convert to PSP `EBOOT.PBP` with a pure-Python popstation converter,
and track already-converted games.
- **Settings** — directories, cover-source API keys, store/server config, and
misc options.
## Requirements
- Linux + Python 3
- `PySide6` — the only Python dependency (`pip install -r requirements.txt`)
- `7z` (p7zip) — used to peek/extract disc images from store archives
- Python 3.10+ on Linux or Windows
- `PySide6`, `curl_cffi`, `pycryptodome` (`pip install -r requirements.txt`)
- `7z` (p7zip / 7-Zip) — used to peek/extract disc images from store archives
- PPSSPP (optional) — to launch discs from the details panel
NoPayStation `.pkg` files are decrypted by the bundled **pure-Python
decryptor** (`app/pkg.py`) — no Wine or `pkg2zip` required
(see [NoPayStation](#4-nopaystation)).
## Install
```bash
@@ -49,21 +59,98 @@ pip install -r requirements.txt
python3 -m app.main
```
## Building binaries
The GUI is packaged with PyInstaller into a self-contained folder. Build
requirements are pinned in `requirements-build.txt` (PySide6 6.8.3 on purpose —
see the note inside).
```bash
python3 -m venv .venv-build
. .venv-build/bin/activate
pip install -r requirements-build.txt
pyinstaller --clean --noconfirm psp-preserve.spec
# output: dist/psp-preserve/ → run dist/psp-preserve/psp-preserve
```
Notes:
- The build bundles the DATs, PS1 loader, and the offline cover bundles. The
cover zips are large (~850 MB), so builds are slow and the output is big.
- `7z` (p7zip-full on Linux, 7-Zip on Windows) is a **runtime** dependency for
reading store archives — it is not bundled.
- User data (credentials, manual covers, PS1 templates, NPS cache, library DB,
cover cache) lives in the OS state dir, never next to the binary.
### CI (Gitea Actions)
`.gitea/workflows/build.yml` builds Linux and Windows artifacts on version-tag
pushes (`v*`) or on manual trigger (**Actions → build → Run workflow**).
Artifacts appear on the Actions run page and can be attached to a release.
Each job needs a [Gitea Actions runner](https://docs.gitea.com/usage/actions/act-runner)
registered with the matching label:
```bash
# Linux runner (Docker mode, run on your server)
./act_runner register \
--instance https://git.ceaf.dev \
--token <registration-token> \
--name linux-builder \
--labels ubuntu-latest:docker://node:20-bookworm
# Windows runner (host mode, run on a Windows machine)
act_runner.exe register \
--instance https://git.ceaf.dev \
--token <registration-token> \
--name win-builder \
--labels windows:host
```
The registration token is generated in Gitea under
**Site Administration → Actions → Runners → Create new runner**.
#### Windows builds without a Windows box (Wine runner)
The `windows` job targets a runner labelled `windows` that builds under Wine on
Linux. Build it from `runner/`:
```bash
docker build -f runner/Dockerfile -t psp-wine-runner .
docker run -d --name psp-wine-runner --restart unless-stopped \
-e GITEA_INSTANCE_URL=https://git.ceaf.dev \
-e GITEA_RUNNER_REGISTRATION_TOKEN=<registration-token> \
-v /mnt/user/appdata/psp-wine-runner:/data \
psp-wine-runner
```
The image ships Wine + Windows CPython 3.12 + PyInstaller, and registers itself
with the `windows:host` label. (If you have a real Windows machine instead,
register a native `act_runner.exe` with the same `windows` label.)
## First-time setup
1. Put your DAT files in `data/dats/`:
- games: No-Intro `Sony - PlayStation Portable`
- videos: No-Intro `Sony - PlayStation Portable (UMD Video)`
2. Ingest them into the local SQLite library:
- games: Redump `Sony - PlayStation Portable` (authoritative hashes)
**and** No-Intro `Sony - PlayStation Portable` (supplies the serials
Redump lacks — keeps serial-based matching working).
- videos: No-Intro `Sony - PlayStation Portable (UMD Video)` (Redump has
no UMD Video DAT).
2. Start the app — any DATs missing from the library are ingested automatically
at startup (no manual step). You can also re-ingest on demand:
```bash
python3 -m app.ingest
```
3. Open the GUI and point **Settings → General → Games/Videos directory** at
your disc folders (or set `PSP_GAMES_ROOTS` / `PSP_VIDEO_ROOTS`).
3. Point **Settings → General → Games/Videos directory** at your disc folders
(or set `PSP_GAMES_ROOTS` / `PSP_VIDEO_ROOTS`).
4. (Optional) PS1 conversion — copy any PSN PS1 classic `EBOOT.PBP` to
`data/base.PBP` (it supplies Sony's encrypted loader). Then set the **PS1
directory**, **PS1 output directory** and **PS1 base.pbp** in
**Settings → General** and use the **PS1** tab (see [PS1 support](#ps1-support)).
Discs that aren't in the No-Intro DATs can be added to `data/supplement.json`
Discs that aren't in the Redump DATs can be added to `data/supplement.json`
(`name`, `region`, `crc32`, `sha1`, `md5`) so the tool still recognizes them.
## Nomenclature
@@ -75,20 +162,32 @@ Discs are matched by hash first (most precise), then serial, then filename.
## Directory layout
- `data/dats/` — clrmamepro DAT files (games + videos).
- `data/supplement.json` — extra discs not in No-Intro.
- `data/supplement.json` — extra discs not in Redump.
- `data/credentials.json` — cover/metadata API keys (IGDB, TheMovieDB,
ScreenScraper). Environment variables take precedence.
ScreenScraper, Google Custom Search). Environment variables take precedence.
- `data/manual_covers/games|videos/` — your own cover images.
- `data/covers/` — checked-in assets: the UMD video/game cover templates,
`libretro-index.json` catalog, `libretro-covers.zip` + `openvgdb-covers.zip`
offline cover bundles, and the legacy cover cache.
- State dir (`~/.local/state/psp-preserve/` by default, or `$XDG_STATE_HOME/psp-preserve`):
`libretro-index.json` + `openvgdb-index.json` catalogs, `libretro-covers.zip` +
`openvgdb-covers.zip` offline cover bundles, and the legacy cover cache.
- `data/base.PBP` — a PSN PS1 classic EBOOT (any game) supplying `STARTDAT`
+ a base `PARAM.SFO`/icon for PS1 conversion (git-ignored, copyrighted).
- `data/loader/DATA.PSP` + `data/loader/KEYS.BIN` — bundled plain-ELF POPS
loader and NPDRM version key (from iPoPS, MIT).
- `data/ps1_templates/` — optional PS1 Classics art templates
(`ICON0.PNG` case frame, `PIC1.PNG` launch background); the app falls back
to drawn art when they're absent.
- `data/nps/` — NoPayStation TSV lists (cached here on first store scan in dev;
a packaged build stores them under the state dir instead).
- State dir (`~/.local/state/psp-preserve/` on Linux, `%LOCALAPPDATA%\PSPPreserve`
on Windows, or `$PSP_STATE_DIR`):
- `library.sqlite` — the catalog + library database.
- `covers/` — downloaded cover cache.
- `data/` — user-writable data in packaged builds (credentials, manual
covers, PS1 templates, NPS TSV cache).
SQLite is kept on a local disk because SQLite locking is unreliable over
network filesystems; the DATs, manual covers and credentials stay in `data/`
(which may be a network share).
network filesystems. Read-only resources (DATs, the POPS loader, icons) are
bundled with the app; user-writable data lives in the state dir.
## Configuration
@@ -104,22 +203,27 @@ QSettings under `PSPPreserve`) taking precedence where noted.
| `PSP_COVERS_DIR` | `$PSP_STATE_DIR/covers` | cover cache |
| `PSP_GAMES_ROOTS` | `/data/umd_games` | comma-separated game disc roots |
| `PSP_VIDEO_ROOTS` | `/data/umd_videos` | comma-separated video disc roots |
| `PSP_PS1_ROOTS` | `/data/ps1` | comma-separated PS1 disc roots |
| `PSP_PS1_OUTPUT_DIR` | `/data/ps1_converted` | converted EBOOT.PBP output folder |
| `PSP_SCAN_WORKERS` | `1` | parallel hashing workers |
| `PSP_STORE_DIR` | `/media/downloads/Sony - PlayStation Portable` | local 7z store folder |
| `PSP_STORE_DIR` | `/data/store` | local 7z store folder |
| `PSP_STORE_SERVER` | `https://psp.example.com` | store server base URL |
| `PSP_STORE_TOKEN` | *(empty)* | store server bearer token |
| `PSP_ARCHIVE_ITEM` | `psp-redump-roms321.com` | archive.org item identifier |
| `PSP_ARCHIVE_LOGGED_IN_SIG` | *(empty)* | archive.org `logged-in-sig` cookie |
| `PSP_ARCHIVE_LOGGED_IN_USER` | *(empty)* | archive.org `logged-in-user` cookie |
| `PSP_NPS_DIR` | `<state>/nps` | NoPayStation output folder |
| `PSP_NPS_TSV_BASE` | `https://nopaystation.com/tsv/` | NoPayStation TSV base URL |
| `PSP_API_TOKEN` | *(empty)* | server API bearer token (unset = open) |
| `PSP_CREDENTIALS` | `$PSP_DATA_DIR/credentials.json` | credentials file |
| `PSP_USER_DATA_DIR` | `<state>/data` (packaged) / `./data` (dev) | user-writable data (credentials, manual covers, PS1 templates, NPS cache) |
| `PSP_CREDENTIALS` | `$PSP_USER_DATA_DIR/credentials.json` | credentials file |
| `PSP_LIBRETRO_COVERS_URL` | *(empty)* | direct URL for `libretro-covers.zip` (else derived from store server) |
| `PSP_OVGDB_COVERS_URL` | *(empty)* | direct URL for `openvgdb-covers.zip` (else derived from store server) |
Cover/metadata keys can also be set as env vars: `IGDB_CLIENT_ID`,
`IGDB_CLIENT_SECRET`, `TMDB_API_KEY`, `SCREENSCRAPER_DEVID`,
`SCREENSCRAPER_DEVPASSWORD`, `SCREENSCRAPER_SSID`, `SCREENSCRAPER_SSPASSWORD`,
`SCREENSCRAPER_SOFTNAME`.
`IGDB_CLIENT_SECRET`, `TMDB_API_KEY`, `GOOGLE_API_KEY`, `GOOGLE_CX`,
`SCREENSCRAPER_DEVID`, `SCREENSCRAPER_DEVPASSWORD`, `SCREENSCRAPER_SSID`,
`SCREENSCRAPER_SSPASSWORD`, `SCREENSCRAPER_SOFTNAME`.
## Cover art
@@ -139,8 +243,11 @@ Games are resolved in this order (offline-first, then network):
3:4 key art, so the app composites it into the PSP game cover template
(`data/covers/UMD_GAME_COVER_TEMPLATE.png`) to restore the full cover.
UMD **videos** have no public box-art API: they use your manual covers, or a
TheMovieDB poster composited into `data/covers/UMD_COVER_TEMPLATE.png`.
UMD **videos** have no public box-art API. The manual **Scrape…** dialog offers
three sources: TheMovieDB posters, Wikipedia image search (keyless), and Google
Images (needs a Google API key + Search Engine ID in **Settings → Cover
sources**). The pick is composited into `data/covers/UMD_COVER_TEMPLATE.png`.
Manual covers in `data/manual_covers/videos/` always win.
### Offline bundles & auto-download
@@ -158,7 +265,47 @@ python3 -m app.openvgdb # OpenVGDB/GameFAQs -> openvgdb-covers.zip
(`app.openvgdb` needs `curl_cffi` to pass GameFAQs' Cloudflare challenge.)
## Store — archive.org and local folder
## PS1 support
The **PS1** tab scans a folder of PlayStation 1 rips and converts them into PSP
`EBOOT.PBP` files with a pure-Python popstation converter (no external tools).
PS1 discs (pending) and converted EBOOTs also show up in the **Library** tab
(filter by **PS1**), where you can convert a disc or copy an EBOOT to the PSP.
- **Source formats** — `.iso`, `.img`, `.bin` (with a matching `.cue` for
multi-track discs).
- **Serial detection** — reads `SYSTEM.CNF` to derive the game ID (e.g.
`SLUS-00001`) and writes it into `PARAM.SFO` + the PSAR header.
- **base.pbp** — conversion needs a PSN PS1 classic `EBOOT.PBP` (any game) to
supply the `STARTDAT` block (boot logo + PGD) and a base `PARAM.SFO`/icon.
Copy one to `data/base.PBP` and point **Settings → General → PS1 base.pbp**
at it. Any PS1 classic works structurally — the converter finds `STARTDAT`
at `PSAR + p1` regardless of PSAR alignment.
- **POPS loader** — the `DATA.PSP` section uses a bundled plain-ELF loader
(`data/loader/DATA.PSP`, from iPoPS, MIT) so the EBOOT boots on modern CFW
without the base PBP's encrypted loader failing to link (`80020148`). The
base PBP's own loader is only used as a fallback when the bundle is missing.
- **Output** — a folder `<out>/<Game ID>/` containing `EBOOT.PBP` (raw-DEFLATE
compressed PSAR) plus a `KEYS.BIN` (NPDRM version key used to decrypt the
`STARTDAT` PGD), ready to copy to `<PSP>/PSP/GAME/<Game ID>/`. Compatible
with CFW POPS (ARK-4/ARK-5).
- **Artwork** — the converter fetches the game's real cover art (libretro /
ScreenScraper / IGDB / Wikipedia, reusing the local cover cache) and builds
official-style PS1 Classics art:
- `ICON0` (80×80) — the cover composited into the bundled case frame
(`data/ps1_templates/ICON0.PNG`) with the PlayStation logo + wordmark;
- `PIC0` (310×180) — a PSN-style info board with the game title, a
description from IGDB, an ESRB rating badge and PS3/PSP platform badges;
- `PIC1` (480×272) — the bundled launch background
(`data/ps1_templates/PIC1.PNG`).
If the template images are missing the app falls back to drawn art, and to
the base icon when no cover is found.
Configure the folders in **Settings → General** (or `PSP_PS1_ROOTS` /
`PSP_PS1_OUTPUT_DIR`). Current limits: single-disc games only (no multi-disc
PBP); disc data is DEFLATE-compressed (level 1 by default, `--complevel 0-9`).
## Store — four sources
The **Store** tab downloads missing discs from a Redump 7z archive collection
(`Name (Region) (tags).7z`, containing the `.iso`/`.cso` inside). It has three
@@ -194,6 +341,48 @@ If the item's files are marked **private** (login required), set the
from a logged-in archive.org browser session). Without them, private files are
skipped — and you'll get a clear error if *everything* in the item is private.
### 4. NoPayStation
The **NoPayStation** source lists PSN content from
[NoPayStation](https://nopaystation.com/), selectable from the content-type
dropdown next to the source selector:
- **PSP Games** (`PSP_GAMES.tsv`) — includes **minis** (also listed separately
in the `Type` column), PC Engine and NeoGeo releases.
- **PS1 Classics** (`PSX_GAMES.tsv`)
- **PSP Themes** (`PSP_THEMES.tsv`)
- **PSP DLC** (`PSP_DLCS.tsv`)
Requirements & behaviour:
1. `.pkg` files are decrypted by the built-in **pure-Python decryptor**
(`app/pkg.py`, powered by `pycryptodome` for speed) — no Wine and no
`pkg2zip` needed.
2. Set **NoPayStation output dir** in **Settings → Store** (default
`$PSP_STATE_DIR/nps`).
3. Downloads land in the NPS folder and appear in the **Library**:
- PSP game → `PSP/<TITLE_ID>/EBOOT.PBP` (`psp_kind="eboot"`).
- PS1 Classic → `PSX/<TITLE_ID>/EBOOT.PBP` (+ `DOCUMENT.DAT`, `KEYS.BIN`).
- Theme → `THEME/<TITLE_ID>/<content_id>.ptf` (the packaged PTF is
EDAT-unwrapped to a raw theme file).
- DLC → `DLC/<GAME_ID>/<content_id>.edat` (installed as-is into the owning
game's folder).
4. Copying to the PSP puts games/PS1 under `/PSP/GAME/` (optionally nested in
a Category Lite folder), themes under `/PSP/THEME/`, and DLC under
`/PSP/GAME/<GAME_ID>/`.
5. The TSV lists are cached in `data/nps/` and refreshed on each store scan
(drop the `.tsv` files there manually to work offline).
> **DRM note:** NPS content is PSN-DRM'd. To *run* it on the PSP you need the
> [`npdrm_free`](https://github.com/qwikrazor87/npdrm_free) plugin (ARK and most
> CFW already ship it). The desktop app only downloads and decrypts; it doesn't
> strip DRM.
>
> **Covers/metadata:** themes and DLC have no cover art or game metadata of
> their own — the app scrapes them like games (IGDB/libretro/etc.), which can
> return unrelated results. Downloaded themes keep their embedded preview as
> the primary cover (used instead of a scraped image when available).
## Server setup
The catalog + store API is a single-file, stdlib-only HTTP server
@@ -234,13 +423,14 @@ Token protection: set `PSP_API_TOKEN`; the client sends
macvlan bridge so it gets its own LAN IP (e.g. `192.168.1.100`, plain HTTP on 8000):
```bash
# in /path/to/psp-preserve/ — adjust the volume paths for your shares
printf 'PSP_API_TOKEN=<long-random-secret>\n' > .env
# in /path/to/psp-preserve/
cp .env.example .env # then edit: LAN IP, share paths, API token
docker compose up -d --build
```
The compose file mounts your Unraid shares into `/data/store`, `/data/umd_games`
and `/data/umd_videos` (the store is mounted read-only).
The compose file reads the LAN IP and Unraid share paths from `.env` and
mounts them into `/data/store`, `/data/umd_games` and `/data/umd_videos`
(the store is mounted read-only).
### TLS (optional)
@@ -260,12 +450,14 @@ token**, then pick **Server** in the Store tab's source selector. Use
Connect a PSP over USB (or mount a UMD) and the **PSP Sync** tab detects it:
- **Games / UMD Videos / Homebrew-PSN / PKG** — tables of what's installed, with
delete-from-PSP (checkboxes and right-click).
- **Games / UMD Videos / Homebrew-PSN / Themes / DLC / PKG** — tables of what's
installed, with delete-from-PSP (checkboxes and right-click).
- **UMD** — reads `PARAM.SFO` from a mounted UMD, shows the title/serial, and
can dump the disc image into your library.
- **Sync from Library** — copy discs from your library onto the PSP, optionally
into a Category Lite category (set in **Settings → PSP → ISO category**).
NoPayStation PSP games install under `/PSP/GAME`, optionally nested in the
**PSP game category**; PS1 Classics use the **PS1 category**.
## Saves sync
@@ -289,6 +481,10 @@ python3 -m app.ingest --dats data/dats --db data/library.sqlite
# (re)build the offline cover bundles (see Cover art)
python3 -m app.libretro
python3 -m app.openvgdb
# convert one PS1 disc to EBOOT.PBP (CLI)
python3 -m app.ps1_convert --source /path/to/game.cue --title "Game" \
--game-id SLUS00001 --base-pbp data/base.PBP --out /out/Game.pbp
```
`python3 -m app.scan --json` prints machine-readable results, and `--quick`
+55 -5
View File
@@ -1,8 +1,33 @@
"""Central configuration, driven by environment variables."""
import os
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parent.parent
# PyInstaller sets ``sys.frozen`` and unpacks bundled files to ``sys._MEIPASS``.
# Read-only resources (DATs, loader, icons) then live in the bundle; otherwise
# they live in the project checkout.
IS_FROZEN = getattr(sys, "frozen", False)
_BUNDLE_DIR = getattr(sys, "_MEIPASS", None)
PROJECT_ROOT = Path(_BUNDLE_DIR).resolve() if _BUNDLE_DIR else Path(__file__).resolve().parent.parent
# Frozen bundles don't ship the system CA store, so Python's urllib would fail
# TLS verification against the store server / NoPayStation / TMDB / archive.org.
# Point OpenSSL at the certifi bundle (a curl_cffi dependency) when frozen only;
# source runs keep using the OS CA store.
if IS_FROZEN:
try:
import certifi
_CA_BUNDLE = certifi.where()
os.environ.setdefault("SSL_CERT_FILE", _CA_BUNDLE)
os.environ.setdefault("REQUESTS_CA_BUNDLE", _CA_BUNDLE)
except Exception: # noqa: BLE001
pass
def resource_path(*parts):
"""Absolute path to a read-only bundled resource (works when frozen)."""
return PROJECT_ROOT.joinpath(*parts)
DATA_DIR = Path(os.environ.get("PSP_DATA_DIR", str(PROJECT_ROOT / "data")))
DATS_DIR = Path(os.environ.get("PSP_DATS_DIR", str(DATA_DIR / "dats")))
@@ -12,9 +37,12 @@ def _default_state_dir():
"""Local state directory (fast, reliable local disk).
SQLite locking is unreliable over network filesystems (SMB/FUSE), so the
library DB and the cover cache live on local storage while DATs, manual
covers and credentials stay in ``DATA_DIR`` (which may be a network share).
library DB and the cover cache live on local storage while read-only DATs
stay bundled with the app.
"""
if os.name == "nt":
base = os.environ.get("LOCALAPPDATA") or os.environ.get("APPDATA") or str(Path.home())
return Path(base) / "PSPPreserve"
xdg = os.environ.get("XDG_STATE_HOME")
base = Path(xdg) if xdg else Path.home() / ".local" / "state"
return base / "psp-preserve"
@@ -23,14 +51,24 @@ def _default_state_dir():
STATE_DIR = Path(os.environ.get("PSP_STATE_DIR", str(_default_state_dir())))
DB_PATH = Path(os.environ.get("PSP_DB_PATH", str(STATE_DIR / "library.sqlite")))
COVERS_DIR = Path(os.environ.get("PSP_COVERS_DIR", str(STATE_DIR / "covers")))
# User-writable data (credentials, manual covers, PS1 templates, NPS TSV cache).
# In dev this is the checkout's ``data/``; when frozen the bundle is read-only,
# so it points at local state that survives app upgrades.
USER_DATA_DIR = Path(os.environ.get(
"PSP_USER_DATA_DIR", str(DATA_DIR if not IS_FROZEN else STATE_DIR / "data")))
LEGACY_DB_PATH = DATA_DIR / "library.sqlite"
LEGACY_COVERS_DIR = DATA_DIR / "covers"
def user_data_path(*parts):
"""Absolute path to a user-writable data file/folder (works when frozen)."""
return USER_DATA_DIR.joinpath(*parts)
# Redump 7z store (browsable in the Store tab). Env override; GUI setting wins.
STORE_ROOT = Path(os.environ.get("PSP_STORE_DIR", "/media/downloads/Sony - PlayStation Portable"))
STORE_ROOT = Path(os.environ.get("PSP_STORE_DIR", "/data/store"))
# Remote store server (HTTPS). Used by the Store tab as the primary source.
STORE_SERVER_URL = os.environ.get("PSP_STORE_SERVER", "https://psp.ceaf.dev")
STORE_SERVER_URL = os.environ.get("PSP_STORE_SERVER", "https://psp.example.com")
STORE_TOKEN = os.environ.get("PSP_STORE_TOKEN", "")
# Optional direct URLs for the offline cover bundle zips. When unset, the GUI
@@ -46,14 +84,26 @@ ARCHIVE_ITEM = os.environ.get("PSP_ARCHIVE_ITEM", "psp-redump-roms321.com")
ARCHIVE_LOGGED_IN_SIG = os.environ.get("PSP_ARCHIVE_LOGGED_IN_SIG", "")
ARCHIVE_LOGGED_IN_USER = os.environ.get("PSP_ARCHIVE_LOGGED_IN_USER", "")
# NoPayStation: where extracted .pkg content lands, and the TSV source base.
# PSP games go to <NPS_OUTPUT_DIR>/PSP/<TITLE_ID>/; PS1 Classics to .../PSX/.
NPS_OUTPUT_DIR = Path(os.environ.get("PSP_NPS_DIR", str(STATE_DIR / "nps")))
NPS_TSV_BASE = os.environ.get("PSP_NPS_TSV_BASE", "https://nopaystation.com/tsv/")
# Scan roots (used by the scanner phase). Comma-separated in env.
GAMES_ROOTS = [Path(p) for p in os.environ.get("PSP_GAMES_ROOTS", "/data/umd_games").split(",") if p.strip()]
VIDEO_ROOTS = [Path(p) for p in os.environ.get("PSP_VIDEO_ROOTS", "/data/umd_videos").split(",") if p.strip()]
# PS1: source folder (raw ISO/BIN/CUE) and converted EBOOT.PBP output folder.
PS1_ROOTS = [Path(p) for p in os.environ.get("PSP_PS1_ROOTS", "/data/ps1").split(",") if p.strip()]
PS1_OUTPUT_DIR = Path(os.environ.get("PSP_PS1_OUTPUT_DIR", "/data/ps1_converted"))
# Only .iso discs are tracked; everything below is explicitly ignored.
DISC_EXTENSIONS = {".iso"}
IGNORED_EXTENSIONS = {".pkg", ".pbp", ".cso"}
# PS1 source images (scanned separately from PSP .iso discs).
PS1_EXTENSIONS = {".cue", ".iso", ".img", ".bin"}
# Parallel hashing workers for the scanner. Keep at 1 on spinning disks
# (concurrent reads cause seek thrash); raise it for SSD / fast arrays.
SCAN_WORKERS = int(os.environ.get("PSP_SCAN_WORKERS", "1"))
+643 -96
View File
@@ -24,19 +24,35 @@ from .db import get_connection
log = logging.getLogger("covers")
_LIBRETRO_BASE = "https://raw.githubusercontent.com/libretro-thumbnails/Sony_-_PlayStation_Portable/master"
_LIBRETRO_KINDS = ("Named_Boxarts",)
_TREE_URL = "https://api.github.com/repos/libretro-thumbnails/Sony_-_PlayStation_Portable/git/trees/master?recursive=1"
_PLATFORMS = {
"games": {"libretro_repo": "Sony_-_PlayStation_Portable", "igdb_id": 38, "ss_id": "61"},
"ps1": {"libretro_repo": "Sony_-_PlayStation", "igdb_id": 7, "ss_id": "57"},
}
def _platform(system):
"""Platform parameters (libretro repo, IGDB id, ScreenScraper id)."""
return _PLATFORMS.get(system) or _PLATFORMS["games"]
def _libretro_base(system="games"):
return ("https://raw.githubusercontent.com/libretro-thumbnails/"
f"{_platform(system)['libretro_repo']}/master")
def _libretro_tree_url(system="games"):
return ("https://api.github.com/repos/libretro-thumbnails/"
f"{_platform(system)['libretro_repo']}/git/trees/master?recursive=1")
_SS_API = "https://api.screenscraper.fr/api2/jeuInfos.php"
_SS_PSP_ID = "61" # Sony PSP
_IGDB_TOKEN_URL = "https://id.twitch.tv/oauth2/token"
_IGDB_API = "https://api.igdb.com/v4/games"
_IGDB_IMAGE = "https://images.igdb.com/igdb/image/upload/"
_IGDB_PSP_ID = 38 # Sony PSP
_TMDB_SEARCH = "https://api.themoviedb.org/3/search/movie"
_TMDB_IMAGE = "https://image.tmdb.org/t/p/"
_MANUAL_DIR = config.DATA_DIR / "manual_covers"
_GOOGLE_CSE_API = "https://www.googleapis.com/customsearch/v1"
_MANUAL_DIR = config.USER_DATA_DIR / "manual_covers"
_OVGDB_ZIP = config.DATA_DIR / "covers" / "openvgdb-covers.zip"
_LIBRETRO_ZIP = config.DATA_DIR / "covers" / "libretro-covers.zip"
@@ -51,7 +67,7 @@ _igdb_token_state = {"token": "", "expires": 0.0} # IGDB OAuth2 token cache
_igdb_lock = threading.Lock()
_igdb_last = 0.0 # monotonic clock of the last IGDB API call
_index_state = {"ts": 0.0, "names": {}, "norm": {}}
_index_state = {} # system -> {"ts", "names", "norm"}
_INDEX_TTL = 86400.0 # refresh the libretro file index once a day
_neg_cache = {} # cache key -> time.time() of the last failed fetch
@@ -186,9 +202,9 @@ def ensure_cover_bundles(server_url=""):
return downloaded
def cover_url(name, kind="Named_Boxarts"):
def cover_url(name, kind="Named_Boxarts", system="games"):
"""Public libretro URL of an image for a catalog name."""
return f"{_LIBRETRO_BASE}/{kind}/{urllib.parse.quote(name)}.png"
return f"{_libretro_base(system)}/{kind}/{urllib.parse.quote(name)}.png"
def cover_path(release_id, system):
@@ -260,7 +276,7 @@ def _candidate_names(name):
yield s
def _fetch_png(name, kind, timeout):
def _fetch_png(name, kind, timeout, system="games"):
"""Fetch a real image from libretro, following symlink text bodies.
GitHub serves a repo symlink as a tiny text body containing the target
@@ -268,7 +284,7 @@ def _fetch_png(name, kind, timeout):
"""
target = name
for _ in range(_MAX_HOPS + 1):
data = _http_get(cover_url(target, kind), timeout)
data = _http_get(cover_url(target, kind, system), timeout)
if data is None:
return None
if _is_image(data):
@@ -283,26 +299,29 @@ def _fetch_png(name, kind, timeout):
# --------------------------------------------------------------------------- #
# libretro filename index + fuzzy matching
# --------------------------------------------------------------------------- #
def _index_path():
return config.COVERS_DIR / "libretro-index.json"
def _index_path(system="games"):
if system == "games":
return config.COVERS_DIR / "libretro-index.json"
return config.COVERS_DIR / f"libretro-index-{system}.json"
def _load_index(force=False):
st = _index_state
def _load_index(force=False, system="games"):
st = _index_state.get(system) or {"ts": 0.0, "names": {}, "norm": {}}
_index_state[system] = st
if not force and st["names"] and time.time() - st["ts"] < _INDEX_TTL:
return st["names"]
path = _index_path()
path = _index_path(system)
if not force and path.exists():
try:
# Load the on-disk index so it doubles as a fallback if the network
# is down, but only return it immediately while it's still fresh.
names = json.loads(path.read_text(encoding="utf-8"))
_set_index(names)
_set_index(names, system)
if time.time() - path.stat().st_mtime < _INDEX_TTL:
return names
except Exception:
pass
data = _http_get(_TREE_URL, timeout=30)
data = _http_get(_libretro_tree_url(system), timeout=30)
if data:
try:
tree = json.loads(data).get("tree", [])
@@ -313,7 +332,7 @@ def _load_index(force=False):
prefix = kind + "/"
if p.startswith(prefix) and p.endswith(".png"):
names[kind].append(p[len(prefix):-4])
_set_index(names)
_set_index(names, system)
try:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(names), encoding="utf-8")
@@ -325,9 +344,10 @@ def _load_index(force=False):
return st["names"] or {kind: [] for kind in _LIBRETRO_KINDS}
def _set_index(names):
def _set_index(names, system="games"):
"""Store the raw filename index plus a pre-normalized copy for fast lookup."""
st = _index_state
st = _index_state.get(system) or {"ts": 0.0, "names": {}, "norm": {}}
_index_state[system] = st
st["names"] = names
st["norm"] = {k: [(_norm(n), n) for n in v] for k, v in names.items()}
st["ts"] = time.time()
@@ -343,11 +363,12 @@ def _plain(s):
return re.sub(r"\s*\([^()]*\)", "", s).strip()
def _index_best(name, kind):
def _index_best(name, kind, system="games"):
"""Return the best-matching repo filename for `name`, or None."""
st = _index_state
st = _index_state.get(system) or {"ts": 0.0, "names": {}, "norm": {}}
if not st["norm"]:
_load_index()
_load_index(system=system)
st = _index_state.get(system) or st
pairs = st["norm"].get(kind, [])
db = _norm(name)
if not pairs or len(db) < 5:
@@ -384,24 +405,28 @@ def _index_best(name, kind):
return best
def _fetch_libretro(name, timeout, indexed_only=False):
# 0. Shipped offline bundle first (fast, no network).
data = fetch_libretro_local(name)
if data is not None:
return data
def _fetch_libretro(name, timeout, system="games", indexed_only=False):
# 0. Shipped offline bundle first (fast, no network) — PSP only for now.
if system == "games":
data = fetch_libretro_local(name)
if data is not None:
return data
# Theme/DLC share the PSP thumbnails repo, so reuse the games index instead
# of downloading a duplicate copy of the same tree under a new system key.
idx_system = system if system in _PLATFORMS else "games"
if not indexed_only:
# 1. Cheap: exact + progressively stripped names (no index needed).
for kind in _LIBRETRO_KINDS:
for cand in _candidate_names(name):
data = _fetch_png(cand, kind, timeout)
data = _fetch_png(cand, kind, timeout, system)
if data is not None:
return data
# 2. Fuzzy/exact via the precomputed index (covers exact names too, so the
# bulk loader skips the wasteful exact pass entirely).
for kind in _LIBRETRO_KINDS:
best = _index_best(name, kind)
best = _index_best(name, kind, idx_system)
if best:
data = _fetch_png(best, kind, timeout)
data = _fetch_png(best, kind, timeout, system)
if data is not None:
return data
return None
@@ -466,6 +491,33 @@ def _ovgdb_name_list():
return _ovgdb_names
_ovgdb_index = None
def _load_ovgdb_index():
"""Load {key: {"title", "url"}} written by app/openvgdb.py, or {}.
Looks in the runtime covers dir first, then the shipped ``data/covers``
copy (so a fresh install still shows readable titles without re-running
the full OpenVGDB scrape).
"""
global _ovgdb_index
if _ovgdb_index is not None:
return _ovgdb_index
_ovgdb_index = {}
for path in (
config.COVERS_DIR / "openvgdb-index.json",
config.LEGACY_COVERS_DIR / "openvgdb-index.json",
):
if path.exists():
try:
_ovgdb_index = json.loads(path.read_text(encoding="utf-8"))
break
except Exception: # noqa: BLE001
_ovgdb_index = {}
return _ovgdb_index
def fetch_openvgdb_local(name):
"""Return a bundled OpenVGDB cover image for `name`, or None."""
names = _ovgdb_name_list()
@@ -549,7 +601,7 @@ def _screenscraper_box_urls(jeu):
return urls
def _fetch_screenscraper(name, timeout):
def _fetch_screenscraper(name, timeout, system="games"):
creds = sources.load()
devid = creds.get("screenscraper_devid")
devpassword = creds.get("screenscraper_devpassword")
@@ -560,7 +612,7 @@ def _fetch_screenscraper(name, timeout):
"devpassword": devpassword,
"softname": creds.get("screenscraper_softname") or "psp-preserve",
"output": "json",
"systemeid": _SS_PSP_ID,
"systemeid": _platform(system)["ss_id"],
"romtype": "rom",
"romnom": name,
}
@@ -759,6 +811,346 @@ def compose_game_cover(key_art_bytes):
return bytes(ba)
_PS1_TEMPLATE_DIR = config.USER_DATA_DIR / "ps1_templates"
def _ps1_template_bytes(name):
"""Read a bundled PS1 art template (``ICON0.PNG`` / ``PIC1.PNG``)."""
try:
p = _PS1_TEMPLATE_DIR / name
if p.is_file():
return p.read_bytes()
except OSError:
pass
return None
def _slot_rect(img):
"""Find the fully-transparent cover slot in an icon template image.
Returns ``(x, y, w, h)`` or ``None`` when the image has no transparent
region (e.g. a flattened template)."""
if img.isNull():
return None
w, h = img.width(), img.height()
min_x = min_y = 1 << 30
max_x = max_y = -1
for y in range(h):
for x in range(w):
if img.pixelColor(x, y).alpha() < 10:
min_x, max_x = min(min_x, x), max(max_x, x)
min_y, max_y = min(min_y, y), max(max_y, y)
if max_x < min_x or max_y < min_y:
return None
return min_x, min_y, max_x - min_x + 1, max_y - min_y + 1
def make_ps1_assets(cover_bytes, title="", description="", esrb=None):
"""Build official-style PS1 Classics art from the cover: (icon0, pic0, pic1).
- ``ICON0`` (80x80): the cover composited into the bundled case template
(PS button logo, ``PlayStation.`` wordmark, gradient frame).
- ``PIC0`` (310x180): the PSN-style info board — title, description,
ESRB rating badge and PS3/PSP platform badges.
- ``PIC1`` (480x272): the bundled launch background (gradient + curve).
Returns ``(None, None, None)`` when the art can't be decoded or Qt is
unavailable.
"""
try:
from PySide6.QtCore import QBuffer, QByteArray, QPointF, QRectF, Qt
from PySide6.QtGui import (QColor, QFont, QFontMetrics, QImage,
QLinearGradient, QPainter, QPainterPath,
QPen, QPolygonF)
except Exception: # noqa: BLE001 - headless API may lack PySide6
return None, None, None
art = QImage.fromData(cover_bytes)
if art.isNull():
return None, None, None
art = art.convertToFormat(QImage.Format_ARGB32)
def _save(img):
ba = QByteArray()
buf = QBuffer(ba)
buf.open(QBuffer.WriteOnly)
img.save(buf, "PNG")
buf.close()
return bytes(ba)
# --- ICON0: cover art inside the case template ---------------------------
icon = QImage(80, 80, QImage.Format_ARGB32)
icon.fill(QColor(0, 0, 0))
p = QPainter(icon)
p.setRenderHint(QPainter.SmoothPixmapTransform, True)
tmpl = QImage.fromData(_ps1_template_bytes("ICON0.PNG") or b"")
if not tmpl.isNull():
tmpl = tmpl.convertToFormat(QImage.Format_ARGB32)
slot = _slot_rect(tmpl) or (0, 0, 80, 80)
sx, sy, sw, sh = slot
# Fit the whole cover inside the slot (no cropping), centred with the
# frame's dark background showing through the letterbox bars.
cover = art.scaled(sw, sh, Qt.KeepAspectRatio, Qt.SmoothTransformation)
p.drawImage(sx + (sw - cover.width()) // 2, sy + (sh - cover.height()) // 2, cover)
p.drawImage(0, 0, tmpl)
else:
cover = art.scaled(80, 80, Qt.KeepAspectRatio, Qt.SmoothTransformation)
p.drawImage((80 - cover.width()) // 2, (80 - cover.height()) // 2, cover)
p.end()
icon0 = _save(icon)
# --- PIC0: PSN-style info board -----------------------------------------
pic0 = _make_pic0(title, description, esrb, QImage, QPainter, QPainterPath,
QColor, QFont, QFontMetrics, QLinearGradient, QPen,
QPolygonF, QPointF, QRectF, Qt, _save)
# --- PIC1: bundled launch background ------------------------------------
pic1 = _make_pic1(QImage, QPainter, QColor, QLinearGradient, Qt, _save)
return icon0, pic0, pic1
def _make_pic1(QImage, QPainter, QColor, QLinearGradient, Qt, _save):
pic = QImage(480, 272, QImage.Format_ARGB32)
raw = _ps1_template_bytes("PIC1.PNG")
tmpl = QImage.fromData(raw) if raw else QImage()
if not tmpl.isNull():
return _save(tmpl.convertToFormat(QImage.Format_ARGB32))
# Fallback: a light lavender gradient with a subtle curve (same vibe as
# the official background) for when the template image is missing.
grad = QLinearGradient(0, 0, 0, 272)
grad.setColorAt(0.0, QColor(235, 232, 238))
grad.setColorAt(1.0, QColor(205, 200, 215))
p = QPainter(pic)
p.fillRect(0, 0, 480, 272, grad)
p.setPen(QPen(QColor(170, 160, 185, 120), 3))
path = QPainterPath()
path.moveTo(-40, 300)
path.quadTo(240, 60, 520, 300)
p.drawPath(path)
p.end()
return _save(pic)
def _make_pic0(title, description, esrb, QImage, QPainter, QPainterPath, QColor,
QFont, QFontMetrics, QLinearGradient, QPen, QPolygonF, QPointF,
QRectF, Qt, _save):
W, H = 310, 180
img = QImage(W, H, QImage.Format_ARGB32)
img.fill(QColor(0, 0, 0, 0))
# Dark gradient backdrop clipped to a rounded card (transparent corners).
grad = QLinearGradient(0, 0, 0, H)
grad.setColorAt(0.0, QColor(38, 38, 40))
grad.setColorAt(1.0, QColor(12, 12, 14))
card = QPainterPath()
card.addRoundedRect(QRectF(0, 0, W, H), 12, 12)
p = QPainter(img)
p.setRenderHint(QPainter.Antialiasing, True)
p.setRenderHint(QPainter.TextAntialiasing, True)
p.fillPath(card, grad)
# Rounded card border.
p.setPen(QPen(QColor(110, 110, 115), 1))
p.setBrush(Qt.NoBrush)
p.drawPath(card)
# PlayStation button glyphs, top-right.
_draw_ps_glyphs(p, W - 34, 12, 22, QRectF, QPointF, QPolygonF, QPen, Qt,
QColor)
# Title (bold, upper-left). Shrink the font until the whole title fits, so
# long names are shown in full instead of being cut with an ellipsis.
f_title = QFont("DejaVu Sans")
f_title.setBold(True)
p.setPen(QColor(255, 255, 255))
title_width = W - 70
fm_title = QFontMetrics(f_title)
full = (title or "").strip()
size = 19
while size > 10 and fm_title.horizontalAdvance(full) > title_width:
size -= 1
f_title.setPixelSize(size)
fm_title = QFontMetrics(f_title)
title_text = full
if fm_title.horizontalAdvance(full) > title_width:
title_text = fm_title.elidedText(full, Qt.ElideRight, title_width)
p.setFont(f_title)
p.drawText(QRectF(18, 16, title_width, 28), Qt.AlignLeft | Qt.AlignVCenter,
title_text)
# Thin divider under the title (like the original PSN info board).
p.setPen(QPen(QColor(255, 255, 255, 55), 1))
p.drawLine(QPointF(18, 46), QPointF(W - 18, 46))
# Description (wrapped to at most 3 lines, last line elided).
if description:
f_desc = QFont("DejaVu Sans")
f_desc.setPixelSize(12)
p.setFont(f_desc)
p.setPen(QColor(215, 215, 215))
fm = QFontMetrics(f_desc)
lines = _wrap_elided(fm, description.strip(), W - 36, 3, Qt)
y = 52
for line in lines:
p.drawText(QRectF(18, y, W - 36, 16), Qt.AlignLeft | Qt.AlignVCenter, line)
y += 16
# ESRB badge (bottom-left) + caption.
if esrb:
badge = esrb.strip()
if badge:
_draw_esrb(p, badge, 18, 140, QRectF, QColor, QFont, QFontMetrics, QPen, Qt)
# Platform badge (bottom-right). Only PSP for now; PS3/PS4 packaging is a
# separate future feature.
_draw_platform_badge(p, "PSP", W - 14 - 30, 146, QColor(212, 175, 55),
QRectF, QColor, QFont, QFontMetrics, Qt)
p.end()
return _save(img)
def _wrap_elided(fm, text, width, max_lines, Qt):
"""Word-wrap ``text`` to ``width`` px, eliding the final line when it runs
past ``max_lines``. Returns a list of up to ``max_lines`` strings."""
words = text.split()
lines = []
cur = ""
for i, w in enumerate(words):
cand = (cur + " " + w).strip() if cur else w
if fm.horizontalAdvance(cand) <= width:
cur = cand
continue
if len(lines) == max_lines - 1:
rest = " ".join([cur] + words[i:]) if cur else " ".join(words[i:])
lines.append(fm.elidedText(rest, Qt.ElideRight, width))
return lines
lines.append(cur)
cur = w
if cur:
lines.append(cur)
return lines[:max_lines]
def _draw_ps_glyphs(p, x, y, size, QRectF, QPointF, QPolygonF, QPen, Qt, QColor):
"""Draw the four PlayStation face-button glyphs in a 2x2 grid, using the
official button colours (green triangle, red circle, blue cross, pink square)."""
cell = size / 2.0
p.setBrush(Qt.NoBrush)
def _pen(color):
pen = QPen(color, max(1.0, size / 14.0))
pen.setCapStyle(Qt.RoundCap)
pen.setJoinStyle(Qt.RoundJoin)
return pen
# Triangle (top-left, green).
p.setPen(_pen(QColor("#3F9B0F")))
tri = QPolygonF([QPointF(x, y + cell), QPointF(x + cell, y), QPointF(x + cell, y + cell)])
p.drawPolygon(tri)
# Circle (top-right, red).
p.setPen(_pen(QColor("#DE1C18")))
p.drawEllipse(QRectF(x + cell, y, cell, cell))
# Cross (bottom-left, blue).
p.setPen(_pen(QColor("#287FC1")))
p.drawLine(QPointF(x + 1, y + cell + 1), QPointF(x + cell - 1, y + size - 1))
p.drawLine(QPointF(x + cell - 1, y + cell + 1), QPointF(x + 1, y + size - 1))
# Square (bottom-right, pink).
p.setPen(_pen(QColor("#C94E9B")))
p.drawRect(QRectF(x + cell, y + cell, cell, cell))
def _draw_esrb(p, badge, x, y, QRectF, QColor, QFont, QFontMetrics, QPen, Qt):
f = QFont("DejaVu Sans")
f.setPixelSize(9)
f.setBold(True)
fm = QFontMetrics(f)
text = badge if badge != "EVERYONE" else "EVERYONE"
# Shrink until it fits the box (the official badge shows the word).
size = 9
while size > 5 and fm.horizontalAdvance(text) > 46:
size -= 1
f.setPixelSize(size)
fm = QFontMetrics(f)
rect = QRectF(x, y, 50, 24)
p.setPen(QPen(QColor(20, 20, 20), 1))
p.setBrush(QColor(255, 255, 255))
p.drawRoundedRect(rect, 3, 3)
p.setPen(QColor(20, 20, 20))
p.setFont(f)
p.drawText(rect, Qt.AlignCenter, text)
# Caption.
fc = QFont("DejaVu Sans")
fc.setPixelSize(7)
p.setFont(fc)
p.setPen(QColor(150, 150, 155))
p.drawText(QRectF(x + 56, y + 6, 120, 14),
Qt.AlignLeft | Qt.AlignVCenter, "ESRB CONTENT RATING")
def _draw_platform_badge(p, label, x, y, color, QRectF, QColor, QFont,
QFontMetrics, Qt):
f = QFont("DejaVu Sans")
f.setPixelSize(10)
f.setBold(True)
rect = QRectF(x, y, 30, 16)
p.setPen(Qt.NoPen)
p.setBrush(QColor(26, 26, 28))
p.drawRoundedRect(rect, 4, 4)
p.setPen(color)
p.setFont(f)
p.drawText(rect, Qt.AlignCenter, label)
def ps1_art(title, release_id, timeout=10):
"""Fetch a PS1 game's cover and build ``(icon0, pic0, pic1)`` PNG bytes.
The EBOOT icon prefers IGDB first (clean box art without the PlayStation
banner that libretro/ScreenScraper thumbnails bake in); if IGDB has
nothing it falls back to the normal cover pipeline (libretro, OpenVGDB,
ScreenScraper, Wikipedia, IGDB). The description and ESRB rating for the
PIC0 info board come from IGDB (best effort).
Returns ``(None, None, None)`` when no art can be produced.
"""
key = release_id or stable_key(title)
path = None
try:
# IGDB first for the icon: clean box art without the banner.
data = _fetch_igdb_cover(title, min(timeout, 6), system="ps1")
if data is not None:
dest = cover_path(key, "ps1")
_write_atomic(dest, data)
path = dest
log.info("ps1 icon cover %r <- igdb", title)
except Exception: # noqa: BLE001
path = None
if path is None:
try:
path = fetch_cover(title, key, "ps1", timeout=timeout)
except Exception: # noqa: BLE001
return None, None, None
if path is None:
return None, None, None
description = ""
esrb = None
try:
meta = get_game_metadata_igdb(title, timeout=timeout, system="ps1")
if meta:
description = meta.get("overview") or ""
esrb = meta.get("esrb")
except Exception: # noqa: BLE001 - metadata is best effort
pass
try:
return make_ps1_assets(path.read_bytes(), title=title,
description=description, esrb=esrb)
except OSError:
return None, None, None
# --------------------------------------------------------------------------- #
# Public entry point
# --------------------------------------------------------------------------- #
@@ -806,19 +1198,20 @@ def fetch_cover(name, release_id, system, timeout=10, quick=False):
if data is not None:
log.info("cover %r <- manual", name)
elif system != "videos":
# 2. Games: libretro (full box art) first, then the shipped OpenVGDB
# bundle (offline), then ScreenScraper (full box art, non-quick),
# then Wikipedia (keyless, non-quick), then IGDB last — IGDB only
# stores cropped 3:4 key art.
data = _fetch_libretro(name, t, indexed_only=quick)
# 2. Games/PS1: libretro (full box art) first, then the shipped
# OpenVGDB bundle (PSP only, offline), then ScreenScraper (non-quick),
# then Wikipedia (keyless, non-quick), then IGDB last.
data = _fetch_libretro(name, t, system=system, indexed_only=quick)
if data is not None:
log.info("cover %r <- libretro", name)
else:
data = fetch_openvgdb_local(name)
if data is not None:
log.info("cover %r <- openvgdb", name)
elif not quick and time.time() < deadline:
data = _fetch_screenscraper(name, t)
data = None
if system == "games":
data = fetch_openvgdb_local(name)
if data is not None:
log.info("cover %r <- openvgdb", name)
if data is None and not quick and time.time() < deadline:
data = _fetch_screenscraper(name, t, system=system)
if data is not None:
log.info("cover %r <- screenscraper", name)
if data is None and not quick and time.time() < deadline:
@@ -826,11 +1219,12 @@ def fetch_cover(name, release_id, system, timeout=10, quick=False):
if data is not None:
log.info("cover %r <- wikipedia", name)
if data is None and time.time() < deadline:
data = _fetch_igdb_cover(name, t)
data = _fetch_igdb_cover(name, t, system=system)
if data is not None:
# IGDB only has cropped 3:4 key art; composite it into the
# full PSP cover template (header bar + artwork).
data = compose_game_cover(data)
if system == "games":
# IGDB only has cropped 3:4 PSP key art; composite it
# into the full PSP cover template (header + artwork).
data = compose_game_cover(data)
log.info("cover %r <- igdb", name)
if data is None:
@@ -918,6 +1312,46 @@ def _search_wikipedia(query, limit=15, timeout=10):
return results
def _search_google_images(query, limit=15, timeout=10):
"""Return [{title, source, thumb_url, image_url}] from Google Images via the
Custom Search JSON API. Requires google_api_key + google_cx; keyless -> [].
"""
key = sources.load().get("google_api_key")
cx = sources.load().get("google_cx")
if not key or not cx:
return []
url = _GOOGLE_CSE_API + "?" + urllib.parse.urlencode({
"key": key,
"cx": cx,
"q": query,
"searchType": "image",
"num": min(max(limit, 1), 10),
"imgSize": "xlarge",
"safe": "off",
})
data = _http_get(url, timeout=timeout)
if not data:
return []
try:
items = json.loads(data).get("items") or []
except Exception: # noqa: BLE001
return []
out = []
for it in items:
link = it.get("link") or ""
thumb = ((it.get("image") or {}).get("thumbnailLink")) or link
if not link:
continue
out.append({
"id": None,
"title": it.get("title") or query,
"thumb_url": thumb,
"image_url": link,
"source": "google",
})
return out[:limit]
def _fetch_wikipedia_cover(name, timeout):
"""Return a game cover image from Wikipedia's File: namespace (keyless)."""
base = plain_title(name)
@@ -995,8 +1429,8 @@ def _wiki_file_search(query, limit, timeout):
return out
def _search_libretro(query, limit):
names = _load_index().get("Named_Boxarts", [])
def _search_libretro(query, limit, system="games"):
names = _load_index(system=system).get("Named_Boxarts", [])
db = _norm(query)
results = []
if not db:
@@ -1022,21 +1456,60 @@ def _search_libretro(query, limit):
results.append({
"title": n,
"source": "libretro",
"thumb_url": cover_url(n),
"image_url": cover_url(n),
"thumb_url": cover_url(n, system=system),
"image_url": cover_url(n, system=system),
})
return results
def _search_openvgdb(query, limit):
"""Return [{title, source, thumb_url, image_url}] from the offline OpenVGDB
bundle, using the display titles stored in openvgdb-index.json."""
names = _ovgdb_name_list()
if not names:
return []
index = _load_ovgdb_index()
db = _norm(query)
if not db:
return []
scored = []
for key in names:
if key == db:
score = 5
elif len(key) >= 4 and key.startswith(db):
score = 4
elif len(key) >= 4 and db in key:
score = 3
elif len(db) >= 4 and len(key) >= 4 and key in db:
score = 2
else:
continue
info = index.get(key) or {}
title = info.get("title") or key
scored.append((score, title, key))
scored.sort(key=lambda t: (-t[0], t[1]))
return [
{
"id": None,
"title": title,
"source": "openvgdb",
"thumb_url": "",
"image_url": "",
"_key": key,
}
for _score, title, key in scored[:limit]
]
# --------------------------------------------------------------------------- #
# Game metadata (details sidebar)
# --------------------------------------------------------------------------- #
_META_TTL = 86400.0
def get_game_metadata(title, timeout=10):
"""Return game metadata from IGDB."""
return get_game_metadata_igdb(title, timeout)
def get_game_metadata(title, timeout=10, system="games"):
"""Return game metadata from IGDB (system picks the platform)."""
return get_game_metadata_igdb(title, timeout, system)
# --------------------------------------------------------------------------- #
@@ -1125,6 +1598,21 @@ def _igdb_date(ts):
return ""
# IGDB ``age_ratings.rating`` values for ESRB (category 1). 6=RP, 7=EC, 8=E,
# 9=E10+, 10=T, 11=M, 12=AO. The badge uses the short letter form.
_ESRB_BADGE = {6: "RP", 7: "EC", 8: "EVERYONE", 9: "EVERYONE 10+",
10: "TEEN", 11: "MATURE", 12: "ADULTS ONLY"}
def _igdb_esrb(g):
for ar in g.get("age_ratings") or []:
if not isinstance(ar, dict):
continue
if ar.get("category") == 1: # ESRB
return _ESRB_BADGE.get(ar.get("rating"))
return None
def _igdb_game_meta(g):
genres = [x.get("name") for x in (g.get("genres") or []) if isinstance(x, dict) and x.get("name")]
devs, pubs = [], []
@@ -1157,32 +1645,35 @@ def _igdb_game_meta(g):
"genres": list(dict.fromkeys(genres)),
"players": None,
"rating": rating,
"esrb": _igdb_esrb(g),
"youtube": yt,
}
def _igdb_search(query, limit=20, timeout=10):
def _igdb_search(query, limit=20, timeout=10, system="games"):
cid, _ = _igdb_creds()
if not cid:
return []
term = plain_title(query) or query
platform_id = _platform(system)["igdb_id"]
q = (
f'search "{term}"; '
"fields name, cover.image_id, first_release_date, summary, genres.name, "
"involved_companies.company.name, involved_companies.developer, involved_companies.publisher, "
"rating, videos.video_id; "
f"where platforms = ({_IGDB_PSP_ID}); "
"rating, videos.video_id, age_ratings.category, age_ratings.rating; "
f"where platforms = ({platform_id}); "
f"limit {max(limit, 1)};"
)
return _igdb_post(q, timeout) or []
def _fetch_igdb_cover(name, timeout):
def _fetch_igdb_cover(name, timeout, system="games"):
cid, _ = _igdb_creds()
if not cid:
return None
term = plain_title(name) or name
q = f'search "{term}"; fields cover.image_id; where platforms = ({_IGDB_PSP_ID}); limit 10;'
platform_id = _platform(system)["igdb_id"]
q = f'search "{term}"; fields cover.image_id; where platforms = ({platform_id}); limit 10;'
games = _igdb_post(q, timeout) or []
tried = set()
for g in games:
@@ -1196,9 +1687,9 @@ def _fetch_igdb_cover(name, timeout):
return None
def search_games_igdb(query, limit=20, timeout=10):
def search_games_igdb(query, limit=20, timeout=10, system="games"):
out = []
for g in _igdb_search(query, limit, timeout):
for g in _igdb_search(query, limit, timeout, system):
cover = (g.get("cover") or {}).get("image_id") or ""
out.append({
"id": g.get("id"),
@@ -1213,8 +1704,8 @@ def search_games_igdb(query, limit=20, timeout=10):
return out[:limit]
def get_game_metadata_igdb(title, timeout=10):
games = _igdb_search(title, limit=10, timeout=timeout)
def get_game_metadata_igdb(title, timeout=10, system="games"):
games = _igdb_search(title, limit=10, timeout=timeout, system=system)
if not games:
return None
games.sort(key=lambda g: -_title_score(title, g.get("name") or ""))
@@ -1228,7 +1719,7 @@ def game_metadata_by_id_igdb(game_id, timeout=10):
q = (
"fields name, cover.image_id, first_release_date, summary, genres.name, "
"involved_companies.company.name, involved_companies.developer, involved_companies.publisher, "
"rating, videos.video_id; "
"rating, videos.video_id, age_ratings.category, age_ratings.rating; "
f"where id = {int(game_id)}; limit 1;"
)
games = _igdb_post(q, timeout) or []
@@ -1334,11 +1825,19 @@ def get_movie_metadata(title, timeout=10):
# Manual linking (pick the correct movie/game when the auto-match is wrong)
# --------------------------------------------------------------------------- #
def search_movies(query, limit=20, timeout=10):
"""Return [{id, title, year, overview, thumb_url, source}] for a movie search.
TheMovieDB first; keyless Wikipedia page titles as a fallback.
"""Return movie candidates for a UMD video: TMDB first, then Wikipedia
images and Google Images (both keyless-optional). A Wikipedia page-title
fallback is added only when nothing else matched (metadata-only linking).
"""
out = []
seen = set()
def add(r):
k = (r.get("source"), _norm(r.get("title") or ""))
if k[1] and k not in seen:
seen.add(k)
out.append(r)
key = sources.load().get("tmdb_api_key")
if key:
url = _TMDB_SEARCH + "?" + urllib.parse.urlencode({"api_key": key, "query": query})
@@ -1348,16 +1847,31 @@ def search_movies(query, limit=20, timeout=10):
results = json.loads(data).get("results") or []
except Exception:
results = []
for r in results:
# Cap TMDB results so Wikipedia/Google image picks still fit.
for r in results[: max(1, min(limit, 6))]:
poster = r.get("poster_path")
out.append({
add({
"id": r.get("id"),
"title": r.get("title") or r.get("name") or "",
"year": (r.get("release_date") or "")[:4],
"overview": (r.get("overview") or "").strip(),
"thumb_url": (_TMDB_IMAGE + "w185" + poster) if poster else "",
"image_url": (_TMDB_IMAGE + "w500" + poster) if poster else "",
"source": "tmdb",
})
# Wikipedia image search (keyless): real poster/cover files.
for r in _search_wikipedia(query, min(limit, 10), timeout):
r["id"] = None
r["movie_title"] = query
add(r)
# Google Images (Custom Search JSON API, when a key + cx are configured).
for r in _search_google_images(query, min(limit, 10), timeout):
r["movie_title"] = query
add(r)
# Last resort: Wikipedia page titles (metadata-only, no image).
if not out:
for page in _wiki_search(query, limit):
out.append({
@@ -1366,6 +1880,8 @@ def search_movies(query, limit=20, timeout=10):
"year": "",
"overview": "",
"thumb_url": "",
"image_url": "",
"movie_title": page,
"source": "wikipedia",
})
return out[:limit]
@@ -1411,40 +1927,71 @@ def movie_metadata_from_wikipedia(page_title, timeout=10):
def search_metadata_candidates(system, query, limit=20, timeout=10):
"""Search candidates the user can link an item to."""
"""Search candidates the user can link an item to.
Games/PS1: IGDB (rich metadata) first, then libretro box art (and the
PSP-only OpenVGDB bundle) so covers can be scraped even without an IGDB key.
"""
if system == "videos":
return search_movies(query, limit, timeout)
results = search_games_igdb(query, limit, timeout)
if results:
return results
# No IGDB results (or no key): fall back to libretro catalog names so the
# user can still link to the correct entry (title-only).
results = _search_libretro(query, limit)
for r in results:
r["source"] = "libretro"
r["id"] = None
return results
results = []
seen = set()
def add(r):
key = (r.get("source"), _norm(r.get("title") or ""))
if key[1] and key not in seen:
seen.add(key)
results.append(r)
# Cap IGDB so libretro/OpenVGDB cover picks still fit in the list.
for r in search_games_igdb(query, max(1, min(limit, 6)), timeout, system):
add(r)
for r in _search_libretro(query, limit, system):
r.setdefault("id", None)
add(r)
if system == "games":
for r in _search_openvgdb(query, limit):
add(r)
return results[:limit]
def _empty_meta(title=""):
return {
"title": title,
"release_date": "",
"overview": "",
"genres": [],
"rating": None,
"youtube": "",
}
def link_metadata(system, candidate, timeout=10):
"""Fetch full metadata for a search candidate the user picked."""
if candidate.get("source") == "wikipedia":
return movie_metadata_from_wikipedia(candidate.get("title") or "", timeout)
if candidate.get("source") == "libretro":
# Catalog-name link only: no metadata backend, just the correct title.
return {
"title": candidate.get("title") or "",
"release_date": "",
"overview": "",
"genres": [],
"rating": None,
"youtube": "",
}
src = candidate.get("source")
if src == "wikipedia":
title = candidate.get("movie_title") or candidate.get("title") or ""
meta = movie_metadata_from_wikipedia(title, timeout) if title else None
if meta:
return meta
if system == "videos":
return get_movie_metadata(title, timeout) or _empty_meta(title)
return _empty_meta(title)
if src in ("libretro", "openvgdb"):
# Cover repositories have no metadata backend; try IGDB by title so the
# user still gets synopsis/genre/etc. when a key is configured.
title = candidate.get("title") or ""
if title:
meta = get_game_metadata_igdb(title, timeout, system)
if meta:
return meta
return _empty_meta(title)
if system == "videos":
if candidate.get("id") is not None:
if src == "tmdb" and candidate.get("id") is not None:
return movie_metadata_by_id(candidate["id"], timeout)
return movie_metadata_from_wikipedia(candidate.get("title") or "", timeout)
if candidate.get("id") is not None and candidate.get("source") == "igdb":
title = candidate.get("movie_title") or candidate.get("title") or ""
return get_movie_metadata(title, timeout) or _empty_meta(title)
if src == "igdb" and candidate.get("id") is not None:
return game_metadata_by_id_igdb(candidate["id"], timeout)
return None
return _empty_meta(candidate.get("title") or "")
+72 -16
View File
@@ -1,23 +1,29 @@
"""Parser for clrmamepro (.dat) files.
"""Parser for clrmamepro / Logiqx XML (.dat) files.
Handles the No-Intro / Redump text DAT format::
Supports both DAT layouts the project uses:
clrmamepro (
name "Sony - PlayStation Portable"
version "2021.10.24"
)
game (
name "WipEout Pure (Europe)"
region "Europe"
serial "ULES-00011"
rom ( name "WipEout Pure (Europe).iso" size 123 crc ABCD1234 md5 ... sha1 ... )
)
* clrmamepro text DAT (No-Intro): parenthesised blocks, e.g.::
``parse_dat_text`` returns ``(header, entries)``. Each entry is a dict with
scalar keys (``name``, ``description``, ``region``, ``serial``, ...) and
repeated nested blocks exposed as lists under their block key (``rom``).
clrmamepro (
name "Sony - PlayStation Portable"
version "2021.10.24"
)
game (
name "WipEout Pure (Europe)"
region "Europe"
serial "ULES-00011"
rom ( name "WipEout Pure (Europe).iso" size 123 crc ABCD1234 md5 ... sha1 ... )
)
* Logiqx XML DAT (Redump): ``<game>`` elements with ``<rom>`` children.
Both ``parse_dat_text`` and ``parse_dat_xml`` return ``(header, entries)``.
Each entry is a dict with scalar keys (``name``, ``description``, ``region``,
``serial``, ...) and repeated nested blocks exposed as lists under their block
key (``rom``).
"""
import re
import xml.etree.ElementTree as ET
from pathlib import Path
_TOKEN_RE = re.compile(
@@ -124,5 +130,55 @@ def parse_dat_text(text):
return header, entries
def _xml_text(element, key):
child = element.find(key)
if child is not None and child.text:
return child.text.strip()
return None
def parse_dat_xml(text):
"""Parse a Logiqx XML DAT (the format redump.org uses).
Returns ``(header, entries)`` in the same shape as :func:`parse_dat_text`.
"""
root = ET.fromstring(text)
header = {}
header_el = root.find("header")
if header_el is not None:
for child in header_el:
if child.text:
header[child.tag] = child.text.strip()
entries = []
for game in root.iter():
if game.tag not in ("game", "machine"):
continue
entry = {"name": (game.get("name") or "").strip()}
for key in ("description", "serial", "region", "category"):
value = _xml_text(game, key)
if value:
entry[key] = value
entry["rom"] = [
{
"name": rom.get("name", ""),
"size": rom.get("size"),
"crc": rom.get("crc"),
"md5": rom.get("md5"),
"sha1": rom.get("sha1"),
"serial": rom.get("serial"),
}
for rom in game.findall("rom")
]
entries.append(entry)
return header, entries
def parse_dat_file(path):
return parse_dat_text(Path(path).read_text(encoding="utf-8-sig"))
text = Path(path).read_text(encoding="utf-8-sig")
stripped = text.lstrip()
if stripped.startswith("<?xml") or stripped.startswith("<"):
return parse_dat_xml(text)
return parse_dat_text(text)
+14
View File
@@ -101,6 +101,20 @@ CREATE TABLE IF NOT EXISTS save_sigs (
sig TEXT NOT NULL,
PRIMARY KEY (save_id, size, mtime_ns)
);
CREATE TABLE IF NOT EXISTS ps1_entries (
id INTEGER PRIMARY KEY,
source_path TEXT NOT NULL UNIQUE,
title TEXT,
serial TEXT,
region TEXT,
disc INTEGER NOT NULL DEFAULT 1,
disc_total INTEGER NOT NULL DEFAULT 1,
output_eboot TEXT,
status TEXT NOT NULL DEFAULT 'pending',
error TEXT,
updated_at TEXT NOT NULL
);
"""
+1618 -237
View File
File diff suppressed because it is too large Load Diff
+30 -16
View File
@@ -23,6 +23,9 @@ _miss_db = {}
_miss_loaded = False
_MISS_TTL = 86400.0 # retry failed metadata after 1 day
# Memoised release-date table (invalidated on save/reset).
_dates_cache = None
def _load_miss_db():
global _miss_loaded
@@ -122,6 +125,7 @@ def save(release_id, system, meta, manual=False, title=None):
)
conn.commit()
conn.close()
_invalidate_dates_cache()
return True
except sqlite3.Error:
return False
@@ -140,6 +144,7 @@ def reset(release_id, system):
)
conn.commit()
conn.close()
_invalidate_dates_cache()
except sqlite3.Error:
pass
@@ -161,23 +166,32 @@ def cached_release_dates():
"""Return {(release_id, system): release_date} from cached metadata only.
Pure DB read — no network — so list views can show a release date without
triggering API fetches for every row.
triggering API fetches for every row. Result is memoised and invalidated
whenever metadata is saved.
"""
out = {}
try:
conn = get_connection(config.DB_PATH)
rows = conn.execute("SELECT release_id, system, meta_json FROM item_info").fetchall()
conn.close()
except sqlite3.Error:
return {}
for r in rows:
global _dates_cache
if _dates_cache is None:
out = {}
try:
meta = json.loads(r["meta_json"] or "{}")
except (TypeError, ValueError):
continue
if isinstance(meta, dict) and meta.get("release_date"):
out[(r["release_id"], r["system"])] = meta["release_date"]
return out
conn = get_connection(config.DB_PATH)
rows = conn.execute("SELECT release_id, system, meta_json FROM item_info").fetchall()
conn.close()
except sqlite3.Error:
return out
for r in rows:
try:
meta = json.loads(r["meta_json"] or "{}")
except (TypeError, ValueError):
continue
if isinstance(meta, dict) and meta.get("release_date"):
out[(r["release_id"], r["system"])] = meta["release_date"]
_dates_cache = out
return _dates_cache
def _invalidate_dates_cache():
global _dates_cache
_dates_cache = None
def get_or_fetch(release_id, system, fetch_title):
@@ -197,7 +211,7 @@ def get_or_fetch(release_id, system, fetch_title):
if system == "videos":
meta = covers.get_movie_metadata(fetch_title)
else:
meta = covers.get_game_metadata(fetch_title)
meta = covers.get_game_metadata(fetch_title, system=system)
except Exception: # noqa: BLE001 - network/API failures must not crash the UI
meta = None
if meta:
+45
View File
@@ -24,6 +24,9 @@ def detect_system(dat_name):
return "videos"
if "PlayStation Portable" in dat_name:
return "games"
# Redump PS1: "Sony - PlayStation" (assumes no PS2/PS3 DATs are present).
if "PlayStation" in dat_name:
return "ps1"
return "unknown"
@@ -37,6 +40,12 @@ def ingest_dat(conn, path):
existing = conn.execute("SELECT id FROM dats WHERE name = ?", (dat_name,)).fetchone()
if existing:
dat_id = existing["id"]
# Files that resolved to a release in this DAT must be re-identified
# against the new catalog (the release ids are about to be recycled).
conn.execute(
"UPDATE files SET release_id = NULL WHERE release_id IN (SELECT id FROM releases WHERE dat_id = ?)",
(dat_id,),
)
conn.execute("DELETE FROM roms WHERE release_id IN (SELECT id FROM releases WHERE dat_id = ?)", (dat_id,))
conn.execute("DELETE FROM releases WHERE dat_id = ?", (dat_id,))
conn.execute(
@@ -79,6 +88,42 @@ def ingest_dat(conn, path):
return dat_name, system, release_count, rom_count
def ensure_catalog(dats_dir=None, conn=None):
"""Ingest any DAT files missing from the DB (idempotent, cheap when current).
Called once at startup so a fresh/reset database still has the catalog and
scanning produces matches instead of everything coming back ``unknown``.
Returns the number of DATs ingested.
"""
dats_dir = Path(dats_dir) if dats_dir else config.DATS_DIR
if not dats_dir.is_dir():
return 0
files = sorted(p for p in dats_dir.glob("*.dat") if p.is_file())
if not files:
return 0
own = conn is None
if own:
conn = get_connection(config.DB_PATH)
try:
known = {r["source_path"] for r in conn.execute("SELECT source_path FROM dats")}
count = 0
for path in files:
if str(path) in known:
continue
try:
ingest_dat(conn, path)
except Exception as exc: # noqa: BLE001 - a bad DAT shouldn't block startup
print(f"warning: could not ingest {path.name}: {exc}", file=sys.stderr)
continue
count += 1
conn.commit()
return count
finally:
if own:
conn.close()
def main(argv=None):
parser = argparse.ArgumentParser(description="Ingest clrmamepro DAT files into SQLite")
parser.add_argument("--dats", help="directory containing .dat files (default: data/dats)")
+4 -2
View File
@@ -48,8 +48,10 @@ def launch_ppsspp(iso, exe=""):
else:
return False, "PPSSPP not found. Install it or set its path in Settings → Saves."
try:
subprocess.Popen(cmd, start_new_session=True, env=_env(),
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
kwargs = {"env": _env(), "stdout": subprocess.DEVNULL, "stderr": subprocess.DEVNULL}
if os.name != "nt":
kwargs["start_new_session"] = True # POSIX-only: detach from our process group
subprocess.Popen(cmd, **kwargs)
return True, ""
except OSError as exc:
return False, str(exc)
+72 -11
View File
@@ -15,8 +15,8 @@ _ARTICLE_RE = re.compile(r"^(.+),\s*(the|a|an)(?:\s+-\s+(.+))?$")
def _normalize(name):
name = name.lower()
name = re.sub(r"\.(iso|cso|pbp|img|bin)$", "", name) # strip disc extension
name = re.sub(r"^\s*\d+\s*-\s*", "", name) # leading No-Intro catalog number
name = re.sub(r"\.(iso|cso|pbp|img|bin|cue)$", "", name) # strip disc extension
name = re.sub(r"^\s*\d+\s*-\s*", "", name) # legacy No-Intro "#### - Name" catalog number
name = re.sub(r"\([^)]*\)", " ", name) # parenthetical tags
name = " ".join(name.split())
m = _ARTICLE_RE.match(name)
@@ -38,7 +38,11 @@ def clear_cache():
def _all_release_candidates(conn):
"""Catalog releases + supplement entries with pre-normalized names."""
"""Catalog releases + supplement entries with pre-normalized names.
Each entry is ``(normalized_name, significant_word_set, candidate)`` so the
fuzzy step can cheaply skip candidates that share no significant word.
"""
key = id(conn)
cands = _releases_cache.get(key)
if cands is not None:
@@ -53,7 +57,11 @@ def _all_release_candidates(conn):
)
]
cands.extend(_supplement_release(e) for e in _load_supplement())
cands = [(_normalize(c["name"]), c) for c in cands]
cands = [
(norm, frozenset(w for w in norm.split() if len(w) >= 3), c)
for c in cands
if (norm := _normalize(c["name"] or ""))
]
_releases_cache[key] = cands
return cands
@@ -103,6 +111,36 @@ def normalize_name(name):
return _normalize(name)
_EUROPEAN_REGIONS = {
"europe", "germany", "france", "spain", "italy", "russia", "netherlands",
"portugal", "poland", "united kingdom", "greece", "norway", "sweden",
"denmark", "finland", "austria", "belgium", "switzerland",
}
def region_matches(region, prefer):
"""True if ``region`` satisfies the user's preferred-region hint."""
if not prefer or not region:
return False
tokens = {t.strip().lower() for t in str(region).split(",") if t.strip()}
if not tokens:
return False
prefer = prefer.strip().lower()
if prefer in tokens:
return True
if prefer == "europe":
return bool(tokens & _EUROPEAN_REGIONS)
return False
def apply_region_preference(candidates, prefer):
"""Narrow ``candidates`` to the preferred region (fall back to all)."""
if not prefer:
return candidates
filtered = [c for c in candidates if region_matches(c.get("region"), prefer)]
return filtered or candidates
def serial_candidates(conn, record):
"""Return all catalog releases matching the disc's serial (size-narrowed)."""
serial = record.get("serial")
@@ -123,7 +161,7 @@ def serial_candidates(conn, record):
return _narrow(conn, [dict(r) for r in rows], record.get("size"))
def match(conn, record):
def match(conn, record, prefer_region=None):
size = record.get("size")
# 1. Hash is the most precise key; serials are shared across revisions.
@@ -141,7 +179,7 @@ def match(conn, record):
if row is not None:
return f"hash:{field}", dict(row)
# 2. Local supplement catalog (dumps not present in No-Intro).
# 2. Local supplement catalog (dumps not present in the DATs).
for entry in _load_supplement():
for field in ("crc32", "sha1", "md5"):
value = entry.get(field)
@@ -163,24 +201,47 @@ def match(conn, record):
for cand in candidates:
if cand["id"] == saved["release_id"]:
return "choice", cand
if prefer_region:
filtered = [c for c in candidates if region_matches(c.get("region"), prefer_region)]
if len(filtered) == 1:
return "region", filtered[0]
return "unknown", None
# 4. Fuzzy fallback (filename + PARAM.SFO title), size-disambiguated.
disk_norm = _normalize(record["name"])
title_norm = _normalize(record.get("title") or "")
# Cheap pre-filter: only run the expensive difflib comparison against
# candidates sharing at least one significant (>=3 char) word with the
# disc's filename or PARAM.SFO title.
target_words = frozenset(w for w in disk_norm.split() if len(w) >= 3) | \
frozenset(w for w in title_norm.split() if len(w) >= 3)
if not target_words:
return "unknown", None
scored = []
for cand_norm, cand in _all_release_candidates(conn):
ratio = max(
SequenceMatcher(None, disk_norm, cand_norm).ratio(),
SequenceMatcher(None, title_norm, cand_norm).ratio(),
)
for cand_norm, cand_words, cand in _all_release_candidates(conn):
if not (target_words & cand_words):
continue
ratio = SequenceMatcher(None, disk_norm, cand_norm).ratio()
if title_norm:
ratio = max(ratio, SequenceMatcher(None, title_norm, cand_norm).ratio())
scored.append((ratio, cand))
best_ratio = max((r for r, _ in scored), default=0.0)
if best_ratio < 0.9:
return "unknown", None
top = [cand for ratio, cand in scored if ratio >= best_ratio - 0.001]
top = _narrow(conn, top, size)
# The same disc can exist in multiple catalogs (e.g. No-Intro + Redump);
# collapse identical entries so they don't read as an ambiguous match.
seen = set()
deduped = []
for cand in top:
key = (cand.get("name"), size)
if key not in seen:
seen.add(key)
deduped.append(cand)
top = deduped
if len(top) == 1:
return f"fuzzy:{best_ratio:.2f}", top[0]
return "unknown", None
+2 -2
View File
@@ -62,7 +62,7 @@ def _reorder_article(title):
return title
def canonical_name(release, serial):
def canonical_name(release, serial, ext=".iso"):
name = release["name"]
region = release.get("region")
if region:
@@ -78,4 +78,4 @@ def canonical_name(release, serial):
parts.append(f"({serial})")
name = _ILLEGAL.sub(" ", " ".join(parts)).strip()
name = re.sub(r"\s+", " ", name)
return f"{name}.iso"
return f"{name}{ext}"
+210
View File
@@ -0,0 +1,210 @@
"""NoPayStation (.pkg) browsing and extraction.
NoPayStation publishes per-platform TSV lists (``https://nopaystation.com/tsv/``)
with one row per downloadable PSN item. Each row links to an encrypted PSN
``.pkg`` (a ``\\x7FPKG`` container, not a ZIP) that is decrypted by the bundled
pure-Python decryptor (``app/pkg.py``) into a ``PSP/GAME/<TITLE_ID>/``-style
folder — no Wine or ``pkg2zip`` required.
"""
from __future__ import annotations
import logging
import shutil
import urllib.parse
import urllib.request
from datetime import datetime
from pathlib import Path
from typing import List
from . import config
log = logging.getLogger("nopaystation")
_TSV_NAMES = {
"psp": "PSP_GAMES.tsv",
"psx": "PSX_GAMES.tsv",
"theme": "PSP_THEMES.tsv",
"dlc": "PSP_DLCS.tsv",
}
# NoPayStation TSVs are tab-separated with a header row whose exact columns
# differ per platform (PSP has "Type"/"RAP" columns, PSX has "Original Name").
# Map header names to canonical keys instead of relying on fixed positions.
_HEADER_ALIASES = {
"title id": "title_id",
"region": "region",
"name": "name",
"pkg direct link": "pkg_url",
"content id": "content_id",
"last modification date": "date",
"file size": "size",
"sha256": "sha256",
"zrif": "zrif",
"rap": "rap",
"download .rap file": "rap_url",
"type": "content_kind",
"original name": "original_name",
}
def tsv_url(content_type: str, base: str = "") -> str:
"""TSV URL for a content type (``psp`` / ``psx`` / ``theme`` / ``dlc``)."""
base = (base or config.NPS_TSV_BASE).rstrip("/")
return f"{base}/{_TSV_NAMES.get(content_type, _TSV_NAMES['psp'])}"
def _tsv_dir() -> Path:
d = config.USER_DATA_DIR / "nps"
d.mkdir(parents=True, exist_ok=True)
return d
def _parse_date(value: str) -> str:
value = (value or "").strip()
if not value:
return ""
for fmt in ("%Y-%m-%d %H:%M:%S", "%Y-%m-%d"):
try:
return datetime.strptime(value, fmt).strftime("%Y-%m-%d")
except ValueError:
continue
return value[:10]
def parse_tsv(text: str, content_type: str) -> List[dict]:
"""Parse a NoPayStation TSV into store item dicts.
The header row drives column mapping (PSP and PSX TSVs differ). Each item
carries the fields the Store page expects (``name``, ``size``,
``release_date``, ``mtime_ns``) plus ``title_id``, ``pkg_url``, ``sha256``,
``content_type`` and the cover ``system`` ("games" for PSP, "ps1" for PSX).
"""
system = {"psp": "games", "psx": "ps1", "theme": "theme", "dlc": "dlc"}.get(content_type, "games")
items: List[dict] = []
header = None
for line in text.splitlines():
line = line.rstrip("\r\n")
if not line.strip():
continue
cols = line.split("\t")
if header is None:
header = [c.strip().lower() for c in cols]
continue
row = {}
for i, key in enumerate(header):
canon = _HEADER_ALIASES.get(key)
if canon and i < len(cols):
row[canon] = cols[i].strip()
name = row.get("name") or ""
pkg_url = row.get("pkg_url") or ""
# NoPayStation marks delisted packages with the literal "MISSING" in the
# URL column — skip them so they can't be queued for download.
if not name or not pkg_url or not pkg_url.lower().startswith(("http://", "https://")):
continue
title_id = row.get("title_id") or ""
try:
size = int(float(row.get("size") or 0))
except (TypeError, ValueError):
size = 0
date = _parse_date(row.get("date") or "")
try:
mtime_ns = int(datetime.strptime(date, "%Y-%m-%d").timestamp() * 1_000_000_000) if date else 0
except ValueError:
mtime_ns = 0
items.append({
"name": name,
"title_id": title_id,
"region": row.get("region") or "",
"pkg_url": pkg_url,
"zrif": row.get("zrif") or "",
"content_id": row.get("content_id") or "",
"size": size,
"sha256": (row.get("sha256") or "").strip().lower(),
"release_date": date,
"mtime_ns": mtime_ns,
"content_type": content_type,
"system": system,
"content_kind": row.get("content_kind") or "",
})
return items
def fetch_items(content_type: str, url: str = "", timeout: int = 60, force: bool = False) -> List[dict]:
"""Download (and cache) a TSV, returning parsed item dicts.
``force`` re-downloads the TSV (used by the Store's Refresh button); the
local cache is only used as a fallback when the network fails.
"""
url = url or tsv_url(content_type)
name = _TSV_NAMES.get(content_type, "PSP_GAMES.tsv")
cache = _tsv_dir() / name
def read_cache():
try:
return parse_tsv(cache.read_text(encoding="utf-8", errors="replace"), content_type)
except OSError:
return None
if not force and cache.is_file():
items = read_cache()
if items is not None:
return items
try:
req = urllib.request.Request(url, headers={"User-Agent": "psp-preserve/0.1"})
with urllib.request.urlopen(req, timeout=timeout) as resp:
data = resp.read()
text = data.decode("utf-8", errors="replace")
cache.write_text(text, encoding="utf-8")
return parse_tsv(text, content_type)
except Exception: # noqa: BLE001 - fall back to the cached list on failure
items = read_cache()
if items is not None:
log.warning("could not refresh %s TSV; using cached copy", content_type)
return items
raise
def _safe_folder(value: str) -> str:
import re
cleaned = re.sub(r"[^A-Za-z0-9]", "", value or "").upper()
return cleaned or "PKG"
def _ensure_keys_bin(folder, kind):
"""PS1 Classics need KEYS.BIN next to EBOOT.PBP to boot on CFW (npdrm)."""
if kind != "psx":
return
src = config.DATA_DIR / "loader" / "KEYS.BIN"
dest = Path(folder) / "KEYS.BIN"
if src.is_file() and not dest.exists():
try:
shutil.copy2(src, dest)
except OSError:
pass
def extract_pkg(pkg_path, title_id: str, dest_dir,
kind_hint: str = None, progress_cb=None, cancel_event=None) -> Path:
"""Decrypt/extract a ``.pkg`` into ``dest_dir/<TITLE_ID>/``.
Uses the bundled pure-Python decryptor (no Wine or pkg2zip). Returns the
destination game folder path.
"""
from . import pkg as pkgmod
from .store import TransferCancelled
pkg_path = Path(pkg_path)
dest = Path(dest_dir) / _safe_folder(title_id)
if dest.is_dir() and any(dest.iterdir()):
return dest
try:
_kind, _tid, folder = pkgmod.extract(pkg_path, dest_dir, kind_hint,
progress_cb, cancel_event)
_ensure_keys_bin(folder, _kind)
return folder
except TransferCancelled:
raise
except Exception as exc: # noqa: BLE001
raise RuntimeError(f"could not decrypt {pkg_path.name}: {exc}") from exc
+15
View File
@@ -17,6 +17,7 @@ tags removed). The GUI reads this bundle with no network access.
"""
import argparse
import concurrent.futures
import json
import logging
import os
import re
@@ -94,6 +95,19 @@ def build_index(sqlite_path):
return best
def write_index(index, path=None):
"""Persist {key: {"title", "url"}} so the GUI can show readable titles when
searching the offline bundle (see covers._search_openvgdb)."""
path = Path(path) if path else (config.COVERS_DIR / "openvgdb-index.json")
data = {k: {"title": t, "url": u} for k, (t, u, _r) in index.items()}
try:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
except OSError as exc: # noqa: BLE001
log.warning("could not write index %s: %s", path, exc)
return path
def _existing_keys():
"""Titles already present in the bundle (for resume)."""
if not BUNDLE.exists():
@@ -152,6 +166,7 @@ def scrape(limit=None, workers=8, rate=3.0):
index = build_index(sqlite_path)
log.info("indexed %d unique PSP covers", len(index))
write_index(index)
existing = _existing_keys()
todo = [(k, v) for k, v in sorted(index.items()) if k not in existing]
+2 -2
View File
@@ -2,9 +2,9 @@
from . import iso_reader, matcher, namer
def identify(conn, record):
def identify(conn, record, prefer_region=None):
"""Return ``{method, release, serial, canonical}`` for a scanned disc record."""
method, release = matcher.match(conn, record)
method, release = matcher.match(conn, record, prefer_region=prefer_region)
return identify_with_release(record, release, method)
+378
View File
@@ -0,0 +1,378 @@
"""Pure-Python PSP/PSX/theme/DLC PSN ``.pkg`` decryption (no Wine / pkg2zip).
Ports the folder-mode extraction used by pkgi-psp's depackager and pkg2zip:
parse the ``\\x7FPKG`` header, derive the PSP/PS3 AES key, AES-CTR-decrypt the
file entries, and write the game's ``EBOOT.PBP`` (+ siblings) into a folder.
Supported content:
* PSP games / minis (``EBOOT.PBP``) — ``DATA.PSP`` stays EDAT-encrypted, so it
runs on CFW with ``npdrm_free`` (ARK ships it), same as pkgi-psp's install.
* PS1 Classics (``EBOOT.PBP`` + ``DOCUMENT.DAT``).
* PSP themes (``.ptf``) — the packaged PTF is EDAT-wrapped and is unwrapped here.
* PSP DLC (``.edat``) — extracted as-is into the owning game's folder.
Theme unwrapping needs ``pycryptodome`` (for AES-CMAC + ECB); everything else
works with either pycryptodome or the built-in pure-Python AES fallback.
"""
from __future__ import annotations
from pathlib import Path
PKG_MAGIC = 0x7F504B47
# Static PSN keys (key type 1 = PSP).
PSP_KEY = bytes.fromhex("07f2c68290b50d2c33818d709b60e62b")
PS3_KEY = bytes.fromhex("2e7b71d7c9c9a14ea3221f188828b8f8")
# PSP KIRK/AMCTL keys used to unwrap theme PTF EDATs (pkgi-psp / libkirk).
_KIRK38 = bytes.fromhex("12468d7e1c42209bba5426835eb03303")
_KIRK39 = bytes.fromhex("c43bb6d653ee67493ea95fbc0ced6f8a")
_KIRK63 = bytes.fromhex("9c9b1372f8c640cf1c62f5d592ddb582")
_AMCTL3 = bytes.fromhex("e350ed1d910a1fd029bb1c3ef34077fb")
_AMCTL4 = bytes.fromhex("135fa47cab395ba476b8cca98f3a0445")
_AMCTL5 = bytes.fromhex("678d7fa32a9ca0d1508ad8385e4b017e")
try:
from Crypto.Cipher import AES as _AES # pycryptodome (fast)
_HAVE_CRYPTO = True
except ImportError: # noqa: BLE001
_HAVE_CRYPTO = False
# --------------------------------------------------------------------------- #
# Pure-Python AES-128 (fallback when pycryptodome is unavailable)
# --------------------------------------------------------------------------- #
_SBOX = bytes.fromhex(
"637c777bf26b6fc53001672bfed7ab76ca82c97dfa5947f0add4a2af9ca472c0"
"b7fd9326363ff7cc34a5e5f171d8311504c723c31896059a071280e2eb27b275"
"09832c1a1b6e5aa0523bd6b329e32f8453d100ed20fcb15b6acbbe394a4c58cf"
"d0efaafb434d338545f9027f503c9fa851a3408f929d38f5bcb6da2110fff3d2"
"cd0c13ec5f974417c4a77e3d645d197360814fdc222a908846eeb814de5e0bdb"
"e0323a0a4906245cc2d3ac629195e479e7c8376d8dd54ea96c56f4ea657aae08"
"ba78252e1ca6b4c6e8dd741f4bbd8b8a703eb5664803f60e613557b986c11d9e"
"e1f8981169d98e949b1e87e9ce5528df8ca1890dbfe6426841992d0fb054bb16"
)
_RCON = (0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1B, 0x36)
def _key_schedule(key):
w = list(key)
for i in range(4, 44):
t = list(w[(i - 1) * 4:(i - 1) * 4 + 4])
if i % 4 == 0:
t = t[1:] + t[:1]
t = [_SBOX[b] for b in t]
t[0] ^= _RCON[i // 4 - 1]
w += [w[(i - 4) * 4 + j] ^ t[j] for j in range(4)]
return [bytes(w[i * 16:(i + 1) * 16]) for i in range(11)]
def _shift_rows(s):
return [s[0], s[5], s[10], s[15], s[4], s[9], s[14], s[3],
s[8], s[13], s[2], s[7], s[12], s[1], s[6], s[11]]
def _xtime(a):
a <<= 1
return (a ^ 0x1B) & 0xFF if a & 0x100 else a & 0xFF
def _mix_columns(s):
out = [0] * 16
for c in range(4):
a0, a1, a2, a3 = s[4 * c], s[4 * c + 1], s[4 * c + 2], s[4 * c + 3]
out[4 * c] = _xtime(a0) ^ (_xtime(a1) ^ a1) ^ a2 ^ a3
out[4 * c + 1] = a0 ^ _xtime(a1) ^ (_xtime(a2) ^ a2) ^ a3
out[4 * c + 2] = a0 ^ a1 ^ _xtime(a2) ^ (_xtime(a3) ^ a3)
out[4 * c + 3] = (_xtime(a0) ^ a0) ^ a1 ^ a2 ^ _xtime(a3)
return out
def _aes_encrypt_block(rk, block):
s = [block[i] ^ rk[0][i] for i in range(16)]
for rnd in range(1, 10):
s = [_SBOX[b] for b in s]
s = _shift_rows(s)
s = _mix_columns(s)
s = [s[i] ^ rk[rnd][i] for i in range(16)]
s = [_SBOX[b] for b in s]
s = _shift_rows(s)
s = [s[i] ^ rk[10][i] for i in range(16)]
return bytes(s)
def _add128(counter, n):
c = bytearray(counter)
for i in range(15, -1, -1):
n += c[i]
c[i] = n & 0xFF
n >>= 8
return bytes(c)
def _ctr_decrypt_py(key, iv, block_index, data):
"""AES-CTR decrypt (pure-Python): counter = iv + block_index, +1 per block."""
rk = _key_schedule(key)
counter = _add128(iv, block_index)
out = bytearray()
for i in range(0, len(data), 16):
ks = _aes_encrypt_block(rk, counter)
out += bytes(b ^ k for b, k in zip(data[i:i + 16], ks))
counter = _add128(counter, 1)
return bytes(out)
def _ctr_decrypt(key, iv, block_index, data):
"""AES-CTR decrypt: 128-bit big-endian counter at ``iv + block_index``."""
if _HAVE_CRYPTO:
nonce = iv[:8]
init = int.from_bytes(iv[8:16], "big") + block_index
return _AES.new(key, _AES.MODE_CTR, nonce=nonce, initial_value=init).decrypt(data)
return _ctr_decrypt_py(key, iv, block_index, data)
def _make_ctr_decryptor(key, iv, block_index):
"""Return a streaming decryptor (callable: bytes -> bytes) for a data run."""
if _HAVE_CRYPTO:
nonce = iv[:8]
init = int.from_bytes(iv[8:16], "big") + block_index
cipher = _AES.new(key, _AES.MODE_CTR, nonce=nonce, initial_value=init)
return cipher.decrypt
rk = _key_schedule(key)
counter = _add128(iv, block_index)
state = {"rk": rk, "counter": counter}
def decrypt(chunk):
out = bytearray()
for i in range(0, len(chunk), 16):
ks = _aes_encrypt_block(state["rk"], state["counter"])
out += bytes(b ^ k for b, k in zip(chunk[i:i + 16], ks))
state["counter"] = _add128(state["counter"], 1)
return bytes(out)
return decrypt
# --------------------------------------------------------------------------- #
# PSP theme PTF EDAT unwrapping (pkgi-psp's unpack_psp_edat)
# --------------------------------------------------------------------------- #
def _need_crypto():
if not _HAVE_CRYPTO:
raise RuntimeError("theme extraction needs pycryptodome (pip install pycryptodome)")
def _ecb_decrypt(key, block):
_need_crypto()
return _AES.new(key, _AES.MODE_ECB).decrypt(block)
def _cmac(key, data):
_need_crypto()
from Crypto.Hash import CMAC
c = CMAC.new(key, ciphermod=_AES)
c.update(data)
return c.digest()
def _kirk_iv(mac, key_header, offset2):
tmp = _ecb_decrypt(_KIRK38, bytes(key_header[0x70:0x80]))
iv = bytes(mac[i] ^ tmp[i] ^ key_header[offset2 + i] ^ _AMCTL3[i] ^ _AMCTL5[i] for i in range(16))
iv = _ecb_decrypt(_KIRK39, iv)
return bytes(b ^ _AMCTL4[i] for i, b in enumerate(iv))
def _kirk_cbc_decrypt(iv, index, data):
prev = bytearray(16) if index == 0 else bytearray(iv[:12]) + index.to_bytes(4, "little")
block = bytearray(iv[:12]) + index.to_bytes(4, "little")
out = bytearray()
for i in range(0, len(data), 16):
v = int.from_bytes(block[12:16], "little") + 1
block[12:16] = v.to_bytes(4, "little")
o = _ecb_decrypt(_KIRK63, bytes(block))
out += bytes(b ^ prev[k] ^ o[k] for k, b in enumerate(data[i:i + 16]))
prev = bytes(block)
return bytes(out)
def _unwrap_theme_ptf(buf):
"""Unwrap a PSP theme PTF EDAT (``buf`` is the fully CTR-decrypted entry)."""
if len(buf) < 0x90 + 0xA0:
raise RuntimeError("theme EDAT is too short")
kh_off = buf[0xC]
key_header = bytearray(buf[kh_off:kh_off + 0xA0])
if bytes(key_header[:4]) != b"\x00PGD":
raise RuntimeError("theme EDAT: bad PGD header")
if int.from_bytes(key_header[4:8], "little") != 1 or int.from_bytes(key_header[8:12], "little") != 1:
raise RuntimeError("theme EDAT: unsupported key/drm type")
mac = _cmac(_KIRK38, bytes(key_header[:0x70]))
# Step 1: reveal the data size/offset fields (bytes 0x30..0x60).
key_header[0x30:0x60] = _kirk_cbc_decrypt(_kirk_iv(mac, key_header, 0x10), 0, bytes(key_header[0x30:0x60]))
data_size = int.from_bytes(key_header[0x44:0x48], "little")
data_offset = int.from_bytes(key_header[0x4c:0x50], "little")
if data_offset != 0x90:
raise RuntimeError("theme EDAT: unexpected data offset")
# Step 2: decrypt the PTF body.
iv = _kirk_iv(mac, key_header, 0x30)
out = bytearray()
nblocks = (data_size + 15) // 16
for i in range(nblocks):
chunk = buf[kh_off + data_offset + i * 16:kh_off + data_offset + (i + 1) * 16]
out += _kirk_cbc_decrypt(iv, i, chunk)
return bytes(out[:data_size])
# --------------------------------------------------------------------------- #
# PKG extraction
# --------------------------------------------------------------------------- #
def extract(pkg_path, dest_dir, kind_hint=None, progress_cb=None, cancel_event=None):
"""Decrypt a PSN pkg into ``dest_dir/<TITLE_ID>/``.
Returns ``(kind, title_id, folder)`` where kind is ``"psp"``, ``"psx"``,
``"theme"`` or ``"dlc"``. ``kind_hint`` ("theme"/"dlc") overrides the
pkg-header content-type detection (NoPayStation DLC ships as two formats).
"""
from .store import TransferCancelled
pkg_path = Path(pkg_path)
dest_dir = Path(dest_dir)
with open(pkg_path, "rb") as fh:
header = fh.read(0x100)
if len(header) < 0xE8 or int.from_bytes(header[0:4], "big") != PKG_MAGIC:
raise RuntimeError(f"not a PKG file: {pkg_path.name}")
meta_offset = int.from_bytes(header[8:12], "big")
meta_count = int.from_bytes(header[12:16], "big")
item_count = int.from_bytes(header[20:24], "big")
enc_offset = int.from_bytes(header[32:40], "big")
iv = header[0x70:0x80]
key_type = header[0xE7] & 7
title_id = header[0x37:0x40].decode("ascii", "replace").rstrip("\x00").strip() or "PKG"
content_type = 0
items_offset = 0
with open(pkg_path, "rb") as fh:
off = meta_offset
for _ in range(meta_count):
fh.seek(off)
m = fh.read(16)
if len(m) < 16:
break
mtype = int.from_bytes(m[0:4], "big")
msize = int.from_bytes(m[4:8], "big")
if mtype == 2:
content_type = int.from_bytes(m[8:12], "big")
elif mtype == 13:
items_offset = int.from_bytes(m[8:12], "big")
off += 8 + msize
if kind_hint in ("theme", "dlc"):
kind = kind_hint
elif content_type == 6:
kind = "psx"
elif content_type in (7, 0xE, 0xF, 0x10):
kind = "psp"
elif content_type == 9:
kind = "theme"
elif content_type == 0xB:
kind = "dlc"
else:
raise RuntimeError(f"unsupported NoPayStation content type 0x{content_type:x}")
# NoPayStation DLC is wrapped in a PS3-format pkg (key type 4 in the header
# but actually encrypted with the PS3 key); everything else uses the PSP key.
if content_type == 0xB:
main_key = PS3_KEY
ps3_key = PS3_KEY
elif key_type == 1:
main_key = PSP_KEY
ps3_key = PS3_KEY
else:
raise RuntimeError(f"unsupported PKG key type {key_type}")
# Decrypt the whole item table in one shot.
with open(pkg_path, "rb") as fh:
fh.seek(enc_offset + items_offset)
table = fh.read(item_count * 32)
table = _ctr_decrypt(main_key, iv, items_offset // 16, table)
entries = []
for i in range(item_count):
e = table[i * 32:(i + 1) * 32]
name_offset = int.from_bytes(e[0:4], "big")
name_size = int.from_bytes(e[4:8], "big")
data_offset = int.from_bytes(e[8:16], "big")
data_size = int.from_bytes(e[16:24], "big")
psp_type = e[24]
flags = e[27]
if flags in (4, 18) or data_size == 0:
continue
item_key = main_key if psp_type == 0x90 else ps3_key
entries.append((name_offset, name_size, data_offset, data_size, item_key))
folder = dest_dir / title_id
folder.mkdir(parents=True, exist_ok=True)
written = []
total = sum(ds for *_, ds, _ in entries) or 1
done = 0
with open(pkg_path, "rb") as fh:
for name_offset, name_size, data_offset, data_size, item_key in entries:
if cancel_event is not None and cancel_event.is_set():
raise TransferCancelled()
fh.seek(enc_offset + name_offset)
name = _ctr_decrypt(item_key, iv, name_offset // 16, fh.read(name_size))
name = name.split(b"\x00", 1)[0].decode("utf-8", "replace")
base = name.rsplit("/", 1)[-1]
if kind == "psx" and base not in ("EBOOT.PBP", "DOCUMENT.DAT"):
continue
if kind == "psp" and base not in ("EBOOT.PBP", "PSP-KEY.EDAT", "CONTENT.DAT"):
continue
if kind == "theme" and not base.lower().endswith(".ptf"):
continue
if kind == "dlc" and not base.lower().endswith(".edat"):
continue
if kind == "dlc" and base.upper() == "ISO.BIN.EDAT":
# Small EDAT wrapper shipped inside PSP DLC packages — not DLC
# content itself, so don't install it (matches pkgi-psp).
continue
if kind == "theme":
# Read the (small) entry fully, CTR-decrypt, then unwrap the EDAT.
fh.seek(enc_offset + data_offset)
raw = _ctr_decrypt(item_key, iv, data_offset // 16, fh.read(data_size))
dest = folder / base
dest.write_bytes(_unwrap_theme_ptf(raw))
done += data_size
else:
dest = folder / base
decrypt = _make_ctr_decryptor(item_key, iv, data_offset // 16)
fh.seek(enc_offset + data_offset)
with open(dest, "wb") as out:
remaining = data_size
while remaining > 0:
if cancel_event is not None and cancel_event.is_set():
raise TransferCancelled()
chunk = fh.read(min(1 << 20, remaining))
if not chunk:
break
out.write(decrypt(chunk))
remaining -= len(chunk)
done += len(chunk)
if progress_cb is not None:
progress_cb("Extracting", min(done / total, 1.0), "")
written.append(dest)
if kind in ("psp", "psx") and not (folder / "EBOOT.PBP").exists():
raise RuntimeError(f"no EBOOT.PBP found in {pkg_path.name}")
if kind == "theme" and not any(p.suffix.lower() == ".ptf" for p in written):
raise RuntimeError(f"no PTF found in {pkg_path.name}")
if kind == "dlc" and not written:
raise RuntimeError(f"no DLC EDAT found in {pkg_path.name}")
return kind, title_id, folder
+158
View File
@@ -0,0 +1,158 @@
"""PS1 library helpers: discover source discs and converted EBOOT.PBPs.
This is the read/scan side of PS1 support (the conversion side lives in
``app.ps1_convert``). It finds raw PS1 disc images (``.cue`` sets and
standalone ``.iso``/``.img``), reads each disc's boot serial, and lists
converted ``EBOOT.PBP`` files in the output folder.
"""
from __future__ import annotations
import re
from pathlib import Path
from typing import List, Optional
from . import config
RAW_SECTOR = 2352
COOKED_SECTOR = 2048
EBOOT_NAME = "EBOOT.PBP"
# SYSTEM.CNF boot line: cdrom:\SLPS_017.62;1 (backslash optional in some rips)
_SERIAL_RE = re.compile(rb"cdrom:\\?([A-Za-z]{4})[_-]?(\d{3})\.(\d{2})")
def _sector_size(path: Path) -> int:
size = path.stat().st_size
if size and size % RAW_SECTOR == 0:
return RAW_SECTOR
return COOKED_SECTOR
def read_serial(image_path: Path) -> Optional[str]:
"""Read the boot serial (e.g. ``SLPS01762``) from a PS1 disc image.
Works on both raw 2352-byte/sector images (BIN/IMG) and cooked 2048-byte
images (ISO). Returns ``None`` if the boot line can't be found.
"""
image_path = Path(image_path)
if not image_path.is_file():
return None
sector_size = _sector_size(image_path)
data_off = 24 if sector_size == RAW_SECTOR else 0
buf = bytearray()
with open(image_path, "rb") as fh:
for _ in range(128): # SYSTEM.CNF is always near the start of the disc
sector = fh.read(sector_size)
if len(sector) < sector_size:
break
buf += sector[data_off:data_off + COOKED_SECTOR]
m = _SERIAL_RE.search(bytes(buf))
if m:
return (m.group(1) + m.group(2) + m.group(3)).decode().upper()
return None
def cue_data_track(cue_path: Path) -> Optional[Path]:
"""Return the first data-track image referenced by a cue sheet."""
from . import ps1_convert
try:
disc = ps1_convert.parse_cue(cue_path.read_text(encoding="utf-8", errors="replace"))
except Exception:
return None
track = disc.data_track
if track is None:
return None
return (cue_path.parent / track.file) if track.file else None
def iter_source_discs(root: Path):
"""Yield PS1 source discs under ``root`` (cue sheets + standalone images)."""
root = Path(root)
if not root.is_dir():
return
for path in sorted(root.rglob("*")):
if not path.is_file():
continue
suffix = path.suffix.lower()
if suffix == ".cue":
yield path
elif suffix in (".iso", ".img"):
# A standalone image: no cue sheet next to it (same stem).
if not (path.parent / f"{path.stem}.cue").exists():
yield path
def find_eboots(root: Path) -> List[Path]:
"""List converted/PSN ``EBOOT.PBP`` files under ``root``."""
root = Path(root)
if not root.is_dir():
return []
return [p for p in sorted(root.rglob("*")) if p.is_file() and p.name.upper() == EBOOT_NAME]
def eboot_title(pbp_path: Path) -> str:
"""Read the game title embedded in an EBOOT.PBP (falls back to folder name)."""
from . import iso_reader, ps1_convert
try:
header = ps1_convert.read_pbp_header(pbp_path)
idx = ps1_convert.SECTION_OFFSET["sfo"] // 4
sfo = ps1_convert.read_pbp_section(pbp_path, header, idx)
parsed = iso_reader.parse_sfo(sfo)
title = parsed.get("TITLE") or parsed.get("TITLE_0") or parsed.get("TITLE_00")
if title:
return str(title)
except Exception:
pass
return pbp_path.parent.name
def scan_library(ps1_dir, output_dir) -> List[dict]:
"""Return a combined view of PS1 source discs and converted EBOOTs.
Each entry is a dict with ``kind`` ("disc" or "eboot"), ``path``, ``name``,
``serial`` and ``status`` ("pending" / "converted").
"""
ps1_dir = Path(ps1_dir)
output_dir = Path(output_dir)
eboots = find_eboots(output_dir)
eboot_names = {e.name.upper() for e in eboots}
eboot_titles = {eboot_title(e).lower() for e in eboots}
entries: List[dict] = []
for src in iter_source_discs(ps1_dir):
if src.suffix.lower() == ".cue":
image = cue_data_track(src) or src
else:
image = src
serial = read_serial(image)
title = src.stem
# A disc is "converted" when an EBOOT's embedded title or filename
# matches the source disc's title/serial.
status = "pending"
if title.lower() in eboot_titles or (serial and any(serial.lower() in t for t in eboot_titles)):
status = "converted"
entries.append({
"kind": "disc",
"path": str(src),
"name": title,
"serial": serial or "",
"status": status,
})
for e in eboots:
entries.append({
"kind": "eboot",
"path": str(e),
"name": eboot_title(e),
"serial": "",
"status": "converted",
})
return entries
+1179
View File
File diff suppressed because it is too large Load Diff
+62 -4
View File
@@ -221,8 +221,48 @@ def _pkg_items(root):
return items
def _theme_items(root):
"""Installed themes (``PSP/THEME/*.ptf``), named from the PTF header."""
items = []
theme_dir = root / "PSP" / "THEME"
if theme_dir.is_dir():
try:
entries = sorted(theme_dir.iterdir())
except OSError:
return items
for path in entries:
if not path.is_file() or path.suffix.lower() != ".ptf":
continue
item = _file_item(path, "theme")
try:
from . import theme as theme_mod
name = theme_mod.ptf_name(path)
except Exception: # noqa: BLE001
name = ""
if name:
item["name"] = name
items.append(item)
return items
def _dlc_items(root):
"""Installed DLC (``PSP/GAME/<GAME_ID>/*.edat``)."""
items = []
game_dir = root / "PSP" / "GAME"
if game_dir.is_dir():
for path in sorted(game_dir.rglob("*")):
if not path.is_file() or path.suffix.lower() != ".edat":
continue
if path.name.upper() == "ISO.BIN.EDAT":
continue # package wrapper, not DLC content
item = _file_item(path, "dlc")
item["id"] = path.parent.name
items.append(item)
return items
def list_installed(root):
"""Return ``{games, videos, homebrew, pkg}`` inventories for a PSP root."""
"""Return ``{games, videos, homebrew, pkg, themes, dlc}`` for a PSP root."""
root = Path(root)
games, videos, homebrew_isos = _iso_items(root)
return {
@@ -230,14 +270,19 @@ def list_installed(root):
"videos": videos,
"homebrew": _homebrew_items(root) + homebrew_isos,
"pkg": _pkg_items(root),
"themes": _theme_items(root),
"dlc": _dlc_items(root),
}
def build_index(inventory):
"""Build a lookup index for matching library discs against the PSP.
Returns ``{"raw": set, "names": set}`` where ``raw`` holds lowercase ISO
stems (exact filename match) and ``names`` holds fuzzy-normalized stems.
Returns ``{"raw", "names", "homebrew_ids", "homebrew_names"}`` where
``raw`` holds lowercase ISO stems (exact filename match), ``names`` holds
fuzzy-normalized stems, and the homebrew keys index ``PSP/GAME/**/EBOOT.PBP``
(homebrew + PS1 Classics, including Category Lite subfolders) by folder id
and embedded title.
"""
raw, names = set(), set()
for group in ("games", "videos"):
@@ -247,7 +292,20 @@ def build_index(inventory):
norm = matcher.normalize_name(stem)
if norm:
names.add(norm)
return {"raw": raw, "names": names}
hb_ids, hb_names = set(), set()
for it in (inventory or {}).get("homebrew", []):
ident = str(it.get("id") or "").strip().lower()
if ident:
hb_ids.add(ident)
stem = Path(it.get("name") or "").stem.strip().lower()
if stem:
hb_names.add(stem)
norm = matcher.normalize_name(Path(it.get("name") or "").stem)
if norm:
hb_names.add(norm)
return {"raw": raw, "names": names,
"homebrew_ids": hb_ids, "homebrew_names": hb_names}
def find_umd_iso(root):
+5 -1
View File
@@ -11,13 +11,15 @@ from pathlib import Path
from . import config
CREDENTIALS_PATH = Path(
os.environ.get("PSP_CREDENTIALS", str(config.DATA_DIR / "credentials.json"))
os.environ.get("PSP_CREDENTIALS", str(config.USER_DATA_DIR / "credentials.json"))
)
FIELDS = [
"igdb_client_id",
"igdb_client_secret",
"tmdb_api_key",
"google_api_key",
"google_cx",
"screenscraper_devid",
"screenscraper_devpassword",
"screenscraper_ssid",
@@ -29,6 +31,8 @@ _ENV_MAP = {
"igdb_client_id": "IGDB_CLIENT_ID",
"igdb_client_secret": "IGDB_CLIENT_SECRET",
"tmdb_api_key": "TMDB_API_KEY",
"google_api_key": "GOOGLE_API_KEY",
"google_cx": "GOOGLE_CX",
"screenscraper_devid": "SCREENSCRAPER_DEVID",
"screenscraper_devpassword": "SCREENSCRAPER_DEVPASSWORD",
"screenscraper_ssid": "SCREENSCRAPER_SSID",
+316
View File
@@ -0,0 +1,316 @@
"""Global dark theme for the GUI (Fusion style + palette + stylesheet).
Applied once at startup from ``gui.main``. The per-widget ``setStyleSheet``
colors sprinkled through ``gui.py`` (``#999`` secondary text, ``#e06c6c`` delete
buttons, ``#14151a`` preview panels) were written for a dark background, so a
single cohesive dark theme here makes the whole UI consistent.
"""
from PySide6.QtGui import QColor, QPalette
from . import config
# Design tokens
BG = "#1e1f24" # window background
SURFACE = "#26272e" # cards, inputs, table headers
RAISED = "#2d2e36" # buttons, combos, menus
BORDER = "#3a3b44"
BORDER_SOFT = "#33343d"
TEXT = "#e6e6ea"
TEXT_DIM = "#9a9ba5"
TEXT_FAINT = "#6f7079"
ACCENT = "#4a9eff"
ACCENT_HOVER = "#6bb0ff"
_STYLESHEET = f"""
QMainWindow {{
background: {BG};
}}
QWidget {{
color: {TEXT};
}}
/* --- Tabs --- */
QTabWidget::pane {{
border: 1px solid {BORDER};
border-radius: 6px;
top: -1px;
}}
QTabBar::tab {{
background: transparent;
color: {TEXT_DIM};
padding: 8px 18px;
border: none;
border-bottom: 2px solid transparent;
margin-right: 2px;
}}
QTabBar::tab:hover:!selected {{
color: {TEXT};
}}
QTabBar::tab:selected {{
color: {TEXT};
border-bottom: 2px solid {ACCENT};
}}
/* --- Buttons --- */
QPushButton {{
background: {RAISED};
border: 1px solid {BORDER};
border-radius: 6px;
padding: 6px 12px;
}}
QPushButton:hover {{
background: #343640;
border-color: {ACCENT_HOVER};
}}
QPushButton:pressed {{
background: {SURFACE};
}}
QPushButton:disabled {{
color: {TEXT_FAINT};
background: {SURFACE};
border-color: {BORDER_SOFT};
}}
QPushButton:checked {{
background: {ACCENT};
border-color: {ACCENT};
color: #101116;
}}
/* --- Inputs --- */
QLineEdit, QSpinBox, QPlainTextEdit, QTextEdit {{
background: {SURFACE};
border: 1px solid {BORDER};
border-radius: 6px;
padding: 5px 8px;
selection-background-color: {ACCENT};
selection-color: #101116;
}}
QLineEdit:focus, QSpinBox:focus, QPlainTextEdit:focus, QTextEdit:focus {{
border-color: {ACCENT};
}}
QLineEdit:disabled, QSpinBox:disabled {{
color: {TEXT_FAINT};
}}
/* --- Combo boxes --- */
QComboBox {{
background: {RAISED};
border: 1px solid {BORDER};
border-radius: 6px;
padding: 4px 10px;
}}
QComboBox:hover {{
border-color: {ACCENT_HOVER};
}}
QComboBox::drop-down {{
border: none;
width: 22px;
}}
QComboBox::down-arrow {{
image: url(__CHEVRON__);
width: 12px;
height: 8px;
margin-right: 6px;
}}
QComboBox QAbstractItemView {{
background: {RAISED};
border: 1px solid {BORDER};
border-radius: 6px;
selection-background-color: rgba(74, 158, 255, 0.25);
selection-color: {TEXT};
outline: 0;
}}
/* --- Tables --- */
QTableWidget {{
background: transparent;
border: none;
gridline-color: {BORDER_SOFT};
}}
QHeaderView::section {{
background: {SURFACE};
color: {TEXT_DIM};
border: none;
border-bottom: 1px solid {BORDER};
padding: 6px 8px;
}}
QTableWidget::item {{
padding: 4px 6px;
}}
QTableWidget::item:selected {{
background: rgba(74, 158, 255, 0.22);
color: {TEXT};
}}
QTableCornerButton::section {{
background: {SURFACE};
border: none;
border-bottom: 1px solid {BORDER};
}}
/* --- Cover grid / lists --- */
QListWidget {{
background: transparent;
border: none;
}}
QListWidget::item {{
border-radius: 6px;
}}
QListWidget::item:hover:!selected {{
background: rgba(255, 255, 255, 0.04);
}}
QListWidget::item:selected {{
background: rgba(74, 158, 255, 0.22);
}}
/* --- Cards (details panels use QScrollArea) --- */
QScrollArea {{
background: {SURFACE};
border: none;
}}
/* --- Menus --- */
QMenu {{
background: {RAISED};
border: 1px solid {BORDER};
border-radius: 6px;
padding: 4px;
}}
QMenu::item {{
padding: 6px 26px 6px 12px;
border-radius: 4px;
}}
QMenu::item:selected {{
background: rgba(74, 158, 255, 0.25);
}}
QMenu::separator {{
height: 1px;
background: {BORDER};
margin: 4px 8px;
}}
/* --- Scrollbars --- */
QScrollBar:vertical {{
background: transparent;
width: 10px;
margin: 0;
}}
QScrollBar::handle:vertical {{
background: {BORDER};
border-radius: 5px;
min-height: 30px;
}}
QScrollBar::handle:vertical:hover {{
background: {ACCENT};
}}
QScrollBar:horizontal {{
background: transparent;
height: 10px;
margin: 0;
}}
QScrollBar::handle:horizontal {{
background: {BORDER};
border-radius: 5px;
min-width: 30px;
}}
QScrollBar::handle:horizontal:hover {{
background: {ACCENT};
}}
QScrollBar::add-line, QScrollBar::sub-line {{
width: 0;
height: 0;
}}
QScrollBar::add-page, QScrollBar::sub-page {{
background: transparent;
}}
/* --- Splitter --- */
QSplitter::handle {{
background: transparent;
}}
QSplitter::handle:horizontal {{
width: 1px;
}}
QSplitter::handle:vertical {{
height: 1px;
}}
/* --- Misc --- */
QToolTip {{
background: {RAISED};
color: {TEXT};
border: 1px solid {BORDER};
padding: 4px 6px;
}}
QStatusBar {{
background: {SURFACE};
color: {TEXT_DIM};
border-top: 1px solid {BORDER};
}}
QProgressBar {{
background: {SURFACE};
border: 1px solid {BORDER};
border-radius: 6px;
text-align: center;
}}
QProgressBar::chunk {{
background: {ACCENT};
border-radius: 5px;
}}
QCheckBox {{
spacing: 8px;
}}
QCheckBox::indicator {{
width: 16px;
height: 16px;
border: 1px solid {BORDER};
border-radius: 4px;
background: {SURFACE};
}}
QCheckBox::indicator:checked {{
background: {ACCENT};
border-color: {ACCENT};
}}
QGroupBox {{
border: 1px solid {BORDER};
border-radius: 6px;
margin-top: 10px;
}}
QGroupBox::title {{
subcontrol-origin: margin;
left: 10px;
padding: 0 4px;
color: {TEXT_DIM};
}}
"""
def _dark_palette():
p = QPalette()
p.setColor(QPalette.Window, QColor(BG))
p.setColor(QPalette.WindowText, QColor(TEXT))
p.setColor(QPalette.Base, QColor(SURFACE))
p.setColor(QPalette.AlternateBase, QColor(RAISED))
p.setColor(QPalette.ToolTipBase, QColor(RAISED))
p.setColor(QPalette.ToolTipText, QColor(TEXT))
p.setColor(QPalette.Text, QColor(TEXT))
p.setColor(QPalette.Button, QColor(RAISED))
p.setColor(QPalette.ButtonText, QColor(TEXT))
p.setColor(QPalette.BrightText, QColor("#ff5c5c"))
p.setColor(QPalette.Link, QColor(ACCENT))
p.setColor(QPalette.Highlight, QColor(ACCENT))
p.setColor(QPalette.HighlightedText, QColor("#101116"))
p.setColor(QPalette.PlaceholderText, QColor(TEXT_FAINT))
faint = QColor(TEXT_FAINT)
for role in (QPalette.WindowText, QPalette.Text, QPalette.ButtonText):
p.setColor(QPalette.Disabled, role, faint)
return p
def apply_theme(app):
"""Apply the dark Fusion theme to ``app`` (call once before showing UI)."""
app.setStyle("Fusion")
app.setPalette(_dark_palette())
chevron = str(config.resource_path("img", "chevron.png"))
app.setStyleSheet(_STYLESHEET.replace("__CHEVRON__", chevron))
+511
View File
@@ -0,0 +1,511 @@
"""PSP theme (``.ptf``) introspection: name + preview image.
PTF themes embed a readable name in their header and a preview image (a GIM
bitmap) in object 0. This module decodes both without any external service:
* the name lives at offset 0x08 (128 UTF-8 bytes);
* the preview is object 0's largest image item, stored as a GIM that may be
Sony-LZR compressed.
GIM decoding covers the common RGBA formats (5650/5551/4444/8888) and 8-bit
palettes; anything exotic simply yields no preview.
"""
from __future__ import annotations
import zlib
from pathlib import Path
PTF_MAGIC = b"\x00PTF"
# --------------------------------------------------------------------------- #
# Sony LZR/RLZ decompression (sceLzrDecompress), ported from PTF-Studio.
# --------------------------------------------------------------------------- #
_LZR_PROBABILITY_SLOTS = 2800
_LZR_INITIAL_PROBABILITY = 0x80
_LZR_RENORM_LIMIT = 0x00FFFFFF
_LZR_U32_MOD = 0x100000000
def _lzr_decompress(inp, max_output, offset=0):
inp = bytes(inp)
end = offset + len(inp) - offset
out = bytearray(max_output)
probs = [_LZR_INITIAL_PROBABILITY] * _LZR_PROBABILITY_SLOTS
out_pos = 0
in_pos = offset + 5
code = int.from_bytes(inp[offset + 1:offset + 5], "big")
range_ = 0xFFFFFFFF
spec = 0
corrupt = False
buf_off = 0
last_byte = 0
last_number_flag = 0
type_ = inp[offset]
if type_ >= 0x80:
type_ -= 0x100
def next_byte():
nonlocal in_pos, corrupt
if in_pos >= end:
corrupt = True
return 0
b = inp[in_pos]
in_pos += 1
return b
if type_ > 7:
raise ValueError("unsupported LZR literal-context mode")
if type_ < 0:
stored = code
if stored > len(out) or in_pos + stored > end:
raise ValueError("malformed stored LZR block")
return inp[in_pos:in_pos + stored]
def renormalize():
nonlocal code, range_
if range_ <= _LZR_RENORM_LIMIT:
code = (code * 256 + next_byte()) % _LZR_U32_MOD
range_ = (range_ * 256) % _LZR_U32_MOD
def decode_bit(prob_index):
nonlocal code, range_, corrupt
if prob_index < 0 or prob_index >= _LZR_PROBABILITY_SLOTS:
corrupt = True
return 0
renormalize()
if corrupt:
return 0
p = probs[prob_index]
bound = (range_ // 256) * p
probs[prob_index] = p - (p // 8)
if code < bound:
range_ = bound
probs[prob_index] += 31
return 1
code -= bound
range_ -= bound
return 0
def decode_bit_speculative(prob_index):
nonlocal code, range_, spec, corrupt
if prob_index < 0 or prob_index >= _LZR_PROBABILITY_SLOTS:
corrupt = True
return 0
if spec <= _LZR_RENORM_LIMIT:
code = (code * 256 + next_byte()) % _LZR_U32_MOD
range_ = (spec * 256) % _LZR_U32_MOD
if corrupt:
return 0
p = probs[prob_index]
bound = (range_ // 256) * p
spec = bound
probs[prob_index] = p - (p // 8)
if code < bound:
range_ = bound
probs[prob_index] += 31
return 1
code -= bound
range_ -= bound
return 0
def decode_number(bits, base, step):
nonlocal code, range_, corrupt, last_number_flag
number = 1
if bits >= 3:
number = number * 2 + decode_bit(base + 3 * step)
if bits >= 4:
number = number * 2 + decode_bit(base + 3 * step)
if bits >= 5:
renormalize()
remaining = bits
while remaining >= 5:
if range_ == 0:
corrupt = True
return 0
number *= 2
range_ = range_ // 2
if code < range_:
number += 1
else:
code -= range_
remaining -= 1
last_number_flag = decode_bit(base)
number = number * 2 + last_number_flag
if bits >= 1:
number = number * 2 + decode_bit(base + step)
if bits >= 2:
number = number * 2 + decode_bit(base + 2 * step)
return number
def selector_context():
return buf_off + 2488
def decode_literal(shift):
nonlocal out_pos, last_byte, corrupt, buf_off
if buf_off > 0:
buf_off -= 1
if out_pos == len(out):
return False
context = ((((out_pos & 7) << 8) + last_byte) // (2 ** shift)) & 7
tree_base = context * 0xFF - 1
node = 1
while node <= 0xFF:
node = node * 2 + decode_bit(tree_base + node)
if corrupt:
return False
out[out_pos] = node & 0xFF
last_byte = node & 0xFF
out_pos += 1
return True
def decode_match():
nonlocal out_pos, last_byte, buf_off, spec, code, range_, corrupt, last_number_flag
slot = selector_context()
spec = range_
length_class = -1
bit = 0
while True:
slot += 8
bit = decode_bit_speculative(slot)
if corrupt:
return -1
length_class += bit
if bit == 0 or length_class >= 6:
break
offset_group = length_class + 2033
offset_bias = 64
if bit != 0 or length_class >= 0:
length_context = (length_class << 5) + (((out_pos * (2 ** length_class)) & 3) << 3) + buf_off + 2552
match_length = decode_number(length_class, length_context, 8)
if corrupt:
return -1
if match_length == 0xFF:
return 1
if last_number_flag != 0 or length_class > 0:
offset_group += 56
offset_bias = 352
else:
match_length = 1
index = 1
offset_class = 0
while True:
offset_class = (index << 4) - offset_bias
prob_index = offset_group + (index << 3)
if prob_index >= _LZR_PROBABILITY_SLOTS:
return -1
bit = decode_bit(prob_index)
if corrupt:
return -1
index = (index << 1) | bit
if offset_class >= 0:
break
if bit != 0 or offset_class > 0:
if bit == 0:
offset_class -= 8
base = offset_class + 2344
if base < 0 or base + 3 >= _LZR_PROBABILITY_SLOTS:
return -1
back_offset = decode_number(offset_class // 8, base, 1)
if corrupt:
return -1
else:
back_offset = 1
if back_offset > out_pos:
return -1
copy_end = out_pos + match_length + 1
if copy_end > len(out):
return -1
buf_off = ((copy_end + 1) & 1) + 6
from_ = out_pos - back_offset
while out_pos < copy_end:
out[out_pos] = out[from_]
out_pos += 1
from_ += 1
last_byte = out[out_pos - 1]
return 0
token_budget = max(1024, max_output * 2 + 1024)
while True:
token_budget -= 1
if token_budget < 0:
raise ValueError("malformed LZR stream did not terminate")
token_is_match = decode_bit(selector_context())
if corrupt:
raise ValueError("truncated LZR stream")
if token_is_match == 0:
if not decode_literal(type_):
raise ValueError("LZR stream exceeds declared output size")
else:
result = decode_match()
if result == 1:
return bytes(out[:out_pos])
if result < 0:
raise ValueError("malformed LZR back-reference")
# --------------------------------------------------------------------------- #
# PTF parsing
# --------------------------------------------------------------------------- #
def _decompress_item(blob, comp, ucsize):
if comp == 2 and len(blob) < ucsize:
return zlib.decompress(blob)
if comp == 1 and len(blob) < ucsize:
return _lzr_decompress(blob, ucsize)
return blob
def parse_ptf(data):
"""Return ``(name, items)`` where items are decompressed item blobs."""
if not isinstance(data, (bytes, bytearray)):
data = data.read_bytes() if hasattr(data, "read_bytes") else bytes(data)
data = bytes(data)
if data[:4] != PTF_MAGIC:
raise ValueError("not a PTF file")
raw_name = data[8:8 + 128]
name = raw_name.split(b"\x00", 1)[0].decode("utf-8", "replace").strip()
def u16(o):
return int.from_bytes(data[o:o + 2], "little")
def u32(o):
return int.from_bytes(data[o:o + 4], "little")
items = []
pos = 0x100
while True:
offset = u32(pos)
pos += 4
if offset == 0 or offset >= len(data):
break
count = u16(offset + 2)
data_off = u32(offset + 8)
p = data_off
for _ in range(count):
file_type = u16(p + 4)
comp = u16(p + 6)
size = u32(p + 8)
ucsize = u32(p + 12)
blob = data[p + 0x20:p + 0x20 + size]
items.append((file_type, _decompress_item(blob, comp, ucsize)))
p += 0x20 + size
return name, items
def ptf_name(path):
"""Theme display name embedded in a ``.ptf`` file."""
try:
with open(path, "rb") as fh:
head = fh.read(8 + 128)
except OSError:
return ""
if head[:4] != PTF_MAGIC:
return ""
name = head[8:8 + 128].split(b"\x00", 1)[0].decode("utf-8", "replace").strip()
return name or Path(path).stem
# --------------------------------------------------------------------------- #
# GIM decoding (RGBA formats only)
# --------------------------------------------------------------------------- #
def _overscan(value, tile):
return value if value % tile == 0 else value + (tile - (value % tile))
def _untile(pixels, w, h, tile_w, tile_h):
over_w = _overscan(w, tile_w)
over_h = _overscan(h, tile_h)
out = [0] * (w * h)
tile_pos = 0
origin_x = 0
origin_y = 0
for y in range(over_h):
for x in range(over_w):
dx = origin_x + (tile_pos % tile_w)
dy = origin_y + (tile_pos // tile_w)
if dx < w and dy < h:
out[dy * w + dx] = pixels[y * over_w + x]
tile_pos += 1
if tile_pos == tile_w * tile_h:
tile_pos = 0
origin_x += tile_w
if origin_x >= over_w:
origin_x = 0
origin_y += tile_h
return out
def _gim_to_rgba(data):
"""Decode a GIM to ``(width, height, rgba_bytes)`` or None if unsupported."""
data = bytes(data)
if len(data) < 0x10:
return None
# Detect byte order from the magic. Sony GIMs store the magic as the byte
# string "MIG.00.1PSP" with little-endian data (LibPSPThemes semantics).
if data[:4] == b"MIG.":
byteorder = "little"
elif data[:4] == b".GIM":
byteorder = "big"
else:
return None
def rd(off, size):
return int.from_bytes(data[off:off + size], byteorder)
image = None
palette = None
pos = 0x10
for _ in range(32):
if pos + 16 > len(data):
break
btype = rd(pos, 2)
next_block = rd(pos + 8, 4)
if btype == 4 or btype == 5:
h = pos + 0x10
fmt = rd(h + 4, 2)
pixel_order = rd(h + 6, 2)
width = rd(h + 8, 2)
height = rd(h + 10, 2)
bpp = rd(h + 12, 2)
next_index = rd(h + 24, 4)
frame_data_end = rd(h + 32, 4)
frame_offset = rd(pos + 0x10 + next_index, 4)
block_content = pos + 0x10
start = block_content + frame_offset
end = block_content + frame_data_end
raw = data[start:end]
pixels = _decode_pixels(raw, fmt, bpp, width, height, pixel_order)
if pixels is None:
return None
if btype == 4:
image = (width, height, pixels)
else:
palette = (fmt, width, height, pixels)
if next_block == 0:
break
pos += next_block
if image is None:
return None
width, height, pixels = image
if palette is not None:
_, pw, ph, pvals = palette
if pw * ph > 0:
pixels = [pvals[p] if p < len(pvals) else 0 for p in pixels]
rgba = bytearray(width * height * 4)
for i, v in enumerate(pixels):
# pixels are packed 0xAABBGGRR
o = i * 4
rgba[o] = v & 0xFF
rgba[o + 1] = (v >> 8) & 0xFF
rgba[o + 2] = (v >> 16) & 0xFF
rgba[o + 3] = (v >> 24) & 0xFF
return width, height, bytes(rgba)
def _decode_pixels(raw, fmt, bpp, width, height, pixel_order):
"""Decode a GIM frame into a list of packed 0xAABBGGRR ints."""
if fmt in (0, 1, 2, 3, 5):
pass
else:
return None # indexed-16 / DXT etc. not supported
if fmt == 5: # p8 indexed
bytes_per_pixel = 1
tile_w = 0x80 // max(bpp, 8)
else:
bytes_per_pixel = max(1, (bpp + 7) // 8)
tile_w = 0x80 // max(bpp, 1)
tile_h = 8
if pixel_order == 1:
w = _overscan(width, tile_w)
h = _overscan(height, tile_h)
else:
w, h = width, height
count = w * h
values = []
pos = 0
for _ in range(count):
v = int.from_bytes(raw[pos:pos + bytes_per_pixel], "little")
pos += bytes_per_pixel
values.append(_expand(fmt, bpp, v))
if pos >= len(raw):
# pad missing trailing pixels with transparent
values.extend([0] * (count - len(values)))
break
if pixel_order == 1:
values = _untile(values, width, height, tile_w, tile_h)
return values[:width * height]
def _expand(fmt, bpp, v):
"""Expand a raw GIM pixel value into packed 0xAABBGGRR."""
if fmt == 3: # rgba8888
r = v & 0xFF
g = (v >> 8) & 0xFF
b = (v >> 16) & 0xFF
a = (v >> 24) & 0xFF
return (a << 24) | (b << 16) | (g << 8) | r
if fmt == 0: # rgba5650
r = (v >> 11) & 0x1F
g = (v >> 5) & 0x3F
b = v & 0x1F
r = (r << 3) | (r >> 2)
g = (g << 2) | (g >> 4)
b = (b << 3) | (b >> 2)
return (0xFF << 24) | (b << 16) | (g << 8) | r
if fmt == 1: # rgba5551
r = (v >> 11) & 0x1F
g = (v >> 6) & 0x1F
b = (v >> 1) & 0x1F
a = 0xFF if v & 1 else 0
r = (r << 3) | (r >> 2)
g = (g << 3) | (g >> 2)
b = (b << 3) | (b >> 2)
return (a << 24) | (b << 16) | (g << 8) | r
if fmt == 2: # rgba4444
r = (v >> 12) & 0xF
g = (v >> 8) & 0xF
b = (v >> 4) & 0xF
a = v & 0xF
r = (r << 4) | r
g = (g << 4) | g
b = (b << 4) | b
a = (a << 4) | a
return (a << 24) | (b << 16) | (g << 8) | r
if fmt == 5: # p8 — value is a palette index (expanded later)
return v & 0xFF
return 0
# --------------------------------------------------------------------------- #
# Public API
# --------------------------------------------------------------------------- #
def theme_preview_rgba(ptf_path):
"""Decode the preview image of a theme PTF.
Returns ``(width, height, rgba_bytes)`` or None if it can't be decoded.
"""
try:
data = Path(ptf_path).read_bytes()
_, items = parse_ptf(data)
except (OSError, ValueError):
return None
# The preview image is the largest decodable GIM item.
best = None
for file_type, blob in items:
if file_type != 5:
continue
decoded = _gim_to_rgba(blob)
if decoded is None:
continue
w, h, rgba = decoded
if best is None or w * h > best[0] * best[1]:
best = (w, h, rgba)
return best
+112
View File
@@ -6,6 +6,7 @@ archives, a remote HTTPS store server, etc.
import json
import logging
import os
import re
import shutil
import tempfile
import urllib.parse
@@ -200,6 +201,62 @@ class FileCopySource(Source):
return dest
class Ps1ConvertSource(Source):
"""Convert a PS1 disc image (.cue/.iso) into an EBOOT.PBP.
``item`` must carry ``source`` (path to the .cue or single image),
``title`` and ``game_id``. ``base_pbp`` is a PSN PS1 classic EBOOT used to
supply the encrypted DATA.PSP loader and STARTDAT block.
"""
label = "PS1 convert"
def __init__(self, base_pbp):
self.base_pbp = Path(base_pbp)
def scan(self, root=None):
return []
def download(self, item, dest_dir, progress_cb=None, cancel_event=None):
dest_dir = Path(dest_dir)
dest_dir.mkdir(parents=True, exist_ok=True)
source = Path(item["source"])
title = str(item.get("title") or source.stem)
game_id = item.get("game_id")
if not game_id:
raise RuntimeError("missing game_id for PS1 conversion")
folder_name = re.sub(r"[^A-Za-z0-9]", "", game_id).upper() or "PS1GAME"
folder = dest_dir / folder_name
eboot = folder / "EBOOT.PBP"
if eboot.exists():
raise FileExistsError(str(eboot))
def cb(done, total):
if progress_cb is not None and total:
progress_cb("Converting", min(done / total, 1.0), "")
from . import covers, ps1_convert
# Pull the game's real cover art (libretro / ScreenScraper / IGDB /
# Wikipedia) and bake it into ICON0 + PIC0 + PIC1 so the XMB shows the
# official-style art, not the default PS1 icon. Best effort.
if progress_cb is not None:
progress_cb("Artwork", 0.0, "")
icon0, pic0, pic1 = covers.ps1_art(title, game_id)
try:
ps1_convert.convert_to_folder(
source, title, game_id, self.base_pbp, dest_dir,
progress_cb=cb, cancel_event=cancel_event,
icon0=icon0, pic0=pic0, pic1=pic1,
)
except ps1_convert.Cancelled:
raise TransferCancelled() from None
return folder
class TreeCopySource(Source):
"""Copy a whole directory (save-data folder) into a destination directory."""
@@ -406,3 +463,58 @@ class LocalShareSource(Source):
return dest
finally:
shutil.rmtree(tmp, ignore_errors=True)
class NoPayStationSource(Source):
"""Browse NoPayStation .pkg lists and install PSP/PSX PSN content.
``content_type`` is ``"psp"`` (PSP_GAMES.tsv), ``"psx"`` (PSX_GAMES.tsv),
``"theme"`` (PSP_THEMES.tsv) or ``"dlc"`` (PSP_DLCS.tsv). Downloading
decrypts the ``.pkg`` with the bundled pure-Python decryptor and writes a
``<TITLE_ID>/`` game folder into ``dest_dir``.
"""
label = "NoPayStation"
def __init__(self, content_type, tsv_url="", force=False):
from . import nopaystation
self.content_type = content_type
self.tsv_url = tsv_url or nopaystation.tsv_url(content_type)
self.force = force
def scan(self, root=None):
from . import nopaystation
return nopaystation.fetch_items(self.content_type, self.tsv_url, force=self.force)
def download(self, item, dest_dir, progress_cb=None, cancel_event=None):
from . import nopaystation
dest_dir = Path(dest_dir)
dest_dir.mkdir(parents=True, exist_ok=True)
# The bundled Python decryptor handles PSP/PSX/themes/DLC directly.
title_id = item.get("title_id") or item.get("content_id") or ""
pkg_name = re.sub(r"[^A-Za-z0-9_-]", "", title_id) or "pkg"
tmp = Path(tempfile.mkdtemp(prefix="nps-download-"))
pkg = tmp / f"{pkg_name}.pkg"
try:
if progress_cb is not None:
progress_cb("Downloading", 0.0, "")
download_url(item["pkg_url"], pkg, None, progress_cb, cancel_event)
kind_hint = self.content_type if self.content_type in ("theme", "dlc") else None
folder = nopaystation.extract_pkg(pkg, title_id, dest_dir,
kind_hint, progress_cb, cancel_event)
if self.content_type in ("theme", "dlc"):
_write_sidecar(folder, item)
return folder
finally:
shutil.rmtree(tmp, ignore_errors=True)
def _write_sidecar(folder, item):
"""Save the store display name beside a downloaded theme/DLC (for the library)."""
try:
meta = {"name": item.get("name") or "", "title_id": item.get("title_id") or "",
"content_id": item.get("content_id") or ""}
(Path(folder) / "meta.json").write_text(json.dumps(meta, ensure_ascii=False), encoding="utf-8")
except OSError:
pass
File diff suppressed because one or more lines are too long
@@ -1,5 +1,5 @@
clrmamepro (
name "Sony - PlayStation Portable"
name "Sony - PlayStation Portable (No-Intro)"
description "Sony - PlayStation Portable"
version "2021.10.24"
homepage "http://github.com/robloach/libretro-dats"
File diff suppressed because it is too large Load Diff
+30
View File
@@ -0,0 +1,30 @@
# Loader assets
The files in this directory are bundled from [iPoPS](https://github.com/julianxhokaxhiu/iPoPS)
by Julian Xhokaxhiu, licensed under the MIT License:
- `DATA.PSP` — a plain-ELF PSP POPS loader (relocatable PRX, 19002 bytes).
- `KEYS.BIN` — a 16-byte NPDRM version key used by CFW to decrypt the
`STARTDAT` PGD.
MIT License
Copyright (c) 2020 Julian Xhokaxhiu
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Binary file not shown.
+1
View File
@@ -0,0 +1 @@
MB\Ω:�� �� �"
+2
View File
@@ -0,0 +1,2 @@
# NoPayStation TSV lists are cached here (downloaded on first store scan).
# You can also drop PSP_GAMES.tsv / PSX_GAMES.tsv here manually to work offline.
+6 -6
View File
@@ -6,7 +6,7 @@ services:
# Plain HTTP for now; see Caddyfile to add Let's Encrypt TLS later.
networks:
br0:
ipv4_address: 192.168.10.100
ipv4_address: ${PSP_LAN_IP}
environment:
PSP_DATA_DIR: /data
PSP_DATS_DIR: /data/dats
@@ -16,14 +16,14 @@ services:
PSP_VIDEO_ROOTS: /data/umd_videos
PSP_STORE_DIR: /data/store
# Secret token the desktop app sends as `Authorization: Bearer <token>`.
# Set via a .env file next to this compose file.
# Set in the .env file next to this compose file.
PSP_API_TOKEN: ${PSP_API_TOKEN:-}
volumes:
- ./data:/data
# Adjust host paths to your Unraid share layout:
- /mnt/user/downloads/Sony - PlayStation Portable:/data/store:ro
- /mnt/user/psp/umd_games:/data/umd_games:ro
- /mnt/user/psp/umd_videos:/data/umd_videos:ro
# Host paths come from .env (see .env.example):
- ${PSP_STORE_HOST}:/data/store:ro
- ${PSP_UMD_GAMES_HOST}:/data/umd_games:ro
- ${PSP_UMD_VIDEOS_HOST}:/data/umd_videos:ro
restart: unless-stopped
networks:
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 7.3 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 273 B

BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 689 KiB

After

Width:  |  Height:  |  Size: 683 KiB

+9
View File
@@ -0,0 +1,9 @@
[Desktop Entry]
Type=Application
Name=PSP Preservation Tool
Comment=Preserve and manage your PSP library
Exec=psp-preserve
Icon=psp-preserve
Terminal=false
Categories=Utility;AudioVideo;
StartupWMClass=psp-preserve
+66
View File
@@ -0,0 +1,66 @@
# -*- mode: python ; coding: utf-8 -*-
"""PyInstaller spec: build the PSP Preservation Tool (onedir).
Bundles the read-only resources (DATs, PS1 loader, offline cover bundles,
icons) under ``_internal/`` next to the executable. User-writable data
(credentials, manual covers, PS1 templates, NPS cache, library DB, cover
cache) lives in the OS state dir at runtime — see ``app/config.py``.
Usage: pyinstaller --clean --noconfirm psp-preserve.spec
Output: dist/psp-preserve/
"""
from PyInstaller.utils.hooks import collect_dynamic_libs, collect_submodules
datas = [
("data/dats", "data/dats"),
("data/loader", "data/loader"),
("data/supplement.json", "data"),
("data/covers/libretro-covers.zip", "data/covers"),
("data/covers/openvgdb-covers.zip", "data/covers"),
("data/covers/libretro-index.json", "data/covers"),
("data/covers/openvgdb-index.json", "data/covers"),
("data/covers/UMD_COVER_TEMPLATE.png", "data/covers"),
("data/covers/UMD_COVER_TEMPLATE_FEATURE.png", "data/covers"),
("data/covers/UMD_GAME_COVER_TEMPLATE.png", "data/covers"),
("img/app-icon.png", "img"),
("img/chevron.png", "img"),
("packaging/psp-preserve.desktop", "."),
]
a = Analysis(
["run.py"],
pathex=[],
binaries=collect_dynamic_libs("curl_cffi"),
datas=datas,
hiddenimports=collect_submodules("curl_cffi"),
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=["tkinter", "unittest", "pydoc"],
noarchive=False,
)
pyz = PYZ(a.pure)
exe = EXE(
pyz,
a.scripts,
[],
exclude_binaries=True,
name="psp-preserve",
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=False,
console=False,
disable_windowed_traceback=False,
)
coll = COLLECT(
exe,
a.binaries,
a.datas,
strip=False,
upx=False,
name="psp-preserve",
)
+7
View File
@@ -0,0 +1,7 @@
# Build dependencies (pinned for reproducible CI builds).
# PySide6 is pinned to 6.8.3: newer 6.11.x bundles a Qt6Core.dll that imports
# icuuc.dll on Windows, which is not shipped and crashes at startup.
pyinstaller>=6.11
PySide6==6.8.3
curl_cffi>=0.7
pycryptodome>=3.20
+1
View File
@@ -1,2 +1,3 @@
PySide6>=6.6
curl_cffi>=0.7
pycryptodome>=3.20
+10
View File
@@ -0,0 +1,10 @@
"""PyInstaller entry point.
Runs the GUI the same way ``python -m app.main`` does, but as a plain script
so PyInstaller can build it (relative imports inside ``app`` won't resolve when
a module file is executed directly, hence this top-level launcher).
"""
from app.main import main
if __name__ == "__main__":
raise SystemExit(main())
+68
View File
@@ -0,0 +1,68 @@
# Wine-based Gitea Actions runner: builds Windows binaries on a Linux host.
#
# Jobs run in "host" mode inside this container, where a Windows CPython 3.12
# (running under Wine) plus PyInstaller produce the .exe. Node.js and git are
# included so the stock JS actions (actions/checkout, actions/upload-artifact)
# work as-is.
FROM debian:bookworm-slim
ENV DEBIAN_FRONTEND=noninteractive \
WINEPREFIX=/root/.wine \
WINEDEBUG=-all \
WINEDLLOVERRIDES="mscoree,mshtml=" \
PYTHON_VERSION=3.12.8
# WineHQ wine-staging (newer than Debian's 8.0, needed for Python 3.12),
# node + git for JS actions, and build/package tooling.
RUN dpkg --add-architecture i386 \
&& apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates curl wget gnupg2 \
&& mkdir -p /etc/apt/keyrings \
&& wget -qO- https://dl.winehq.org/wine-builds/winehq.key | gpg --dearmor -o /etc/apt/keyrings/winehq.gpg \
&& echo "deb [signed-by=/etc/apt/keyrings/winehq.gpg] https://dl.winehq.org/wine-builds/debian/ bookworm main" > /etc/apt/sources.list.d/winehq.list \
&& apt-get update \
&& apt-get install -y --install-recommends winehq-staging \
&& apt-get install -y --no-install-recommends unzip zip xvfb xauth cabextract nodejs git \
&& apt-get clean && rm -rf /var/lib/apt/lists/*
# winetricks (standalone script; not packaged in Debian main).
RUN wget -qO /usr/local/bin/winetricks \
"https://raw.githubusercontent.com/Winetricks/winetricks/master/src/winetricks" \
&& chmod +x /usr/local/bin/winetricks
# Gitea Actions runner (linux-amd64 binary).
RUN wget -qO /usr/local/bin/act_runner \
"https://gitea.com/gitea/act_runner/releases/download/v0.6.1/act_runner-0.6.1-linux-amd64" \
&& chmod +x /usr/local/bin/act_runner
# MSVC runtime required by PySide6's Qt DLLs (also initializes the Wine prefix).
RUN xvfb-run -a winetricks --unattended vcrun2022
# Windows CPython 3.12 (embeddable zip) + pip.
RUN wget -qO /tmp/pyembed.zip \
"https://www.python.org/ftp/python/${PYTHON_VERSION}/python-${PYTHON_VERSION}-embed-amd64.zip" \
&& mkdir -p /root/.wine/drive_c/Python312 \
&& cd /root/.wine/drive_c/Python312 \
&& unzip -q /tmp/pyembed.zip \
&& rm -f /tmp/pyembed.zip \
&& printf 'python312.zip\n.\nimport site\nLib\\site-packages\n' > python312._pth \
&& wine "C:\\Python312\\python.exe" --version
# Convenience wrapper: winpython -> wine C:\Python312\python.exe
RUN printf '#!/bin/sh\nexec wine "C:\\Python312\\python.exe" "$@"\n' > /usr/local/bin/winpython \
&& chmod +x /usr/local/bin/winpython
# Bootstrap pip, then pre-install the Windows build deps so CI jobs start fast.
RUN wget -qO /tmp/get-pip.py https://bootstrap.pypa.io/get-pip.py \
&& wine "C:\\Python312\\python.exe" /tmp/get-pip.py \
&& rm -f /tmp/get-pip.py
COPY requirements-build.txt /tmp/requirements-build.txt
RUN winpython -m pip install --upgrade pip \
&& winpython -m pip install -r /tmp/requirements-build.txt \
&& rm -f /tmp/requirements-build.txt
COPY runner/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
WORKDIR /data
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
+14
View File
@@ -0,0 +1,14 @@
#!/bin/sh
set -e
# Register on first run, then run the daemon (host mode).
if [ ! -f /data/.runner ]; then
echo "Registering Wine runner with Gitea…"
act_runner register --no-interactive \
--instance "${GITEA_INSTANCE_URL:?GITEA_INSTANCE_URL is required}" \
--token "${GITEA_RUNNER_REGISTRATION_TOKEN:?GITEA_RUNNER_REGISTRATION_TOKEN is required}" \
--name "${RUNNER_NAME:-windows-builder}" \
--labels "${RUNNER_LABELS:-windows:host}"
fi
exec act_runner daemon