quoije/psp-preservation-tool
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
35db22e3c5f61c734f15b0fc63dff5e5abf68a81
psp-preservation-tool/README.md
T

22 KiB
Raw Blame History

PSP Preservation Tool

the name is terrible, i know.

grrr AI Generated

A cross-platform (Linux + Windows) desktop GUI — PySide6 — for preserving PSP UMD games and videos as .iso files. It scans your discs, verifies them against the Redump/No-Intro catalogs, reports what's missing, and retags/renames discs to a canonical scheme. It fetches cover art and metadata, syncs discs and save-data with a real PSP, can pull missing discs from a Redump 7z archive collection, downloads PSN content from NoPayStation, and converts PlayStation 1 ISO/CUE/BIN rips into PSP EBOOT.PBP homebrew.

PSP Preservation Tool

  • Dark theme with a blue accent — the Library tab in cover view. *

Features

  • Library — scan folders of .iso discs, verify against the DAT catalog, browse with cover art, search/filter, see per-game details (developer, publisher, genre, release date, synopsis, trailer), and launch discs in PPSSPP.
  • Store — browse and download discs from four sources: a local 7z folder, your own HTTP store server, an archive.org item, or NoPayStation (PSP/PS1 PSN .pkg content) (see Store).
  • Queue — background transfers with progress, cancel, per-row delete, etc.
  • PSP Sync — detect a connected PSP (USB) or a mounted UMD, inventory what's installed (games / UMD videos / homebrew / PKG), copy discs to the PSP, delete from the PSP, and dump a UMD to the library.
  • Saves — two-way sync of save-data between the PSP, PPSSPP, and a cloud folder.
  • PS1 — scan a folder of PlayStation 1 discs (ISO/BIN/CUE), read each disc's serial, convert to PSP EBOOT.PBP with a pure-Python popstation converter, and track already-converted games.
  • Settings — directories, cover-source API keys, store/server config, and misc options.

Requirements

  • Python 3.10+ on Linux or Windows
  • PySide6, curl_cffi, pycryptodome (pip install -r requirements.txt)
  • 7z (p7zip / 7-Zip) — used to peek/extract disc images from store archives
  • PPSSPP (optional) — to launch discs from the details panel

NoPayStation .pkg files are decrypted by the bundled pure-Python decryptor (app/pkg.py) — no Wine or pkg2zip required (see NoPayStation).

Install

python3 -m venv .venv          # keep the venv on a local disk, not an SMB mount
. .venv/bin/activate
pip install -r requirements.txt

Run the GUI

python3 -m app.main

Building binaries

The GUI is packaged with PyInstaller into a self-contained folder. Build requirements are pinned in requirements-build.txt (PySide6 6.8.3 on purpose — see the note inside).

python3 -m venv .venv-build
. .venv-build/bin/activate
pip install -r requirements-build.txt
pyinstaller --clean --noconfirm psp-preserve.spec
# output: dist/psp-preserve/  → run dist/psp-preserve/psp-preserve

Notes:

  • The build bundles the DATs, PS1 loader, and the offline cover bundles. The cover zips are large (~850 MB), so builds are slow and the output is big.
  • 7z (p7zip-full on Linux, 7-Zip on Windows) is a runtime dependency for reading store archives — it is not bundled.
  • User data (credentials, manual covers, PS1 templates, NPS cache, library DB, cover cache) lives in the OS state dir, never next to the binary.

CI (Gitea Actions)

.gitea/workflows/build.yml builds Linux and Windows artifacts on version-tag pushes (v*) or on manual trigger (Actions → build → Run workflow). Artifacts appear on the Actions run page and can be attached to a release.

Each job needs a Gitea Actions runner registered with the matching label:

# Linux runner (Docker mode, run on your server)
./act_runner register \
  --instance https://git.ceaf.dev \
  --token <registration-token> \
  --name linux-builder \
  --labels ubuntu-latest:docker://node:20-bookworm

# Windows runner (host mode, run on a Windows machine)
act_runner.exe register \
  --instance https://git.ceaf.dev \
  --token <registration-token> \
  --name win-builder \
  --labels windows:host

The registration token is generated in Gitea under Site Administration → Actions → Runners → Create new runner.

Windows builds without a Windows box (Wine runner)

The windows job targets a runner labelled windows that builds under Wine on Linux. Build it from runner/:

docker build -f runner/Dockerfile -t psp-wine-runner .
docker run -d --name psp-wine-runner --restart unless-stopped \
  -e GITEA_INSTANCE_URL=https://git.ceaf.dev \
  -e GITEA_RUNNER_REGISTRATION_TOKEN=<registration-token> \
  -v /mnt/user/appdata/psp-wine-runner:/data \
  psp-wine-runner

The image ships Wine + Windows CPython 3.12 + PyInstaller, and registers itself with the windows:host label. (If you have a real Windows machine instead, register a native act_runner.exe with the same windows label.)

First-time setup

  1. Put your DAT files in data/dats/:

    • games: Redump Sony - PlayStation Portable (authoritative hashes) and No-Intro Sony - PlayStation Portable (supplies the serials Redump lacks — keeps serial-based matching working).
    • videos: No-Intro Sony - PlayStation Portable (UMD Video) (Redump has no UMD Video DAT).

  2. Start the app — any DATs missing from the library are ingested automatically at startup (no manual step). You can also re-ingest on demand:

    python3 -m app.ingest

  3. Point Settings → General → Games/Videos directory at your disc folders (or set PSP_GAMES_ROOTS / PSP_VIDEO_ROOTS).

  4. (Optional) PS1 conversion — copy any PSN PS1 classic EBOOT.PBP to data/base.PBP (it supplies Sony's encrypted loader). Then set the PS1 directory, PS1 output directory and PS1 base.pbp in Settings → General and use the PS1 tab (see PS1 support).

Discs that aren't in the Redump DATs can be added to data/supplement.json (name, region, crc32, sha1, md5) so the tool still recognizes them.

Nomenclature

Title (Region) (tags) (Serial).iso — leading articles stay in front (The Sims 2), and revision/disc tags are preserved ((v1.03), (Disc 1)). Discs are matched by hash first (most precise), then serial, then filename.

Directory layout

  • data/dats/ — clrmamepro DAT files (games + videos).
  • data/supplement.json — extra discs not in Redump.
  • data/credentials.json — cover/metadata API keys (IGDB, TheMovieDB, ScreenScraper, Google Custom Search). Environment variables take precedence.
  • data/manual_covers/games|videos/ — your own cover images.
  • data/covers/ — checked-in assets: the UMD video/game cover templates, libretro-index.json + openvgdb-index.json catalogs, libretro-covers.zip + openvgdb-covers.zip offline cover bundles, and the legacy cover cache.
  • data/base.PBP — a PSN PS1 classic EBOOT (any game) supplying STARTDAT
    • a base PARAM.SFO/icon for PS1 conversion (git-ignored, copyrighted).
  • data/loader/DATA.PSP + data/loader/KEYS.BIN — bundled plain-ELF POPS loader and NPDRM version key (from iPoPS, MIT).
  • data/ps1_templates/ — optional PS1 Classics art templates (ICON0.PNG case frame, PIC1.PNG launch background); the app falls back to drawn art when they're absent.
  • data/nps/ — NoPayStation TSV lists (cached here on first store scan in dev; a packaged build stores them under the state dir instead).
  • State dir (~/.local/state/psp-preserve/ on Linux, %LOCALAPPDATA%\PSPPreserve on Windows, or $PSP_STATE_DIR):
    • library.sqlite — the catalog + library database.
    • covers/ — downloaded cover cache.
    • data/ — user-writable data in packaged builds (credentials, manual covers, PS1 templates, NPS TSV cache).

SQLite is kept on a local disk because SQLite locking is unreliable over network filesystems. Read-only resources (DATs, the POPS loader, icons) are bundled with the app; user-writable data lives in the state dir.

Configuration

Everything is driven by environment variables, with GUI settings (stored in QSettings under PSPPreserve) taking precedence where noted.

Variable Default Purpose
PSP_DATA_DIR ./data data root (DATs, credentials, manual covers)
PSP_DATS_DIR $PSP_DATA_DIR/dats DAT folder
PSP_STATE_DIR ~/.local/state/psp-preserve local state root
PSP_DB_PATH $PSP_STATE_DIR/library.sqlite SQLite library
PSP_COVERS_DIR $PSP_STATE_DIR/covers cover cache
PSP_GAMES_ROOTS /data/umd_games comma-separated game disc roots
PSP_VIDEO_ROOTS /data/umd_videos comma-separated video disc roots
PSP_PS1_ROOTS /data/ps1 comma-separated PS1 disc roots
PSP_PS1_OUTPUT_DIR /data/ps1_converted converted EBOOT.PBP output folder
PSP_SCAN_WORKERS 1 parallel hashing workers
PSP_STORE_DIR /data/store local 7z store folder
PSP_STORE_SERVER https://psp.example.com store server base URL
PSP_STORE_TOKEN (empty) store server bearer token
PSP_ARCHIVE_ITEM psp-redump-roms321.com archive.org item identifier
PSP_ARCHIVE_LOGGED_IN_SIG (empty) archive.org logged-in-sig cookie
PSP_ARCHIVE_LOGGED_IN_USER (empty) archive.org logged-in-user cookie
PSP_NPS_DIR <state>/nps NoPayStation output folder
PSP_NPS_TSV_BASE https://nopaystation.com/tsv/ NoPayStation TSV base URL
PSP_API_TOKEN (empty) server API bearer token (unset = open)
PSP_USER_DATA_DIR <state>/data (packaged) / ./data (dev) user-writable data (credentials, manual covers, PS1 templates, NPS cache)
PSP_CREDENTIALS $PSP_USER_DATA_DIR/credentials.json credentials file
PSP_LIBRETRO_COVERS_URL (empty) direct URL for libretro-covers.zip (else derived from store server)
PSP_OVGDB_COVERS_URL (empty) direct URL for openvgdb-covers.zip (else derived from store server)

Cover/metadata keys can also be set as env vars: IGDB_CLIENT_ID, IGDB_CLIENT_SECRET, TMDB_API_KEY, GOOGLE_API_KEY, GOOGLE_CX, SCREENSCRAPER_DEVID, SCREENSCRAPER_DEVPASSWORD, SCREENSCRAPER_SSID, SCREENSCRAPER_SSPASSWORD, SCREENSCRAPER_SOFTNAME.

Cover art

Games are resolved in this order (offline-first, then network):

  1. Manual covers — images you drop in data/manual_covers/games/ (fuzzy-matched; strips (Region), handles Title, The).
  2. libretro-thumbnails (offline bundle) — full PSP box art from data/covers/libretro-covers.zip. If that zip is absent, it downloads from GitHub and caches instead.
  3. OpenVGDB (offline bundle) — full covers from data/covers/openvgdb-covers.zip.
  4. ScreenScraper — full box art + rich metadata; needs a free dev ID/password.
  5. Wikipedia — keyless fallback.
  6. IGDB — needs a Twitch/IGDB client ID + secret. IGDB only stores cropped 3:4 key art, so the app composites it into the PSP game cover template (data/covers/UMD_GAME_COVER_TEMPLATE.png) to restore the full cover.

UMD videos have no public box-art API. The manual Scrape… dialog offers three sources: TheMovieDB posters, Wikipedia image search (keyless), and Google Images (needs a Google API key + Search Engine ID in Settings → Cover sources). The pick is composited into data/covers/UMD_COVER_TEMPLATE.png. Manual covers in data/manual_covers/videos/ always win.

Offline bundles & auto-download

The two bundles (libretro-covers.zip, openvgdb-covers.zip) are shipped in data/covers/, so a fresh install works with no internet and no API keys. If either is missing at startup, the app downloads it from the store server's /api/covers/<name>.zip endpoint (see Server setup).

Rebuild them yourself:

python3 -m app.libretro          # libretro-thumbnails -> libretro-covers.zip
python3 -m app.openvgdb          # OpenVGDB/GameFAQs  -> openvgdb-covers.zip

(app.openvgdb needs curl_cffi to pass GameFAQs' Cloudflare challenge.)

PS1 support

The PS1 tab scans a folder of PlayStation 1 rips and converts them into PSP EBOOT.PBP files with a pure-Python popstation converter (no external tools). PS1 discs (pending) and converted EBOOTs also show up in the Library tab (filter by PS1), where you can convert a disc or copy an EBOOT to the PSP.

  • Source formats — .iso, .img, .bin (with a matching .cue for multi-track discs).
  • Serial detection — reads SYSTEM.CNF to derive the game ID (e.g. SLUS-00001) and writes it into PARAM.SFO + the PSAR header.
  • base.pbp — conversion needs a PSN PS1 classic EBOOT.PBP (any game) to supply the STARTDAT block (boot logo + PGD) and a base PARAM.SFO/icon. Copy one to data/base.PBP and point Settings → General → PS1 base.pbp at it. Any PS1 classic works structurally — the converter finds STARTDAT at PSAR + p1 regardless of PSAR alignment.
  • POPS loader — the DATA.PSP section uses a bundled plain-ELF loader (data/loader/DATA.PSP, from iPoPS, MIT) so the EBOOT boots on modern CFW without the base PBP's encrypted loader failing to link (80020148). The base PBP's own loader is only used as a fallback when the bundle is missing.
  • Output — a folder <out>/<Game ID>/ containing EBOOT.PBP (raw-DEFLATE compressed PSAR) plus a KEYS.BIN (NPDRM version key used to decrypt the STARTDAT PGD), ready to copy to <PSP>/PSP/GAME/<Game ID>/. Compatible with CFW POPS (ARK-4/ARK-5).
  • Artwork — the converter fetches the game's real cover art (libretro / ScreenScraper / IGDB / Wikipedia, reusing the local cover cache) and builds official-style PS1 Classics art:
    • ICON0 (80×80) — the cover composited into the bundled case frame (data/ps1_templates/ICON0.PNG) with the PlayStation logo + wordmark;
    • PIC0 (310×180) — a PSN-style info board with the game title, a description from IGDB, an ESRB rating badge and PS3/PSP platform badges;
    • PIC1 (480×272) — the bundled launch background (data/ps1_templates/PIC1.PNG). If the template images are missing the app falls back to drawn art, and to the base icon when no cover is found.

Configure the folders in Settings → General (or PSP_PS1_ROOTS / PSP_PS1_OUTPUT_DIR). Current limits: single-disc games only (no multi-disc PBP); disc data is DEFLATE-compressed (level 1 by default, --complevel 0-9).

Store — four sources

The Store tab downloads missing discs from a Redump 7z archive collection (Name (Region) (tags).7z, containing the .iso/.cso inside). It has three sources, selectable in the tab:

1. Local folder

Point Settings → Store → Local store dir (or PSP_STORE_DIR) at a folder of .7z/.zip archives. The app scans the folder, lists the archives, and on download:

  1. reads the archive listing with 7z to find the disc image inside,
  2. copies the archive to a temp dir and extracts the image with 7z,
  3. moves the image into your games/videos directory.

2. Server (HTTP API)

See Server setup. The app talks to GET /api/store (list) and GET /api/store/download/<name> (stream), using your PSP_API_TOKEN as Authorization: Bearer <token>.

3. Archive.org

Point Settings → Store → Archive.org item (or PSP_ARCHIVE_ITEM) at an archive.org item identifier (or a full details/download URL). The app:

  1. lists the item via https://archive.org/metadata/<item>,
  2. downloads archives from https://archive.org/download/<item>/<file>,
  3. extracts the disc image and moves it into your library.

If the item's files are marked private (login required), set the logged-in-sig and logged-in-user cookies in Settings → Store (grab them from a logged-in archive.org browser session). Without them, private files are skipped — and you'll get a clear error if everything in the item is private.

4. NoPayStation

The NoPayStation source lists PSN content from NoPayStation, selectable from the content-type dropdown next to the source selector:

  • PSP Games (PSP_GAMES.tsv) — includes minis (also listed separately in the Type column), PC Engine and NeoGeo releases.
  • PS1 Classics (PSX_GAMES.tsv)
  • PSP Themes (PSP_THEMES.tsv)
  • PSP DLC (PSP_DLCS.tsv)

Requirements & behaviour:

  1. .pkg files are decrypted by the built-in pure-Python decryptor (app/pkg.py, powered by pycryptodome for speed) — no Wine and no pkg2zip needed.
  2. Set NoPayStation output dir in Settings → Store (default $PSP_STATE_DIR/nps).
  3. Downloads land in the NPS folder and appear in the Library:
    • PSP game → PSP/<TITLE_ID>/EBOOT.PBP (psp_kind="eboot").
    • PS1 Classic → PSX/<TITLE_ID>/EBOOT.PBP (+ DOCUMENT.DAT, KEYS.BIN).
    • Theme → THEME/<TITLE_ID>/<content_id>.ptf (the packaged PTF is EDAT-unwrapped to a raw theme file).
    • DLC → DLC/<GAME_ID>/<content_id>.edat (installed as-is into the owning game's folder).
  4. Copying to the PSP puts games/PS1 under /PSP/GAME/ (optionally nested in a Category Lite folder), themes under /PSP/THEME/, and DLC under /PSP/GAME/<GAME_ID>/.
  5. The TSV lists are cached in data/nps/ and refreshed on each store scan (drop the .tsv files there manually to work offline).

DRM note: NPS content is PSN-DRM'd. To run it on the PSP you need the npdrm_free plugin (ARK and most CFW already ship it). The desktop app only downloads and decrypts; it doesn't strip DRM.

Covers/metadata: themes and DLC have no cover art or game metadata of their own — the app scrapes them like games (IGDB/libretro/etc.), which can return unrelated results. Downloaded themes keep their embedded preview as the primary cover (used instead of a scraped image when available).

Server setup

The catalog + store API is a single-file, stdlib-only HTTP server (app/api.py) — no pip install needed on the server.

PSP_DATA_DIR=/data \
PSP_DATS_DIR=/data/dats \
PSP_DB_PATH=/data/library.sqlite \
PSP_COVERS_DIR=/data/covers \
PSP_GAMES_ROOTS=/data/umd_games \
PSP_VIDEO_ROOTS=/data/umd_videos \
PSP_STORE_DIR=/data/store \
PSP_API_TOKEN=<long-random-secret> \
python3 -m app.api --host 0.0.0.0 --port 8000

Endpoints:

Endpoint Description
GET /api/summary release counts per system
GET /api/releases?system=&q=&limit=&offset= search the catalog
GET /api/releases/<id> one release + its rom hashes
GET /api/library?system=&refresh= your scanned library
GET /api/cover/<id> cover image (fetched on demand)
GET /api/covers/<name>.zip offline cover bundle (public)
GET /api/store list store archives (token)
GET /api/store/download/<name> stream one archive (token)

Token protection: set PSP_API_TOKEN; the client sends Authorization: Bearer <token>. Leave it unset to allow anonymous access (not recommended).

Docker (Unraid)

docker-compose.yml builds the API image and attaches it to Unraid's br0 macvlan bridge so it gets its own LAN IP (e.g. 192.168.1.100, plain HTTP on 8000):

# in /path/to/psp-preserve/
cp .env.example .env   # then edit: LAN IP, share paths, API token
docker compose up -d --build

The compose file reads the LAN IP and Unraid share paths from .env and mounts them into /data/store, /data/umd_games and /data/umd_videos (the store is mounted read-only).

TLS (optional)

Caddyfile is a ready-made reverse proxy for https://psp.example.com with automatic Let's Encrypt certificates. To enable it, add the caddy service to the compose file, give it its own br0 IP, point it at the API, and forward ports 80/443 on your router to it.

Client

In the GUI: Settings → Store → enter the Server URL (e.g. http://<server-ip>:8000 or https://psp.example.com) and the same API token, then pick Server in the Store tab's source selector. Use Archive.org or Local folder the same way.

PSP Sync

Connect a PSP over USB (or mount a UMD) and the PSP Sync tab detects it:

  • Games / UMD Videos / Homebrew-PSN / Themes / DLC / PKG — tables of what's installed, with delete-from-PSP (checkboxes and right-click).
  • UMD — reads PARAM.SFO from a mounted UMD, shows the title/serial, and can dump the disc image into your library.
  • Sync from Library — copy discs from your library onto the PSP, optionally into a Category Lite category (set in Settings → PSP → ISO category). NoPayStation PSP games install under /PSP/GAME, optionally nested in the PSP game category; PS1 Classics use the PS1 category.

Saves sync

The Saves tab shows save-data across three locations — PSP (USB), PPSSPP, and a Cloud folder — and syncs both ways. Save signatures are cached (keyed by size + mtime) so rescanning is fast, and an Auto-sync option runs the sync automatically whenever a PSP is connected.

CLI tools

# scan + report (dry-run)
python3 -m app.scan --roots /path/to/umd_games,/path/to/umd_videos

# scan + apply renames
python3 -m app.scan --roots /path/to/umd_games,/path/to/umd_videos --apply

# ingest DAT files into the library DB
python3 -m app.ingest --dats data/dats --db data/library.sqlite

# (re)build the offline cover bundles (see Cover art)
python3 -m app.libretro
python3 -m app.openvgdb

# convert one PS1 disc to EBOOT.PBP (CLI)
python3 -m app.ps1_convert --source /path/to/game.cue --title "Game" \
  --game-id SLUS00001 --base-pbp data/base.PBP --out /out/Game.pbp

python3 -m app.scan --json prints machine-readable results, and --quick skips full-file hashing (serial/title match only).

Reference in New Issue View Git Blame Copy Permalink