Files
psp-preservation-tool/README.md
T

421 lines
19 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# PSP Preservation Tool
the name is terrible, i know.
![AI Generated](https://raw.githubusercontent.com/dotnetdave/ai-usage-badges/refs/heads/main/badges/png/1x/ai-generated.png)
Native Linux GUI (PySide6) for preserving PSP UMD **games** and **videos** as
`.iso` files. It scans your discs, verifies them against the full Redump catalog
(Redump DATs), reports what's missing, and retags/renames discs to a canonical
scheme. It also fetches cover art and metadata, syncs discs and save-data with a
real PSP, can pull missing discs from your own Redump 7z archive collection, and
converts PlayStation 1 ISO/CUE/BIN rips into PSP `EBOOT.PBP` homebrew.
![PSP Preservation Tool](img/qt-gui.png)
* Use default app theme for Qt.
## Features
- **Library** — scan folders of `.iso` discs, verify against the DAT catalog,
browse with cover art, search/filter, see per-game details (developer,
publisher, genre, release date, synopsis, trailer), and launch discs in PPSSPP.
- **Store** — browse and download discs from four sources: a local 7z folder,
your own HTTP store server, an archive.org item, or NoPayStation (PSP/PS1
PSN `.pkg` content) (see [Store](#store--archiveorg-and-local-folder)).
- **Queue** — background transfers with progress, cancel, per-row delete, etc.
- **PSP Sync** — detect a connected PSP (USB) or a mounted UMD, inventory what's
installed (games / UMD videos / homebrew / PKG), copy discs to the PSP, delete
from the PSP, and dump a UMD to the library.
- **Saves** — two-way sync of save-data between the PSP, PPSSPP, and a cloud
folder.
- **PS1** — scan a folder of PlayStation 1 discs (ISO/BIN/CUE), read each disc's
serial, convert to PSP `EBOOT.PBP` with a pure-Python popstation converter,
and track already-converted games.
- **Settings** — directories, cover-source API keys, store/server config, and
misc options.
## Requirements
- Linux + Python 3
- `PySide6` + `pycryptodome` (`pip install -r requirements.txt`)
- `7z` (p7zip) — used to peek/extract disc images from store archives
- PPSSPP (optional) — to launch discs from the details panel
NoPayStation `.pkg` files are decrypted by the bundled **pure-Python
decryptor** (`app/pkg.py`) — no Wine or `pkg2zip` required
(see [NoPayStation](#4-nopaystation)).
## Install
```bash
python3 -m venv .venv # keep the venv on a local disk, not an SMB mount
. .venv/bin/activate
pip install -r requirements.txt
```
## Run the GUI
```bash
python3 -m app.main
```
## First-time setup
1. Put your DAT files in `data/dats/`:
- games: Redump `Sony - PlayStation Portable` (authoritative hashes)
**and** No-Intro `Sony - PlayStation Portable` (supplies the serials
Redump lacks — keeps serial-based matching working).
- videos: No-Intro `Sony - PlayStation Portable (UMD Video)` (Redump has
no UMD Video DAT).
2. Ingest them into the local SQLite library:
```bash
python3 -m app.ingest
```
3. Open the GUI and point **Settings → General → Games/Videos directory** at
your disc folders (or set `PSP_GAMES_ROOTS` / `PSP_VIDEO_ROOTS`).
4. (Optional) PS1 conversion — copy any PSN PS1 classic `EBOOT.PBP` to
`data/base.PBP` (it supplies Sony's encrypted loader). Then set the **PS1
directory**, **PS1 output directory** and **PS1 base.pbp** in
**Settings → General** and use the **PS1** tab (see [PS1 support](#ps1-support)).
Discs that aren't in the Redump DATs can be added to `data/supplement.json`
(`name`, `region`, `crc32`, `sha1`, `md5`) so the tool still recognizes them.
## Nomenclature
`Title (Region) (tags) (Serial).iso` — leading articles stay in front
(`The Sims 2`), and revision/disc tags are preserved (`(v1.03)`, `(Disc 1)`).
Discs are matched by hash first (most precise), then serial, then filename.
## Directory layout
- `data/dats/` — clrmamepro DAT files (games + videos).
- `data/supplement.json` — extra discs not in Redump.
- `data/credentials.json` — cover/metadata API keys (IGDB, TheMovieDB,
ScreenScraper, Google Custom Search). Environment variables take precedence.
- `data/manual_covers/games|videos/` — your own cover images.
- `data/covers/` — checked-in assets: the UMD video/game cover templates,
`libretro-index.json` + `openvgdb-index.json` catalogs, `libretro-covers.zip` +
`openvgdb-covers.zip` offline cover bundles, and the legacy cover cache.
- `data/base.PBP` — a PSN PS1 classic EBOOT (any game) supplying `STARTDAT`
+ a base `PARAM.SFO`/icon for PS1 conversion (git-ignored, copyrighted).
- `data/loader/DATA.PSP` + `data/loader/KEYS.BIN` — bundled plain-ELF POPS
loader and NPDRM version key (from iPoPS, MIT).
- `data/ps1_templates/` — optional PS1 Classics art templates
(`ICON0.PNG` case frame, `PIC1.PNG` launch background); the app falls back
to drawn art when they're absent.
- `data/nps/` — NoPayStation TSV lists (cached here on first store scan; drop
`PSP_GAMES.tsv` / `PSX_GAMES.tsv` manually to work offline).
- `data/bin/pkg2zip.exe` — bundled `pkg2zip` binary (MIT, by mmozeiko) kept
only as an optional fallback decryptor; the app normally uses its own
pure-Python decryptor (`app/pkg.py`) and never needs Wine.
See `data/bin/ATTRIBUTION.md`.
- State dir (`~/.local/state/psp-preserve/` by default, or `$XDG_STATE_HOME/psp-preserve`):
- `library.sqlite` — the catalog + library database.
- `covers/` — downloaded cover cache.
SQLite is kept on a local disk because SQLite locking is unreliable over
network filesystems; the DATs, manual covers and credentials stay in `data/`
(which may be a network share).
## Configuration
Everything is driven by environment variables, with GUI settings (stored in
QSettings under `PSPPreserve`) taking precedence where noted.
| Variable | Default | Purpose |
|---|---|---|
| `PSP_DATA_DIR` | `./data` | data root (DATs, credentials, manual covers) |
| `PSP_DATS_DIR` | `$PSP_DATA_DIR/dats` | DAT folder |
| `PSP_STATE_DIR` | `~/.local/state/psp-preserve` | local state root |
| `PSP_DB_PATH` | `$PSP_STATE_DIR/library.sqlite` | SQLite library |
| `PSP_COVERS_DIR` | `$PSP_STATE_DIR/covers` | cover cache |
| `PSP_GAMES_ROOTS` | `/data/umd_games` | comma-separated game disc roots |
| `PSP_VIDEO_ROOTS` | `/data/umd_videos` | comma-separated video disc roots |
| `PSP_PS1_ROOTS` | `/data/ps1` | comma-separated PS1 disc roots |
| `PSP_PS1_OUTPUT_DIR` | `/data/ps1_converted` | converted EBOOT.PBP output folder |
| `PSP_SCAN_WORKERS` | `1` | parallel hashing workers |
| `PSP_STORE_DIR` | `/media/downloads/Sony - PlayStation Portable` | local 7z store folder |
| `PSP_STORE_SERVER` | `https://psp.example.com` | store server base URL |
| `PSP_STORE_TOKEN` | *(empty)* | store server bearer token |
| `PSP_ARCHIVE_ITEM` | `psp-redump-roms321.com` | archive.org item identifier |
| `PSP_ARCHIVE_LOGGED_IN_SIG` | *(empty)* | archive.org `logged-in-sig` cookie |
| `PSP_ARCHIVE_LOGGED_IN_USER` | *(empty)* | archive.org `logged-in-user` cookie |
| `PSP_NPS_DIR` | `<state>/nps` | NoPayStation output folder |
| `PSP_NPS_TSV_BASE` | `https://nopaystation.com/tsv/` | NoPayStation TSV base URL |
| `PSP_PKG2ZIP_PATH` | *(empty)* | pkg2zip binary path (else auto-detected) |
| `PSP_API_TOKEN` | *(empty)* | server API bearer token (unset = open) |
| `PSP_CREDENTIALS` | `$PSP_DATA_DIR/credentials.json` | credentials file |
| `PSP_LIBRETRO_COVERS_URL` | *(empty)* | direct URL for `libretro-covers.zip` (else derived from store server) |
| `PSP_OVGDB_COVERS_URL` | *(empty)* | direct URL for `openvgdb-covers.zip` (else derived from store server) |
Cover/metadata keys can also be set as env vars: `IGDB_CLIENT_ID`,
`IGDB_CLIENT_SECRET`, `TMDB_API_KEY`, `GOOGLE_API_KEY`, `GOOGLE_CX`,
`SCREENSCRAPER_DEVID`, `SCREENSCRAPER_DEVPASSWORD`, `SCREENSCRAPER_SSID`,
`SCREENSCRAPER_SSPASSWORD`, `SCREENSCRAPER_SOFTNAME`.
## Cover art
Games are resolved in this order (offline-first, then network):
1. **Manual covers** — images you drop in `data/manual_covers/games/`
(fuzzy-matched; strips `(Region)`, handles `Title, The`).
2. **libretro-thumbnails (offline bundle)** — full PSP box art from
`data/covers/libretro-covers.zip`. If that zip is absent, it downloads from
GitHub and caches instead.
3. **OpenVGDB (offline bundle)** — full covers from
`data/covers/openvgdb-covers.zip`.
4. **ScreenScraper** — full box art + rich metadata; needs a free dev
ID/password.
5. **Wikipedia** — keyless fallback.
6. **IGDB** — needs a Twitch/IGDB client ID + secret. IGDB only stores cropped
3:4 key art, so the app composites it into the PSP game cover template
(`data/covers/UMD_GAME_COVER_TEMPLATE.png`) to restore the full cover.
UMD **videos** have no public box-art API. The manual **Scrape…** dialog offers
three sources: TheMovieDB posters, Wikipedia image search (keyless), and Google
Images (needs a Google API key + Search Engine ID in **Settings → Cover
sources**). The pick is composited into `data/covers/UMD_COVER_TEMPLATE.png`.
Manual covers in `data/manual_covers/videos/` always win.
### Offline bundles & auto-download
The two bundles (`libretro-covers.zip`, `openvgdb-covers.zip`) are shipped in
`data/covers/`, so a fresh install works with **no internet and no API keys**.
If either is missing at startup, the app downloads it from the store server's
`/api/covers/<name>.zip` endpoint (see [Server setup](#server-setup)).
Rebuild them yourself:
```bash
python3 -m app.libretro # libretro-thumbnails -> libretro-covers.zip
python3 -m app.openvgdb # OpenVGDB/GameFAQs -> openvgdb-covers.zip
```
(`app.openvgdb` needs `curl_cffi` to pass GameFAQs' Cloudflare challenge.)
## PS1 support
The **PS1** tab scans a folder of PlayStation 1 rips and converts them into PSP
`EBOOT.PBP` files with a pure-Python popstation converter (no external tools).
PS1 discs (pending) and converted EBOOTs also show up in the **Library** tab
(filter by **PS1**), where you can convert a disc or copy an EBOOT to the PSP.
- **Source formats** — `.iso`, `.img`, `.bin` (with a matching `.cue` for
multi-track discs).
- **Serial detection** — reads `SYSTEM.CNF` to derive the game ID (e.g.
`SLUS-00001`) and writes it into `PARAM.SFO` + the PSAR header.
- **base.pbp** — conversion needs a PSN PS1 classic `EBOOT.PBP` (any game) to
supply the `STARTDAT` block (boot logo + PGD) and a base `PARAM.SFO`/icon.
Copy one to `data/base.PBP` and point **Settings → General → PS1 base.pbp**
at it. Any PS1 classic works structurally — the converter finds `STARTDAT`
at `PSAR + p1` regardless of PSAR alignment.
- **POPS loader** — the `DATA.PSP` section uses a bundled plain-ELF loader
(`data/loader/DATA.PSP`, from iPoPS, MIT) so the EBOOT boots on modern CFW
without the base PBP's encrypted loader failing to link (`80020148`). The
base PBP's own loader is only used as a fallback when the bundle is missing.
- **Output** — a folder `<out>/<Game ID>/` containing `EBOOT.PBP` (raw-DEFLATE
compressed PSAR) plus a `KEYS.BIN` (NPDRM version key used to decrypt the
`STARTDAT` PGD), ready to copy to `<PSP>/PSP/GAME/<Game ID>/`. Compatible
with CFW POPS (ARK-4/ARK-5).
- **Artwork** — the converter fetches the game's real cover art (libretro /
ScreenScraper / IGDB / Wikipedia, reusing the local cover cache) and builds
official-style PS1 Classics art:
- `ICON0` (80×80) — the cover composited into the bundled case frame
(`data/ps1_templates/ICON0.PNG`) with the PlayStation logo + wordmark;
- `PIC0` (310×180) — a PSN-style info board with the game title, a
description from IGDB, an ESRB rating badge and PS3/PSP platform badges;
- `PIC1` (480×272) — the bundled launch background
(`data/ps1_templates/PIC1.PNG`).
If the template images are missing the app falls back to drawn art, and to
the base icon when no cover is found.
Configure the folders in **Settings → General** (or `PSP_PS1_ROOTS` /
`PSP_PS1_OUTPUT_DIR`). Current limits: single-disc games only (no multi-disc
PBP); disc data is DEFLATE-compressed (level 1 by default, `--complevel 0-9`).
## Store — archive.org and local folder
The **Store** tab downloads missing discs from a Redump 7z archive collection
(`Name (Region) (tags).7z`, containing the `.iso`/`.cso` inside). It has three
sources, selectable in the tab:
### 1. Local folder
Point **Settings → Store → Local store dir** (or `PSP_STORE_DIR`) at a folder of
`.7z`/`.zip` archives. The app scans the folder, lists the archives, and on
download:
1. reads the archive listing with `7z` to find the disc image inside,
2. copies the archive to a temp dir and extracts the image with `7z`,
3. moves the image into your games/videos directory.
### 2. Server (HTTP API)
See [Server setup](#server-setup). The app talks to `GET /api/store` (list) and
`GET /api/store/download/<name>` (stream), using your `PSP_API_TOKEN` as
`Authorization: Bearer <token>`.
### 3. Archive.org
Point **Settings → Store → Archive.org item** (or `PSP_ARCHIVE_ITEM`) at an
archive.org item identifier (or a full `details`/`download` URL). The app:
1. lists the item via `https://archive.org/metadata/<item>`,
2. downloads archives from `https://archive.org/download/<item>/<file>`,
3. extracts the disc image and moves it into your library.
If the item's files are marked **private** (login required), set the
`logged-in-sig` and `logged-in-user` cookies in **Settings → Store** (grab them
from a logged-in archive.org browser session). Without them, private files are
skipped — and you'll get a clear error if *everything* in the item is private.
### 4. NoPayStation
The **NoPayStation** source lists PSN content from
[NoPayStation](https://nopaystation.com/), selectable from the content-type
dropdown next to the source selector:
- **PSP Games** (`PSP_GAMES.tsv`) — includes **minis** (also listed separately
in the `Type` column), PC Engine and NeoGeo releases.
- **PS1 Classics** (`PSX_GAMES.tsv`)
- **PSP Themes** (`PSP_THEMES.tsv`)
- **PSP DLC** (`PSP_DLCS.tsv`)
Requirements & behaviour:
1. `.pkg` files are decrypted by the built-in **pure-Python decryptor**
(`app/pkg.py`, powered by `pycryptodome` for speed) — no Wine and no
`pkg2zip` install needed. A bundled `pkg2zip.exe` is kept as a fallback
only, and only runs if Python decryption fails and Wine is present.
2. Set **NoPayStation output dir** in **Settings → Store** (default
`$PSP_STATE_DIR/nps`).
3. Downloads land in the NPS folder and appear in the **Library**:
- PSP game → `PSP/<TITLE_ID>/EBOOT.PBP` (`psp_kind="eboot"`).
- PS1 Classic → `PSX/<TITLE_ID>/EBOOT.PBP` (+ `DOCUMENT.DAT`, `KEYS.BIN`).
- Theme → `THEME/<TITLE_ID>/<content_id>.ptf` (the packaged PTF is
EDAT-unwrapped to a raw theme file).
- DLC → `DLC/<GAME_ID>/<content_id>.edat` (installed as-is into the owning
game's folder).
4. Copying to the PSP puts games/PS1 under `/PSP/GAME/` (optionally nested in
a Category Lite folder), themes under `/PSP/THEME/`, and DLC under
`/PSP/GAME/<GAME_ID>/`.
5. The TSV lists are cached in `data/nps/` and refreshed on each store scan
(drop the `.tsv` files there manually to work offline).
> **DRM note:** NPS content is PSN-DRM'd. To *run* it on the PSP you need the
> [`npdrm_free`](https://github.com/qwikrazor87/npdrm_free) plugin (ARK and most
> CFW already ship it). The desktop app only downloads and decrypts; it doesn't
> strip DRM.
>
> **Covers/metadata:** themes and DLC have no cover art or game metadata, so
> the app never scrapes IGDB/libretro/etc. for them (their Library rows show a
> placeholder icon instead of a mis-matched game cover).
## Server setup
The catalog + store API is a single-file, stdlib-only HTTP server
(`app/api.py`) — no pip install needed on the server.
```bash
PSP_DATA_DIR=/data \
PSP_DATS_DIR=/data/dats \
PSP_DB_PATH=/data/library.sqlite \
PSP_COVERS_DIR=/data/covers \
PSP_GAMES_ROOTS=/data/umd_games \
PSP_VIDEO_ROOTS=/data/umd_videos \
PSP_STORE_DIR=/data/store \
PSP_API_TOKEN=<long-random-secret> \
python3 -m app.api --host 0.0.0.0 --port 8000
```
Endpoints:
| Endpoint | Description |
|---|---|
| `GET /api/summary` | release counts per system |
| `GET /api/releases?system=&q=&limit=&offset=` | search the catalog |
| `GET /api/releases/<id>` | one release + its rom hashes |
| `GET /api/library?system=&refresh=` | your scanned library |
| `GET /api/cover/<id>` | cover image (fetched on demand) |
| `GET /api/covers/<name>.zip` | offline cover bundle (public) |
| `GET /api/store` | list store archives *(token)* |
| `GET /api/store/download/<name>` | stream one archive *(token)* |
Token protection: set `PSP_API_TOKEN`; the client sends
`Authorization: Bearer <token>`. Leave it unset to allow anonymous access
(not recommended).
### Docker (Unraid)
`docker-compose.yml` builds the API image and attaches it to Unraid's `br0`
macvlan bridge so it gets its own LAN IP (e.g. `192.168.1.100`, plain HTTP on 8000):
```bash
# in /path/to/psp-preserve/ — adjust the volume paths for your shares
printf 'PSP_API_TOKEN=<long-random-secret>\n' > .env
docker compose up -d --build
```
The compose file mounts your Unraid shares into `/data/store`, `/data/umd_games`
and `/data/umd_videos` (the store is mounted read-only).
### TLS (optional)
`Caddyfile` is a ready-made reverse proxy for `https://psp.example.com` with
automatic Let's Encrypt certificates. To enable it, add the `caddy` service to
the compose file, give it its own `br0` IP, point it at the API, and forward
ports 80/443 on your router to it.
### Client
In the GUI: **Settings → Store** → enter the **Server URL** (e.g.
`http://<server-ip>:8000` or `https://psp.example.com`) and the same **API
token**, then pick **Server** in the Store tab's source selector. Use
**Archive.org** or **Local folder** the same way.
## PSP Sync
Connect a PSP over USB (or mount a UMD) and the **PSP Sync** tab detects it:
- **Games / UMD Videos / Homebrew-PSN / Themes / DLC / PKG** — tables of what's
installed, with delete-from-PSP (checkboxes and right-click).
- **UMD** — reads `PARAM.SFO` from a mounted UMD, shows the title/serial, and
can dump the disc image into your library.
- **Sync from Library** — copy discs from your library onto the PSP, optionally
into a Category Lite category (set in **Settings → PSP → ISO category**).
NoPayStation PSP games install under `/PSP/GAME`, optionally nested in the
**PSP game category**; PS1 Classics use the **PS1 category**.
## Saves sync
The **Saves** tab shows save-data across three locations — **PSP** (USB),
**PPSSPP**, and a **Cloud folder** — and syncs both ways. Save signatures are
cached (keyed by size + mtime) so rescanning is fast, and an **Auto-sync**
option runs the sync automatically whenever a PSP is connected.
## CLI tools
```bash
# scan + report (dry-run)
python3 -m app.scan --roots /path/to/umd_games,/path/to/umd_videos
# scan + apply renames
python3 -m app.scan --roots /path/to/umd_games,/path/to/umd_videos --apply
# ingest DAT files into the library DB
python3 -m app.ingest --dats data/dats --db data/library.sqlite
# (re)build the offline cover bundles (see Cover art)
python3 -m app.libretro
python3 -m app.openvgdb
# convert one PS1 disc to EBOOT.PBP (CLI)
python3 -m app.ps1_convert --source /path/to/game.cue --title "Game" \
--game-id SLUS00001 --base-pbp data/base.PBP --out /out/Game.pbp
```
`python3 -m app.scan --json` prints machine-readable results, and `--quick`
skips full-file hashing (serial/title match only).