405 lines
18 KiB
Markdown
405 lines
18 KiB
Markdown
# PSP Preservation Tool
|
||
the name is terrible, i know.
|
||
|
||
|
||
|
||
Native Linux GUI (PySide6) for preserving PSP UMD **games** and **videos** as
|
||
`.iso` files. It scans your discs, verifies them against the full Redump catalog
|
||
(Redump DATs), reports what's missing, and retags/renames discs to a canonical
|
||
scheme. It also fetches cover art and metadata, syncs discs and save-data with a
|
||
real PSP, can pull missing discs from your own Redump 7z archive collection, and
|
||
converts PlayStation 1 ISO/CUE/BIN rips into PSP `EBOOT.PBP` homebrew.
|
||
|
||
|
||
* Use default app theme for Qt.
|
||
|
||
## Features
|
||
|
||
- **Library** — scan folders of `.iso` discs, verify against the DAT catalog,
|
||
browse with cover art, search/filter, see per-game details (developer,
|
||
publisher, genre, release date, synopsis, trailer), and launch discs in PPSSPP.
|
||
- **Store** — browse and download discs from four sources: a local 7z folder,
|
||
your own HTTP store server, an archive.org item, or NoPayStation (PSP/PS1
|
||
PSN `.pkg` content) (see [Store](#store--archiveorg-and-local-folder)).
|
||
- **Queue** — background transfers with progress, cancel, per-row delete, etc.
|
||
- **PSP Sync** — detect a connected PSP (USB) or a mounted UMD, inventory what's
|
||
installed (games / UMD videos / homebrew / PKG), copy discs to the PSP, delete
|
||
from the PSP, and dump a UMD to the library.
|
||
- **Saves** — two-way sync of save-data between the PSP, PPSSPP, and a cloud
|
||
folder.
|
||
- **PS1** — scan a folder of PlayStation 1 discs (ISO/BIN/CUE), read each disc's
|
||
serial, convert to PSP `EBOOT.PBP` with a pure-Python popstation converter,
|
||
and track already-converted games.
|
||
- **Settings** — directories, cover-source API keys, store/server config, and
|
||
misc options.
|
||
|
||
## Requirements
|
||
|
||
- Linux + Python 3
|
||
- `PySide6` + `pycryptodome` (`pip install -r requirements.txt`)
|
||
- `7z` (p7zip) — used to peek/extract disc images from store archives
|
||
- PPSSPP (optional) — to launch discs from the details panel
|
||
|
||
NoPayStation `.pkg` files are decrypted by the bundled **pure-Python
|
||
decryptor** (`app/pkg.py`) — no Wine or `pkg2zip` required
|
||
(see [NoPayStation](#4-nopaystation)).
|
||
|
||
## Install
|
||
|
||
```bash
|
||
python3 -m venv .venv # keep the venv on a local disk, not an SMB mount
|
||
. .venv/bin/activate
|
||
pip install -r requirements.txt
|
||
```
|
||
|
||
## Run the GUI
|
||
|
||
```bash
|
||
python3 -m app.main
|
||
```
|
||
|
||
## First-time setup
|
||
|
||
1. Put your DAT files in `data/dats/`:
|
||
- games: Redump `Sony - PlayStation Portable` (authoritative hashes)
|
||
**and** No-Intro `Sony - PlayStation Portable` (supplies the serials
|
||
Redump lacks — keeps serial-based matching working).
|
||
- videos: No-Intro `Sony - PlayStation Portable (UMD Video)` (Redump has
|
||
no UMD Video DAT).
|
||
2. Ingest them into the local SQLite library:
|
||
|
||
```bash
|
||
python3 -m app.ingest
|
||
```
|
||
|
||
3. Open the GUI and point **Settings → General → Games/Videos directory** at
|
||
your disc folders (or set `PSP_GAMES_ROOTS` / `PSP_VIDEO_ROOTS`).
|
||
4. (Optional) PS1 conversion — copy any PSN PS1 classic `EBOOT.PBP` to
|
||
`data/base.PBP` (it supplies Sony's encrypted loader). Then set the **PS1
|
||
directory**, **PS1 output directory** and **PS1 base.pbp** in
|
||
**Settings → General** and use the **PS1** tab (see [PS1 support](#ps1-support)).
|
||
|
||
Discs that aren't in the Redump DATs can be added to `data/supplement.json`
|
||
(`name`, `region`, `crc32`, `sha1`, `md5`) so the tool still recognizes them.
|
||
|
||
## Nomenclature
|
||
|
||
`Title (Region) (tags) (Serial).iso` — leading articles stay in front
|
||
(`The Sims 2`), and revision/disc tags are preserved (`(v1.03)`, `(Disc 1)`).
|
||
Discs are matched by hash first (most precise), then serial, then filename.
|
||
|
||
## Directory layout
|
||
|
||
- `data/dats/` — clrmamepro DAT files (games + videos).
|
||
- `data/supplement.json` — extra discs not in Redump.
|
||
- `data/credentials.json` — cover/metadata API keys (IGDB, TheMovieDB,
|
||
ScreenScraper, Google Custom Search). Environment variables take precedence.
|
||
- `data/manual_covers/games|videos/` — your own cover images.
|
||
- `data/covers/` — checked-in assets: the UMD video/game cover templates,
|
||
`libretro-index.json` + `openvgdb-index.json` catalogs, `libretro-covers.zip` +
|
||
`openvgdb-covers.zip` offline cover bundles, and the legacy cover cache.
|
||
- `data/base.PBP` — a PSN PS1 classic EBOOT (any game) supplying `STARTDAT`
|
||
+ a base `PARAM.SFO`/icon for PS1 conversion (git-ignored, copyrighted).
|
||
- `data/loader/DATA.PSP` + `data/loader/KEYS.BIN` — bundled plain-ELF POPS
|
||
loader and NPDRM version key (from iPoPS, MIT).
|
||
- `data/ps1_templates/` — optional PS1 Classics art templates
|
||
(`ICON0.PNG` case frame, `PIC1.PNG` launch background); the app falls back
|
||
to drawn art when they're absent.
|
||
- `data/nps/` — NoPayStation TSV lists (cached here on first store scan; drop
|
||
`PSP_GAMES.tsv` / `PSX_GAMES.tsv` manually to work offline).
|
||
- `data/bin/pkg2zip.exe` — bundled `pkg2zip` binary (MIT, by mmozeiko) kept
|
||
only as an optional fallback decryptor; the app normally uses its own
|
||
pure-Python decryptor (`app/pkg.py`) and never needs Wine.
|
||
See `data/bin/ATTRIBUTION.md`.
|
||
- State dir (`~/.local/state/psp-preserve/` by default, or `$XDG_STATE_HOME/psp-preserve`):
|
||
- `library.sqlite` — the catalog + library database.
|
||
- `covers/` — downloaded cover cache.
|
||
|
||
SQLite is kept on a local disk because SQLite locking is unreliable over
|
||
network filesystems; the DATs, manual covers and credentials stay in `data/`
|
||
(which may be a network share).
|
||
|
||
## Configuration
|
||
|
||
Everything is driven by environment variables, with GUI settings (stored in
|
||
QSettings under `PSPPreserve`) taking precedence where noted.
|
||
|
||
| Variable | Default | Purpose |
|
||
|---|---|---|
|
||
| `PSP_DATA_DIR` | `./data` | data root (DATs, credentials, manual covers) |
|
||
| `PSP_DATS_DIR` | `$PSP_DATA_DIR/dats` | DAT folder |
|
||
| `PSP_STATE_DIR` | `~/.local/state/psp-preserve` | local state root |
|
||
| `PSP_DB_PATH` | `$PSP_STATE_DIR/library.sqlite` | SQLite library |
|
||
| `PSP_COVERS_DIR` | `$PSP_STATE_DIR/covers` | cover cache |
|
||
| `PSP_GAMES_ROOTS` | `/data/umd_games` | comma-separated game disc roots |
|
||
| `PSP_VIDEO_ROOTS` | `/data/umd_videos` | comma-separated video disc roots |
|
||
| `PSP_PS1_ROOTS` | `/data/ps1` | comma-separated PS1 disc roots |
|
||
| `PSP_PS1_OUTPUT_DIR` | `/data/ps1_converted` | converted EBOOT.PBP output folder |
|
||
| `PSP_SCAN_WORKERS` | `1` | parallel hashing workers |
|
||
| `PSP_STORE_DIR` | `/media/downloads/Sony - PlayStation Portable` | local 7z store folder |
|
||
| `PSP_STORE_SERVER` | `https://psp.example.com` | store server base URL |
|
||
| `PSP_STORE_TOKEN` | *(empty)* | store server bearer token |
|
||
| `PSP_ARCHIVE_ITEM` | `psp-redump-roms321.com` | archive.org item identifier |
|
||
| `PSP_ARCHIVE_LOGGED_IN_SIG` | *(empty)* | archive.org `logged-in-sig` cookie |
|
||
| `PSP_ARCHIVE_LOGGED_IN_USER` | *(empty)* | archive.org `logged-in-user` cookie |
|
||
| `PSP_NPS_DIR` | `<state>/nps` | NoPayStation output folder |
|
||
| `PSP_NPS_TSV_BASE` | `https://nopaystation.com/tsv/` | NoPayStation TSV base URL |
|
||
| `PSP_PKG2ZIP_PATH` | *(empty)* | pkg2zip binary path (else auto-detected) |
|
||
| `PSP_API_TOKEN` | *(empty)* | server API bearer token (unset = open) |
|
||
| `PSP_CREDENTIALS` | `$PSP_DATA_DIR/credentials.json` | credentials file |
|
||
| `PSP_LIBRETRO_COVERS_URL` | *(empty)* | direct URL for `libretro-covers.zip` (else derived from store server) |
|
||
| `PSP_OVGDB_COVERS_URL` | *(empty)* | direct URL for `openvgdb-covers.zip` (else derived from store server) |
|
||
|
||
Cover/metadata keys can also be set as env vars: `IGDB_CLIENT_ID`,
|
||
`IGDB_CLIENT_SECRET`, `TMDB_API_KEY`, `GOOGLE_API_KEY`, `GOOGLE_CX`,
|
||
`SCREENSCRAPER_DEVID`, `SCREENSCRAPER_DEVPASSWORD`, `SCREENSCRAPER_SSID`,
|
||
`SCREENSCRAPER_SSPASSWORD`, `SCREENSCRAPER_SOFTNAME`.
|
||
|
||
## Cover art
|
||
|
||
Games are resolved in this order (offline-first, then network):
|
||
|
||
1. **Manual covers** — images you drop in `data/manual_covers/games/`
|
||
(fuzzy-matched; strips `(Region)`, handles `Title, The`).
|
||
2. **libretro-thumbnails (offline bundle)** — full PSP box art from
|
||
`data/covers/libretro-covers.zip`. If that zip is absent, it downloads from
|
||
GitHub and caches instead.
|
||
3. **OpenVGDB (offline bundle)** — full covers from
|
||
`data/covers/openvgdb-covers.zip`.
|
||
4. **ScreenScraper** — full box art + rich metadata; needs a free dev
|
||
ID/password.
|
||
5. **Wikipedia** — keyless fallback.
|
||
6. **IGDB** — needs a Twitch/IGDB client ID + secret. IGDB only stores cropped
|
||
3:4 key art, so the app composites it into the PSP game cover template
|
||
(`data/covers/UMD_GAME_COVER_TEMPLATE.png`) to restore the full cover.
|
||
|
||
UMD **videos** have no public box-art API. The manual **Scrape…** dialog offers
|
||
three sources: TheMovieDB posters, Wikipedia image search (keyless), and Google
|
||
Images (needs a Google API key + Search Engine ID in **Settings → Cover
|
||
sources**). The pick is composited into `data/covers/UMD_COVER_TEMPLATE.png`.
|
||
Manual covers in `data/manual_covers/videos/` always win.
|
||
|
||
### Offline bundles & auto-download
|
||
|
||
The two bundles (`libretro-covers.zip`, `openvgdb-covers.zip`) are shipped in
|
||
`data/covers/`, so a fresh install works with **no internet and no API keys**.
|
||
If either is missing at startup, the app downloads it from the store server's
|
||
`/api/covers/<name>.zip` endpoint (see [Server setup](#server-setup)).
|
||
|
||
Rebuild them yourself:
|
||
|
||
```bash
|
||
python3 -m app.libretro # libretro-thumbnails -> libretro-covers.zip
|
||
python3 -m app.openvgdb # OpenVGDB/GameFAQs -> openvgdb-covers.zip
|
||
```
|
||
|
||
(`app.openvgdb` needs `curl_cffi` to pass GameFAQs' Cloudflare challenge.)
|
||
|
||
## PS1 support
|
||
|
||
The **PS1** tab scans a folder of PlayStation 1 rips and converts them into PSP
|
||
`EBOOT.PBP` files with a pure-Python popstation converter (no external tools).
|
||
PS1 discs (pending) and converted EBOOTs also show up in the **Library** tab
|
||
(filter by **PS1**), where you can convert a disc or copy an EBOOT to the PSP.
|
||
|
||
- **Source formats** — `.iso`, `.img`, `.bin` (with a matching `.cue` for
|
||
multi-track discs).
|
||
- **Serial detection** — reads `SYSTEM.CNF` to derive the game ID (e.g.
|
||
`SLUS-00001`) and writes it into `PARAM.SFO` + the PSAR header.
|
||
- **base.pbp** — conversion needs a PSN PS1 classic `EBOOT.PBP` (any game) to
|
||
supply the `STARTDAT` block (boot logo + PGD) and a base `PARAM.SFO`/icon.
|
||
Copy one to `data/base.PBP` and point **Settings → General → PS1 base.pbp**
|
||
at it. Any PS1 classic works structurally — the converter finds `STARTDAT`
|
||
at `PSAR + p1` regardless of PSAR alignment.
|
||
- **POPS loader** — the `DATA.PSP` section uses a bundled plain-ELF loader
|
||
(`data/loader/DATA.PSP`, from iPoPS, MIT) so the EBOOT boots on modern CFW
|
||
without the base PBP's encrypted loader failing to link (`80020148`). The
|
||
base PBP's own loader is only used as a fallback when the bundle is missing.
|
||
- **Output** — a folder `<out>/<Game ID>/` containing `EBOOT.PBP` (raw-DEFLATE
|
||
compressed PSAR) plus a `KEYS.BIN` (NPDRM version key used to decrypt the
|
||
`STARTDAT` PGD), ready to copy to `<PSP>/PSP/GAME/<Game ID>/`. Compatible
|
||
with CFW POPS (ARK-4/ARK-5).
|
||
- **Artwork** — the converter fetches the game's real cover art (libretro /
|
||
ScreenScraper / IGDB / Wikipedia, reusing the local cover cache) and builds
|
||
official-style PS1 Classics art:
|
||
- `ICON0` (80×80) — the cover composited into the bundled case frame
|
||
(`data/ps1_templates/ICON0.PNG`) with the PlayStation logo + wordmark;
|
||
- `PIC0` (310×180) — a PSN-style info board with the game title, a
|
||
description from IGDB, an ESRB rating badge and PS3/PSP platform badges;
|
||
- `PIC1` (480×272) — the bundled launch background
|
||
(`data/ps1_templates/PIC1.PNG`).
|
||
If the template images are missing the app falls back to drawn art, and to
|
||
the base icon when no cover is found.
|
||
|
||
Configure the folders in **Settings → General** (or `PSP_PS1_ROOTS` /
|
||
`PSP_PS1_OUTPUT_DIR`). Current limits: single-disc games only (no multi-disc
|
||
PBP); disc data is DEFLATE-compressed (level 1 by default, `--complevel 0-9`).
|
||
|
||
## Store — archive.org and local folder
|
||
|
||
The **Store** tab downloads missing discs from a Redump 7z archive collection
|
||
(`Name (Region) (tags).7z`, containing the `.iso`/`.cso` inside). It has three
|
||
sources, selectable in the tab:
|
||
|
||
### 1. Local folder
|
||
|
||
Point **Settings → Store → Local store dir** (or `PSP_STORE_DIR`) at a folder of
|
||
`.7z`/`.zip` archives. The app scans the folder, lists the archives, and on
|
||
download:
|
||
|
||
1. reads the archive listing with `7z` to find the disc image inside,
|
||
2. copies the archive to a temp dir and extracts the image with `7z`,
|
||
3. moves the image into your games/videos directory.
|
||
|
||
### 2. Server (HTTP API)
|
||
|
||
See [Server setup](#server-setup). The app talks to `GET /api/store` (list) and
|
||
`GET /api/store/download/<name>` (stream), using your `PSP_API_TOKEN` as
|
||
`Authorization: Bearer <token>`.
|
||
|
||
### 3. Archive.org
|
||
|
||
Point **Settings → Store → Archive.org item** (or `PSP_ARCHIVE_ITEM`) at an
|
||
archive.org item identifier (or a full `details`/`download` URL). The app:
|
||
|
||
1. lists the item via `https://archive.org/metadata/<item>`,
|
||
2. downloads archives from `https://archive.org/download/<item>/<file>`,
|
||
3. extracts the disc image and moves it into your library.
|
||
|
||
If the item's files are marked **private** (login required), set the
|
||
`logged-in-sig` and `logged-in-user` cookies in **Settings → Store** (grab them
|
||
from a logged-in archive.org browser session). Without them, private files are
|
||
skipped — and you'll get a clear error if *everything* in the item is private.
|
||
|
||
### 4. NoPayStation
|
||
|
||
The **NoPayStation** source lists PSN content from
|
||
[NoPayStation](https://nopaystation.com/) — **PSP games** (`PSP_GAMES.tsv`) and
|
||
**PS1 Classics** (`PSX_GAMES.tsv`), selectable from the content-type dropdown
|
||
next to the source selector.
|
||
|
||
Requirements & behaviour:
|
||
|
||
1. `.pkg` files are decrypted by the built-in **pure-Python decryptor**
|
||
(`app/pkg.py`, powered by `pycryptodome` for speed) — no Wine and no
|
||
`pkg2zip` install needed. A bundled `pkg2zip.exe` is kept as a fallback
|
||
only, and only runs if Python decryption fails and Wine is present.
|
||
2. Set **NoPayStation output dir** in **Settings → Store** (default
|
||
`$PSP_STATE_DIR/nps`).
|
||
3. Downloading a PSP game writes `PSP/<TITLE_ID>/EBOOT.PBP` (plus
|
||
`PSP-KEY.EDAT` / `CONTENT.DAT` when the package ships them); a PS1 Classic
|
||
writes `PSX/<TITLE_ID>/EBOOT.PBP` (+ `DOCUMENT.DAT` and `KEYS.BIN`). Both
|
||
then show up in the **Library** (PS1 Classics under **PS1**, PSP games
|
||
under **Games**) and can be copied to the PSP like any other item.
|
||
4. The TSV lists are cached in `data/nps/` and refreshed on each store scan
|
||
(drop the `.tsv` files there manually to work offline).
|
||
|
||
> **DRM note:** NPS content is PSN-DRM'd. To *run* it on the PSP you need the
|
||
> [`npdrm_free`](https://github.com/qwikrazor87/npdrm_free) plugin (ARK and most
|
||
> CFW already ship it). The desktop app only downloads and decrypts; it doesn't
|
||
> strip DRM.
|
||
|
||
## Server setup
|
||
|
||
The catalog + store API is a single-file, stdlib-only HTTP server
|
||
(`app/api.py`) — no pip install needed on the server.
|
||
|
||
```bash
|
||
PSP_DATA_DIR=/data \
|
||
PSP_DATS_DIR=/data/dats \
|
||
PSP_DB_PATH=/data/library.sqlite \
|
||
PSP_COVERS_DIR=/data/covers \
|
||
PSP_GAMES_ROOTS=/data/umd_games \
|
||
PSP_VIDEO_ROOTS=/data/umd_videos \
|
||
PSP_STORE_DIR=/data/store \
|
||
PSP_API_TOKEN=<long-random-secret> \
|
||
python3 -m app.api --host 0.0.0.0 --port 8000
|
||
```
|
||
|
||
Endpoints:
|
||
|
||
| Endpoint | Description |
|
||
|---|---|
|
||
| `GET /api/summary` | release counts per system |
|
||
| `GET /api/releases?system=&q=&limit=&offset=` | search the catalog |
|
||
| `GET /api/releases/<id>` | one release + its rom hashes |
|
||
| `GET /api/library?system=&refresh=` | your scanned library |
|
||
| `GET /api/cover/<id>` | cover image (fetched on demand) |
|
||
| `GET /api/covers/<name>.zip` | offline cover bundle (public) |
|
||
| `GET /api/store` | list store archives *(token)* |
|
||
| `GET /api/store/download/<name>` | stream one archive *(token)* |
|
||
|
||
Token protection: set `PSP_API_TOKEN`; the client sends
|
||
`Authorization: Bearer <token>`. Leave it unset to allow anonymous access
|
||
(not recommended).
|
||
|
||
### Docker (Unraid)
|
||
|
||
`docker-compose.yml` builds the API image and attaches it to Unraid's `br0`
|
||
macvlan bridge so it gets its own LAN IP (e.g. `192.168.1.100`, plain HTTP on 8000):
|
||
|
||
```bash
|
||
# in /path/to/psp-preserve/ — adjust the volume paths for your shares
|
||
printf 'PSP_API_TOKEN=<long-random-secret>\n' > .env
|
||
docker compose up -d --build
|
||
```
|
||
|
||
The compose file mounts your Unraid shares into `/data/store`, `/data/umd_games`
|
||
and `/data/umd_videos` (the store is mounted read-only).
|
||
|
||
### TLS (optional)
|
||
|
||
`Caddyfile` is a ready-made reverse proxy for `https://psp.example.com` with
|
||
automatic Let's Encrypt certificates. To enable it, add the `caddy` service to
|
||
the compose file, give it its own `br0` IP, point it at the API, and forward
|
||
ports 80/443 on your router to it.
|
||
|
||
### Client
|
||
|
||
In the GUI: **Settings → Store** → enter the **Server URL** (e.g.
|
||
`http://<server-ip>:8000` or `https://psp.example.com`) and the same **API
|
||
token**, then pick **Server** in the Store tab's source selector. Use
|
||
**Archive.org** or **Local folder** the same way.
|
||
|
||
## PSP Sync
|
||
|
||
Connect a PSP over USB (or mount a UMD) and the **PSP Sync** tab detects it:
|
||
|
||
- **Games / UMD Videos / Homebrew-PSN / PKG** — tables of what's installed, with
|
||
delete-from-PSP (checkboxes and right-click).
|
||
- **UMD** — reads `PARAM.SFO` from a mounted UMD, shows the title/serial, and
|
||
can dump the disc image into your library.
|
||
- **Sync from Library** — copy discs from your library onto the PSP, optionally
|
||
into a Category Lite category (set in **Settings → PSP → ISO category**).
|
||
|
||
## Saves sync
|
||
|
||
The **Saves** tab shows save-data across three locations — **PSP** (USB),
|
||
**PPSSPP**, and a **Cloud folder** — and syncs both ways. Save signatures are
|
||
cached (keyed by size + mtime) so rescanning is fast, and an **Auto-sync**
|
||
option runs the sync automatically whenever a PSP is connected.
|
||
|
||
## CLI tools
|
||
|
||
```bash
|
||
# scan + report (dry-run)
|
||
python3 -m app.scan --roots /path/to/umd_games,/path/to/umd_videos
|
||
|
||
# scan + apply renames
|
||
python3 -m app.scan --roots /path/to/umd_games,/path/to/umd_videos --apply
|
||
|
||
# ingest DAT files into the library DB
|
||
python3 -m app.ingest --dats data/dats --db data/library.sqlite
|
||
|
||
# (re)build the offline cover bundles (see Cover art)
|
||
python3 -m app.libretro
|
||
python3 -m app.openvgdb
|
||
|
||
# convert one PS1 disc to EBOOT.PBP (CLI)
|
||
python3 -m app.ps1_convert --source /path/to/game.cue --title "Game" \
|
||
--game-id SLUS00001 --base-pbp data/base.PBP --out /out/Game.pbp
|
||
```
|
||
|
||
`python3 -m app.scan --json` prints machine-readable results, and `--quick`
|
||
skips full-file hashing (serial/title match only).
|