340 lines
15 KiB
Markdown
340 lines
15 KiB
Markdown
# PSP Preservation Tool
|
|
|
|
|
|
|
|
Native Linux GUI (PySide6) for preserving PSP UMD **games** and **videos** as
|
|
`.iso` files. It scans your discs, verifies them against the full Redump catalog
|
|
(Redump DATs), reports what's missing, and retags/renames discs to a canonical
|
|
scheme. It also fetches cover art and metadata, syncs discs and save-data with a
|
|
real PSP, can pull missing discs from your own Redump 7z archive collection, and
|
|
converts PlayStation 1 ISO/CUE/BIN rips into PSP `EBOOT.PBP` homebrew.
|
|
|
|
|
|
* Use default app theme for Qt.
|
|
|
|
## Features
|
|
|
|
- **Library** — scan folders of `.iso` discs, verify against the DAT catalog,
|
|
browse with cover art, search/filter, see per-game details (developer,
|
|
publisher, genre, release date, synopsis, trailer), and launch discs in PPSSPP.
|
|
- **Store** — browse and download discs from three sources: a local 7z folder,
|
|
your own HTTP store server, or an archive.org item (see
|
|
[Store](#store--archiveorg-and-local-folder)).
|
|
- **Queue** — background transfers with progress, cancel, per-row delete, etc.
|
|
- **PSP Sync** — detect a connected PSP (USB) or a mounted UMD, inventory what's
|
|
installed (games / UMD videos / homebrew / PKG), copy discs to the PSP, delete
|
|
from the PSP, and dump a UMD to the library.
|
|
- **Saves** — two-way sync of save-data between the PSP, PPSSPP, and a cloud
|
|
folder.
|
|
- **PS1** — scan a folder of PlayStation 1 discs (ISO/BIN/CUE), read each disc's
|
|
serial, convert to PSP `EBOOT.PBP` with a pure-Python popstation converter,
|
|
and track already-converted games.
|
|
- **Settings** — directories, cover-source API keys, store/server config, and
|
|
misc options.
|
|
|
|
## Requirements
|
|
|
|
- Linux + Python 3
|
|
- `PySide6` — the only Python dependency (`pip install -r requirements.txt`)
|
|
- `7z` (p7zip) — used to peek/extract disc images from store archives
|
|
- PPSSPP (optional) — to launch discs from the details panel
|
|
|
|
## Install
|
|
|
|
```bash
|
|
python3 -m venv .venv # keep the venv on a local disk, not an SMB mount
|
|
. .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
## Run the GUI
|
|
|
|
```bash
|
|
python3 -m app.main
|
|
```
|
|
|
|
## First-time setup
|
|
|
|
1. Put your DAT files in `data/dats/`:
|
|
- games: Redump `Sony - PlayStation Portable` (authoritative hashes)
|
|
**and** No-Intro `Sony - PlayStation Portable` (supplies the serials
|
|
Redump lacks — keeps serial-based matching working).
|
|
- videos: No-Intro `Sony - PlayStation Portable (UMD Video)` (Redump has
|
|
no UMD Video DAT).
|
|
2. Ingest them into the local SQLite library:
|
|
|
|
```bash
|
|
python3 -m app.ingest
|
|
```
|
|
|
|
3. Open the GUI and point **Settings → General → Games/Videos directory** at
|
|
your disc folders (or set `PSP_GAMES_ROOTS` / `PSP_VIDEO_ROOTS`).
|
|
4. (Optional) PS1 conversion — copy any PSN PS1 classic `EBOOT.PBP` to
|
|
`data/base.PBP` (it supplies Sony's encrypted loader). Then set the **PS1
|
|
directory**, **PS1 output directory** and **PS1 base.pbp** in
|
|
**Settings → General** and use the **PS1** tab (see [PS1 support](#ps1-support)).
|
|
|
|
Discs that aren't in the Redump DATs can be added to `data/supplement.json`
|
|
(`name`, `region`, `crc32`, `sha1`, `md5`) so the tool still recognizes them.
|
|
|
|
## Nomenclature
|
|
|
|
`Title (Region) (tags) (Serial).iso` — leading articles stay in front
|
|
(`The Sims 2`), and revision/disc tags are preserved (`(v1.03)`, `(Disc 1)`).
|
|
Discs are matched by hash first (most precise), then serial, then filename.
|
|
|
|
## Directory layout
|
|
|
|
- `data/dats/` — clrmamepro DAT files (games + videos).
|
|
- `data/supplement.json` — extra discs not in Redump.
|
|
- `data/credentials.json` — cover/metadata API keys (IGDB, TheMovieDB,
|
|
ScreenScraper, Google Custom Search). Environment variables take precedence.
|
|
- `data/manual_covers/games|videos/` — your own cover images.
|
|
- `data/covers/` — checked-in assets: the UMD video/game cover templates,
|
|
`libretro-index.json` + `openvgdb-index.json` catalogs, `libretro-covers.zip` +
|
|
`openvgdb-covers.zip` offline cover bundles, and the legacy cover cache.
|
|
- `data/base.PBP` — a PSN PS1 classic EBOOT (any game) used as the encrypted
|
|
loader when converting PS1 discs to `EBOOT.PBP` (git-ignored, copyrighted).
|
|
- State dir (`~/.local/state/psp-preserve/` by default, or `$XDG_STATE_HOME/psp-preserve`):
|
|
- `library.sqlite` — the catalog + library database.
|
|
- `covers/` — downloaded cover cache.
|
|
|
|
SQLite is kept on a local disk because SQLite locking is unreliable over
|
|
network filesystems; the DATs, manual covers and credentials stay in `data/`
|
|
(which may be a network share).
|
|
|
|
## Configuration
|
|
|
|
Everything is driven by environment variables, with GUI settings (stored in
|
|
QSettings under `PSPPreserve`) taking precedence where noted.
|
|
|
|
| Variable | Default | Purpose |
|
|
|---|---|---|
|
|
| `PSP_DATA_DIR` | `./data` | data root (DATs, credentials, manual covers) |
|
|
| `PSP_DATS_DIR` | `$PSP_DATA_DIR/dats` | DAT folder |
|
|
| `PSP_STATE_DIR` | `~/.local/state/psp-preserve` | local state root |
|
|
| `PSP_DB_PATH` | `$PSP_STATE_DIR/library.sqlite` | SQLite library |
|
|
| `PSP_COVERS_DIR` | `$PSP_STATE_DIR/covers` | cover cache |
|
|
| `PSP_GAMES_ROOTS` | `/data/umd_games` | comma-separated game disc roots |
|
|
| `PSP_VIDEO_ROOTS` | `/data/umd_videos` | comma-separated video disc roots |
|
|
| `PSP_PS1_ROOTS` | `/data/ps1` | comma-separated PS1 disc roots |
|
|
| `PSP_PS1_OUTPUT_DIR` | `/data/ps1_converted` | converted EBOOT.PBP output folder |
|
|
| `PSP_SCAN_WORKERS` | `1` | parallel hashing workers |
|
|
| `PSP_STORE_DIR` | `/media/downloads/Sony - PlayStation Portable` | local 7z store folder |
|
|
| `PSP_STORE_SERVER` | `https://psp.example.com` | store server base URL |
|
|
| `PSP_STORE_TOKEN` | *(empty)* | store server bearer token |
|
|
| `PSP_ARCHIVE_ITEM` | `psp-redump-roms321.com` | archive.org item identifier |
|
|
| `PSP_ARCHIVE_LOGGED_IN_SIG` | *(empty)* | archive.org `logged-in-sig` cookie |
|
|
| `PSP_ARCHIVE_LOGGED_IN_USER` | *(empty)* | archive.org `logged-in-user` cookie |
|
|
| `PSP_API_TOKEN` | *(empty)* | server API bearer token (unset = open) |
|
|
| `PSP_CREDENTIALS` | `$PSP_DATA_DIR/credentials.json` | credentials file |
|
|
| `PSP_LIBRETRO_COVERS_URL` | *(empty)* | direct URL for `libretro-covers.zip` (else derived from store server) |
|
|
| `PSP_OVGDB_COVERS_URL` | *(empty)* | direct URL for `openvgdb-covers.zip` (else derived from store server) |
|
|
|
|
Cover/metadata keys can also be set as env vars: `IGDB_CLIENT_ID`,
|
|
`IGDB_CLIENT_SECRET`, `TMDB_API_KEY`, `GOOGLE_API_KEY`, `GOOGLE_CX`,
|
|
`SCREENSCRAPER_DEVID`, `SCREENSCRAPER_DEVPASSWORD`, `SCREENSCRAPER_SSID`,
|
|
`SCREENSCRAPER_SSPASSWORD`, `SCREENSCRAPER_SOFTNAME`.
|
|
|
|
## Cover art
|
|
|
|
Games are resolved in this order (offline-first, then network):
|
|
|
|
1. **Manual covers** — images you drop in `data/manual_covers/games/`
|
|
(fuzzy-matched; strips `(Region)`, handles `Title, The`).
|
|
2. **libretro-thumbnails (offline bundle)** — full PSP box art from
|
|
`data/covers/libretro-covers.zip`. If that zip is absent, it downloads from
|
|
GitHub and caches instead.
|
|
3. **OpenVGDB (offline bundle)** — full covers from
|
|
`data/covers/openvgdb-covers.zip`.
|
|
4. **ScreenScraper** — full box art + rich metadata; needs a free dev
|
|
ID/password.
|
|
5. **Wikipedia** — keyless fallback.
|
|
6. **IGDB** — needs a Twitch/IGDB client ID + secret. IGDB only stores cropped
|
|
3:4 key art, so the app composites it into the PSP game cover template
|
|
(`data/covers/UMD_GAME_COVER_TEMPLATE.png`) to restore the full cover.
|
|
|
|
UMD **videos** have no public box-art API. The manual **Scrape…** dialog offers
|
|
three sources: TheMovieDB posters, Wikipedia image search (keyless), and Google
|
|
Images (needs a Google API key + Search Engine ID in **Settings → Cover
|
|
sources**). The pick is composited into `data/covers/UMD_COVER_TEMPLATE.png`.
|
|
Manual covers in `data/manual_covers/videos/` always win.
|
|
|
|
### Offline bundles & auto-download
|
|
|
|
The two bundles (`libretro-covers.zip`, `openvgdb-covers.zip`) are shipped in
|
|
`data/covers/`, so a fresh install works with **no internet and no API keys**.
|
|
If either is missing at startup, the app downloads it from the store server's
|
|
`/api/covers/<name>.zip` endpoint (see [Server setup](#server-setup)).
|
|
|
|
Rebuild them yourself:
|
|
|
|
```bash
|
|
python3 -m app.libretro # libretro-thumbnails -> libretro-covers.zip
|
|
python3 -m app.openvgdb # OpenVGDB/GameFAQs -> openvgdb-covers.zip
|
|
```
|
|
|
|
(`app.openvgdb` needs `curl_cffi` to pass GameFAQs' Cloudflare challenge.)
|
|
|
|
## PS1 support
|
|
|
|
The **PS1** tab scans a folder of PlayStation 1 rips and converts them into PSP
|
|
`EBOOT.PBP` files with a pure-Python popstation converter (no external tools).
|
|
PS1 discs (pending) and converted EBOOTs also show up in the **Library** tab
|
|
(filter by **PS1**), where you can convert a disc or copy an EBOOT to the PSP.
|
|
|
|
- **Source formats** — `.iso`, `.img`, `.bin` (with a matching `.cue` for
|
|
multi-track discs).
|
|
- **Serial detection** — reads `SYSTEM.CNF` to derive the game ID (e.g.
|
|
`SLUS-00001`) and writes it into `PARAM.SFO` + the PSAR header.
|
|
- **base.pbp** — conversion needs a PSN PS1 classic `EBOOT.PBP` (any game) to
|
|
supply Sony's encrypted `DATA.PSP` loader. Copy one to `data/base.PBP` and
|
|
point **Settings → General → PS1 base.pbp** at it.
|
|
- **Output** — a PSN-style folder `<out>/<Game ID>/` containing `EBOOT.PBP`,
|
|
`KEYS.BIN` and `DOCUMENT.DAT`, ready to copy to
|
|
`<PSP>/PSP/GAME/<Game ID>/`.
|
|
|
|
Configure the folders in **Settings → General** (or `PSP_PS1_ROOTS` /
|
|
`PSP_PS1_OUTPUT_DIR`). Current limits: single-disc games only (no multi-disc
|
|
PBP), and images are stored uncompressed (LZRC compression is on the roadmap).
|
|
|
|
## Store — archive.org and local folder
|
|
|
|
The **Store** tab downloads missing discs from a Redump 7z archive collection
|
|
(`Name (Region) (tags).7z`, containing the `.iso`/`.cso` inside). It has three
|
|
sources, selectable in the tab:
|
|
|
|
### 1. Local folder
|
|
|
|
Point **Settings → Store → Local store dir** (or `PSP_STORE_DIR`) at a folder of
|
|
`.7z`/`.zip` archives. The app scans the folder, lists the archives, and on
|
|
download:
|
|
|
|
1. reads the archive listing with `7z` to find the disc image inside,
|
|
2. copies the archive to a temp dir and extracts the image with `7z`,
|
|
3. moves the image into your games/videos directory.
|
|
|
|
### 2. Server (HTTP API)
|
|
|
|
See [Server setup](#server-setup). The app talks to `GET /api/store` (list) and
|
|
`GET /api/store/download/<name>` (stream), using your `PSP_API_TOKEN` as
|
|
`Authorization: Bearer <token>`.
|
|
|
|
### 3. Archive.org
|
|
|
|
Point **Settings → Store → Archive.org item** (or `PSP_ARCHIVE_ITEM`) at an
|
|
archive.org item identifier (or a full `details`/`download` URL). The app:
|
|
|
|
1. lists the item via `https://archive.org/metadata/<item>`,
|
|
2. downloads archives from `https://archive.org/download/<item>/<file>`,
|
|
3. extracts the disc image and moves it into your library.
|
|
|
|
If the item's files are marked **private** (login required), set the
|
|
`logged-in-sig` and `logged-in-user` cookies in **Settings → Store** (grab them
|
|
from a logged-in archive.org browser session). Without them, private files are
|
|
skipped — and you'll get a clear error if *everything* in the item is private.
|
|
|
|
## Server setup
|
|
|
|
The catalog + store API is a single-file, stdlib-only HTTP server
|
|
(`app/api.py`) — no pip install needed on the server.
|
|
|
|
```bash
|
|
PSP_DATA_DIR=/data \
|
|
PSP_DATS_DIR=/data/dats \
|
|
PSP_DB_PATH=/data/library.sqlite \
|
|
PSP_COVERS_DIR=/data/covers \
|
|
PSP_GAMES_ROOTS=/data/umd_games \
|
|
PSP_VIDEO_ROOTS=/data/umd_videos \
|
|
PSP_STORE_DIR=/data/store \
|
|
PSP_API_TOKEN=<long-random-secret> \
|
|
python3 -m app.api --host 0.0.0.0 --port 8000
|
|
```
|
|
|
|
Endpoints:
|
|
|
|
| Endpoint | Description |
|
|
|---|---|
|
|
| `GET /api/summary` | release counts per system |
|
|
| `GET /api/releases?system=&q=&limit=&offset=` | search the catalog |
|
|
| `GET /api/releases/<id>` | one release + its rom hashes |
|
|
| `GET /api/library?system=&refresh=` | your scanned library |
|
|
| `GET /api/cover/<id>` | cover image (fetched on demand) |
|
|
| `GET /api/covers/<name>.zip` | offline cover bundle (public) |
|
|
| `GET /api/store` | list store archives *(token)* |
|
|
| `GET /api/store/download/<name>` | stream one archive *(token)* |
|
|
|
|
Token protection: set `PSP_API_TOKEN`; the client sends
|
|
`Authorization: Bearer <token>`. Leave it unset to allow anonymous access
|
|
(not recommended).
|
|
|
|
### Docker (Unraid)
|
|
|
|
`docker-compose.yml` builds the API image and attaches it to Unraid's `br0`
|
|
macvlan bridge so it gets its own LAN IP (e.g. `192.168.1.100`, plain HTTP on 8000):
|
|
|
|
```bash
|
|
# in /path/to/psp-preserve/ — adjust the volume paths for your shares
|
|
printf 'PSP_API_TOKEN=<long-random-secret>\n' > .env
|
|
docker compose up -d --build
|
|
```
|
|
|
|
The compose file mounts your Unraid shares into `/data/store`, `/data/umd_games`
|
|
and `/data/umd_videos` (the store is mounted read-only).
|
|
|
|
### TLS (optional)
|
|
|
|
`Caddyfile` is a ready-made reverse proxy for `https://psp.example.com` with
|
|
automatic Let's Encrypt certificates. To enable it, add the `caddy` service to
|
|
the compose file, give it its own `br0` IP, point it at the API, and forward
|
|
ports 80/443 on your router to it.
|
|
|
|
### Client
|
|
|
|
In the GUI: **Settings → Store** → enter the **Server URL** (e.g.
|
|
`http://<server-ip>:8000` or `https://psp.example.com`) and the same **API
|
|
token**, then pick **Server** in the Store tab's source selector. Use
|
|
**Archive.org** or **Local folder** the same way.
|
|
|
|
## PSP Sync
|
|
|
|
Connect a PSP over USB (or mount a UMD) and the **PSP Sync** tab detects it:
|
|
|
|
- **Games / UMD Videos / Homebrew-PSN / PKG** — tables of what's installed, with
|
|
delete-from-PSP (checkboxes and right-click).
|
|
- **UMD** — reads `PARAM.SFO` from a mounted UMD, shows the title/serial, and
|
|
can dump the disc image into your library.
|
|
- **Sync from Library** — copy discs from your library onto the PSP, optionally
|
|
into a Category Lite category (set in **Settings → PSP → ISO category**).
|
|
|
|
## Saves sync
|
|
|
|
The **Saves** tab shows save-data across three locations — **PSP** (USB),
|
|
**PPSSPP**, and a **Cloud folder** — and syncs both ways. Save signatures are
|
|
cached (keyed by size + mtime) so rescanning is fast, and an **Auto-sync**
|
|
option runs the sync automatically whenever a PSP is connected.
|
|
|
|
## CLI tools
|
|
|
|
```bash
|
|
# scan + report (dry-run)
|
|
python3 -m app.scan --roots /path/to/umd_games,/path/to/umd_videos
|
|
|
|
# scan + apply renames
|
|
python3 -m app.scan --roots /path/to/umd_games,/path/to/umd_videos --apply
|
|
|
|
# ingest DAT files into the library DB
|
|
python3 -m app.ingest --dats data/dats --db data/library.sqlite
|
|
|
|
# (re)build the offline cover bundles (see Cover art)
|
|
python3 -m app.libretro
|
|
python3 -m app.openvgdb
|
|
|
|
# convert one PS1 disc to EBOOT.PBP (CLI)
|
|
python3 -m app.ps1_convert --source /path/to/game.cue --title "Game" \
|
|
--game-id SLUS00001 --base-pbp data/base.PBP --out /out/Game.pbp
|
|
```
|
|
|
|
`python3 -m app.scan --json` prints machine-readable results, and `--quick`
|
|
skips full-file hashing (serial/title match only).
|