# PSP Preservation Tool
the name is terrible, i know.

![AI Generated](https://raw.githubusercontent.com/dotnetdave/ai-usage-badges/refs/heads/main/badges/png/1x/ai-generated.png)

A cross-platform (Linux + Windows) desktop GUI — PySide6 — for preserving PSP
UMD **games** and **videos** as `.iso` files. It scans your discs, verifies them
against the Redump/No-Intro catalogs, reports what's missing, and retags/renames
discs to a canonical scheme. It fetches cover art and metadata, syncs discs and
save-data with a real PSP, can pull missing discs from a Redump 7z archive
collection, downloads PSN content from NoPayStation, and converts PlayStation 1
ISO/CUE/BIN rips into PSP `EBOOT.PBP` homebrew.

![PSP Preservation Tool](img/qt-gui.png)
* Dark theme with a blue accent — the Library tab in cover view. *

## Features

- **Library** — scan folders of `.iso` discs, verify against the DAT catalog,
  browse with cover art, search/filter, see per-game details (developer,
  publisher, genre, release date, synopsis, trailer), and launch discs in PPSSPP.
- **Store** — browse and download discs from four sources: a local 7z folder,
  your own HTTP store server, an archive.org item, or NoPayStation (PSP/PS1
  PSN `.pkg` content) (see [Store](#store--archiveorg-and-local-folder)).
- **Queue** — background transfers with progress, cancel, per-row delete, etc.
- **PSP Sync** — detect a connected PSP (USB) or a mounted UMD, inventory what's
  installed (games / UMD videos / homebrew / PKG), copy discs to the PSP, delete
  from the PSP, and dump a UMD to the library.
- **Saves** — two-way sync of save-data between the PSP, PPSSPP, and a cloud
  folder.
- **PS1** — scan a folder of PlayStation 1 discs (ISO/BIN/CUE), read each disc's
  serial, convert to PSP `EBOOT.PBP` with a pure-Python popstation converter,
  and track already-converted games.
- **Settings** — directories, cover-source API keys, store/server config, and
  misc options.

## Requirements

- Python 3.10+ on Linux or Windows
- `PySide6`, `curl_cffi`, `pycryptodome` (`pip install -r requirements.txt`)
- `7z` (p7zip / 7-Zip) — used to peek/extract disc images from store archives
- PPSSPP (optional) — to launch discs from the details panel

NoPayStation `.pkg` files are decrypted by the bundled **pure-Python
decryptor** (`app/pkg.py`) — no Wine or `pkg2zip` required
(see [NoPayStation](#4-nopaystation)).

## Install

```bash
python3 -m venv .venv          # keep the venv on a local disk, not an SMB mount
. .venv/bin/activate
pip install -r requirements.txt
```

## Run the GUI

```bash
python3 -m app.main
```

## Building binaries

The GUI is packaged with PyInstaller into a self-contained folder. Build
requirements are pinned in `requirements-build.txt` (PySide6 6.8.3 on purpose —
see the note inside).

```bash
python3 -m venv .venv-build
. .venv-build/bin/activate
pip install -r requirements-build.txt
pyinstaller --clean --noconfirm psp-preserve.spec
# output: dist/psp-preserve/  → run dist/psp-preserve/psp-preserve
```

Notes:

- The build bundles the DATs, PS1 loader, and the offline cover bundles. The
  cover zips are large (~850 MB), so builds are slow and the output is big.
- `7z` (p7zip-full on Linux, 7-Zip on Windows) is a **runtime** dependency for
  reading store archives — it is not bundled.
- User data (credentials, manual covers, PS1 templates, NPS cache, library DB,
  cover cache) lives in the OS state dir, never next to the binary.

### CI (Gitea Actions)

`.gitea/workflows/build.yml` builds Linux and Windows artifacts on version-tag
pushes (`v*`) or on manual trigger (**Actions → build → Run workflow**).
Artifacts appear on the Actions run page and can be attached to a release.

Each job needs a [Gitea Actions runner](https://docs.gitea.com/usage/actions/act-runner)
registered with the matching label:

```bash
# Linux runner (Docker mode, run on your server)
./act_runner register \
  --instance https://git.ceaf.dev \
  --token <registration-token> \
  --name linux-builder \
  --labels ubuntu-latest:docker://node:20-bookworm

# Windows runner (host mode, run on a Windows machine)
act_runner.exe register \
  --instance https://git.ceaf.dev \
  --token <registration-token> \
  --name win-builder \
  --labels windows:host
```

The registration token is generated in Gitea under
**Site Administration → Actions → Runners → Create new runner**.

## First-time setup

1. Put your DAT files in `data/dats/`:
   - games: Redump `Sony - PlayStation Portable` (authoritative hashes)
     **and** No-Intro `Sony - PlayStation Portable` (supplies the serials
     Redump lacks — keeps serial-based matching working).
   - videos: No-Intro `Sony - PlayStation Portable (UMD Video)` (Redump has
     no UMD Video DAT).
2. Start the app — any DATs missing from the library are ingested automatically
   at startup (no manual step). You can also re-ingest on demand:

   ```bash
   python3 -m app.ingest
   ```

3. Point **Settings → General → Games/Videos directory** at your disc folders
   (or set `PSP_GAMES_ROOTS` / `PSP_VIDEO_ROOTS`).
4. (Optional) PS1 conversion — copy any PSN PS1 classic `EBOOT.PBP` to
   `data/base.PBP` (it supplies Sony's encrypted loader). Then set the **PS1
   directory**, **PS1 output directory** and **PS1 base.pbp** in
   **Settings → General** and use the **PS1** tab (see [PS1 support](#ps1-support)).

Discs that aren't in the Redump DATs can be added to `data/supplement.json`
(`name`, `region`, `crc32`, `sha1`, `md5`) so the tool still recognizes them.

## Nomenclature

`Title (Region) (tags) (Serial).iso` — leading articles stay in front
(`The Sims 2`), and revision/disc tags are preserved (`(v1.03)`, `(Disc 1)`).
Discs are matched by hash first (most precise), then serial, then filename.

## Directory layout

- `data/dats/` — clrmamepro DAT files (games + videos).
- `data/supplement.json` — extra discs not in Redump.
- `data/credentials.json` — cover/metadata API keys (IGDB, TheMovieDB,
  ScreenScraper, Google Custom Search). Environment variables take precedence.
- `data/manual_covers/games|videos/` — your own cover images.
- `data/covers/` — checked-in assets: the UMD video/game cover templates,
  `libretro-index.json` + `openvgdb-index.json` catalogs, `libretro-covers.zip` +
  `openvgdb-covers.zip` offline cover bundles, and the legacy cover cache.
- `data/base.PBP` — a PSN PS1 classic EBOOT (any game) supplying `STARTDAT`
  + a base `PARAM.SFO`/icon for PS1 conversion (git-ignored, copyrighted).
- `data/loader/DATA.PSP` + `data/loader/KEYS.BIN` — bundled plain-ELF POPS
  loader and NPDRM version key (from iPoPS, MIT).
- `data/ps1_templates/` — optional PS1 Classics art templates
  (`ICON0.PNG` case frame, `PIC1.PNG` launch background); the app falls back
  to drawn art when they're absent.
- `data/nps/` — NoPayStation TSV lists (cached here on first store scan in dev;
  a packaged build stores them under the state dir instead).
- State dir (`~/.local/state/psp-preserve/` on Linux, `%LOCALAPPDATA%\PSPPreserve`
  on Windows, or `$PSP_STATE_DIR`):
  - `library.sqlite` — the catalog + library database.
  - `covers/` — downloaded cover cache.
  - `data/` — user-writable data in packaged builds (credentials, manual
    covers, PS1 templates, NPS TSV cache).

SQLite is kept on a local disk because SQLite locking is unreliable over
network filesystems. Read-only resources (DATs, the POPS loader, icons) are
bundled with the app; user-writable data lives in the state dir.

## Configuration

Everything is driven by environment variables, with GUI settings (stored in
QSettings under `PSPPreserve`) taking precedence where noted.

| Variable | Default | Purpose |
|---|---|---|
| `PSP_DATA_DIR` | `./data` | data root (DATs, credentials, manual covers) |
| `PSP_DATS_DIR` | `$PSP_DATA_DIR/dats` | DAT folder |
| `PSP_STATE_DIR` | `~/.local/state/psp-preserve` | local state root |
| `PSP_DB_PATH` | `$PSP_STATE_DIR/library.sqlite` | SQLite library |
| `PSP_COVERS_DIR` | `$PSP_STATE_DIR/covers` | cover cache |
| `PSP_GAMES_ROOTS` | `/data/umd_games` | comma-separated game disc roots |
| `PSP_VIDEO_ROOTS` | `/data/umd_videos` | comma-separated video disc roots |
| `PSP_PS1_ROOTS` | `/data/ps1` | comma-separated PS1 disc roots |
| `PSP_PS1_OUTPUT_DIR` | `/data/ps1_converted` | converted EBOOT.PBP output folder |
| `PSP_SCAN_WORKERS` | `1` | parallel hashing workers |
| `PSP_STORE_DIR` | `/data/store` | local 7z store folder |
| `PSP_STORE_SERVER` | `https://psp.example.com` | store server base URL |
| `PSP_STORE_TOKEN` | *(empty)* | store server bearer token |
| `PSP_ARCHIVE_ITEM` | `psp-redump-roms321.com` | archive.org item identifier |
| `PSP_ARCHIVE_LOGGED_IN_SIG` | *(empty)* | archive.org `logged-in-sig` cookie |
| `PSP_ARCHIVE_LOGGED_IN_USER` | *(empty)* | archive.org `logged-in-user` cookie |
| `PSP_NPS_DIR` | `<state>/nps` | NoPayStation output folder |
| `PSP_NPS_TSV_BASE` | `https://nopaystation.com/tsv/` | NoPayStation TSV base URL |
| `PSP_API_TOKEN` | *(empty)* | server API bearer token (unset = open) |
| `PSP_USER_DATA_DIR` | `<state>/data` (packaged) / `./data` (dev) | user-writable data (credentials, manual covers, PS1 templates, NPS cache) |
| `PSP_CREDENTIALS` | `$PSP_USER_DATA_DIR/credentials.json` | credentials file |
| `PSP_LIBRETRO_COVERS_URL` | *(empty)* | direct URL for `libretro-covers.zip` (else derived from store server) |
| `PSP_OVGDB_COVERS_URL` | *(empty)* | direct URL for `openvgdb-covers.zip` (else derived from store server) |

Cover/metadata keys can also be set as env vars: `IGDB_CLIENT_ID`,
`IGDB_CLIENT_SECRET`, `TMDB_API_KEY`, `GOOGLE_API_KEY`, `GOOGLE_CX`,
`SCREENSCRAPER_DEVID`, `SCREENSCRAPER_DEVPASSWORD`, `SCREENSCRAPER_SSID`,
`SCREENSCRAPER_SSPASSWORD`, `SCREENSCRAPER_SOFTNAME`.

## Cover art

Games are resolved in this order (offline-first, then network):

1. **Manual covers** — images you drop in `data/manual_covers/games/`
   (fuzzy-matched; strips `(Region)`, handles `Title, The`).
2. **libretro-thumbnails (offline bundle)** — full PSP box art from
   `data/covers/libretro-covers.zip`. If that zip is absent, it downloads from
   GitHub and caches instead.
3. **OpenVGDB (offline bundle)** — full covers from
   `data/covers/openvgdb-covers.zip`.
4. **ScreenScraper** — full box art + rich metadata; needs a free dev
   ID/password.
5. **Wikipedia** — keyless fallback.
6. **IGDB** — needs a Twitch/IGDB client ID + secret. IGDB only stores cropped
   3:4 key art, so the app composites it into the PSP game cover template
   (`data/covers/UMD_GAME_COVER_TEMPLATE.png`) to restore the full cover.

UMD **videos** have no public box-art API. The manual **Scrape…** dialog offers
three sources: TheMovieDB posters, Wikipedia image search (keyless), and Google
Images (needs a Google API key + Search Engine ID in **Settings → Cover
sources**). The pick is composited into `data/covers/UMD_COVER_TEMPLATE.png`.
Manual covers in `data/manual_covers/videos/` always win.

### Offline bundles & auto-download

The two bundles (`libretro-covers.zip`, `openvgdb-covers.zip`) are shipped in
`data/covers/`, so a fresh install works with **no internet and no API keys**.
If either is missing at startup, the app downloads it from the store server's
`/api/covers/<name>.zip` endpoint (see [Server setup](#server-setup)).

Rebuild them yourself:

```bash
python3 -m app.libretro          # libretro-thumbnails -> libretro-covers.zip
python3 -m app.openvgdb          # OpenVGDB/GameFAQs  -> openvgdb-covers.zip
```

(`app.openvgdb` needs `curl_cffi` to pass GameFAQs' Cloudflare challenge.)

## PS1 support

The **PS1** tab scans a folder of PlayStation 1 rips and converts them into PSP
`EBOOT.PBP` files with a pure-Python popstation converter (no external tools).
PS1 discs (pending) and converted EBOOTs also show up in the **Library** tab
(filter by **PS1**), where you can convert a disc or copy an EBOOT to the PSP.

- **Source formats** — `.iso`, `.img`, `.bin` (with a matching `.cue` for
  multi-track discs).
- **Serial detection** — reads `SYSTEM.CNF` to derive the game ID (e.g.
  `SLUS-00001`) and writes it into `PARAM.SFO` + the PSAR header.
- **base.pbp** — conversion needs a PSN PS1 classic `EBOOT.PBP` (any game) to
  supply the `STARTDAT` block (boot logo + PGD) and a base `PARAM.SFO`/icon.
  Copy one to `data/base.PBP` and point **Settings → General → PS1 base.pbp**
  at it. Any PS1 classic works structurally — the converter finds `STARTDAT`
  at `PSAR + p1` regardless of PSAR alignment.
- **POPS loader** — the `DATA.PSP` section uses a bundled plain-ELF loader
  (`data/loader/DATA.PSP`, from iPoPS, MIT) so the EBOOT boots on modern CFW
  without the base PBP's encrypted loader failing to link (`80020148`). The
  base PBP's own loader is only used as a fallback when the bundle is missing.
- **Output** — a folder `<out>/<Game ID>/` containing `EBOOT.PBP` (raw-DEFLATE
  compressed PSAR) plus a `KEYS.BIN` (NPDRM version key used to decrypt the
  `STARTDAT` PGD), ready to copy to `<PSP>/PSP/GAME/<Game ID>/`. Compatible
  with CFW POPS (ARK-4/ARK-5).
- **Artwork** — the converter fetches the game's real cover art (libretro /
  ScreenScraper / IGDB / Wikipedia, reusing the local cover cache) and builds
  official-style PS1 Classics art:
  - `ICON0` (80×80) — the cover composited into the bundled case frame
    (`data/ps1_templates/ICON0.PNG`) with the PlayStation logo + wordmark;
  - `PIC0` (310×180) — a PSN-style info board with the game title, a
    description from IGDB, an ESRB rating badge and PS3/PSP platform badges;
  - `PIC1` (480×272) — the bundled launch background
    (`data/ps1_templates/PIC1.PNG`).
  If the template images are missing the app falls back to drawn art, and to
  the base icon when no cover is found.

Configure the folders in **Settings → General** (or `PSP_PS1_ROOTS` /
`PSP_PS1_OUTPUT_DIR`). Current limits: single-disc games only (no multi-disc
PBP); disc data is DEFLATE-compressed (level 1 by default, `--complevel 0-9`).

## Store — four sources

The **Store** tab downloads missing discs from a Redump 7z archive collection
(`Name (Region) (tags).7z`, containing the `.iso`/`.cso` inside). It has three
sources, selectable in the tab:

### 1. Local folder

Point **Settings → Store → Local store dir** (or `PSP_STORE_DIR`) at a folder of
`.7z`/`.zip` archives. The app scans the folder, lists the archives, and on
download:

1. reads the archive listing with `7z` to find the disc image inside,
2. copies the archive to a temp dir and extracts the image with `7z`,
3. moves the image into your games/videos directory.

### 2. Server (HTTP API)

See [Server setup](#server-setup). The app talks to `GET /api/store` (list) and
`GET /api/store/download/<name>` (stream), using your `PSP_API_TOKEN` as
`Authorization: Bearer <token>`.

### 3. Archive.org

Point **Settings → Store → Archive.org item** (or `PSP_ARCHIVE_ITEM`) at an
archive.org item identifier (or a full `details`/`download` URL). The app:

1. lists the item via `https://archive.org/metadata/<item>`,
2. downloads archives from `https://archive.org/download/<item>/<file>`,
3. extracts the disc image and moves it into your library.

If the item's files are marked **private** (login required), set the
`logged-in-sig` and `logged-in-user` cookies in **Settings → Store** (grab them
from a logged-in archive.org browser session). Without them, private files are
skipped — and you'll get a clear error if *everything* in the item is private.

### 4. NoPayStation

The **NoPayStation** source lists PSN content from
[NoPayStation](https://nopaystation.com/), selectable from the content-type
dropdown next to the source selector:

- **PSP Games** (`PSP_GAMES.tsv`) — includes **minis** (also listed separately
  in the `Type` column), PC Engine and NeoGeo releases.
- **PS1 Classics** (`PSX_GAMES.tsv`)
- **PSP Themes** (`PSP_THEMES.tsv`)
- **PSP DLC** (`PSP_DLCS.tsv`)

Requirements & behaviour:

1. `.pkg` files are decrypted by the built-in **pure-Python decryptor**
   (`app/pkg.py`, powered by `pycryptodome` for speed) — no Wine and no
   `pkg2zip` needed.
2. Set **NoPayStation output dir** in **Settings → Store** (default
   `$PSP_STATE_DIR/nps`).
3. Downloads land in the NPS folder and appear in the **Library**:
   - PSP game → `PSP/<TITLE_ID>/EBOOT.PBP` (`psp_kind="eboot"`).
   - PS1 Classic → `PSX/<TITLE_ID>/EBOOT.PBP` (+ `DOCUMENT.DAT`, `KEYS.BIN`).
   - Theme → `THEME/<TITLE_ID>/<content_id>.ptf` (the packaged PTF is
     EDAT-unwrapped to a raw theme file).
   - DLC → `DLC/<GAME_ID>/<content_id>.edat` (installed as-is into the owning
     game's folder).
4. Copying to the PSP puts games/PS1 under `/PSP/GAME/` (optionally nested in
   a Category Lite folder), themes under `/PSP/THEME/`, and DLC under
   `/PSP/GAME/<GAME_ID>/`.
5. The TSV lists are cached in `data/nps/` and refreshed on each store scan
   (drop the `.tsv` files there manually to work offline).

> **DRM note:** NPS content is PSN-DRM'd. To *run* it on the PSP you need the
> [`npdrm_free`](https://github.com/qwikrazor87/npdrm_free) plugin (ARK and most
> CFW already ship it). The desktop app only downloads and decrypts; it doesn't
> strip DRM.
>
> **Covers/metadata:** themes and DLC have no cover art or game metadata of
> their own — the app scrapes them like games (IGDB/libretro/etc.), which can
> return unrelated results. Downloaded themes keep their embedded preview as
> the primary cover (used instead of a scraped image when available).

## Server setup

The catalog + store API is a single-file, stdlib-only HTTP server
(`app/api.py`) — no pip install needed on the server.

```bash
PSP_DATA_DIR=/data \
PSP_DATS_DIR=/data/dats \
PSP_DB_PATH=/data/library.sqlite \
PSP_COVERS_DIR=/data/covers \
PSP_GAMES_ROOTS=/data/umd_games \
PSP_VIDEO_ROOTS=/data/umd_videos \
PSP_STORE_DIR=/data/store \
PSP_API_TOKEN=<long-random-secret> \
python3 -m app.api --host 0.0.0.0 --port 8000
```

Endpoints:

| Endpoint | Description |
|---|---|
| `GET /api/summary` | release counts per system |
| `GET /api/releases?system=&q=&limit=&offset=` | search the catalog |
| `GET /api/releases/<id>` | one release + its rom hashes |
| `GET /api/library?system=&refresh=` | your scanned library |
| `GET /api/cover/<id>` | cover image (fetched on demand) |
| `GET /api/covers/<name>.zip` | offline cover bundle (public) |
| `GET /api/store` | list store archives *(token)* |
| `GET /api/store/download/<name>` | stream one archive *(token)* |

Token protection: set `PSP_API_TOKEN`; the client sends
`Authorization: Bearer <token>`. Leave it unset to allow anonymous access
(not recommended).

### Docker (Unraid)

`docker-compose.yml` builds the API image and attaches it to Unraid's `br0`
macvlan bridge so it gets its own LAN IP (e.g. `192.168.1.100`, plain HTTP on 8000):

```bash
# in /path/to/psp-preserve/
cp .env.example .env   # then edit: LAN IP, share paths, API token
docker compose up -d --build
```

The compose file reads the LAN IP and Unraid share paths from `.env` and
mounts them into `/data/store`, `/data/umd_games` and `/data/umd_videos`
(the store is mounted read-only).

### TLS (optional)

`Caddyfile` is a ready-made reverse proxy for `https://psp.example.com` with
automatic Let's Encrypt certificates. To enable it, add the `caddy` service to
the compose file, give it its own `br0` IP, point it at the API, and forward
ports 80/443 on your router to it.

### Client

In the GUI: **Settings → Store** → enter the **Server URL** (e.g.
`http://<server-ip>:8000` or `https://psp.example.com`) and the same **API
token**, then pick **Server** in the Store tab's source selector. Use
**Archive.org** or **Local folder** the same way.

## PSP Sync

Connect a PSP over USB (or mount a UMD) and the **PSP Sync** tab detects it:

- **Games / UMD Videos / Homebrew-PSN / Themes / DLC / PKG** — tables of what's
  installed, with delete-from-PSP (checkboxes and right-click).
- **UMD** — reads `PARAM.SFO` from a mounted UMD, shows the title/serial, and
  can dump the disc image into your library.
- **Sync from Library** — copy discs from your library onto the PSP, optionally
  into a Category Lite category (set in **Settings → PSP → ISO category**).
  NoPayStation PSP games install under `/PSP/GAME`, optionally nested in the
  **PSP game category**; PS1 Classics use the **PS1 category**.

## Saves sync

The **Saves** tab shows save-data across three locations — **PSP** (USB),
**PPSSPP**, and a **Cloud folder** — and syncs both ways. Save signatures are
cached (keyed by size + mtime) so rescanning is fast, and an **Auto-sync**
option runs the sync automatically whenever a PSP is connected.

## CLI tools

```bash
# scan + report (dry-run)
python3 -m app.scan --roots /path/to/umd_games,/path/to/umd_videos

# scan + apply renames
python3 -m app.scan --roots /path/to/umd_games,/path/to/umd_videos --apply

# ingest DAT files into the library DB
python3 -m app.ingest --dats data/dats --db data/library.sqlite

# (re)build the offline cover bundles (see Cover art)
python3 -m app.libretro
python3 -m app.openvgdb

# convert one PS1 disc to EBOOT.PBP (CLI)
python3 -m app.ps1_convert --source /path/to/game.cue --title "Game" \
  --game-id SLUS00001 --base-pbp data/base.PBP --out /out/Game.pbp
```

`python3 -m app.scan --json` prints machine-readable results, and `--quick`
skips full-file hashing (serial/title match only).