# PSP Preservation Tool

![AI Generated](https://raw.githubusercontent.com/dotnetdave/ai-usage-badges/refs/heads/main/badges/png/1x/ai-generated.png)

Native Linux GUI (PySide6) for preserving PSP UMD **games** and **videos** as
`.iso` files. It scans your discs, verifies them against the full Redump catalog
(Redump DATs), reports what's missing, and retags/renames discs to a canonical
scheme. It also fetches cover art and metadata, syncs discs and save-data with a
real PSP, can pull missing discs from your own Redump 7z archive collection, and
converts PlayStation 1 ISO/CUE/BIN rips into PSP `EBOOT.PBP` homebrew.

![PSP Preservation Tool](img/qt-gui.png)
* Use default app theme for Qt.

## Features

- **Library** — scan folders of `.iso` discs, verify against the DAT catalog,
  browse with cover art, search/filter, see per-game details (developer,
  publisher, genre, release date, synopsis, trailer), and launch discs in PPSSPP.
- **Store** — browse and download discs from three sources: a local 7z folder,
  your own HTTP store server, or an archive.org item (see
  [Store](#store--archiveorg-and-local-folder)).
- **Queue** — background transfers with progress, cancel, per-row delete, etc.
- **PSP Sync** — detect a connected PSP (USB) or a mounted UMD, inventory what's
  installed (games / UMD videos / homebrew / PKG), copy discs to the PSP, delete
  from the PSP, and dump a UMD to the library.
- **Saves** — two-way sync of save-data between the PSP, PPSSPP, and a cloud
  folder.
- **PS1** — scan a folder of PlayStation 1 discs (ISO/BIN/CUE), read each disc's
  serial, convert to PSP `EBOOT.PBP` with a pure-Python popstation converter,
  and track already-converted games.
- **Settings** — directories, cover-source API keys, store/server config, and
  misc options.

## Requirements

- Linux + Python 3
- `PySide6` — the only Python dependency (`pip install -r requirements.txt`)
- `7z` (p7zip) — used to peek/extract disc images from store archives
- PPSSPP (optional) — to launch discs from the details panel

## Install

```bash
python3 -m venv .venv          # keep the venv on a local disk, not an SMB mount
. .venv/bin/activate
pip install -r requirements.txt
```

## Run the GUI

```bash
python3 -m app.main
```

## First-time setup

1. Put your DAT files in `data/dats/`:
   - games: Redump `Sony - PlayStation Portable` (authoritative hashes)
     **and** No-Intro `Sony - PlayStation Portable` (supplies the serials
     Redump lacks — keeps serial-based matching working).
   - videos: No-Intro `Sony - PlayStation Portable (UMD Video)` (Redump has
     no UMD Video DAT).
2. Ingest them into the local SQLite library:

   ```bash
   python3 -m app.ingest
   ```

3. Open the GUI and point **Settings → General → Games/Videos directory** at
   your disc folders (or set `PSP_GAMES_ROOTS` / `PSP_VIDEO_ROOTS`).
4. (Optional) PS1 conversion — copy any PSN PS1 classic `EBOOT.PBP` to
   `data/base.PBP` (it supplies Sony's encrypted loader). Then set the **PS1
   directory**, **PS1 output directory** and **PS1 base.pbp** in
   **Settings → General** and use the **PS1** tab (see [PS1 support](#ps1-support)).

Discs that aren't in the Redump DATs can be added to `data/supplement.json`
(`name`, `region`, `crc32`, `sha1`, `md5`) so the tool still recognizes them.

## Nomenclature

`Title (Region) (tags) (Serial).iso` — leading articles stay in front
(`The Sims 2`), and revision/disc tags are preserved (`(v1.03)`, `(Disc 1)`).
Discs are matched by hash first (most precise), then serial, then filename.

## Directory layout

- `data/dats/` — clrmamepro DAT files (games + videos).
- `data/supplement.json` — extra discs not in Redump.
- `data/credentials.json` — cover/metadata API keys (IGDB, TheMovieDB,
  ScreenScraper, Google Custom Search). Environment variables take precedence.
- `data/manual_covers/games|videos/` — your own cover images.
- `data/covers/` — checked-in assets: the UMD video/game cover templates,
  `libretro-index.json` + `openvgdb-index.json` catalogs, `libretro-covers.zip` +
  `openvgdb-covers.zip` offline cover bundles, and the legacy cover cache.
- `data/base.PBP` — a PSN PS1 classic EBOOT (any game) supplying `STARTDAT`
  + a base `PARAM.SFO`/icon for PS1 conversion (git-ignored, copyrighted).
- `data/loader/DATA.PSP` + `data/loader/KEYS.BIN` — bundled plain-ELF POPS
  loader and NPDRM version key (from iPoPS, MIT).
- State dir (`~/.local/state/psp-preserve/` by default, or `$XDG_STATE_HOME/psp-preserve`):
  - `library.sqlite` — the catalog + library database.
  - `covers/` — downloaded cover cache.

SQLite is kept on a local disk because SQLite locking is unreliable over
network filesystems; the DATs, manual covers and credentials stay in `data/`
(which may be a network share).

## Configuration

Everything is driven by environment variables, with GUI settings (stored in
QSettings under `PSPPreserve`) taking precedence where noted.

| Variable | Default | Purpose |
|---|---|---|
| `PSP_DATA_DIR` | `./data` | data root (DATs, credentials, manual covers) |
| `PSP_DATS_DIR` | `$PSP_DATA_DIR/dats` | DAT folder |
| `PSP_STATE_DIR` | `~/.local/state/psp-preserve` | local state root |
| `PSP_DB_PATH` | `$PSP_STATE_DIR/library.sqlite` | SQLite library |
| `PSP_COVERS_DIR` | `$PSP_STATE_DIR/covers` | cover cache |
| `PSP_GAMES_ROOTS` | `/data/umd_games` | comma-separated game disc roots |
| `PSP_VIDEO_ROOTS` | `/data/umd_videos` | comma-separated video disc roots |
| `PSP_PS1_ROOTS` | `/data/ps1` | comma-separated PS1 disc roots |
| `PSP_PS1_OUTPUT_DIR` | `/data/ps1_converted` | converted EBOOT.PBP output folder |
| `PSP_SCAN_WORKERS` | `1` | parallel hashing workers |
| `PSP_STORE_DIR` | `/media/downloads/Sony - PlayStation Portable` | local 7z store folder |
| `PSP_STORE_SERVER` | `https://psp.example.com` | store server base URL |
| `PSP_STORE_TOKEN` | *(empty)* | store server bearer token |
| `PSP_ARCHIVE_ITEM` | `psp-redump-roms321.com` | archive.org item identifier |
| `PSP_ARCHIVE_LOGGED_IN_SIG` | *(empty)* | archive.org `logged-in-sig` cookie |
| `PSP_ARCHIVE_LOGGED_IN_USER` | *(empty)* | archive.org `logged-in-user` cookie |
| `PSP_API_TOKEN` | *(empty)* | server API bearer token (unset = open) |
| `PSP_CREDENTIALS` | `$PSP_DATA_DIR/credentials.json` | credentials file |
| `PSP_LIBRETRO_COVERS_URL` | *(empty)* | direct URL for `libretro-covers.zip` (else derived from store server) |
| `PSP_OVGDB_COVERS_URL` | *(empty)* | direct URL for `openvgdb-covers.zip` (else derived from store server) |

Cover/metadata keys can also be set as env vars: `IGDB_CLIENT_ID`,
`IGDB_CLIENT_SECRET`, `TMDB_API_KEY`, `GOOGLE_API_KEY`, `GOOGLE_CX`,
`SCREENSCRAPER_DEVID`, `SCREENSCRAPER_DEVPASSWORD`, `SCREENSCRAPER_SSID`,
`SCREENSCRAPER_SSPASSWORD`, `SCREENSCRAPER_SOFTNAME`.

## Cover art

Games are resolved in this order (offline-first, then network):

1. **Manual covers** — images you drop in `data/manual_covers/games/`
   (fuzzy-matched; strips `(Region)`, handles `Title, The`).
2. **libretro-thumbnails (offline bundle)** — full PSP box art from
   `data/covers/libretro-covers.zip`. If that zip is absent, it downloads from
   GitHub and caches instead.
3. **OpenVGDB (offline bundle)** — full covers from
   `data/covers/openvgdb-covers.zip`.
4. **ScreenScraper** — full box art + rich metadata; needs a free dev
   ID/password.
5. **Wikipedia** — keyless fallback.
6. **IGDB** — needs a Twitch/IGDB client ID + secret. IGDB only stores cropped
   3:4 key art, so the app composites it into the PSP game cover template
   (`data/covers/UMD_GAME_COVER_TEMPLATE.png`) to restore the full cover.

UMD **videos** have no public box-art API. The manual **Scrape…** dialog offers
three sources: TheMovieDB posters, Wikipedia image search (keyless), and Google
Images (needs a Google API key + Search Engine ID in **Settings → Cover
sources**). The pick is composited into `data/covers/UMD_COVER_TEMPLATE.png`.
Manual covers in `data/manual_covers/videos/` always win.

### Offline bundles & auto-download

The two bundles (`libretro-covers.zip`, `openvgdb-covers.zip`) are shipped in
`data/covers/`, so a fresh install works with **no internet and no API keys**.
If either is missing at startup, the app downloads it from the store server's
`/api/covers/<name>.zip` endpoint (see [Server setup](#server-setup)).

Rebuild them yourself:

```bash
python3 -m app.libretro          # libretro-thumbnails -> libretro-covers.zip
python3 -m app.openvgdb          # OpenVGDB/GameFAQs  -> openvgdb-covers.zip
```

(`app.openvgdb` needs `curl_cffi` to pass GameFAQs' Cloudflare challenge.)

## PS1 support

The **PS1** tab scans a folder of PlayStation 1 rips and converts them into PSP
`EBOOT.PBP` files with a pure-Python popstation converter (no external tools).
PS1 discs (pending) and converted EBOOTs also show up in the **Library** tab
(filter by **PS1**), where you can convert a disc or copy an EBOOT to the PSP.

- **Source formats** — `.iso`, `.img`, `.bin` (with a matching `.cue` for
  multi-track discs).
- **Serial detection** — reads `SYSTEM.CNF` to derive the game ID (e.g.
  `SLUS-00001`) and writes it into `PARAM.SFO` + the PSAR header.
- **base.pbp** — conversion needs a PSN PS1 classic `EBOOT.PBP` (any game) to
  supply the `STARTDAT` block (boot logo + PGD) and a base `PARAM.SFO`/icon.
  Copy one to `data/base.PBP` and point **Settings → General → PS1 base.pbp**
  at it. Any PS1 classic works structurally — the converter finds `STARTDAT`
  at `PSAR + p1` regardless of PSAR alignment.
- **POPS loader** — the `DATA.PSP` section uses a bundled plain-ELF loader
  (`data/loader/DATA.PSP`, from iPoPS, MIT) so the EBOOT boots on modern CFW
  without the base PBP's encrypted loader failing to link (`80020148`). The
  base PBP's own loader is only used as a fallback when the bundle is missing.
- **Output** — a folder `<out>/<Game ID>/` containing `EBOOT.PBP` (raw-DEFLATE
  compressed PSAR) plus a `KEYS.BIN` (NPDRM version key used to decrypt the
  `STARTDAT` PGD), ready to copy to `<PSP>/PSP/GAME/<Game ID>/`. Compatible
  with CFW POPS (ARK-4/ARK-5).
- **Artwork** — the converter fetches the game's real cover art (libretro /
  ScreenScraper / IGDB / Wikipedia, reusing the local cover cache) and bakes
  it into the EBOOT's `ICON0` (144×80 XMB icon) and `PIC1` (480×272 launch
  background), so the PSP shows the actual game rather than the default PS1
  icon. Falls back to the base icon when no art is found.

Configure the folders in **Settings → General** (or `PSP_PS1_ROOTS` /
`PSP_PS1_OUTPUT_DIR`). Current limits: single-disc games only (no multi-disc
PBP); disc data is DEFLATE-compressed (level 1 by default, `--complevel 0-9`).

## Store — archive.org and local folder

The **Store** tab downloads missing discs from a Redump 7z archive collection
(`Name (Region) (tags).7z`, containing the `.iso`/`.cso` inside). It has three
sources, selectable in the tab:

### 1. Local folder

Point **Settings → Store → Local store dir** (or `PSP_STORE_DIR`) at a folder of
`.7z`/`.zip` archives. The app scans the folder, lists the archives, and on
download:

1. reads the archive listing with `7z` to find the disc image inside,
2. copies the archive to a temp dir and extracts the image with `7z`,
3. moves the image into your games/videos directory.

### 2. Server (HTTP API)

See [Server setup](#server-setup). The app talks to `GET /api/store` (list) and
`GET /api/store/download/<name>` (stream), using your `PSP_API_TOKEN` as
`Authorization: Bearer <token>`.

### 3. Archive.org

Point **Settings → Store → Archive.org item** (or `PSP_ARCHIVE_ITEM`) at an
archive.org item identifier (or a full `details`/`download` URL). The app:

1. lists the item via `https://archive.org/metadata/<item>`,
2. downloads archives from `https://archive.org/download/<item>/<file>`,
3. extracts the disc image and moves it into your library.

If the item's files are marked **private** (login required), set the
`logged-in-sig` and `logged-in-user` cookies in **Settings → Store** (grab them
from a logged-in archive.org browser session). Without them, private files are
skipped — and you'll get a clear error if *everything* in the item is private.

## Server setup

The catalog + store API is a single-file, stdlib-only HTTP server
(`app/api.py`) — no pip install needed on the server.

```bash
PSP_DATA_DIR=/data \
PSP_DATS_DIR=/data/dats \
PSP_DB_PATH=/data/library.sqlite \
PSP_COVERS_DIR=/data/covers \
PSP_GAMES_ROOTS=/data/umd_games \
PSP_VIDEO_ROOTS=/data/umd_videos \
PSP_STORE_DIR=/data/store \
PSP_API_TOKEN=<long-random-secret> \
python3 -m app.api --host 0.0.0.0 --port 8000
```

Endpoints:

| Endpoint | Description |
|---|---|
| `GET /api/summary` | release counts per system |
| `GET /api/releases?system=&q=&limit=&offset=` | search the catalog |
| `GET /api/releases/<id>` | one release + its rom hashes |
| `GET /api/library?system=&refresh=` | your scanned library |
| `GET /api/cover/<id>` | cover image (fetched on demand) |
| `GET /api/covers/<name>.zip` | offline cover bundle (public) |
| `GET /api/store` | list store archives *(token)* |
| `GET /api/store/download/<name>` | stream one archive *(token)* |

Token protection: set `PSP_API_TOKEN`; the client sends
`Authorization: Bearer <token>`. Leave it unset to allow anonymous access
(not recommended).

### Docker (Unraid)

`docker-compose.yml` builds the API image and attaches it to Unraid's `br0`
macvlan bridge so it gets its own LAN IP (e.g. `192.168.1.100`, plain HTTP on 8000):

```bash
# in /path/to/psp-preserve/ — adjust the volume paths for your shares
printf 'PSP_API_TOKEN=<long-random-secret>\n' > .env
docker compose up -d --build
```

The compose file mounts your Unraid shares into `/data/store`, `/data/umd_games`
and `/data/umd_videos` (the store is mounted read-only).

### TLS (optional)

`Caddyfile` is a ready-made reverse proxy for `https://psp.example.com` with
automatic Let's Encrypt certificates. To enable it, add the `caddy` service to
the compose file, give it its own `br0` IP, point it at the API, and forward
ports 80/443 on your router to it.

### Client

In the GUI: **Settings → Store** → enter the **Server URL** (e.g.
`http://<server-ip>:8000` or `https://psp.example.com`) and the same **API
token**, then pick **Server** in the Store tab's source selector. Use
**Archive.org** or **Local folder** the same way.

## PSP Sync

Connect a PSP over USB (or mount a UMD) and the **PSP Sync** tab detects it:

- **Games / UMD Videos / Homebrew-PSN / PKG** — tables of what's installed, with
  delete-from-PSP (checkboxes and right-click).
- **UMD** — reads `PARAM.SFO` from a mounted UMD, shows the title/serial, and
  can dump the disc image into your library.
- **Sync from Library** — copy discs from your library onto the PSP, optionally
  into a Category Lite category (set in **Settings → PSP → ISO category**).

## Saves sync

The **Saves** tab shows save-data across three locations — **PSP** (USB),
**PPSSPP**, and a **Cloud folder** — and syncs both ways. Save signatures are
cached (keyed by size + mtime) so rescanning is fast, and an **Auto-sync**
option runs the sync automatically whenever a PSP is connected.

## CLI tools

```bash
# scan + report (dry-run)
python3 -m app.scan --roots /path/to/umd_games,/path/to/umd_videos

# scan + apply renames
python3 -m app.scan --roots /path/to/umd_games,/path/to/umd_videos --apply

# ingest DAT files into the library DB
python3 -m app.ingest --dats data/dats --db data/library.sqlite

# (re)build the offline cover bundles (see Cover art)
python3 -m app.libretro
python3 -m app.openvgdb

# convert one PS1 disc to EBOOT.PBP (CLI)
python3 -m app.ps1_convert --source /path/to/game.cue --title "Game" \
  --game-id SLUS00001 --base-pbp data/base.PBP --out /out/Game.pbp
```

`python3 -m app.scan --json` prints machine-readable results, and `--quick`
skips full-file hashing (serial/title match only).