# PSP Preservation Tool

![AI Generated](https://raw.githubusercontent.com/dotnetdave/ai-usage-badges/refs/heads/main/badges/png/1x/ai-generated.png)

Native Linux GUI (PySide6) for preserving PSP UMD **games** and **videos** as
`.iso` files. It scans your discs, verifies them against the full Redump catalog
(Redump DATs), reports what's missing, and retags/renames discs to a canonical
scheme. It also fetches cover art and metadata, syncs discs and save-data with a
real PSP, can pull missing discs from your own Redump 7z archive collection, and
converts PlayStation 1 ISO/CUE/BIN rips into PSP `EBOOT.PBP` homebrew.

![PSP Preservation Tool](img/qt-gui.png)
* Use default app theme for Qt.

## Features

- **Library** — scan folders of `.iso` discs, verify against the DAT catalog,
  browse with cover art, search/filter, see per-game details (developer,
  publisher, genre, release date, synopsis, trailer), and launch discs in PPSSPP.
- **Store** — browse and download discs from three sources: a local 7z folder,
  your own HTTP store server, or an archive.org item (see
  [Store](#store--archiveorg-and-local-folder)).
- **Queue** — background transfers with progress, cancel, per-row delete, etc.
- **PSP Sync** — detect a connected PSP (USB) or a mounted UMD, inventory what's
  installed (games / UMD videos / homebrew / PKG), copy discs to the PSP, delete
  from the PSP, and dump a UMD to the library.
- **Saves** — two-way sync of save-data between the PSP, PPSSPP, and a cloud
  folder.
- **PS1** — scan a folder of PlayStation 1 discs (ISO/BIN/CUE), read each disc's
  serial, convert to PSP `EBOOT.PBP` with a pure-Python popstation converter,
  and track already-converted games.
- **Settings** — directories, cover-source API keys, store/server config, and
  misc options.

## Requirements

- Linux + Python 3
- `PySide6` — the only Python dependency (`pip install -r requirements.txt`)
- `7z` (p7zip) — used to peek/extract disc images from store archives
- PPSSPP (optional) — to launch discs from the details panel

## Install

```bash
python3 -m venv .venv          # keep the venv on a local disk, not an SMB mount
. .venv/bin/activate
pip install -r requirements.txt
```

## Run the GUI

```bash
python3 -m app.main
```

## First-time setup

1. Put your DAT files in `data/dats/`:
   - games: Redump `Sony - PlayStation Portable` (authoritative hashes)
     **and** No-Intro `Sony - PlayStation Portable` (supplies the serials
     Redump lacks — keeps serial-based matching working).
   - videos: No-Intro `Sony - PlayStation Portable (UMD Video)` (Redump has
     no UMD Video DAT).
2. Ingest them into the local SQLite library:

   ```bash
   python3 -m app.ingest
   ```

3. Open the GUI and point **Settings → General → Games/Videos directory** at
   your disc folders (or set `PSP_GAMES_ROOTS` / `PSP_VIDEO_ROOTS`).
4. (Optional) PS1 conversion — copy any PSN PS1 classic `EBOOT.PBP` to
   `data/base.PBP` (it supplies Sony's encrypted loader). Then set the **PS1
   directory**, **PS1 output directory** and **PS1 base.pbp** in
   **Settings → General** and use the **PS1** tab (see [PS1 support](#ps1-support)).

Discs that aren't in the Redump DATs can be added to `data/supplement.json`
(`name`, `region`, `crc32`, `sha1`, `md5`) so the tool still recognizes them.

## Nomenclature

`Title (Region) (tags) (Serial).iso` — leading articles stay in front
(`The Sims 2`), and revision/disc tags are preserved (`(v1.03)`, `(Disc 1)`).
Discs are matched by hash first (most precise), then serial, then filename.

## Directory layout

- `data/dats/` — clrmamepro DAT files (games + videos).
- `data/supplement.json` — extra discs not in Redump.
- `data/credentials.json` — cover/metadata API keys (IGDB, TheMovieDB,
  ScreenScraper, Google Custom Search). Environment variables take precedence.
- `data/manual_covers/games|videos/` — your own cover images.
- `data/covers/` — checked-in assets: the UMD video/game cover templates,
  `libretro-index.json` + `openvgdb-index.json` catalogs, `libretro-covers.zip` +
  `openvgdb-covers.zip` offline cover bundles, and the legacy cover cache.
- `data/base.PBP` — a PSN PS1 classic EBOOT (any game) used as the encrypted
  loader when converting PS1 discs to `EBOOT.PBP` (git-ignored, copyrighted).
- State dir (`~/.local/state/psp-preserve/` by default, or `$XDG_STATE_HOME/psp-preserve`):
  - `library.sqlite` — the catalog + library database.
  - `covers/` — downloaded cover cache.

SQLite is kept on a local disk because SQLite locking is unreliable over
network filesystems; the DATs, manual covers and credentials stay in `data/`
(which may be a network share).

## Configuration

Everything is driven by environment variables, with GUI settings (stored in
QSettings under `PSPPreserve`) taking precedence where noted.

| Variable | Default | Purpose |
|---|---|---|
| `PSP_DATA_DIR` | `./data` | data root (DATs, credentials, manual covers) |
| `PSP_DATS_DIR` | `$PSP_DATA_DIR/dats` | DAT folder |
| `PSP_STATE_DIR` | `~/.local/state/psp-preserve` | local state root |
| `PSP_DB_PATH` | `$PSP_STATE_DIR/library.sqlite` | SQLite library |
| `PSP_COVERS_DIR` | `$PSP_STATE_DIR/covers` | cover cache |
| `PSP_GAMES_ROOTS` | `/data/umd_games` | comma-separated game disc roots |
| `PSP_VIDEO_ROOTS` | `/data/umd_videos` | comma-separated video disc roots |
| `PSP_PS1_ROOTS` | `/data/ps1` | comma-separated PS1 disc roots |
| `PSP_PS1_OUTPUT_DIR` | `/data/ps1_converted` | converted EBOOT.PBP output folder |
| `PSP_SCAN_WORKERS` | `1` | parallel hashing workers |
| `PSP_STORE_DIR` | `/media/downloads/Sony - PlayStation Portable` | local 7z store folder |
| `PSP_STORE_SERVER` | `https://psp.example.com` | store server base URL |
| `PSP_STORE_TOKEN` | *(empty)* | store server bearer token |
| `PSP_ARCHIVE_ITEM` | `psp-redump-roms321.com` | archive.org item identifier |
| `PSP_ARCHIVE_LOGGED_IN_SIG` | *(empty)* | archive.org `logged-in-sig` cookie |
| `PSP_ARCHIVE_LOGGED_IN_USER` | *(empty)* | archive.org `logged-in-user` cookie |
| `PSP_API_TOKEN` | *(empty)* | server API bearer token (unset = open) |
| `PSP_CREDENTIALS` | `$PSP_DATA_DIR/credentials.json` | credentials file |
| `PSP_LIBRETRO_COVERS_URL` | *(empty)* | direct URL for `libretro-covers.zip` (else derived from store server) |
| `PSP_OVGDB_COVERS_URL` | *(empty)* | direct URL for `openvgdb-covers.zip` (else derived from store server) |

Cover/metadata keys can also be set as env vars: `IGDB_CLIENT_ID`,
`IGDB_CLIENT_SECRET`, `TMDB_API_KEY`, `GOOGLE_API_KEY`, `GOOGLE_CX`,
`SCREENSCRAPER_DEVID`, `SCREENSCRAPER_DEVPASSWORD`, `SCREENSCRAPER_SSID`,
`SCREENSCRAPER_SSPASSWORD`, `SCREENSCRAPER_SOFTNAME`.

## Cover art

Games are resolved in this order (offline-first, then network):

1. **Manual covers** — images you drop in `data/manual_covers/games/`
   (fuzzy-matched; strips `(Region)`, handles `Title, The`).
2. **libretro-thumbnails (offline bundle)** — full PSP box art from
   `data/covers/libretro-covers.zip`. If that zip is absent, it downloads from
   GitHub and caches instead.
3. **OpenVGDB (offline bundle)** — full covers from
   `data/covers/openvgdb-covers.zip`.
4. **ScreenScraper** — full box art + rich metadata; needs a free dev
   ID/password.
5. **Wikipedia** — keyless fallback.
6. **IGDB** — needs a Twitch/IGDB client ID + secret. IGDB only stores cropped
   3:4 key art, so the app composites it into the PSP game cover template
   (`data/covers/UMD_GAME_COVER_TEMPLATE.png`) to restore the full cover.

UMD **videos** have no public box-art API. The manual **Scrape…** dialog offers
three sources: TheMovieDB posters, Wikipedia image search (keyless), and Google
Images (needs a Google API key + Search Engine ID in **Settings → Cover
sources**). The pick is composited into `data/covers/UMD_COVER_TEMPLATE.png`.
Manual covers in `data/manual_covers/videos/` always win.

### Offline bundles & auto-download

The two bundles (`libretro-covers.zip`, `openvgdb-covers.zip`) are shipped in
`data/covers/`, so a fresh install works with **no internet and no API keys**.
If either is missing at startup, the app downloads it from the store server's
`/api/covers/<name>.zip` endpoint (see [Server setup](#server-setup)).

Rebuild them yourself:

```bash
python3 -m app.libretro          # libretro-thumbnails -> libretro-covers.zip
python3 -m app.openvgdb          # OpenVGDB/GameFAQs  -> openvgdb-covers.zip
```

(`app.openvgdb` needs `curl_cffi` to pass GameFAQs' Cloudflare challenge.)

## PS1 support

The **PS1** tab scans a folder of PlayStation 1 rips and converts them into PSP
`EBOOT.PBP` files with a pure-Python popstation converter (no external tools).

- **Source formats** — `.iso`, `.img`, `.bin` (with a matching `.cue` for
  multi-track discs).
- **Serial detection** — reads `SYSTEM.CNF` to derive the game ID (e.g.
  `SLUS-00001`) and writes it into `PARAM.SFO` + the PSAR header.
- **base.pbp** — conversion needs a PSN PS1 classic `EBOOT.PBP` (any game) to
  supply Sony's encrypted `DATA.PSP` loader. Copy one to `data/base.PBP` and
  point **Settings → General → PS1 base.pbp** at it.
- **Output** — `Title (Game ID).pbp` in the **PS1 output directory**. To run on
  a PSP, copy each one to `<PSP>/PSP/GAME/<Game ID>/EBOOT.PBP`.

Configure the folders in **Settings → General** (or `PSP_PS1_ROOTS` /
`PSP_PS1_OUTPUT_DIR`). Current limits: single-disc games only (no multi-disc
PBP), and images are stored uncompressed (LZRC compression is on the roadmap).

## Store — archive.org and local folder

The **Store** tab downloads missing discs from a Redump 7z archive collection
(`Name (Region) (tags).7z`, containing the `.iso`/`.cso` inside). It has three
sources, selectable in the tab:

### 1. Local folder

Point **Settings → Store → Local store dir** (or `PSP_STORE_DIR`) at a folder of
`.7z`/`.zip` archives. The app scans the folder, lists the archives, and on
download:

1. reads the archive listing with `7z` to find the disc image inside,
2. copies the archive to a temp dir and extracts the image with `7z`,
3. moves the image into your games/videos directory.

### 2. Server (HTTP API)

See [Server setup](#server-setup). The app talks to `GET /api/store` (list) and
`GET /api/store/download/<name>` (stream), using your `PSP_API_TOKEN` as
`Authorization: Bearer <token>`.

### 3. Archive.org

Point **Settings → Store → Archive.org item** (or `PSP_ARCHIVE_ITEM`) at an
archive.org item identifier (or a full `details`/`download` URL). The app:

1. lists the item via `https://archive.org/metadata/<item>`,
2. downloads archives from `https://archive.org/download/<item>/<file>`,
3. extracts the disc image and moves it into your library.

If the item's files are marked **private** (login required), set the
`logged-in-sig` and `logged-in-user` cookies in **Settings → Store** (grab them
from a logged-in archive.org browser session). Without them, private files are
skipped — and you'll get a clear error if *everything* in the item is private.

## Server setup

The catalog + store API is a single-file, stdlib-only HTTP server
(`app/api.py`) — no pip install needed on the server.

```bash
PSP_DATA_DIR=/data \
PSP_DATS_DIR=/data/dats \
PSP_DB_PATH=/data/library.sqlite \
PSP_COVERS_DIR=/data/covers \
PSP_GAMES_ROOTS=/data/umd_games \
PSP_VIDEO_ROOTS=/data/umd_videos \
PSP_STORE_DIR=/data/store \
PSP_API_TOKEN=<long-random-secret> \
python3 -m app.api --host 0.0.0.0 --port 8000
```

Endpoints:

| Endpoint | Description |
|---|---|
| `GET /api/summary` | release counts per system |
| `GET /api/releases?system=&q=&limit=&offset=` | search the catalog |
| `GET /api/releases/<id>` | one release + its rom hashes |
| `GET /api/library?system=&refresh=` | your scanned library |
| `GET /api/cover/<id>` | cover image (fetched on demand) |
| `GET /api/covers/<name>.zip` | offline cover bundle (public) |
| `GET /api/store` | list store archives *(token)* |
| `GET /api/store/download/<name>` | stream one archive *(token)* |

Token protection: set `PSP_API_TOKEN`; the client sends
`Authorization: Bearer <token>`. Leave it unset to allow anonymous access
(not recommended).

### Docker (Unraid)

`docker-compose.yml` builds the API image and attaches it to Unraid's `br0`
macvlan bridge so it gets its own LAN IP (e.g. `192.168.1.100`, plain HTTP on 8000):

```bash
# in /path/to/psp-preserve/ — adjust the volume paths for your shares
printf 'PSP_API_TOKEN=<long-random-secret>\n' > .env
docker compose up -d --build
```

The compose file mounts your Unraid shares into `/data/store`, `/data/umd_games`
and `/data/umd_videos` (the store is mounted read-only).

### TLS (optional)

`Caddyfile` is a ready-made reverse proxy for `https://psp.example.com` with
automatic Let's Encrypt certificates. To enable it, add the `caddy` service to
the compose file, give it its own `br0` IP, point it at the API, and forward
ports 80/443 on your router to it.

### Client

In the GUI: **Settings → Store** → enter the **Server URL** (e.g.
`http://<server-ip>:8000` or `https://psp.example.com`) and the same **API
token**, then pick **Server** in the Store tab's source selector. Use
**Archive.org** or **Local folder** the same way.

## PSP Sync

Connect a PSP over USB (or mount a UMD) and the **PSP Sync** tab detects it:

- **Games / UMD Videos / Homebrew-PSN / PKG** — tables of what's installed, with
  delete-from-PSP (checkboxes and right-click).
- **UMD** — reads `PARAM.SFO` from a mounted UMD, shows the title/serial, and
  can dump the disc image into your library.
- **Sync from Library** — copy discs from your library onto the PSP, optionally
  into a Category Lite category (set in **Settings → PSP → ISO category**).

## Saves sync

The **Saves** tab shows save-data across three locations — **PSP** (USB),
**PPSSPP**, and a **Cloud folder** — and syncs both ways. Save signatures are
cached (keyed by size + mtime) so rescanning is fast, and an **Auto-sync**
option runs the sync automatically whenever a PSP is connected.

## CLI tools

```bash
# scan + report (dry-run)
python3 -m app.scan --roots /path/to/umd_games,/path/to/umd_videos

# scan + apply renames
python3 -m app.scan --roots /path/to/umd_games,/path/to/umd_videos --apply

# ingest DAT files into the library DB
python3 -m app.ingest --dats data/dats --db data/library.sqlite

# (re)build the offline cover bundles (see Cover art)
python3 -m app.libretro
python3 -m app.openvgdb

# convert one PS1 disc to EBOOT.PBP (CLI)
python3 -m app.ps1_convert --source /path/to/game.cue --title "Game" \
  --game-id SLUS00001 --base-pbp data/base.PBP --out /out/Game.pbp
```

`python3 -m app.scan --json` prints machine-readable results, and `--quick`
skips full-file hashing (serial/title match only).