Compare commits

...
14 Commits
Author SHA1 Message Date
quoije 02b4adb1c1 PS1: match popstation PSAR header byte-for-byte; drop fake KEYS.BIN/DOCUMENT.DAT 2026-09-07 16:20:39 -04:00
quoije a9c371c221 PS1: output PSN-style game folders (EBOOT.PBP + KEYS.BIN + DOCUMENT.DAT) 2026-09-07 16:20:39 -04:00
quoije 4d5c48cb94 Add PS1 games to the Library with covers, metadata, and EBOOT actions 2026-09-07 16:20:39 -04:00
quoije 777d316c0f Docs: document PS1 support, cover sources, and new env vars 2026-09-07 16:20:39 -04:00
quoije 625e2f8718 UMD videos: add Wikipedia + Google image cover sources to scrape 2026-09-07 16:20:39 -04:00
quoije 7d8b449846 Scrape: restore full-size movie poster URL for UMD video covers 2026-09-07 16:20:39 -04:00
quoije 27e482d344 Library: refresh details cover box after a scrape 2026-09-07 16:20:39 -04:00
quoije 5d619e5f59 Scrape: include libretro + OpenVGDB covers alongside IGDB 2026-09-07 16:20:39 -04:00
quoije 743d5e8e90 Phase 2: PS1 library scan, settings, conversion queue, and PS1 tab 2026-09-07 16:20:39 -04:00
quoije bb1b8dea66 Use Redump + No-Intro DATs together for games; dedupe fuzzy matches 2026-09-07 16:20:39 -04:00
quoije 2d599141f9 Add Logiqx XML DAT parsing (Redump) and clear stale file matches on re-ingest 2026-09-07 16:20:39 -04:00
quoije 922c01d17c Target Redump DATs/catalog instead of No-Intro 2026-09-07 16:20:39 -04:00
quoije 128b62e5b8 Generate PS1 TOC in EBOOT converter 2026-09-07 16:20:39 -04:00
quoije da3429d918 Add pure-Python PS1 to EBOOT.PBP converter (phase 1) 2026-09-07 16:20:39 -04:00
19 changed files with 19390 additions and 149 deletions
+4
View File
@@ -6,3 +6,7 @@ data/*.sqlite
data/*.sqlite-*
data/credentials.json
session-transcript-2026-09-01.md
# Copyrighted PSN EBOOTs (base.pbp) used only for local PS1 conversion testing
data/*.PBP
data/*.pbp
+60 -15
View File
@@ -3,10 +3,11 @@
![AI Generated](https://raw.githubusercontent.com/dotnetdave/ai-usage-badges/refs/heads/main/badges/png/1x/ai-generated.png)
Native Linux GUI (PySide6) for preserving PSP UMD **games** and **videos** as
`.iso` files. It scans your discs, verifies them against a full catalog
(No-Intro DATs), reports what's missing, and retags/renames discs to a canonical
`.iso` files. It scans your discs, verifies them against the full Redump catalog
(Redump DATs), reports what's missing, and retags/renames discs to a canonical
scheme. It also fetches cover art and metadata, syncs discs and save-data with a
real PSP, and can pull missing discs from your own Redump 7z archive collection.
real PSP, can pull missing discs from your own Redump 7z archive collection, and
converts PlayStation 1 ISO/CUE/BIN rips into PSP `EBOOT.PBP` homebrew.
![PSP Preservation Tool](img/qt-gui.png)
* Use default app theme for Qt.
@@ -25,6 +26,9 @@ real PSP, and can pull missing discs from your own Redump 7z archive collection.
from the PSP, and dump a UMD to the library.
- **Saves** — two-way sync of save-data between the PSP, PPSSPP, and a cloud
folder.
- **PS1** — scan a folder of PlayStation 1 discs (ISO/BIN/CUE), read each disc's
serial, convert to PSP `EBOOT.PBP` with a pure-Python popstation converter,
and track already-converted games.
- **Settings** — directories, cover-source API keys, store/server config, and
misc options.
@@ -52,8 +56,11 @@ python3 -m app.main
## First-time setup
1. Put your DAT files in `data/dats/`:
- games: No-Intro `Sony - PlayStation Portable`
- videos: No-Intro `Sony - PlayStation Portable (UMD Video)`
- games: Redump `Sony - PlayStation Portable` (authoritative hashes)
**and** No-Intro `Sony - PlayStation Portable` (supplies the serials
Redump lacks — keeps serial-based matching working).
- videos: No-Intro `Sony - PlayStation Portable (UMD Video)` (Redump has
no UMD Video DAT).
2. Ingest them into the local SQLite library:
```bash
@@ -62,8 +69,12 @@ python3 -m app.main
3. Open the GUI and point **Settings → General → Games/Videos directory** at
your disc folders (or set `PSP_GAMES_ROOTS` / `PSP_VIDEO_ROOTS`).
4. (Optional) PS1 conversion — copy any PSN PS1 classic `EBOOT.PBP` to
`data/base.PBP` (it supplies Sony's encrypted loader). Then set the **PS1
directory**, **PS1 output directory** and **PS1 base.pbp** in
**Settings → General** and use the **PS1** tab (see [PS1 support](#ps1-support)).
Discs that aren't in the No-Intro DATs can be added to `data/supplement.json`
Discs that aren't in the Redump DATs can be added to `data/supplement.json`
(`name`, `region`, `crc32`, `sha1`, `md5`) so the tool still recognizes them.
## Nomenclature
@@ -75,13 +86,15 @@ Discs are matched by hash first (most precise), then serial, then filename.
## Directory layout
- `data/dats/` — clrmamepro DAT files (games + videos).
- `data/supplement.json` — extra discs not in No-Intro.
- `data/supplement.json` — extra discs not in Redump.
- `data/credentials.json` — cover/metadata API keys (IGDB, TheMovieDB,
ScreenScraper). Environment variables take precedence.
ScreenScraper, Google Custom Search). Environment variables take precedence.
- `data/manual_covers/games|videos/` — your own cover images.
- `data/covers/` — checked-in assets: the UMD video/game cover templates,
`libretro-index.json` catalog, `libretro-covers.zip` + `openvgdb-covers.zip`
offline cover bundles, and the legacy cover cache.
`libretro-index.json` + `openvgdb-index.json` catalogs, `libretro-covers.zip` +
`openvgdb-covers.zip` offline cover bundles, and the legacy cover cache.
- `data/base.PBP` — a PSN PS1 classic EBOOT (any game) used as the encrypted
loader when converting PS1 discs to `EBOOT.PBP` (git-ignored, copyrighted).
- State dir (`~/.local/state/psp-preserve/` by default, or `$XDG_STATE_HOME/psp-preserve`):
- `library.sqlite` — the catalog + library database.
- `covers/` — downloaded cover cache.
@@ -104,6 +117,8 @@ QSettings under `PSPPreserve`) taking precedence where noted.
| `PSP_COVERS_DIR` | `$PSP_STATE_DIR/covers` | cover cache |
| `PSP_GAMES_ROOTS` | `/data/umd_games` | comma-separated game disc roots |
| `PSP_VIDEO_ROOTS` | `/data/umd_videos` | comma-separated video disc roots |
| `PSP_PS1_ROOTS` | `/data/ps1` | comma-separated PS1 disc roots |
| `PSP_PS1_OUTPUT_DIR` | `/data/ps1_converted` | converted EBOOT.PBP output folder |
| `PSP_SCAN_WORKERS` | `1` | parallel hashing workers |
| `PSP_STORE_DIR` | `/media/downloads/Sony - PlayStation Portable` | local 7z store folder |
| `PSP_STORE_SERVER` | `https://psp.example.com` | store server base URL |
@@ -117,9 +132,9 @@ QSettings under `PSPPreserve`) taking precedence where noted.
| `PSP_OVGDB_COVERS_URL` | *(empty)* | direct URL for `openvgdb-covers.zip` (else derived from store server) |
Cover/metadata keys can also be set as env vars: `IGDB_CLIENT_ID`,
`IGDB_CLIENT_SECRET`, `TMDB_API_KEY`, `SCREENSCRAPER_DEVID`,
`SCREENSCRAPER_DEVPASSWORD`, `SCREENSCRAPER_SSID`, `SCREENSCRAPER_SSPASSWORD`,
`SCREENSCRAPER_SOFTNAME`.
`IGDB_CLIENT_SECRET`, `TMDB_API_KEY`, `GOOGLE_API_KEY`, `GOOGLE_CX`,
`SCREENSCRAPER_DEVID`, `SCREENSCRAPER_DEVPASSWORD`, `SCREENSCRAPER_SSID`,
`SCREENSCRAPER_SSPASSWORD`, `SCREENSCRAPER_SOFTNAME`.
## Cover art
@@ -139,8 +154,11 @@ Games are resolved in this order (offline-first, then network):
3:4 key art, so the app composites it into the PSP game cover template
(`data/covers/UMD_GAME_COVER_TEMPLATE.png`) to restore the full cover.
UMD **videos** have no public box-art API: they use your manual covers, or a
TheMovieDB poster composited into `data/covers/UMD_COVER_TEMPLATE.png`.
UMD **videos** have no public box-art API. The manual **Scrape…** dialog offers
three sources: TheMovieDB posters, Wikipedia image search (keyless), and Google
Images (needs a Google API key + Search Engine ID in **Settings → Cover
sources**). The pick is composited into `data/covers/UMD_COVER_TEMPLATE.png`.
Manual covers in `data/manual_covers/videos/` always win.
### Offline bundles & auto-download
@@ -158,6 +176,29 @@ python3 -m app.openvgdb # OpenVGDB/GameFAQs -> openvgdb-covers.zip
(`app.openvgdb` needs `curl_cffi` to pass GameFAQs' Cloudflare challenge.)
## PS1 support
The **PS1** tab scans a folder of PlayStation 1 rips and converts them into PSP
`EBOOT.PBP` files with a pure-Python popstation converter (no external tools).
PS1 discs (pending) and converted EBOOTs also show up in the **Library** tab
(filter by **PS1**), where you can convert a disc or copy an EBOOT to the PSP.
- **Source formats** — `.iso`, `.img`, `.bin` (with a matching `.cue` for
multi-track discs).
- **Serial detection** — reads `SYSTEM.CNF` to derive the game ID (e.g.
`SLUS-00001`) and writes it into `PARAM.SFO` + the PSAR header.
- **base.pbp** — conversion needs a PSN PS1 classic `EBOOT.PBP` (any game) to
supply Sony's encrypted `DATA.PSP` loader. Copy one to `data/base.PBP` and
point **Settings → General → PS1 base.pbp** at it.
- **Output** — a folder `<out>/<Game ID>/` containing a single `EBOOT.PBP`
(the popstation way: uncompressed level-0 PSAR, no `KEYS.BIN`/`DOCUMENT.DAT`),
ready to copy to `<PSP>/PSP/GAME/<Game ID>/`. Compatible with CFW POPS
(ARK-4).
Configure the folders in **Settings → General** (or `PSP_PS1_ROOTS` /
`PSP_PS1_OUTPUT_DIR`). Current limits: single-disc games only (no multi-disc
PBP), and images are stored uncompressed (LZRC compression is on the roadmap).
## Store — archive.org and local folder
The **Store** tab downloads missing discs from a Redump 7z archive collection
@@ -289,6 +330,10 @@ python3 -m app.ingest --dats data/dats --db data/library.sqlite
# (re)build the offline cover bundles (see Cover art)
python3 -m app.libretro
python3 -m app.openvgdb
# convert one PS1 disc to EBOOT.PBP (CLI)
python3 -m app.ps1_convert --source /path/to/game.cue --title "Game" \
--game-id SLUS00001 --base-pbp data/base.PBP --out /out/Game.pbp
```
`python3 -m app.scan --json` prints machine-readable results, and `--quick`
+7
View File
@@ -50,10 +50,17 @@ ARCHIVE_LOGGED_IN_USER = os.environ.get("PSP_ARCHIVE_LOGGED_IN_USER", "")
GAMES_ROOTS = [Path(p) for p in os.environ.get("PSP_GAMES_ROOTS", "/data/umd_games").split(",") if p.strip()]
VIDEO_ROOTS = [Path(p) for p in os.environ.get("PSP_VIDEO_ROOTS", "/data/umd_videos").split(",") if p.strip()]
# PS1: source folder (raw ISO/BIN/CUE) and converted EBOOT.PBP output folder.
PS1_ROOTS = [Path(p) for p in os.environ.get("PSP_PS1_ROOTS", "/data/ps1").split(",") if p.strip()]
PS1_OUTPUT_DIR = Path(os.environ.get("PSP_PS1_OUTPUT_DIR", "/data/ps1_converted"))
# Only .iso discs are tracked; everything below is explicitly ignored.
DISC_EXTENSIONS = {".iso"}
IGNORED_EXTENSIONS = {".pkg", ".pbp", ".cso"}
# PS1 source images (scanned separately from PSP .iso discs).
PS1_EXTENSIONS = {".cue", ".iso", ".img", ".bin"}
# Parallel hashing workers for the scanner. Keep at 1 on spinning disks
# (concurrent reads cause seek thrash); raise it for SSD / fast arrays.
SCAN_WORKERS = int(os.environ.get("PSP_SCAN_WORKERS", "1"))
+268 -80
View File
@@ -24,18 +24,34 @@ from .db import get_connection
log = logging.getLogger("covers")
_LIBRETRO_BASE = "https://raw.githubusercontent.com/libretro-thumbnails/Sony_-_PlayStation_Portable/master"
_LIBRETRO_KINDS = ("Named_Boxarts",)
_TREE_URL = "https://api.github.com/repos/libretro-thumbnails/Sony_-_PlayStation_Portable/git/trees/master?recursive=1"
_PLATFORMS = {
"games": {"libretro_repo": "Sony_-_PlayStation_Portable", "igdb_id": 38, "ss_id": "61"},
"ps1": {"libretro_repo": "Sony_-_PlayStation", "igdb_id": 7, "ss_id": "57"},
}
def _platform(system):
"""Platform parameters (libretro repo, IGDB id, ScreenScraper id)."""
return _PLATFORMS.get(system) or _PLATFORMS["games"]
def _libretro_base(system="games"):
return ("https://raw.githubusercontent.com/libretro-thumbnails/"
f"{_platform(system)['libretro_repo']}/master")
def _libretro_tree_url(system="games"):
return ("https://api.github.com/repos/libretro-thumbnails/"
f"{_platform(system)['libretro_repo']}/git/trees/master?recursive=1")
_SS_API = "https://api.screenscraper.fr/api2/jeuInfos.php"
_SS_PSP_ID = "61" # Sony PSP
_IGDB_TOKEN_URL = "https://id.twitch.tv/oauth2/token"
_IGDB_API = "https://api.igdb.com/v4/games"
_IGDB_IMAGE = "https://images.igdb.com/igdb/image/upload/"
_IGDB_PSP_ID = 38 # Sony PSP
_TMDB_SEARCH = "https://api.themoviedb.org/3/search/movie"
_TMDB_IMAGE = "https://image.tmdb.org/t/p/"
_GOOGLE_CSE_API = "https://www.googleapis.com/customsearch/v1"
_MANUAL_DIR = config.DATA_DIR / "manual_covers"
_OVGDB_ZIP = config.DATA_DIR / "covers" / "openvgdb-covers.zip"
_LIBRETRO_ZIP = config.DATA_DIR / "covers" / "libretro-covers.zip"
@@ -51,7 +67,7 @@ _igdb_token_state = {"token": "", "expires": 0.0} # IGDB OAuth2 token cache
_igdb_lock = threading.Lock()
_igdb_last = 0.0 # monotonic clock of the last IGDB API call
_index_state = {"ts": 0.0, "names": {}, "norm": {}}
_index_state = {} # system -> {"ts", "names", "norm"}
_INDEX_TTL = 86400.0 # refresh the libretro file index once a day
_neg_cache = {} # cache key -> time.time() of the last failed fetch
@@ -186,9 +202,9 @@ def ensure_cover_bundles(server_url=""):
return downloaded
def cover_url(name, kind="Named_Boxarts"):
def cover_url(name, kind="Named_Boxarts", system="games"):
"""Public libretro URL of an image for a catalog name."""
return f"{_LIBRETRO_BASE}/{kind}/{urllib.parse.quote(name)}.png"
return f"{_libretro_base(system)}/{kind}/{urllib.parse.quote(name)}.png"
def cover_path(release_id, system):
@@ -260,7 +276,7 @@ def _candidate_names(name):
yield s
def _fetch_png(name, kind, timeout):
def _fetch_png(name, kind, timeout, system="games"):
"""Fetch a real image from libretro, following symlink text bodies.
GitHub serves a repo symlink as a tiny text body containing the target
@@ -268,7 +284,7 @@ def _fetch_png(name, kind, timeout):
"""
target = name
for _ in range(_MAX_HOPS + 1):
data = _http_get(cover_url(target, kind), timeout)
data = _http_get(cover_url(target, kind, system), timeout)
if data is None:
return None
if _is_image(data):
@@ -283,26 +299,29 @@ def _fetch_png(name, kind, timeout):
# --------------------------------------------------------------------------- #
# libretro filename index + fuzzy matching
# --------------------------------------------------------------------------- #
def _index_path():
def _index_path(system="games"):
if system == "games":
return config.COVERS_DIR / "libretro-index.json"
return config.COVERS_DIR / f"libretro-index-{system}.json"
def _load_index(force=False):
st = _index_state
def _load_index(force=False, system="games"):
st = _index_state.get(system) or {"ts": 0.0, "names": {}, "norm": {}}
_index_state[system] = st
if not force and st["names"] and time.time() - st["ts"] < _INDEX_TTL:
return st["names"]
path = _index_path()
path = _index_path(system)
if not force and path.exists():
try:
# Load the on-disk index so it doubles as a fallback if the network
# is down, but only return it immediately while it's still fresh.
names = json.loads(path.read_text(encoding="utf-8"))
_set_index(names)
_set_index(names, system)
if time.time() - path.stat().st_mtime < _INDEX_TTL:
return names
except Exception:
pass
data = _http_get(_TREE_URL, timeout=30)
data = _http_get(_libretro_tree_url(system), timeout=30)
if data:
try:
tree = json.loads(data).get("tree", [])
@@ -313,7 +332,7 @@ def _load_index(force=False):
prefix = kind + "/"
if p.startswith(prefix) and p.endswith(".png"):
names[kind].append(p[len(prefix):-4])
_set_index(names)
_set_index(names, system)
try:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(names), encoding="utf-8")
@@ -325,9 +344,10 @@ def _load_index(force=False):
return st["names"] or {kind: [] for kind in _LIBRETRO_KINDS}
def _set_index(names):
def _set_index(names, system="games"):
"""Store the raw filename index plus a pre-normalized copy for fast lookup."""
st = _index_state
st = _index_state.get(system) or {"ts": 0.0, "names": {}, "norm": {}}
_index_state[system] = st
st["names"] = names
st["norm"] = {k: [(_norm(n), n) for n in v] for k, v in names.items()}
st["ts"] = time.time()
@@ -343,11 +363,12 @@ def _plain(s):
return re.sub(r"\s*\([^()]*\)", "", s).strip()
def _index_best(name, kind):
def _index_best(name, kind, system="games"):
"""Return the best-matching repo filename for `name`, or None."""
st = _index_state
st = _index_state.get(system) or {"ts": 0.0, "names": {}, "norm": {}}
if not st["norm"]:
_load_index()
_load_index(system=system)
st = _index_state.get(system) or st
pairs = st["norm"].get(kind, [])
db = _norm(name)
if not pairs or len(db) < 5:
@@ -384,8 +405,9 @@ def _index_best(name, kind):
return best
def _fetch_libretro(name, timeout, indexed_only=False):
# 0. Shipped offline bundle first (fast, no network).
def _fetch_libretro(name, timeout, system="games", indexed_only=False):
# 0. Shipped offline bundle first (fast, no network) — PSP only for now.
if system == "games":
data = fetch_libretro_local(name)
if data is not None:
return data
@@ -393,15 +415,15 @@ def _fetch_libretro(name, timeout, indexed_only=False):
# 1. Cheap: exact + progressively stripped names (no index needed).
for kind in _LIBRETRO_KINDS:
for cand in _candidate_names(name):
data = _fetch_png(cand, kind, timeout)
data = _fetch_png(cand, kind, timeout, system)
if data is not None:
return data
# 2. Fuzzy/exact via the precomputed index (covers exact names too, so the
# bulk loader skips the wasteful exact pass entirely).
for kind in _LIBRETRO_KINDS:
best = _index_best(name, kind)
best = _index_best(name, kind, system)
if best:
data = _fetch_png(best, kind, timeout)
data = _fetch_png(best, kind, timeout, system)
if data is not None:
return data
return None
@@ -466,6 +488,33 @@ def _ovgdb_name_list():
return _ovgdb_names
_ovgdb_index = None
def _load_ovgdb_index():
"""Load {key: {"title", "url"}} written by app/openvgdb.py, or {}.
Looks in the runtime covers dir first, then the shipped ``data/covers``
copy (so a fresh install still shows readable titles without re-running
the full OpenVGDB scrape).
"""
global _ovgdb_index
if _ovgdb_index is not None:
return _ovgdb_index
_ovgdb_index = {}
for path in (
config.COVERS_DIR / "openvgdb-index.json",
config.LEGACY_COVERS_DIR / "openvgdb-index.json",
):
if path.exists():
try:
_ovgdb_index = json.loads(path.read_text(encoding="utf-8"))
break
except Exception: # noqa: BLE001
_ovgdb_index = {}
return _ovgdb_index
def fetch_openvgdb_local(name):
"""Return a bundled OpenVGDB cover image for `name`, or None."""
names = _ovgdb_name_list()
@@ -549,7 +598,7 @@ def _screenscraper_box_urls(jeu):
return urls
def _fetch_screenscraper(name, timeout):
def _fetch_screenscraper(name, timeout, system="games"):
creds = sources.load()
devid = creds.get("screenscraper_devid")
devpassword = creds.get("screenscraper_devpassword")
@@ -560,7 +609,7 @@ def _fetch_screenscraper(name, timeout):
"devpassword": devpassword,
"softname": creds.get("screenscraper_softname") or "psp-preserve",
"output": "json",
"systemeid": _SS_PSP_ID,
"systemeid": _platform(system)["ss_id"],
"romtype": "rom",
"romnom": name,
}
@@ -806,19 +855,20 @@ def fetch_cover(name, release_id, system, timeout=10, quick=False):
if data is not None:
log.info("cover %r <- manual", name)
elif system != "videos":
# 2. Games: libretro (full box art) first, then the shipped OpenVGDB
# bundle (offline), then ScreenScraper (full box art, non-quick),
# then Wikipedia (keyless, non-quick), then IGDB last — IGDB only
# stores cropped 3:4 key art.
data = _fetch_libretro(name, t, indexed_only=quick)
# 2. Games/PS1: libretro (full box art) first, then the shipped
# OpenVGDB bundle (PSP only, offline), then ScreenScraper (non-quick),
# then Wikipedia (keyless, non-quick), then IGDB last.
data = _fetch_libretro(name, t, system=system, indexed_only=quick)
if data is not None:
log.info("cover %r <- libretro", name)
else:
data = None
if system == "games":
data = fetch_openvgdb_local(name)
if data is not None:
log.info("cover %r <- openvgdb", name)
elif not quick and time.time() < deadline:
data = _fetch_screenscraper(name, t)
if data is None and not quick and time.time() < deadline:
data = _fetch_screenscraper(name, t, system=system)
if data is not None:
log.info("cover %r <- screenscraper", name)
if data is None and not quick and time.time() < deadline:
@@ -826,10 +876,11 @@ def fetch_cover(name, release_id, system, timeout=10, quick=False):
if data is not None:
log.info("cover %r <- wikipedia", name)
if data is None and time.time() < deadline:
data = _fetch_igdb_cover(name, t)
data = _fetch_igdb_cover(name, t, system=system)
if data is not None:
# IGDB only has cropped 3:4 key art; composite it into the
# full PSP cover template (header bar + artwork).
if system == "games":
# IGDB only has cropped 3:4 PSP key art; composite it
# into the full PSP cover template (header + artwork).
data = compose_game_cover(data)
log.info("cover %r <- igdb", name)
@@ -918,6 +969,46 @@ def _search_wikipedia(query, limit=15, timeout=10):
return results
def _search_google_images(query, limit=15, timeout=10):
"""Return [{title, source, thumb_url, image_url}] from Google Images via the
Custom Search JSON API. Requires google_api_key + google_cx; keyless -> [].
"""
key = sources.load().get("google_api_key")
cx = sources.load().get("google_cx")
if not key or not cx:
return []
url = _GOOGLE_CSE_API + "?" + urllib.parse.urlencode({
"key": key,
"cx": cx,
"q": query,
"searchType": "image",
"num": min(max(limit, 1), 10),
"imgSize": "xlarge",
"safe": "off",
})
data = _http_get(url, timeout=timeout)
if not data:
return []
try:
items = json.loads(data).get("items") or []
except Exception: # noqa: BLE001
return []
out = []
for it in items:
link = it.get("link") or ""
thumb = ((it.get("image") or {}).get("thumbnailLink")) or link
if not link:
continue
out.append({
"id": None,
"title": it.get("title") or query,
"thumb_url": thumb,
"image_url": link,
"source": "google",
})
return out[:limit]
def _fetch_wikipedia_cover(name, timeout):
"""Return a game cover image from Wikipedia's File: namespace (keyless)."""
base = plain_title(name)
@@ -995,8 +1086,8 @@ def _wiki_file_search(query, limit, timeout):
return out
def _search_libretro(query, limit):
names = _load_index().get("Named_Boxarts", [])
def _search_libretro(query, limit, system="games"):
names = _load_index(system=system).get("Named_Boxarts", [])
db = _norm(query)
results = []
if not db:
@@ -1022,21 +1113,60 @@ def _search_libretro(query, limit):
results.append({
"title": n,
"source": "libretro",
"thumb_url": cover_url(n),
"image_url": cover_url(n),
"thumb_url": cover_url(n, system=system),
"image_url": cover_url(n, system=system),
})
return results
def _search_openvgdb(query, limit):
"""Return [{title, source, thumb_url, image_url}] from the offline OpenVGDB
bundle, using the display titles stored in openvgdb-index.json."""
names = _ovgdb_name_list()
if not names:
return []
index = _load_ovgdb_index()
db = _norm(query)
if not db:
return []
scored = []
for key in names:
if key == db:
score = 5
elif len(key) >= 4 and key.startswith(db):
score = 4
elif len(key) >= 4 and db in key:
score = 3
elif len(db) >= 4 and len(key) >= 4 and key in db:
score = 2
else:
continue
info = index.get(key) or {}
title = info.get("title") or key
scored.append((score, title, key))
scored.sort(key=lambda t: (-t[0], t[1]))
return [
{
"id": None,
"title": title,
"source": "openvgdb",
"thumb_url": "",
"image_url": "",
"_key": key,
}
for _score, title, key in scored[:limit]
]
# --------------------------------------------------------------------------- #
# Game metadata (details sidebar)
# --------------------------------------------------------------------------- #
_META_TTL = 86400.0
def get_game_metadata(title, timeout=10):
"""Return game metadata from IGDB."""
return get_game_metadata_igdb(title, timeout)
def get_game_metadata(title, timeout=10, system="games"):
"""Return game metadata from IGDB (system picks the platform)."""
return get_game_metadata_igdb(title, timeout, system)
# --------------------------------------------------------------------------- #
@@ -1161,28 +1291,30 @@ def _igdb_game_meta(g):
}
def _igdb_search(query, limit=20, timeout=10):
def _igdb_search(query, limit=20, timeout=10, system="games"):
cid, _ = _igdb_creds()
if not cid:
return []
term = plain_title(query) or query
platform_id = _platform(system)["igdb_id"]
q = (
f'search "{term}"; '
"fields name, cover.image_id, first_release_date, summary, genres.name, "
"involved_companies.company.name, involved_companies.developer, involved_companies.publisher, "
"rating, videos.video_id; "
f"where platforms = ({_IGDB_PSP_ID}); "
f"where platforms = ({platform_id}); "
f"limit {max(limit, 1)};"
)
return _igdb_post(q, timeout) or []
def _fetch_igdb_cover(name, timeout):
def _fetch_igdb_cover(name, timeout, system="games"):
cid, _ = _igdb_creds()
if not cid:
return None
term = plain_title(name) or name
q = f'search "{term}"; fields cover.image_id; where platforms = ({_IGDB_PSP_ID}); limit 10;'
platform_id = _platform(system)["igdb_id"]
q = f'search "{term}"; fields cover.image_id; where platforms = ({platform_id}); limit 10;'
games = _igdb_post(q, timeout) or []
tried = set()
for g in games:
@@ -1196,9 +1328,9 @@ def _fetch_igdb_cover(name, timeout):
return None
def search_games_igdb(query, limit=20, timeout=10):
def search_games_igdb(query, limit=20, timeout=10, system="games"):
out = []
for g in _igdb_search(query, limit, timeout):
for g in _igdb_search(query, limit, timeout, system):
cover = (g.get("cover") or {}).get("image_id") or ""
out.append({
"id": g.get("id"),
@@ -1213,8 +1345,8 @@ def search_games_igdb(query, limit=20, timeout=10):
return out[:limit]
def get_game_metadata_igdb(title, timeout=10):
games = _igdb_search(title, limit=10, timeout=timeout)
def get_game_metadata_igdb(title, timeout=10, system="games"):
games = _igdb_search(title, limit=10, timeout=timeout, system=system)
if not games:
return None
games.sort(key=lambda g: -_title_score(title, g.get("name") or ""))
@@ -1334,11 +1466,19 @@ def get_movie_metadata(title, timeout=10):
# Manual linking (pick the correct movie/game when the auto-match is wrong)
# --------------------------------------------------------------------------- #
def search_movies(query, limit=20, timeout=10):
"""Return [{id, title, year, overview, thumb_url, source}] for a movie search.
TheMovieDB first; keyless Wikipedia page titles as a fallback.
"""Return movie candidates for a UMD video: TMDB first, then Wikipedia
images and Google Images (both keyless-optional). A Wikipedia page-title
fallback is added only when nothing else matched (metadata-only linking).
"""
out = []
seen = set()
def add(r):
k = (r.get("source"), _norm(r.get("title") or ""))
if k[1] and k not in seen:
seen.add(k)
out.append(r)
key = sources.load().get("tmdb_api_key")
if key:
url = _TMDB_SEARCH + "?" + urllib.parse.urlencode({"api_key": key, "query": query})
@@ -1348,16 +1488,31 @@ def search_movies(query, limit=20, timeout=10):
results = json.loads(data).get("results") or []
except Exception:
results = []
for r in results:
# Cap TMDB results so Wikipedia/Google image picks still fit.
for r in results[: max(1, min(limit, 6))]:
poster = r.get("poster_path")
out.append({
add({
"id": r.get("id"),
"title": r.get("title") or r.get("name") or "",
"year": (r.get("release_date") or "")[:4],
"overview": (r.get("overview") or "").strip(),
"thumb_url": (_TMDB_IMAGE + "w185" + poster) if poster else "",
"image_url": (_TMDB_IMAGE + "w500" + poster) if poster else "",
"source": "tmdb",
})
# Wikipedia image search (keyless): real poster/cover files.
for r in _search_wikipedia(query, min(limit, 10), timeout):
r["id"] = None
r["movie_title"] = query
add(r)
# Google Images (Custom Search JSON API, when a key + cx are configured).
for r in _search_google_images(query, min(limit, 10), timeout):
r["movie_title"] = query
add(r)
# Last resort: Wikipedia page titles (metadata-only, no image).
if not out:
for page in _wiki_search(query, limit):
out.append({
@@ -1366,6 +1521,8 @@ def search_movies(query, limit=20, timeout=10):
"year": "",
"overview": "",
"thumb_url": "",
"image_url": "",
"movie_title": page,
"source": "wikipedia",
})
return out[:limit]
@@ -1411,40 +1568,71 @@ def movie_metadata_from_wikipedia(page_title, timeout=10):
def search_metadata_candidates(system, query, limit=20, timeout=10):
"""Search candidates the user can link an item to."""
"""Search candidates the user can link an item to.
Games/PS1: IGDB (rich metadata) first, then libretro box art (and the
PSP-only OpenVGDB bundle) so covers can be scraped even without an IGDB key.
"""
if system == "videos":
return search_movies(query, limit, timeout)
results = search_games_igdb(query, limit, timeout)
if results:
return results
# No IGDB results (or no key): fall back to libretro catalog names so the
# user can still link to the correct entry (title-only).
results = _search_libretro(query, limit)
for r in results:
r["source"] = "libretro"
r["id"] = None
return results
results = []
seen = set()
def add(r):
key = (r.get("source"), _norm(r.get("title") or ""))
if key[1] and key not in seen:
seen.add(key)
results.append(r)
# Cap IGDB so libretro/OpenVGDB cover picks still fit in the list.
for r in search_games_igdb(query, max(1, min(limit, 6)), timeout, system):
add(r)
for r in _search_libretro(query, limit, system):
r.setdefault("id", None)
add(r)
if system == "games":
for r in _search_openvgdb(query, limit):
add(r)
return results[:limit]
def link_metadata(system, candidate, timeout=10):
"""Fetch full metadata for a search candidate the user picked."""
if candidate.get("source") == "wikipedia":
return movie_metadata_from_wikipedia(candidate.get("title") or "", timeout)
if candidate.get("source") == "libretro":
# Catalog-name link only: no metadata backend, just the correct title.
def _empty_meta(title=""):
return {
"title": candidate.get("title") or "",
"title": title,
"release_date": "",
"overview": "",
"genres": [],
"rating": None,
"youtube": "",
}
def link_metadata(system, candidate, timeout=10):
"""Fetch full metadata for a search candidate the user picked."""
src = candidate.get("source")
if src == "wikipedia":
title = candidate.get("movie_title") or candidate.get("title") or ""
meta = movie_metadata_from_wikipedia(title, timeout) if title else None
if meta:
return meta
if system == "videos":
if candidate.get("id") is not None:
return get_movie_metadata(title, timeout) or _empty_meta(title)
return _empty_meta(title)
if src in ("libretro", "openvgdb"):
# Cover repositories have no metadata backend; try IGDB by title so the
# user still gets synopsis/genre/etc. when a key is configured.
title = candidate.get("title") or ""
if title:
meta = get_game_metadata_igdb(title, timeout, system)
if meta:
return meta
return _empty_meta(title)
if system == "videos":
if src == "tmdb" and candidate.get("id") is not None:
return movie_metadata_by_id(candidate["id"], timeout)
return movie_metadata_from_wikipedia(candidate.get("title") or "", timeout)
if candidate.get("id") is not None and candidate.get("source") == "igdb":
title = candidate.get("movie_title") or candidate.get("title") or ""
return get_movie_metadata(title, timeout) or _empty_meta(title)
if src == "igdb" and candidate.get("id") is not None:
return game_metadata_by_id_igdb(candidate["id"], timeout)
return None
return _empty_meta(candidate.get("title") or "")
+62 -6
View File
@@ -1,6 +1,8 @@
"""Parser for clrmamepro (.dat) files.
"""Parser for clrmamepro / Logiqx XML (.dat) files.
Handles the No-Intro / Redump text DAT format::
Supports both DAT layouts the project uses:
* clrmamepro text DAT (No-Intro): parenthesised blocks, e.g.::
clrmamepro (
name "Sony - PlayStation Portable"
@@ -13,11 +15,15 @@ Handles the No-Intro / Redump text DAT format::
rom ( name "WipEout Pure (Europe).iso" size 123 crc ABCD1234 md5 ... sha1 ... )
)
``parse_dat_text`` returns ``(header, entries)``. Each entry is a dict with
scalar keys (``name``, ``description``, ``region``, ``serial``, ...) and
repeated nested blocks exposed as lists under their block key (``rom``).
* Logiqx XML DAT (Redump): ``<game>`` elements with ``<rom>`` children.
Both ``parse_dat_text`` and ``parse_dat_xml`` return ``(header, entries)``.
Each entry is a dict with scalar keys (``name``, ``description``, ``region``,
``serial``, ...) and repeated nested blocks exposed as lists under their block
key (``rom``).
"""
import re
import xml.etree.ElementTree as ET
from pathlib import Path
_TOKEN_RE = re.compile(
@@ -124,5 +130,55 @@ def parse_dat_text(text):
return header, entries
def _xml_text(element, key):
child = element.find(key)
if child is not None and child.text:
return child.text.strip()
return None
def parse_dat_xml(text):
"""Parse a Logiqx XML DAT (the format redump.org uses).
Returns ``(header, entries)`` in the same shape as :func:`parse_dat_text`.
"""
root = ET.fromstring(text)
header = {}
header_el = root.find("header")
if header_el is not None:
for child in header_el:
if child.text:
header[child.tag] = child.text.strip()
entries = []
for game in root.iter():
if game.tag not in ("game", "machine"):
continue
entry = {"name": (game.get("name") or "").strip()}
for key in ("description", "serial", "region", "category"):
value = _xml_text(game, key)
if value:
entry[key] = value
entry["rom"] = [
{
"name": rom.get("name", ""),
"size": rom.get("size"),
"crc": rom.get("crc"),
"md5": rom.get("md5"),
"sha1": rom.get("sha1"),
"serial": rom.get("serial"),
}
for rom in game.findall("rom")
]
entries.append(entry)
return header, entries
def parse_dat_file(path):
return parse_dat_text(Path(path).read_text(encoding="utf-8-sig"))
text = Path(path).read_text(encoding="utf-8-sig")
stripped = text.lstrip()
if stripped.startswith("<?xml") or stripped.startswith("<"):
return parse_dat_xml(text)
return parse_dat_text(text)
+14
View File
@@ -101,6 +101,20 @@ CREATE TABLE IF NOT EXISTS save_sigs (
sig TEXT NOT NULL,
PRIMARY KEY (save_id, size, mtime_ns)
);
CREATE TABLE IF NOT EXISTS ps1_entries (
id INTEGER PRIMARY KEY,
source_path TEXT NOT NULL UNIQUE,
title TEXT,
serial TEXT,
region TEXT,
disc INTEGER NOT NULL DEFAULT 1,
disc_total INTEGER NOT NULL DEFAULT 1,
output_eboot TEXT,
status TEXT NOT NULL DEFAULT 'pending',
error TEXT,
updated_at TEXT NOT NULL
);
"""
+285 -16
View File
@@ -14,13 +14,13 @@ from PySide6.QtCore import QBuffer, QByteArray, QObject, QRect, QSettings, QSize
from PySide6.QtGui import QColor, QDesktopServices, QFont, QIcon, QImage, QPainter, QPixmap
from PySide6.QtWidgets import (
QAbstractItemView, QApplication, QCheckBox, QComboBox, QDialog, QDialogButtonBox,
QFileDialog, QFormLayout, QGridLayout, QHBoxLayout, QHeaderView, QLabel, QLineEdit,
QFileDialog, QFormLayout, QGridLayout, QHBoxLayout, QHeaderView, QInputDialog, QLabel, QLineEdit,
QListWidget, QListWidgetItem, QMainWindow, QMenu, QMessageBox, QPlainTextEdit,
QProgressBar, QProgressDialog, QPushButton, QScrollArea, QSpinBox, QSplitter, QStackedWidget,
QTabWidget, QTableWidget, QTableWidgetItem, QVBoxLayout, QWidget,
)
from . import config, covers, info, iso_reader, launcher, matcher, namer, pipeline, psp, queue, saves, scanner, sources, store, transfers
from . import config, covers, info, iso_reader, launcher, matcher, namer, pipeline, ps1, psp, queue, saves, scanner, sources, store, transfers
from .db import get_connection, init_db
log = logging.getLogger("gui")
@@ -559,6 +559,7 @@ class MetadataWorker(QObject):
class GameDetailsPanel(QWidget):
edit_requested = Signal() # emitted when the user wants to hand-tag the item
link_requested = Signal() # emitted when the user wants to pick a movie/game to link
convert_requested = Signal() # emitted when the user wants to convert a PS1 disc to EBOOT
copy_requested = Signal() # emitted when the user wants to copy the current item to the PSP
delete_requested = Signal() # emitted when the user wants to delete the current item
play_requested = Signal() # emitted when the user wants to launch the item in PPSSPP
@@ -569,6 +570,7 @@ class GameDetailsPanel(QWidget):
self.setFixedWidth(300)
self._youtube_url = ""
self._meta = None
self._copy_available = False
outer = QVBoxLayout(self)
outer.setContentsMargins(12, 12, 0, 12)
@@ -627,6 +629,12 @@ class GameDetailsPanel(QWidget):
self.play_btn.setToolTip("Launch this game in PPSSPP")
self.layout.addWidget(self.play_btn)
self.convert_btn = QPushButton("\u2699 Convert to EBOOT")
self.convert_btn.clicked.connect(self.convert_requested.emit)
self.convert_btn.setVisible(False)
self.convert_btn.setToolTip("Convert this PS1 disc to an EBOOT.PBP")
self.layout.addWidget(self.convert_btn)
btn_grid = QGridLayout()
btn_grid.setHorizontalSpacing(6)
btn_grid.setVerticalSpacing(6)
@@ -658,14 +666,28 @@ class GameDetailsPanel(QWidget):
def set_item(self, item):
self.system = item.get("system")
self._ps1_kind = item.get("ps1_kind")
path = item.get("path") or ""
self.play_btn.setEnabled(item.get("system") == "games" and bool(path) and Path(path).suffix.lower() == ".iso")
is_ps1 = item.get("system") == "ps1"
is_ps1_disc = is_ps1 and item.get("ps1_kind") == "disc"
self.play_btn.setEnabled(
(item.get("system") == "games" and bool(path) and Path(path).suffix.lower() == ".iso")
or (is_ps1 and item.get("ps1_kind") == "eboot" and bool(path))
)
self.play_btn.setVisible(not is_ps1_disc)
self.convert_btn.setVisible(is_ps1_disc)
self.copy_btn.setEnabled(self._copy_available and not is_ps1_disc)
self.copy_btn.setToolTip(
"Copy this EBOOT to /PSP/GAME" if is_ps1 else "Copy this disc to the connected PSP"
)
self.del_psp_btn.setEnabled(bool(item.get("on_psp")))
self.title.setText(item.get("title_override") or item.get("name") or "Unknown")
parts = []
if item.get("serial"):
parts.append(item["serial"])
if item.get("region"):
if is_ps1 and item.get("ps1_status"):
parts.append(item["ps1_status"])
elif item.get("region"):
parts.append(item["region"])
self.subtitle.setText(" · ".join(parts))
self._clear_meta()
@@ -676,7 +698,8 @@ class GameDetailsPanel(QWidget):
self._set_cover(item.get("release_id"), item.get("system"))
def set_copy_available(self, available):
self.copy_btn.setEnabled(bool(available))
self._copy_available = bool(available)
self.copy_btn.setEnabled(self._copy_available and getattr(self, "_ps1_kind", None) != "disc")
def set_psp_installed(self, on_psp):
self.del_psp_btn.setEnabled(bool(on_psp))
@@ -824,6 +847,15 @@ class ScrapeSearchWorker(QObject):
def run(self):
found = covers.search_metadata_candidates(self.system, self.query, limit=20, timeout=10)
for r in found:
src = r.get("source")
if src == "openvgdb":
r["thumb_bytes"] = covers.fetch_openvgdb_local(r.get("title") or "")
elif src == "libretro":
data = covers.fetch_libretro_local(r.get("title") or "")
r["thumb_bytes"] = data if data else (
covers.download_image(r.get("thumb_url"), timeout=8) if r.get("thumb_url") else None
)
else:
url = r.get("thumb_url")
r["thumb_bytes"] = covers.download_image(url, timeout=8) if url else None
self.results.emit(found)
@@ -849,16 +881,26 @@ class ScrapeFetchWorker(QObject):
def _scrape_cover(system, candidate, rid, timeout=20):
"""Download the cover for a metadata candidate; returns bytes or None."""
src = candidate.get("source")
title = candidate.get("title") or ""
# Cover repositories ship offline bundles — read them directly (no network).
if src == "openvgdb" and title:
data = covers.fetch_openvgdb_local(title)
if data:
return data
if src == "libretro" and title:
data = covers.fetch_libretro_local(title)
if data:
return data
url = candidate.get("image_url")
if url:
data = covers.download_image(url, timeout)
if data:
if system == "videos":
return compose_umd_cover(data)
if candidate.get("source") == "igdb":
if candidate.get("source") == "igdb" and system == "games":
return covers.compose_game_cover(data)
return data
title = candidate.get("title") or ""
if title:
path = covers.fetch_cover(title, rid, system, timeout=timeout)
if path is not None:
@@ -955,6 +997,8 @@ class ScrapeDialog(QDialog):
icon = QIcon(pm)
year = r.get("year") or (r.get("release_date") or "")[:4]
label = f"{r['title']} ({year})" if year else r["title"]
if r.get("source") not in (None, "igdb", "tmdb"):
label += f" [{r['source']}]"
item = QListWidgetItem(icon, label)
if r.get("overview"):
item.setToolTip(r["overview"])
@@ -1011,7 +1055,7 @@ class SettingsPage(QWidget):
def __init__(self, games_dir, videos_dir, store_dir, ppsspp_dir, cloud_dir, ppsspp_exe,
store_server_url, store_token, store_archive_id, store_archive_sig, store_archive_user,
iso_category, parent=None):
iso_category, ps1_dir, ps1_output_dir, ps1_base_pbp, parent=None):
super().__init__(parent)
outer = QVBoxLayout(self)
outer.setContentsMargins(0, 0, 0, 0)
@@ -1025,6 +1069,9 @@ class SettingsPage(QWidget):
page, form = self._new_form()
self.games_edit = self._dir_row(form, "Games directory:", games_dir)
self.videos_edit = self._dir_row(form, "Videos directory:", videos_dir)
self.ps1_edit = self._dir_row(form, "PS1 directory:", ps1_dir)
self.ps1_out_edit = self._dir_row(form, "PS1 output directory:", ps1_output_dir)
self.ps1_base_edit = self._file_row(form, "PS1 base.pbp (PSN classic):", ps1_base_pbp)
self.auto_scan = QCheckBox("Scan directories automatically on startup")
self.auto_scan.setChecked(str(s.value("auto_scan", False)).lower() == "true")
form.addRow(self.auto_scan)
@@ -1094,6 +1141,8 @@ class SettingsPage(QWidget):
form.addRow(QLabel("<b>Videos — metadata &amp; posters</b>"))
add_field("TheMovieDB API key:", "tmdb_api_key", password=True)
add_field("Google API key:", "google_api_key", password=True)
add_field("Google Search Engine ID (cx):", "google_cx")
hint = QLabel(
"Manual UMD covers: drop images into "
@@ -1156,7 +1205,7 @@ class SettingsPage(QWidget):
return edit
def _browse_file(self, edit):
path, _ = QFileDialog.getOpenFileName(self, "Select PPSSPP executable")
path, _ = QFileDialog.getOpenFileName(self, "Select file")
if path:
edit.setText(path)
@@ -1210,6 +1259,15 @@ class SettingsPage(QWidget):
def iso_category(self):
return self.iso_category_edit.text().strip()
def ps1_dir(self):
return self.ps1_edit.text().strip()
def ps1_output_dir(self):
return self.ps1_out_edit.text().strip()
def ps1_base_pbp(self):
return self.ps1_base_edit.text().strip()
def _save(self):
sources.save({key: edit.text() for key, edit in self.fields.items()})
covers.clear_miss_cache()
@@ -2989,7 +3047,7 @@ class PspSyncPage(QWidget):
self._workers = []
def build_library_items():
def build_library_items(ps1_dir="", output_dir=""):
"""Build the library item list from the DB with a single joined query."""
items = []
seen = set()
@@ -3058,16 +3116,59 @@ def build_library_items():
conn.commit()
finally:
conn.close()
items.extend(_ps1_library_items(ps1_dir, output_dir, overrides, dates))
return items
def _ps1_library_items(ps1_dir, output_dir, overrides, dates):
"""PS1 discs (pending) + converted EBOOTs, merged into the library list."""
if not ps1_dir:
return []
try:
entries = ps1.scan_library(ps1_dir, output_dir)
except Exception: # noqa: BLE001
return []
out = []
for e in entries:
# A converted disc is represented by its EBOOT entry (one row per game).
if e["kind"] == "disc" and e.get("status") == "converted":
continue
name = e["name"]
rid = covers.stable_key(name)
path = Path(e["path"])
try:
size = path.stat().st_size if path.exists() else 0
except OSError:
size = 0
out.append({
"name": name,
"release_id": rid,
"system": "ps1",
"canonical": name,
"serial": e.get("serial") or "",
"region": None,
"title_override": overrides.get(("ps1", str(rid))),
"release_date": dates.get((str(rid), "ps1")) or "",
"path": str(path),
"size": size,
"ps1_kind": e["kind"],
"ps1_status": e.get("status") or "pending",
})
return out
class LibraryWorker(QObject):
result = Signal(list)
finished = Signal()
def __init__(self, ps1_dir="", output_dir=""):
super().__init__()
self.ps1_dir = ps1_dir
self.output_dir = output_dir
def run(self):
try:
items = build_library_items()
items = build_library_items(self.ps1_dir, self.output_dir)
except Exception: # noqa: BLE001
items = []
self.result.emit(items)
@@ -3220,6 +3321,113 @@ class SavesPage(QWidget):
self._workers = []
class Ps1ScanWorker(QObject):
results = Signal(list)
failed = Signal(str)
def __init__(self, ps1_dir, output_dir):
super().__init__()
self.ps1_dir = ps1_dir
self.output_dir = output_dir
def run(self):
try:
self.results.emit(ps1.scan_library(self.ps1_dir, self.output_dir))
except Exception as exc: # noqa: BLE001
self.failed.emit(str(exc))
class Ps1Page(QWidget):
"""PS1 tab: list raw source discs, converted EBOOTs and PSN games."""
convert_requested = Signal(dict)
def __init__(self, parent=None):
super().__init__(parent)
self._ps1_dir = ""
self._output_dir = ""
self._entries = []
self._thread = None
self._worker = None
layout = QVBoxLayout(self)
bar = QHBoxLayout()
self.status = QLabel("PS1 discs — set the PS1 folder in Settings → General.")
bar.addWidget(self.status, 1)
self.refresh_btn = QPushButton("Refresh")
self.refresh_btn.clicked.connect(self.refresh)
self.convert_btn = QPushButton("Convert selected")
self.convert_btn.clicked.connect(self._convert_selected)
bar.addWidget(self.refresh_btn)
bar.addWidget(self.convert_btn)
layout.addLayout(bar)
self.table = QTableWidget(0, 4)
self.table.setHorizontalHeaderLabels(["Type", "Name", "Serial", "Status"])
self.table.setSelectionBehavior(QAbstractItemView.SelectRows)
self.table.setEditTriggers(QAbstractItemView.NoEditTriggers)
self.table.verticalHeader().setVisible(False)
self.table.horizontalHeader().setSectionResizeMode(1, QHeaderView.Stretch)
layout.addWidget(self.table, 1)
def set_dirs(self, ps1_dir, output_dir):
self._ps1_dir = ps1_dir
self._output_dir = output_dir
def refresh(self):
if not self._ps1_dir:
self.status.setText("PS1 folder not set — configure it in Settings → General.")
self.table.setRowCount(0)
self._entries = []
return
if self._thread is not None and _thread_running(self._thread):
return
self.status.setText("Scanning PS1 folders…")
self._worker = Ps1ScanWorker(self._ps1_dir, self._output_dir)
self._thread = QThread(self)
self._worker.moveToThread(self._thread)
self._thread.started.connect(self._worker.run)
self._worker.results.connect(self._on_results)
self._worker.failed.connect(self._on_failed)
self._worker.results.connect(self._thread.quit)
self._worker.failed.connect(self._thread.quit)
self._thread.start()
def _on_results(self, entries):
self._entries = entries
self.table.setRowCount(len(entries))
for row, e in enumerate(entries):
kind = "Disc" if e["kind"] == "disc" else "EBOOT"
self.table.setItem(row, 0, QTableWidgetItem(kind))
self.table.setItem(row, 1, QTableWidgetItem(e["name"]))
self.table.setItem(row, 2, QTableWidgetItem(e.get("serial") or ""))
self.table.setItem(row, 3, QTableWidgetItem(e.get("status") or "pending"))
pending = sum(1 for e in entries if e["kind"] == "disc" and e["status"] != "converted")
self.status.setText(f"{len(entries)} item(s) — {pending} ready to convert")
def _on_failed(self, message):
self.status.setText(f"PS1 scan failed: {message}")
def _convert_selected(self):
row = self.table.currentRow()
if row < 0 or row >= len(self._entries):
return
e = self._entries[row]
if e["kind"] != "disc":
QMessageBox.information(self, "PS1", "Select a raw PS1 disc (Disc row), not an EBOOT.")
return
if e["status"] == "converted":
QMessageBox.information(self, "PS1", "This disc already has a converted EBOOT.")
return
game_id = e.get("serial") or ""
if not game_id:
text, ok = QInputDialog.getText(self, "PS1", "Game ID (e.g. SLUS00594):")
if not ok or not text.strip():
return
game_id = text.strip()
self.convert_requested.emit({"source": e["path"], "title": e["name"], "game_id": game_id})
class MainWindow(QMainWindow):
def __init__(self):
super().__init__()
@@ -3259,6 +3467,9 @@ class MainWindow(QMainWindow):
self._auto_sync_saves = str(self.settings.value("auto_sync_saves", "false")).lower() == "true"
self._ppsspp_exe = str(self.settings.value("ppsspp_exe", "") or "")
self._iso_category = str(self.settings.value("iso_category", "") or "")
self._ps1_dir = self._first_existing(config.PS1_ROOTS) or str(self.settings.value("ps1_dir", "") or "")
self._ps1_output_dir = str(self.settings.value("ps1_output_dir", str(config.PS1_OUTPUT_DIR)) or "")
self._ps1_base_pbp = str(self.settings.value("ps1_base_pbp", "") or "")
log.info("games dir: %s", self._games_dir or "(none)")
log.info("videos dir: %s", self._videos_dir or "(none)")
log.info("store: %s", self._store_server_url or self._store_dir or "(none)")
@@ -3291,6 +3502,10 @@ class MainWindow(QMainWindow):
self.saves_page = SavesPage()
self.saves_page.sync_requested.connect(self._on_saves_sync)
self.tabs.addTab(self.saves_page, "Saves")
self.ps1_page = Ps1Page()
self.ps1_page.set_dirs(self._ps1_dir, self._ps1_output_dir)
self.ps1_page.convert_requested.connect(self._on_ps1_convert)
self.tabs.addTab(self.ps1_page, "PS1")
self.tabs.addTab(self._build_settings_tab(), "Settings")
self.tabs.currentChanged.connect(self._on_tab_changed)
self._on_psp_device_changed(self.psp_page.root())
@@ -3321,7 +3536,7 @@ class MainWindow(QMainWindow):
self.library_search.textChanged.connect(self._apply_library_filter)
bar.addWidget(self.library_search, 1)
self.library_type = QComboBox()
self.library_type.addItems(["All types", "Games", "Videos"])
self.library_type.addItems(["All types", "Games", "Videos", "PS1"])
self.library_type.currentTextChanged.connect(self._apply_library_filter)
bar.addWidget(self.library_type)
self.library_psp = QComboBox()
@@ -3371,6 +3586,7 @@ class MainWindow(QMainWindow):
self.cover_grid.scrape_requested.connect(self._scrape_grid_item)
self.details_panel.edit_requested.connect(self._edit_details)
self.details_panel.link_requested.connect(self._scrape_item)
self.details_panel.convert_requested.connect(self._convert_current_item)
self.details_panel.copy_requested.connect(self._copy_current_to_psp)
self.details_panel.delete_requested.connect(self._delete_current)
self.details_panel.delete_psp_requested.connect(self._delete_current_from_psp)
@@ -3461,7 +3677,8 @@ class MainWindow(QMainWindow):
self.settings_page = SettingsPage(self._games_dir, self._videos_dir, self._store_dir,
self._ppsspp_saves_dir, self._cloud_saves_dir, self._ppsspp_exe,
self._store_server_url, self._store_token, self._store_archive_id,
self._store_archive_sig, self._store_archive_user, self._iso_category)
self._store_archive_sig, self._store_archive_user, self._iso_category,
self._ps1_dir, self._ps1_output_dir, self._ps1_base_pbp)
self.settings_page.saved.connect(self._on_settings_saved)
return self.settings_page
@@ -3490,6 +3707,13 @@ class MainWindow(QMainWindow):
self._auto_sync_saves = self.settings_page.auto_sync_saves()
self._ppsspp_exe = self.settings_page.ppsspp_exe()
self._iso_category = self.settings_page.iso_category()
self._ps1_dir = self.settings_page.ps1_dir()
self._ps1_output_dir = self.settings_page.ps1_output_dir()
self._ps1_base_pbp = self.settings_page.ps1_base_pbp()
self.settings.setValue("ps1_dir", self._ps1_dir)
self.settings.setValue("ps1_output_dir", self._ps1_output_dir)
self.settings.setValue("ps1_base_pbp", self._ps1_base_pbp)
self.ps1_page.set_dirs(self._ps1_dir, self._ps1_output_dir)
self.settings.setValue("ppsspp_saves_dir", self._ppsspp_saves_dir)
self.settings.setValue("cloud_saves_dir", self._cloud_saves_dir)
self.settings.setValue("auto_sync_saves", self._auto_sync_saves)
@@ -3499,7 +3723,8 @@ class MainWindow(QMainWindow):
self._update_dir_status()
def _on_tab_changed(self, index):
pass # store builds eagerly and caches its view; no per-tab work needed
if hasattr(self, "ps1_page") and index == self.tabs.indexOf(self.ps1_page):
self.ps1_page.refresh()
def _update_saves_locations(self):
if not hasattr(self, "saves_page"):
@@ -3530,7 +3755,7 @@ class MainWindow(QMainWindow):
self.statusBar().showMessage("Loading library…")
log.info("loading library…")
thread = QThread(self)
worker = LibraryWorker()
worker = LibraryWorker(self._ps1_dir, self._ps1_output_dir)
worker.moveToThread(thread)
thread.started.connect(worker.run)
worker.result.connect(self._on_library_loaded)
@@ -3568,6 +3793,8 @@ class MainWindow(QMainWindow):
continue
if typ == "Videos" and it["system"] != "videos":
continue
if typ == "PS1" and it["system"] != "ps1":
continue
if q and q not in (it["canonical"] or it["name"]).lower():
continue
if psp_f == "On PSP" and not it.get("on_psp"):
@@ -3593,7 +3820,8 @@ class MainWindow(QMainWindow):
title.setIcon(self._psp_dot_icon())
title.setToolTip("Installed on PSP")
self.library_table.setItem(row, 0, title)
self.library_table.setItem(row, 1, QTableWidgetItem("Game" if it["system"] == "games" else "Video"))
kind_label = {"games": "Game", "videos": "Video", "ps1": "PS1"}.get(it["system"], "Game")
self.library_table.setItem(row, 1, QTableWidgetItem(kind_label))
self.library_table.setItem(row, 2, QTableWidgetItem(it.get("region") or ""))
self.library_table.setItem(row, 3, QTableWidgetItem(it.get("serial") or ""))
size = it.get("size")
@@ -3716,6 +3944,10 @@ class MainWindow(QMainWindow):
meta["manual"] = True
self.details_panel.set_item(item)
self.details_panel.set_metadata(meta)
# Always refresh the details panel's cover box: a cover-only scrape
# (no metadata backend) skips set_item above, and `rid` may fall back
# to a stable key when the release has no database id.
self.details_panel._set_cover(rid, system)
self.cover_grid.update_item_label(item)
self.cover_grid.refresh_item(item)
@@ -3724,6 +3956,18 @@ class MainWindow(QMainWindow):
self.details_panel.set_item(it)
self._scrape_item()
def _convert_current_item(self):
item = getattr(self, "_current_item", None)
if item is None or item.get("system") != "ps1" or item.get("ps1_kind") != "disc":
return
game_id = item.get("serial") or ""
if not game_id:
text, ok = QInputDialog.getText(self, "PS1", "Game ID (e.g. SLUS00594):")
if not ok or not text.strip():
return
game_id = text.strip()
self._on_ps1_convert({"source": item.get("path"), "title": item.get("name"), "game_id": game_id})
def _on_store_download(self, items):
if not items:
return
@@ -3805,6 +4049,8 @@ class MainWindow(QMainWindow):
return
if item.get("system") == "videos":
return # UMD Video discs can't be played in PPSSPP
if item.get("system") == "ps1" and item.get("ps1_kind") != "eboot":
return # raw PS1 discs are converted, not played
path = item.get("path")
if not path or not Path(path).exists():
QMessageBox.warning(self, "Missing file", f"File not found:\n{path}")
@@ -3877,6 +4123,15 @@ class MainWindow(QMainWindow):
psp_root = Path(self._psp_root)
if system == "videos":
dest_dir = psp_root / "ISO" / "VIDEO"
elif system == "ps1":
# PS1 EBOOTs install as a whole game folder under /PSP/GAME.
src_folder = Path(item["path"]).parent
dest_dir = psp_root / "PSP" / "GAME"
log.info("copying ps1 game to psp: %s -> %s", src_folder, dest_dir / src_folder.name)
self.queue.enqueue(transfers.TreeCopySource(str(src_folder)), dest_dir,
{"dest_name": src_folder.name}, item.get("name") or src_folder.name,
kind="psp-install")
return True
else:
cat = "".join(c for c in (self._iso_category or "").strip() if c not in "/\\")
dest_dir = (psp_root / "ISO" / cat) if cat else (psp_root / "ISO")
@@ -3984,6 +4239,17 @@ class MainWindow(QMainWindow):
if hasattr(self, "psp_page"):
self.psp_page.refresh_inventory()
def _on_ps1_convert(self, item):
if not self._ps1_base_pbp or not Path(self._ps1_base_pbp).is_file():
QMessageBox.warning(self, "PS1", "Set a valid PS1 base.pbp path in Settings → General first.")
return
if not item.get("game_id"):
QMessageBox.warning(self, "PS1", "No game ID for this disc.")
return
out_dir = Path(self._ps1_output_dir) if self._ps1_output_dir else Path.cwd() / "ps1_converted"
source = transfers.Ps1ConvertSource(self._ps1_base_pbp)
self.queue.enqueue(source, out_dir, item, item.get("title") or item["source"], kind="ps1-convert")
def _on_queue_job_done(self, job):
if self._closing:
return
@@ -4004,6 +4270,9 @@ class MainWindow(QMainWindow):
QTimer.singleShot(0, self._refresh_psp)
if kind == "save-sync":
QTimer.singleShot(0, self._refresh_saves)
if kind == "ps1-convert":
QTimer.singleShot(0, self.ps1_page.refresh)
QTimer.singleShot(0, self.load_library)
# -- Common ------------------------------------------------------------ #
@staticmethod
+1 -1
View File
@@ -197,7 +197,7 @@ def get_or_fetch(release_id, system, fetch_title):
if system == "videos":
meta = covers.get_movie_metadata(fetch_title)
else:
meta = covers.get_game_metadata(fetch_title)
meta = covers.get_game_metadata(fetch_title, system=system)
except Exception: # noqa: BLE001 - network/API failures must not crash the UI
meta = None
if meta:
+9
View File
@@ -24,6 +24,9 @@ def detect_system(dat_name):
return "videos"
if "PlayStation Portable" in dat_name:
return "games"
# Redump PS1: "Sony - PlayStation" (assumes no PS2/PS3 DATs are present).
if "PlayStation" in dat_name:
return "ps1"
return "unknown"
@@ -37,6 +40,12 @@ def ingest_dat(conn, path):
existing = conn.execute("SELECT id FROM dats WHERE name = ?", (dat_name,)).fetchone()
if existing:
dat_id = existing["id"]
# Files that resolved to a release in this DAT must be re-identified
# against the new catalog (the release ids are about to be recycled).
conn.execute(
"UPDATE files SET release_id = NULL WHERE release_id IN (SELECT id FROM releases WHERE dat_id = ?)",
(dat_id,),
)
conn.execute("DELETE FROM roms WHERE release_id IN (SELECT id FROM releases WHERE dat_id = ?)", (dat_id,))
conn.execute("DELETE FROM releases WHERE dat_id = ?", (dat_id,))
conn.execute(
+13 -3
View File
@@ -15,8 +15,8 @@ _ARTICLE_RE = re.compile(r"^(.+),\s*(the|a|an)(?:\s+-\s+(.+))?$")
def _normalize(name):
name = name.lower()
name = re.sub(r"\.(iso|cso|pbp|img|bin)$", "", name) # strip disc extension
name = re.sub(r"^\s*\d+\s*-\s*", "", name) # leading No-Intro catalog number
name = re.sub(r"\.(iso|cso|pbp|img|bin|cue)$", "", name) # strip disc extension
name = re.sub(r"^\s*\d+\s*-\s*", "", name) # legacy No-Intro "#### - Name" catalog number
name = re.sub(r"\([^)]*\)", " ", name) # parenthetical tags
name = " ".join(name.split())
m = _ARTICLE_RE.match(name)
@@ -141,7 +141,7 @@ def match(conn, record):
if row is not None:
return f"hash:{field}", dict(row)
# 2. Local supplement catalog (dumps not present in No-Intro).
# 2. Local supplement catalog (dumps not present in the DATs).
for entry in _load_supplement():
for field in ("crc32", "sha1", "md5"):
value = entry.get(field)
@@ -181,6 +181,16 @@ def match(conn, record):
return "unknown", None
top = [cand for ratio, cand in scored if ratio >= best_ratio - 0.001]
top = _narrow(conn, top, size)
# The same disc can exist in multiple catalogs (e.g. No-Intro + Redump);
# collapse identical entries so they don't read as an ambiguous match.
seen = set()
deduped = []
for cand in top:
key = (cand.get("name"), size)
if key not in seen:
seen.add(key)
deduped.append(cand)
top = deduped
if len(top) == 1:
return f"fuzzy:{best_ratio:.2f}", top[0]
return "unknown", None
+2 -2
View File
@@ -62,7 +62,7 @@ def _reorder_article(title):
return title
def canonical_name(release, serial):
def canonical_name(release, serial, ext=".iso"):
name = release["name"]
region = release.get("region")
if region:
@@ -78,4 +78,4 @@ def canonical_name(release, serial):
parts.append(f"({serial})")
name = _ILLEGAL.sub(" ", " ".join(parts)).strip()
name = re.sub(r"\s+", " ", name)
return f"{name}.iso"
return f"{name}{ext}"
+15
View File
@@ -17,6 +17,7 @@ tags removed). The GUI reads this bundle with no network access.
"""
import argparse
import concurrent.futures
import json
import logging
import os
import re
@@ -94,6 +95,19 @@ def build_index(sqlite_path):
return best
def write_index(index, path=None):
"""Persist {key: {"title", "url"}} so the GUI can show readable titles when
searching the offline bundle (see covers._search_openvgdb)."""
path = Path(path) if path else (config.COVERS_DIR / "openvgdb-index.json")
data = {k: {"title": t, "url": u} for k, (t, u, _r) in index.items()}
try:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
except OSError as exc: # noqa: BLE001
log.warning("could not write index %s: %s", path, exc)
return path
def _existing_keys():
"""Titles already present in the bundle (for resume)."""
if not BUNDLE.exists():
@@ -152,6 +166,7 @@ def scrape(limit=None, workers=8, rate=3.0):
index = build_index(sqlite_path)
log.info("indexed %d unique PSP covers", len(index))
write_index(index)
existing = _existing_keys()
todo = [(k, v) for k, v in sorted(index.items()) if k not in existing]
+158
View File
@@ -0,0 +1,158 @@
"""PS1 library helpers: discover source discs and converted EBOOT.PBPs.
This is the read/scan side of PS1 support (the conversion side lives in
``app.ps1_convert``). It finds raw PS1 disc images (``.cue`` sets and
standalone ``.iso``/``.img``), reads each disc's boot serial, and lists
converted ``EBOOT.PBP`` files in the output folder.
"""
from __future__ import annotations
import re
from pathlib import Path
from typing import List, Optional
from . import config
RAW_SECTOR = 2352
COOKED_SECTOR = 2048
EBOOT_NAME = "EBOOT.PBP"
# SYSTEM.CNF boot line: cdrom:\SLPS_017.62;1 (backslash optional in some rips)
_SERIAL_RE = re.compile(rb"cdrom:\\?([A-Za-z]{4})[_-]?(\d{3})\.(\d{2})")
def _sector_size(path: Path) -> int:
size = path.stat().st_size
if size and size % RAW_SECTOR == 0:
return RAW_SECTOR
return COOKED_SECTOR
def read_serial(image_path: Path) -> Optional[str]:
"""Read the boot serial (e.g. ``SLPS01762``) from a PS1 disc image.
Works on both raw 2352-byte/sector images (BIN/IMG) and cooked 2048-byte
images (ISO). Returns ``None`` if the boot line can't be found.
"""
image_path = Path(image_path)
if not image_path.is_file():
return None
sector_size = _sector_size(image_path)
data_off = 24 if sector_size == RAW_SECTOR else 0
buf = bytearray()
with open(image_path, "rb") as fh:
for _ in range(128): # SYSTEM.CNF is always near the start of the disc
sector = fh.read(sector_size)
if len(sector) < sector_size:
break
buf += sector[data_off:data_off + COOKED_SECTOR]
m = _SERIAL_RE.search(bytes(buf))
if m:
return (m.group(1) + m.group(2) + m.group(3)).decode().upper()
return None
def cue_data_track(cue_path: Path) -> Optional[Path]:
"""Return the first data-track image referenced by a cue sheet."""
from . import ps1_convert
try:
disc = ps1_convert.parse_cue(cue_path.read_text(encoding="utf-8", errors="replace"))
except Exception:
return None
track = disc.data_track
if track is None:
return None
return (cue_path.parent / track.file) if track.file else None
def iter_source_discs(root: Path):
"""Yield PS1 source discs under ``root`` (cue sheets + standalone images)."""
root = Path(root)
if not root.is_dir():
return
for path in sorted(root.rglob("*")):
if not path.is_file():
continue
suffix = path.suffix.lower()
if suffix == ".cue":
yield path
elif suffix in (".iso", ".img"):
# A standalone image: no cue sheet next to it (same stem).
if not (path.parent / f"{path.stem}.cue").exists():
yield path
def find_eboots(root: Path) -> List[Path]:
"""List converted/PSN ``EBOOT.PBP`` files under ``root``."""
root = Path(root)
if not root.is_dir():
return []
return [p for p in sorted(root.rglob("*")) if p.is_file() and p.name.upper() == EBOOT_NAME]
def eboot_title(pbp_path: Path) -> str:
"""Read the game title embedded in an EBOOT.PBP (falls back to folder name)."""
from . import iso_reader, ps1_convert
try:
header = ps1_convert.read_pbp_header(pbp_path)
idx = ps1_convert.SECTION_OFFSET["sfo"] // 4
sfo = ps1_convert.read_pbp_section(pbp_path, header, idx)
parsed = iso_reader.parse_sfo(sfo)
title = parsed.get("TITLE") or parsed.get("TITLE_0") or parsed.get("TITLE_00")
if title:
return str(title)
except Exception:
pass
return pbp_path.parent.name
def scan_library(ps1_dir, output_dir) -> List[dict]:
"""Return a combined view of PS1 source discs and converted EBOOTs.
Each entry is a dict with ``kind`` ("disc" or "eboot"), ``path``, ``name``,
``serial`` and ``status`` ("pending" / "converted").
"""
ps1_dir = Path(ps1_dir)
output_dir = Path(output_dir)
eboots = find_eboots(output_dir)
eboot_names = {e.name.upper() for e in eboots}
eboot_titles = {eboot_title(e).lower() for e in eboots}
entries: List[dict] = []
for src in iter_source_discs(ps1_dir):
if src.suffix.lower() == ".cue":
image = cue_data_track(src) or src
else:
image = src
serial = read_serial(image)
title = src.stem
# A disc is "converted" when an EBOOT's embedded title or filename
# matches the source disc's title/serial.
status = "pending"
if title.lower() in eboot_titles or (serial and any(serial.lower() in t for t in eboot_titles)):
status = "converted"
entries.append({
"kind": "disc",
"path": str(src),
"name": title,
"serial": serial or "",
"status": status,
})
for e in eboots:
entries.append({
"kind": "eboot",
"path": str(e),
"name": eboot_title(e),
"serial": "",
"status": "converted",
})
return entries
+899
View File
@@ -0,0 +1,899 @@
"""PS1 -> PSP EBOOT.PBP conversion (pure Python).
Converts a PlayStation 1 disc image (``.iso``, ``.img``, or a ``.cue`` +
``.bin`` set) into the ``EBOOT.PBP`` "PS1 classic" container that PSP POPS
firmware can launch.
This is an independent, MIT-clean reimplementation of the popstation-md
algorithm (same format that PSXPackager/RupertAvery writes), implemented from
the published container format rather than by copying any GPL source.
Format reference (offsets are byte-exact):
* PBP header - 40 bytes: b"\\x00PBP" magic + version + 8 uint32 section offsets
(SFO, ICON0, ICON1, PIC0, PIC1, SND0, DATA.PSP, DATA.PSAR).
* PSAR - the DATA.PSAR section holding the disc image. Starts with
b"PSISOIMG0000", game id at +0x400, TOC at +0x800,
ISO block index at +0x4000, disc data at +0x100000.
A ``base.pbp`` (any PSN PS1 classic ``EBOOT.PBP``) is REQUIRED to supply two
blobs that we cannot synthesise:
* DATA.PSP - the encrypted POPS loader (header section 8).
* STARTDAT - the trailing encrypted block (boot.png + PGD).
Without a base.pbp the resulting EBOOT will not boot on real hardware; the
converter refuses to run without one.
Compression: only level 0 (uncompressed) is implemented. LZRC (levels 1-9) is a
planned fast-follow.
"""
from __future__ import annotations
import argparse
import io
import os
import re
import struct
import zlib
from dataclasses import dataclass, field
from pathlib import Path
from typing import Iterator, List, Optional, Tuple
# --------------------------------------------------------------------------
# Constants
# --------------------------------------------------------------------------
PBP_MAGIC = b"\x00PBP"
PBP_HEADER_SIZE = 0x28
PBP_HEADER_U32 = 10 # magic + version + 8 section offsets
# Section index -> header offset (in bytes) within the 40-byte header.
SECTION_OFFSET = {
"sfo": 0x08,
"icon0": 0x0C,
"icon1": 0x10,
"pic0": 0x14,
"pic1": 0x18,
"snd0": 0x1C,
"data_psp": 0x20,
"psar": 0x24,
}
SECTION_ORDER = ["sfo", "icon0", "icon1", "pic0", "pic1", "snd0", "data_psp", "psar"]
# PSAR (DATA.PSAR) layout.
PSAR_MAGIC = b"PSISOIMG0000"
PSAR_GAMEID_OFFSET = 0x400
PSAR_TOC_OFFSET = 0x800
PSAR_INDEX_OFFSET = 0x4000
PSAR_ISO_OFFSET = 0x100000
PSAR_ALIGNMENT = 0x10000
# One disc "block" in the ISO index = 16 raw 2352-byte sectors.
RAW_SECTOR = 0x930 # 2352
COOKED_SECTOR = 0x800 # 2048
BLOCK_SIZE = RAW_SECTOR * 16 # 0x9300
INDEX_ENTRY_SIZE = 32 # offset u32 + length u32 + 6 unused u32
SYNC_PATTERN = b"\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00"
# SFO field formats (PSP PARAM.SFO).
FMT_U32 = 0x0004
FMT_STRING = 0x0204
SFO_DEFAULTS = {
"BOOTABLE": (FMT_U32, 1),
"CATEGORY": (FMT_STRING, "ME"),
"DISC_VERSION": (FMT_STRING, "1.00"),
"LICENSE": (FMT_STRING, "Library programs Copyright(C) Sony ComputerEntertainment Inc."),
"PARENTAL_LEVEL": (FMT_U32, 1),
"PSP_SYSTEM_VER": (FMT_STRING, "3.71"),
"REGION": (FMT_U32, 0x8000),
}
class ConversionError(Exception):
"""Raised when a disc cannot be converted."""
class Cancelled(Exception):
"""Raised internally when the caller requests cancellation."""
# --------------------------------------------------------------------------
# CUE sheet parsing
# --------------------------------------------------------------------------
@dataclass
class CueTrack:
number: int
mode: str # "MODE1/2048", "MODE1/2352", "MODE2/2352", "AUDIO"
file: str
file_type: str # "BINARY" | "WAVE" | ...
indexes: List[Tuple[int, int]] = field(default_factory=list) # (index, msf)
sector_size: int = 0
@property
def pregap_sectors(self) -> int:
"""Sectors between INDEX 00 and INDEX 01 (0 if no pregap)."""
idx = dict(self.indexes)
if 0 in idx and 1 in idx:
return max(0, idx[1] - idx[0])
return 0
@dataclass
class CueDisc:
tracks: List[CueTrack]
@property
def data_track(self) -> Optional[CueTrack]:
for t in self.tracks:
if t.mode != "AUDIO":
return t
return None
def _msf_to_sectors(msf: str) -> int:
m, s, f = (int(x) for x in msf.split(":"))
return (m * 60 + s) * 75 + f
def _sector_size_for_mode(mode: str) -> int:
if mode in ("MODE1/2048",):
return COOKED_SECTOR
if mode in ("MODE1/2352", "MODE2/2352", "AUDIO"):
return RAW_SECTOR
# CDG / MODE2/2336 etc. fall back to raw.
return RAW_SECTOR
def parse_cue(text: str) -> CueDisc:
"""Parse a CUE sheet into a :class:`CueDisc`.
Handles the common PS1 layouts: one or more ``FILE`` lines, ``TRACK``
lines (``MODE1/2048``, ``MODE1/2352``, ``MODE2/2352``, ``AUDIO``) and
``INDEX`` lines (``INDEX 00`` pregap + ``INDEX 01`` start).
"""
tracks: List[CueTrack] = []
current_file: Optional[str] = None
current_type = "BINARY"
current_track: Optional[CueTrack] = None
file_re = re.compile(r'^FILE\s+"([^"]+)"\s+(\w+)', re.I)
track_re = re.compile(r'^\s*TRACK\s+(\d{1,2})\s+(\S+)', re.I)
index_re = re.compile(r'^\s*INDEX\s+(\d{1,2})\s+(\d{1,2}:\d{2}:\d{2})', re.I)
for raw in text.splitlines():
line = raw.strip()
if not line or line.startswith(("REM", "CATALOG", "CDTEXTFILE", "PERFORMER", "TITLE", "ISRC", "FLAGS", "POSTGAP", "PREGAP")):
if line.startswith("PREGAP"):
# PREGAP <mm:ss:ff> is equivalent to an INDEX 00 offset.
m = re.search(r'PREGAP\s+(\d{1,2}:\d{2}:\d{2})', line, re.I)
if m and current_track is not None:
current_track.indexes.append((0, _msf_to_sectors(m.group(1))))
continue
m = file_re.match(line)
if m:
current_file = m.group(1)
current_type = m.group(2).upper()
current_track = None
continue
m = track_re.match(line)
if m:
number = int(m.group(1))
mode = m.group(2).upper()
current_track = CueTrack(
number=number,
mode=mode,
file=current_file or "",
file_type=current_type,
sector_size=_sector_size_for_mode(mode),
)
tracks.append(current_track)
continue
m = index_re.match(line)
if m and current_track is not None:
current_track.indexes.append((int(m.group(1)), _msf_to_sectors(m.group(2))))
if not tracks:
raise ConversionError("CUE sheet contains no tracks")
for t in tracks:
if not t.file:
raise ConversionError(f"track {t.number} has no FILE")
return CueDisc(tracks)
# --------------------------------------------------------------------------
# Sector normalisation (cooked 2048 -> raw 2352 Mode 2 Form 1)
# --------------------------------------------------------------------------
def _to_bcd(value: int) -> int:
return ((value // 10) << 4) | (value % 10)
def _sectors_to_msf(sectors: int) -> Tuple[int, int, int]:
"""Convert a sector count to (minutes, seconds, frames) at 75 frames/s."""
total_seconds = sectors // 75
return total_seconds // 60, total_seconds % 60, sectors % 75
def _expand_cooked_sector(data: bytes, lba: int) -> bytes:
"""Expand a 2048-byte cooked sector to a 2352-byte raw Mode2 Form1 sector.
Rebuilds the 12-byte sync pattern, the 4-byte MSF header (BCD), the
8-byte subheader, and appends 280 bytes of EDC/ECC (zero-filled; the PSP
POPS loader does not verify EDC/ECC for converted games).
"""
assert len(data) == COOKED_SECTOR
frame = lba + 150 # 150-sector lead-in offset
msf = bytes([
_to_bcd((frame // 75 // 60) % 100),
_to_bcd((frame // 75) % 60),
_to_bcd(frame % 75),
0x02, # mode 2
])
# Subheader: file=2, channel=0, submode=0, coding=0; repeated once.
subheader = b"\x00\x00\x08\x00" * 2
return SYNC_PATTERN + msf + subheader + data + (b"\x00" * 280)
def iter_raw_sectors(
path: Path,
sector_size: int,
pregap_sectors: int = 0,
progress_cb=None,
cancel_event=None,
) -> Iterator[bytes]:
"""Yield 2352-byte raw sectors for one track file.
``sector_size`` is 2048 (cooked; expanded on the fly) or 2352 (raw).
``pregap_sectors`` leading zero sectors are emitted first.
"""
def _cancelled():
return cancel_event is not None and cancel_event.is_set()
if pregap_sectors:
for _ in range(pregap_sectors):
yield b"\x00" * RAW_SECTOR
size = path.stat().st_size
with open(path, "rb") as fh:
lba = 0
done = 0
while True:
if _cancelled():
raise Cancelled()
chunk = fh.read(sector_size)
if not chunk:
break
if len(chunk) != sector_size:
chunk = chunk.ljust(sector_size, b"\x00")
if sector_size == COOKED_SECTOR:
yield _expand_cooked_sector(chunk, lba)
else:
yield chunk
lba += 1
done += sector_size
if progress_cb:
progress_cb(done, size)
if progress_cb:
progress_cb(size, size)
# --------------------------------------------------------------------------
# PARAM.SFO writer
# --------------------------------------------------------------------------
def build_param_sfo(entries: List[Tuple[str, int, object]]) -> bytes:
"""Build a PSP PARAM.SFO from ``(key, fmt, value)`` entries.
``fmt`` is ``FMT_U32`` (uint32) or ``FMT_STRING`` (utf-8 string).
Layout matches the PSP format: header, index table, key table, data table.
"""
header_size = 20
index_size = len(entries) * 16
key_table_start = header_size + index_size
key_table = b"".join(key.encode("utf-8") + b"\x00" for key, _, _ in entries)
key_pad = (4 - (len(key_table) % 4)) % 4
data_table_start = key_table_start + len(key_table) + key_pad
data_table = b""
index_table = b""
key_offset = 0
data_offset = 0
for key, fmt, value in entries:
if fmt == FMT_U32:
raw = struct.pack("<I", int(value))
length = 4
max_len = 4
elif fmt == FMT_STRING:
s = str(value)
raw = s.encode("utf-8") + b"\x00"
length = len(s)
max_len = len(s) + 1
if len(raw) < max_len:
raw = raw.ljust(max_len, b"\x00")
else:
raise ConversionError(f"unsupported SFO format 0x{fmt:04x}")
index_table += struct.pack(
"<HHIII", key_offset, fmt, length, max_len, data_offset
)
data_table += raw
key_offset += len(key) + 1
data_offset += len(raw)
return (
b"\x00PSF"
+ struct.pack("<IIII", 0x00000101, key_table_start, data_table_start, len(entries))
+ index_table
+ key_table
+ (b"\x00" * key_pad)
+ data_table
)
def default_ps1_sfo(title: str, game_id: str) -> bytes:
"""A complete PARAM.SFO for a converted PS1 game."""
entries = []
for key, (fmt, value) in SFO_DEFAULTS.items():
entries.append((key, fmt, value))
entries.append(("DISC_ID", FMT_STRING, game_id))
entries.append(("TITLE", FMT_STRING, title))
return build_param_sfo(entries)
# --------------------------------------------------------------------------
# Minimal PNG generator (no external image deps)
# --------------------------------------------------------------------------
def _png(width: int, height: int, rgb: Tuple[int, int, int]) -> bytes:
def chunk(typ: bytes, data: bytes) -> bytes:
payload = typ + data
return struct.pack(">I", len(data)) + payload + struct.pack(">I", zlib.crc32(payload) & 0xFFFFFFFF)
ihdr = struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0) # 8-bit RGB
raw = b"".join(b"\x00" + bytes(rgb) * width for _ in range(height))
return (
b"\x89PNG\r\n\x1a\n"
+ chunk(b"IHDR", ihdr)
+ chunk(b"IDAT", zlib.compress(raw, 9))
+ chunk(b"IEND", b"")
)
def default_icon0_png() -> bytes:
return _png(144, 80, (24, 24, 24))
def default_pic1_png() -> bytes:
return _png(480, 272, (10, 10, 16))
# --------------------------------------------------------------------------
# PBP reading (used both for base.pbp extraction and self-test validation)
# --------------------------------------------------------------------------
def read_pbp_header(path: Path) -> List[int]:
with open(path, "rb") as fh:
data = fh.read(PBP_HEADER_SIZE)
if len(data) < PBP_HEADER_SIZE or data[:4] != PBP_MAGIC:
raise ConversionError(f"{path} is not a PBP file (bad magic)")
return list(struct.unpack("<10I", data))
def read_pbp_section(path: Path, header: List[int], index: int) -> bytes:
start = header[index]
end = header[index + 1] if index + 1 < len(header) else None
with open(path, "rb") as fh:
fh.seek(start)
data = fh.read() if end is None else fh.read(end - start)
return data
def extract_data_psp(base_pbp: Path) -> bytes:
"""Extract the encrypted POPS loader (DATA.PSP section) from a base PBP."""
header = read_pbp_header(base_pbp)
idx = SECTION_OFFSET["data_psp"] // 4 # header array index (2 + section index)
data = read_pbp_section(base_pbp, header, idx)
if not data:
raise ConversionError(f"{base_pbp} has no DATA.PSP section")
return data
def extract_startdat(base_pbp: Path) -> bytes:
"""Extract the trailing STARTDAT block (boot.png + PGD) from a base PBP.
The STARTDAT offset is discovered the same way the reference writer does:
read the u32 at ``psar + 0x0C``, add ``0x50000``, and expect b"STARTDAT"
there. Everything from that marker to EOF is copied verbatim.
"""
header = read_pbp_header(base_pbp)
psar_offset = header[SECTION_OFFSET["psar"] // 4] # header array index
with open(base_pbp, "rb") as fh:
fh.seek(psar_offset + 0x0C)
(x,) = struct.unpack("<I", fh.read(4))
marker_off = x + 0x50000
fh.seek(marker_off)
if fh.read(8) != b"STARTDAT":
raise ConversionError(f"cannot find STARTDAT in {base_pbp} (not a valid PS1 base PBP?)")
fh.seek(marker_off)
return fh.read()
raise ConversionError("unreachable")
# --------------------------------------------------------------------------
# PBP writer
# --------------------------------------------------------------------------
def _format_psar_game_id(game_id: str) -> str:
"""Format a PS1 serial for the PSAR header field.
The PBP's PARAM.SFO uses the plain 9-char id (``SLPS01762``), while the
PSAR header stores the region-delineated form (``_SLPS_01762``).
"""
m = re.match(r"^([A-Za-z]{4})[_-]?(\d{5})$", game_id)
if m:
return f"_{m.group(1).upper()}_{m.group(2)}"
return game_id
def _pad_to(out, target: int) -> None:
"""Write zero bytes until the stream reaches ``target`` (absolute offset)."""
delta = target - out.tell()
if delta < 0:
raise ConversionError(f"internal error: overshot offset by {-delta} bytes")
if delta:
out.write(b"\x00" * delta)
# Track type values used in the PS1 TOC (PSAR +0x800).
_TOC_DATA = 0x41
_TOC_AUDIO = 0x01
_TOC_A0 = 0xA0
_TOC_A1 = 0xA1
_TOC_A2 = 0xA2
_TOC_ENTRY = 10 # bytes per TOC entry
_TOC_LEAD_IN = 150 # 2-second lead-in, in sectors
def build_toc(disc: "DiscImage") -> bytes:
"""Generate the PS1 table-of-contents stored at PSAR +0x800.
Emits the standard CD-style TOC: an A0 entry (first track), an A1 entry
(last track), an A2 entry (lead-out), then one 10-byte entry per track.
Track addresses are BCD-encoded MSF; a 150-sector (2 s) lead-in is added
to the relative address field, matching the reference implementation.
"""
tracks = disc.tracks_info()
total = disc.total_sectors()
entries = [bytearray(_TOC_ENTRY) for _ in range(len(tracks) + 3)]
first_num, first_data, _ = tracks[0]
last_num, last_data, _ = tracks[-1]
e = entries[0]
e[0] = _TOC_DATA if first_data else _TOC_AUDIO
e[2] = _TOC_A0
e[7] = _to_bcd(first_num)
e[8] = _to_bcd(0x20)
e = entries[1]
e[0] = _TOC_DATA if last_data else _TOC_AUDIO
e[2] = _TOC_A1
e[7] = _to_bcd(last_num)
e = entries[2]
e[0] = _TOC_AUDIO
e[2] = _TOC_A2
mm, ss, ff = _sectors_to_msf(total)
e[7], e[8], e[9] = _to_bcd(mm), _to_bcd(ss), _to_bcd(ff)
for i, (num, is_data, start) in enumerate(tracks):
e = entries[3 + i]
e[0] = _TOC_DATA if is_data else _TOC_AUDIO
e[2] = _to_bcd(num)
mm, ss, ff = _sectors_to_msf(start)
e[3], e[4], e[5] = _to_bcd(mm), _to_bcd(ss), _to_bcd(ff)
mm, ss, ff = _sectors_to_msf(start + _TOC_LEAD_IN)
e[7], e[8], e[9] = _to_bcd(mm), _to_bcd(ss), _to_bcd(ff)
return b"".join(bytes(e) for e in entries)
def _build_psar_header(disc: "DiscImage", game_id: str, title: str) -> bytes:
"""Build the fixed PSAR header region 0x400..0x4000, byte-identical to the
popstation level-0 writer.
Layout (offsets relative to the PSAR start):
* 0x400 data1 (0xE20 bytes): game id at +0, TOC at +0x400, 0x10 at +0x7FE
* 0x1220 p2 offset (u32) = padded ISO size + 0x100000 + 0x2D31
* 0x1224 data2 (0x2DDC bytes): FF 07 at +4, title at +8, 0x03 at +0x88
"""
data1_size = 0xE20
data2_size = 0x2DDC
p2_delta = 0x2D31
block = bytearray(data1_size + 4 + data2_size) # 0x3C00 = 0x400..0x4000
gid = _format_psar_game_id(game_id).encode("ascii")[:11]
block[: len(gid)] = gid
toc = build_toc(disc)
block[0x400: 0x400 + len(toc)] = toc
block[0x7FE] = 0x10
p2 = disc.padded_size() + PSAR_ISO_OFFSET + p2_delta
block[data1_size: data1_size + 4] = struct.pack("<I", p2)
data2 = data1_size + 4
block[data2 + 4: data2 + 6] = b"\xFF\x07"
title_bytes = title.encode("ascii", "replace")[:0x80]
block[data2 + 8: data2 + 8 + len(title_bytes)] = title_bytes
block[data2 + 0x88] = 0x03
return bytes(block)
def _write_psar(
out: io.BufferedWriter,
disc: "DiscImage",
game_id: str,
title: str,
progress_cb=None,
cancel_event=None,
) -> None:
"""Write the DATA.PSAR section (uncompressed, single-disc).
All offsets are relative to the start of the PSAR section.
"""
base = out.tell()
iso_size = disc.padded_size()
# 0x000: PSAR signature; 0x00C: disc size hint.
out.write(PSAR_MAGIC)
out.write(struct.pack("<I", iso_size + PSAR_ISO_OFFSET))
_pad_to(out, base + PSAR_GAMEID_OFFSET)
out.write(_build_psar_header(disc, game_id, title))
_pad_to(out, base + PSAR_INDEX_OFFSET)
# ISO block index: (offset, length) per 0x9300 block. Uncompressed =>
# length is always BLOCK_SIZE and offsets increase linearly.
block_count = iso_size // BLOCK_SIZE
offset = 0
for _ in range(block_count):
out.write(struct.pack("<II", offset, BLOCK_SIZE))
out.write(b"\x00" * (INDEX_ENTRY_SIZE - 8))
offset += BLOCK_SIZE
_pad_to(out, base + PSAR_ISO_OFFSET)
# Disc data, block by block.
buf = bytearray()
for sector in disc.iter_raw(progress_cb=progress_cb, cancel_event=cancel_event):
buf.extend(sector)
if len(buf) == BLOCK_SIZE:
out.write(buf)
buf = bytearray()
if buf:
out.write(buf.ljust(BLOCK_SIZE, b"\x00"))
class DiscImage:
"""A normalised view of a PS1 disc as a stream of raw 2352-byte sectors."""
def __init__(self, cue: Optional[CueDisc], path: Path):
self.cue = cue
self.path = path
def total_sectors(self) -> int:
if self.cue is None:
size = self.path.stat().st_size
if size % RAW_SECTOR == 0:
return size // RAW_SECTOR
return (size + COOKED_SECTOR - 1) // COOKED_SECTOR
total = 0
base_dir = self.path.parent
for track in self.cue.tracks:
fpath = base_dir / track.file
if fpath.is_file():
total += track.pregap_sectors + (fpath.stat().st_size // track.sector_size)
return total
def padded_size(self) -> int:
size = self.total_sectors() * RAW_SECTOR
if size % BLOCK_SIZE:
size += BLOCK_SIZE - (size % BLOCK_SIZE)
return size
def tracks_info(self) -> List[Tuple[int, bool, int]]:
"""Return ``(track_number, is_data, absolute_start_sector)`` per track.
``absolute_start_sector`` is the disc-wide sector of each track's
INDEX 01 (i.e. accounting for every preceding track's pregap + data).
A cue-less single image is treated as one data track starting at 0.
"""
if self.cue is None:
return [(1, True, 0)]
base_dir = self.path.parent
out: List[Tuple[int, bool, int]] = []
cursor = 0
for track in self.cue.tracks:
idx01 = dict(track.indexes).get(1, 0)
out.append((track.number, track.mode != "AUDIO", cursor + idx01))
fpath = base_dir / track.file
data_sectors = fpath.stat().st_size // track.sector_size if fpath.is_file() else 0
cursor += track.pregap_sectors + data_sectors
return out
def iter_raw(self, progress_cb=None, cancel_event=None) -> Iterator[bytes]:
if self.cue is None:
yield from iter_raw_sectors(
self.path,
RAW_SECTOR if self.path.stat().st_size % RAW_SECTOR == 0 else COOKED_SECTOR,
progress_cb=progress_cb,
cancel_event=cancel_event,
)
return
base_dir = self.path.parent
for track in self.cue.tracks:
fpath = base_dir / track.file
if not fpath.is_file():
raise ConversionError(f"track file missing: {fpath}")
yield from iter_raw_sectors(
fpath,
track.sector_size,
pregap_sectors=track.pregap_sectors,
progress_cb=progress_cb,
cancel_event=cancel_event,
)
def _aligned_up(value: int, alignment: int) -> int:
return value + ((alignment - (value % alignment)) % alignment)
def convert(
source: Path,
title: str,
game_id: str,
base_pbp: Path,
out_path: Path,
progress_cb=None,
cancel_event=None,
) -> Path:
"""Convert a PS1 disc image to ``out_path`` (an ``EBOOT.PBP``).
``source`` may be a ``.cue`` (multi-track supported) or a single
``.iso``/``.img``/``.bin``. ``base_pbp`` must be a PSN PS1 classic EBOOT.
"""
if not base_pbp.is_file():
raise ConversionError(f"base.pbp not found: {base_pbp}")
if not source.is_file():
raise ConversionError(f"disc image not found: {source}")
game_id = game_id.strip()
if not game_id:
raise ConversionError("a game id (e.g. SLUS00594) is required")
if source.suffix.lower() == ".cue":
disc: DiscImage = DiscImage(parse_cue(source.read_text(encoding="utf-8", errors="replace")), source)
else:
disc = DiscImage(None, source)
# Resources required from the base PBP.
data_psp = extract_data_psp(base_pbp)
startdat = extract_startdat(base_pbp)
sfo = default_ps1_sfo(title, game_id)
icon0 = default_icon0_png()
pic1 = default_pic1_png()
# Sections in header order; empty sections keep the previous offset.
sections = {
"sfo": sfo,
"icon0": icon0,
"icon1": b"",
"pic0": b"",
"pic1": pic1,
"snd0": b"",
"data_psp": data_psp,
# psar written separately, sized after everything else.
"psar": None,
}
header = [0] * PBP_HEADER_U32
header[0] = struct.unpack("<I", PBP_MAGIC)[0]
header[1] = 0x00010000
current = PBP_HEADER_SIZE
for i, key in enumerate(SECTION_ORDER):
header[2 + i] = current
if key != "psar":
current += len(sections[key])
# DATA.PSAR must start on a 0x10000 boundary. It is the final section, so
# its length is implicit (everything up to STARTDAT/EOF).
psar_offset = _aligned_up(header[2 + SECTION_ORDER.index("psar")], PSAR_ALIGNMENT)
header[2 + SECTION_ORDER.index("psar")] = psar_offset
out_path.parent.mkdir(parents=True, exist_ok=True)
tmp = out_path.with_suffix(out_path.suffix + ".tmp")
try:
with open(tmp, "wb") as fh:
fh.write(struct.pack("<10I", *header))
for key in SECTION_ORDER:
if key == "psar":
break
if sections[key]:
fh.write(sections[key])
# Pad up to the aligned PSAR offset.
while fh.tell() < psar_offset:
fh.write(b"\x00")
_write_psar(fh, disc, game_id, title, progress_cb=progress_cb, cancel_event=cancel_event)
# STARTDAT goes at the very end.
fh.write(startdat)
os.replace(tmp, out_path)
except BaseException:
try:
tmp.unlink(missing_ok=True)
except OSError:
pass
raise
return out_path
def convert_to_folder(
source: Path,
title: str,
game_id: str,
base_pbp: Path,
out_dir: Path,
progress_cb=None,
cancel_event=None,
) -> Path:
"""Convert a PS1 disc into a game folder ready for /PSP/GAME.
Writes ``<out_dir>/<GAME_ID>/EBOOT.PBP`` — the same single-file output the
popstation tool produces (uncompressed/unencrypted level-0 PSAR). No
KEYS.BIN or DOCUMENT.DAT is written: popstation doesn't emit them and CFW
(ARK-4 POPS) doesn't need them.
"""
folder_name = re.sub(r"[^A-Za-z0-9]", "", game_id).upper() or "PS1GAME"
folder = Path(out_dir) / folder_name
folder.mkdir(parents=True, exist_ok=True)
convert(source, title, game_id, base_pbp, folder / "EBOOT.PBP",
progress_cb=progress_cb, cancel_event=cancel_event)
return folder
# --------------------------------------------------------------------------
# Self-test / CLI
# --------------------------------------------------------------------------
def _self_test() -> None:
"""Validate the parts that do not need hardware: SFO round-trip, PNG
generation, PBP header round-trip, and cue parsing."""
import tempfile
# CUE parsing.
cue = parse_cue(
'FILE "Game (Track 1).bin" BINARY\n'
" TRACK 01 MODE2/2352\n"
" INDEX 01 00:00:00\n"
" TRACK 02 AUDIO\n"
" INDEX 00 00:02:00\n"
" INDEX 01 00:04:00\n"
'FILE "Game (Track 2).bin" BINARY\n'
" TRACK 03 AUDIO\n"
" INDEX 01 00:00:00\n"
)
assert len(cue.tracks) == 3, "expected 3 tracks"
assert cue.tracks[0].mode == "MODE2/2352"
assert cue.tracks[1].pregap_sectors == 150, "expected 2s (150 sector) pregap"
assert cue.data_track is cue.tracks[0]
# TOC generation: 1 data (10 sectors) + 1 audio track (20 sectors + 2s pregap).
with tempfile.TemporaryDirectory() as td:
d = Path(td)
(d / "t1.bin").write_bytes(b"\x00" * (10 * RAW_SECTOR))
(d / "t2.bin").write_bytes(b"\x00" * (20 * RAW_SECTOR))
cue2 = parse_cue(
'FILE "t1.bin" BINARY\n TRACK 01 MODE2/2352\n INDEX 01 00:00:00\n'
'FILE "t2.bin" BINARY\n TRACK 02 AUDIO\n INDEX 00 00:00:00\n INDEX 01 00:02:00\n'
)
toc = build_toc(DiscImage(cue2, d / "t.cue"))
assert len(toc) == (2 + 3) * 10, len(toc)
assert toc[0] == 0x41 and toc[2] == 0xA0 and toc[7] == 0x01 # A0: first track
assert toc[10] == 0x01 and toc[12] == 0xA1 and toc[17] == 0x02 # A1: last track
assert toc[22] == 0xA2 # A2: lead-out
assert toc[30] == 0x41 and toc[32] == 0x01 # track 1: data, number 1
assert toc[40] == 0x01 and toc[42] == 0x02 # track 2: audio, number 2
# track 2 relative address = start(160) + 150 lead-in = 310 = 00:04:10
assert toc[47] == 0x00 and toc[48] == 0x04 and toc[49] == 0x10
# SFO round-trip through the project's own parser.
from . import iso_reader # local import to keep the module importable standalone
sfo = default_ps1_sfo("Final Fantasy VII", "SCUS94163")
parsed = iso_reader.parse_sfo(sfo)
assert parsed.get("TITLE") == "Final Fantasy VII", parsed
assert parsed.get("DISC_ID") == "SCUS94163", parsed
assert parsed.get("CATEGORY") == "ME", parsed
assert parsed.get("BOOTABLE") == 1, parsed
# PNG generation produces valid PNG signatures + IHDR dimensions.
icon = default_icon0_png()
assert icon[:8] == b"\x89PNG\r\n\x1a\n"
assert struct.unpack(">II", icon[16:24]) == (144, 80)
pic = default_pic1_png()
assert struct.unpack(">II", pic[16:24]) == (480, 272)
# PBP header round-trip: write a synthetic PBP and read it back.
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "test.pbp"
header = [0] * 10
header[0] = struct.unpack("<I", PBP_MAGIC)[0]
header[1] = 0x00010000
header[2] = 0x28
header[3] = 0x28 + len(sfo)
with open(p, "wb") as fh:
fh.write(struct.pack("<10I", *header))
fh.write(sfo)
assert read_pbp_header(p)[0] == header[0]
assert read_pbp_section(p, header, 2) == sfo
print("ps1_convert self-test OK")
def main(argv: Optional[List[str]] = None) -> int:
ap = argparse.ArgumentParser(description="Convert a PS1 disc image to a PSP EBOOT.PBP")
ap.add_argument("--source", help="path to .cue or a single .iso/.img/.bin")
ap.add_argument("--title", help="game title shown on the PSP")
ap.add_argument("--game-id", help="PS1 game id, e.g. SLUS00594")
ap.add_argument("--base-pbp", help="path to a PSN PS1 classic EBOOT.PBP (required)")
ap.add_argument("--out", help="output EBOOT.PBP path (single file)")
ap.add_argument("--out-dir", help="output folder: writes <dir>/<GAME_ID>/EBOOT.PBP")
ap.add_argument("--test", action="store_true", help="run self-test and exit")
args = ap.parse_args(argv)
if args.test:
_self_test()
return 0
if not all([args.source, args.title, args.game_id, args.base_pbp]) or (not args.out and not args.out_dir):
ap.error("--source --title --game-id --base-pbp and --out (or --out-dir) are all required (or use --test)")
try:
if args.out_dir:
folder = convert_to_folder(
Path(args.source), args.title, args.game_id,
Path(args.base_pbp), Path(args.out_dir),
)
print(f"wrote {folder}")
else:
convert(
Path(args.source), args.title, args.game_id,
Path(args.base_pbp), Path(args.out),
)
print(f"wrote {args.out}")
return 0
except ConversionError as exc:
print(f"error: {exc}")
return 1
if __name__ == "__main__":
raise SystemExit(main())
+4
View File
@@ -18,6 +18,8 @@ FIELDS = [
"igdb_client_id",
"igdb_client_secret",
"tmdb_api_key",
"google_api_key",
"google_cx",
"screenscraper_devid",
"screenscraper_devpassword",
"screenscraper_ssid",
@@ -29,6 +31,8 @@ _ENV_MAP = {
"igdb_client_id": "IGDB_CLIENT_ID",
"igdb_client_secret": "IGDB_CLIENT_SECRET",
"tmdb_api_key": "TMDB_API_KEY",
"google_api_key": "GOOGLE_API_KEY",
"google_cx": "GOOGLE_CX",
"screenscraper_devid": "SCREENSCRAPER_DEVID",
"screenscraper_devpassword": "SCREENSCRAPER_DEVPASSWORD",
"screenscraper_ssid": "SCREENSCRAPER_SSID",
+49
View File
@@ -6,6 +6,7 @@ archives, a remote HTTPS store server, etc.
import json
import logging
import os
import re
import shutil
import tempfile
import urllib.parse
@@ -200,6 +201,54 @@ class FileCopySource(Source):
return dest
class Ps1ConvertSource(Source):
"""Convert a PS1 disc image (.cue/.iso) into an EBOOT.PBP.
``item`` must carry ``source`` (path to the .cue or single image),
``title`` and ``game_id``. ``base_pbp`` is a PSN PS1 classic EBOOT used to
supply the encrypted DATA.PSP loader and STARTDAT block.
"""
label = "PS1 convert"
def __init__(self, base_pbp):
self.base_pbp = Path(base_pbp)
def scan(self, root=None):
return []
def download(self, item, dest_dir, progress_cb=None, cancel_event=None):
dest_dir = Path(dest_dir)
dest_dir.mkdir(parents=True, exist_ok=True)
source = Path(item["source"])
title = str(item.get("title") or source.stem)
game_id = item.get("game_id")
if not game_id:
raise RuntimeError("missing game_id for PS1 conversion")
folder_name = re.sub(r"[^A-Za-z0-9]", "", game_id).upper() or "PS1GAME"
folder = dest_dir / folder_name
eboot = folder / "EBOOT.PBP"
if eboot.exists():
raise FileExistsError(str(eboot))
def cb(done, total):
if progress_cb is not None and total:
progress_cb("Converting", min(done / total, 1.0), "")
from . import ps1_convert
try:
ps1_convert.convert_to_folder(
source, title, game_id, self.base_pbp, dest_dir,
progress_cb=cb, cancel_event=cancel_event,
)
except ps1_convert.Cancelled:
raise TransferCancelled() from None
return folder
class TreeCopySource(Source):
"""Copy a whole directory (save-data folder) into a destination directory."""
File diff suppressed because one or more lines are too long
@@ -1,5 +1,5 @@
clrmamepro (
name "Sony - PlayStation Portable"
name "Sony - PlayStation Portable (No-Intro)"
description "Sony - PlayStation Portable"
version "2021.10.24"
homepage "http://github.com/robloach/libretro-dats"
File diff suppressed because it is too large Load Diff