library: only show converted PS1 EBOOTs

Pending PS1 source discs stay in the PS1 tab; the Library now lists only
converted EBOOT entries.

README.md

@@ -93,8 +93,10 @@ Discs are matched by hash first (most precise), then serial, then filename.
 - `data/covers/` — checked-in assets: the UMD video/game cover templates,
   `libretro-index.json` + `openvgdb-index.json` catalogs, `libretro-covers.zip` +
   `openvgdb-covers.zip` offline cover bundles, and the legacy cover cache.
-- `data/base.PBP` — a PSN PS1 classic EBOOT (any game) used as the encrypted
+- `data/base.PBP` — a PSN PS1 classic EBOOT (any game) supplying `STARTDAT`
-  loader when converting PS1 discs to `EBOOT.PBP` (git-ignored, copyrighted).
+  + a base `PARAM.SFO`/icon for PS1 conversion (git-ignored, copyrighted).
+- `data/loader/DATA.PSP` + `data/loader/KEYS.BIN` — bundled plain-ELF POPS
+  loader and NPDRM version key (from iPoPS, MIT).
 - State dir (`~/.local/state/psp-preserve/` by default, or `$XDG_STATE_HOME/psp-preserve`):
   - `library.sqlite` — the catalog + library database.
   - `covers/` — downloaded cover cache.
@@ -188,16 +190,22 @@ PS1 discs (pending) and converted EBOOTs also show up in the **Library** tab
 - **Serial detection** — reads `SYSTEM.CNF` to derive the game ID (e.g.
   `SLUS-00001`) and writes it into `PARAM.SFO` + the PSAR header.
 - **base.pbp** — conversion needs a PSN PS1 classic `EBOOT.PBP` (any game) to
-  supply Sony's encrypted `DATA.PSP` loader. Copy one to `data/base.PBP` and
+  supply the `STARTDAT` block (boot logo + PGD) and a base `PARAM.SFO`/icon.
-  point **Settings → General → PS1 base.pbp** at it.
+  Copy one to `data/base.PBP` and point **Settings → General → PS1 base.pbp**
-- **Output** — a folder `<out>/<Game ID>/` containing a single `EBOOT.PBP`
+  at it. Any PS1 classic works structurally — the converter finds `STARTDAT`
-  (the popstation way: uncompressed level-0 PSAR, no `KEYS.BIN`/`DOCUMENT.DAT`),
+  at `PSAR + p1` regardless of PSAR alignment.
-  ready to copy to `<PSP>/PSP/GAME/<Game ID>/`. Compatible with CFW POPS
+- **POPS loader** — the `DATA.PSP` section uses a bundled plain-ELF loader
-  (ARK-4).
+  (`data/loader/DATA.PSP`, from iPoPS, MIT) so the EBOOT boots on modern CFW
+  without the base PBP's encrypted loader failing to link (`80020148`). The
+  base PBP's own loader is only used as a fallback when the bundle is missing.
+- **Output** — a folder `<out>/<Game ID>/` containing `EBOOT.PBP` (raw-DEFLATE
+  compressed PSAR) plus a `KEYS.BIN` (NPDRM version key used to decrypt the
+  `STARTDAT` PGD), ready to copy to `<PSP>/PSP/GAME/<Game ID>/`. Compatible
+  with CFW POPS (ARK-4/ARK-5).
 
 Configure the folders in **Settings → General** (or `PSP_PS1_ROOTS` /
 `PSP_PS1_OUTPUT_DIR`). Current limits: single-disc games only (no multi-disc
-PBP), and images are stored uncompressed (LZRC compression is on the roadmap).
+PBP); disc data is DEFLATE-compressed (level 1 by default, `--complevel 0-9`).
 
 ## Store — archive.org and local folder
 

app/gui.py

@@ -1055,7 +1055,7 @@ class SettingsPage(QWidget):
 
     def __init__(self, games_dir, videos_dir, store_dir, ppsspp_dir, cloud_dir, ppsspp_exe,
                  store_server_url, store_token, store_archive_id, store_archive_sig, store_archive_user,
-                 iso_category, ps1_dir, ps1_output_dir, ps1_base_pbp, parent=None):
+                 iso_category, ps1_category, ps1_dir, ps1_output_dir, ps1_base_pbp, parent=None):
         super().__init__(parent)
         outer = QVBoxLayout(self)
         outer.setContentsMargins(0, 0, 0, 0)
@@ -1107,6 +1107,9 @@ class SettingsPage(QWidget):
         self.iso_category_edit = QLineEdit(iso_category)
         self.iso_category_edit.setPlaceholderText("e.g. Homebrew (blank = ISO root)")
         form.addRow("ISO category (Category Lite):", self.iso_category_edit)
+        self.ps1_category_edit = QLineEdit(ps1_category)
+        self.ps1_category_edit.setPlaceholderText("e.g. PS1 (blank = PSP/GAME root)")
+        form.addRow("PS1 category (Category Lite):", self.ps1_category_edit)
         self.tabs.addTab(page, "PSP")
 
         # -- Cover sources ------------------------------------------------ #
@@ -1259,6 +1262,9 @@ class SettingsPage(QWidget):
     def iso_category(self):
         return self.iso_category_edit.text().strip()
 
+    def ps1_category(self):
+        return self.ps1_category_edit.text().strip()
+
     def ps1_dir(self):
         return self.ps1_edit.text().strip()
 
@@ -3121,7 +3127,7 @@ def build_library_items(ps1_dir="", output_dir=""):
 
 
 def _ps1_library_items(ps1_dir, output_dir, overrides, dates):
-    """PS1 discs (pending) + converted EBOOTs, merged into the library list."""
+    """Converted PS1 EBOOTs, merged into the library list."""
     if not ps1_dir:
         return []
     try:
@@ -3130,8 +3136,9 @@ def _ps1_library_items(ps1_dir, output_dir, overrides, dates):
         return []
     out = []
     for e in entries:
-        # A converted disc is represented by its EBOOT entry (one row per game).
+        # Only converted EBOOTs are shown in the Library; pending source discs
-        if e["kind"] == "disc" and e.get("status") == "converted":
+        # live in the PS1 tab, where they can be converted.
+        if e["kind"] != "eboot":
             continue
         name = e["name"]
         rid = covers.stable_key(name)
@@ -3467,6 +3474,7 @@ class MainWindow(QMainWindow):
         self._auto_sync_saves = str(self.settings.value("auto_sync_saves", "false")).lower() == "true"
         self._ppsspp_exe = str(self.settings.value("ppsspp_exe", "") or "")
         self._iso_category = str(self.settings.value("iso_category", "") or "")
+        self._ps1_category = str(self.settings.value("ps1_category", "") or "")
         self._ps1_dir = self._first_existing(config.PS1_ROOTS) or str(self.settings.value("ps1_dir", "") or "")
         self._ps1_output_dir = str(self.settings.value("ps1_output_dir", str(config.PS1_OUTPUT_DIR)) or "")
         self._ps1_base_pbp = str(self.settings.value("ps1_base_pbp", "") or "")
@@ -3678,6 +3686,7 @@ class MainWindow(QMainWindow):
                                           self._ppsspp_saves_dir, self._cloud_saves_dir, self._ppsspp_exe,
                                           self._store_server_url, self._store_token, self._store_archive_id,
                                           self._store_archive_sig, self._store_archive_user, self._iso_category,
+                                          self._ps1_category,
                                           self._ps1_dir, self._ps1_output_dir, self._ps1_base_pbp)
         self.settings_page.saved.connect(self._on_settings_saved)
         return self.settings_page
@@ -3707,6 +3716,7 @@ class MainWindow(QMainWindow):
         self._auto_sync_saves = self.settings_page.auto_sync_saves()
         self._ppsspp_exe = self.settings_page.ppsspp_exe()
         self._iso_category = self.settings_page.iso_category()
+        self._ps1_category = self.settings_page.ps1_category()
         self._ps1_dir = self.settings_page.ps1_dir()
         self._ps1_output_dir = self.settings_page.ps1_output_dir()
         self._ps1_base_pbp = self.settings_page.ps1_base_pbp()
@@ -3719,6 +3729,7 @@ class MainWindow(QMainWindow):
         self.settings.setValue("auto_sync_saves", self._auto_sync_saves)
         self.settings.setValue("ppsspp_exe", self._ppsspp_exe)
         self.settings.setValue("iso_category", self._iso_category)
+        self.settings.setValue("ps1_category", self._ps1_category)
         self._update_saves_locations()
         self._update_dir_status()
 
@@ -4124,9 +4135,11 @@ class MainWindow(QMainWindow):
         if system == "videos":
             dest_dir = psp_root / "ISO" / "VIDEO"
         elif system == "ps1":
-            # PS1 EBOOTs install as a whole game folder under /PSP/GAME.
+            # PS1 EBOOTs install as a whole game folder under /PSP/GAME
+            # (optionally nested in a Category Lite folder).
             src_folder = Path(item["path"]).parent
-            dest_dir = psp_root / "PSP" / "GAME"
+            ps1_cat = "".join(c for c in (self._ps1_category or "").strip() if c not in "/\\")
+            dest_dir = (psp_root / "PSP" / "GAME" / ps1_cat) if ps1_cat else (psp_root / "PSP" / "GAME")
             log.info("copying ps1 game to psp: %s -> %s", src_folder, dest_dir / src_folder.name)
             self.queue.enqueue(transfers.TreeCopySource(str(src_folder)), dest_dir,
                                {"dest_name": src_folder.name}, item.get("name") or src_folder.name,
@@ -4249,6 +4262,7 @@ class MainWindow(QMainWindow):
         out_dir = Path(self._ps1_output_dir) if self._ps1_output_dir else Path.cwd() / "ps1_converted"
         source = transfers.Ps1ConvertSource(self._ps1_base_pbp)
         self.queue.enqueue(source, out_dir, item, item.get("title") or item["source"], kind="ps1-convert")
+        self.tabs.setCurrentWidget(self.queue_page)
 
     def _on_queue_job_done(self, job):
         if self._closing:

app/ps1_convert.py

@@ -15,22 +15,27 @@ Format reference (offsets are byte-exact):
                     b"PSISOIMG0000", game id at +0x400, TOC at +0x800,
                     ISO block index at +0x4000, disc data at +0x100000.
 
-A ``base.pbp`` (any PSN PS1 classic ``EBOOT.PBP``) is REQUIRED to supply two
+A ``base.pbp`` (any PSN PS1 classic ``EBOOT.PBP``) supplies the blobs that we
-blobs that we cannot synthesise:
+cannot synthesise:
 
-    * DATA.PSP  - the encrypted POPS loader (header section 8).
     * STARTDAT  - the trailing encrypted block (boot.png + PGD).
+    * DATA.PSP  - fallback only: the encrypted POPS loader, used when the
+                  bundled plain-ELF loader (``data/loader/DATA.PSP``, from
+                  iPoPS/MIT) is unavailable.
 
 Without a base.pbp the resulting EBOOT will not boot on real hardware; the
 converter refuses to run without one.
 
-Compression: only level 0 (uncompressed) is implemented. LZRC (levels 1-9) is a
+Compression: the disc data is split into 0x9300-byte blocks and deflated
-planned fast-follow.
+(raw DEFLATE, no zlib wrapper) at the configured level (default 1). POPS on
+real PSP hardware refuses to boot uncompressed (level-0) PSARs, so level 0
+is kept only for diagnostics; a valid conversion always uses level >= 1.
 """
 
 from __future__ import annotations
 
 import argparse
+import hashlib
 import io
 import os
 import re
@@ -67,7 +72,9 @@ PSAR_GAMEID_OFFSET = 0x400
 PSAR_TOC_OFFSET = 0x800
 PSAR_INDEX_OFFSET = 0x4000
 PSAR_ISO_OFFSET = 0x100000
-PSAR_ALIGNMENT = 0x10000
+# popstation aligns the DATA.PSAR section to a 0x50000 boundary (the base PBP
+# also places it there). Some POPS loaders rely on this alignment.
+PSAR_ALIGNMENT = 0x50000
 
 # One disc "block" in the ISO index = 16 raw 2352-byte sectors.
 RAW_SECTOR = 0x930       # 2352
@@ -77,6 +84,14 @@ INDEX_ENTRY_SIZE = 32    # offset u32 + length u32 + 6 unused u32
 
 SYNC_PATTERN = b"\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00"
 
+# Bundled plain-ELF POPS loader + NPDRM key (from iPoPS, MIT). Modern CFW
+# (ARK/PRO/ME) loads this ELF directly as the DATA.PSP section, sidestepping
+# the base PBP's encrypted ~PSP loader which frequently fails to link on
+# 6.60/6.61 (error 80020148 "game could not be started").
+_LOADER_DIR = Path(__file__).resolve().parent.parent / "data" / "loader"
+BUNDLED_DATA_PSP = _LOADER_DIR / "DATA.PSP"
+BUNDLED_KEYS_BIN = _LOADER_DIR / "KEYS.BIN"
+
 # SFO field formats (PSP PARAM.SFO).
 FMT_U32 = 0x0004
 FMT_STRING = 0x0204
@@ -112,10 +127,17 @@ class CueTrack:
     file_type: str       # "BINARY" | "WAVE" | ...
     indexes: List[Tuple[int, int]] = field(default_factory=list)  # (index, msf)
     sector_size: int = 0
+    external_pregap: int = 0  # sectors from a PREGAP command (not in the file)
 
     @property
     def pregap_sectors(self) -> int:
-        """Sectors between INDEX 00 and INDEX 01 (0 if no pregap)."""
+        """Sectors between INDEX 00 and INDEX 01 (the in-file pregap).
+
+        This is informational only: when both INDEX 00 and INDEX 01 are
+        present in the same FILE, the pregap data is already inside the file
+        and must not be re-emitted. A ``PREGAP`` command, by contrast, is a
+        pregap that is *not* present in the file (see ``external_pregap``).
+        """
         idx = dict(self.indexes)
         if 0 in idx and 1 in idx:
             return max(0, idx[1] - idx[0])
@@ -168,10 +190,11 @@ def parse_cue(text: str) -> CueDisc:
         line = raw.strip()
         if not line or line.startswith(("REM", "CATALOG", "CDTEXTFILE", "PERFORMER", "TITLE", "ISRC", "FLAGS", "POSTGAP", "PREGAP")):
             if line.startswith("PREGAP"):
-                # PREGAP <mm:ss:ff> is equivalent to an INDEX 00 offset.
+                # PREGAP <mm:ss:ff> is a pregap NOT stored in the track file;
+                # the writer must emit that many leading zero sectors.
                 m = re.search(r'PREGAP\s+(\d{1,2}:\d{2}:\d{2})', line, re.I)
                 if m and current_track is not None:
-                    current_track.indexes.append((0, _msf_to_sectors(m.group(1))))
+                    current_track.external_pregap = _msf_to_sectors(m.group(1))
             continue
 
         m = file_re.match(line)
@@ -348,6 +371,93 @@ def default_ps1_sfo(title: str, game_id: str) -> bytes:
     return build_param_sfo(entries)
 
 
+def extract_sfo(base_pbp: Path) -> bytes:
+    """Return the PARAM.SFO section of a base PBP."""
+    header = read_pbp_header(base_pbp)
+    idx = SECTION_OFFSET["sfo"] // 4
+    return read_pbp_section(base_pbp, header, idx)
+
+
+def extract_icon0(base_pbp: Path) -> bytes:
+    """Return the ICON0.PNG section of a base PBP."""
+    header = read_pbp_header(base_pbp)
+    idx = SECTION_OFFSET["icon0"] // 4
+    return read_pbp_section(base_pbp, header, idx)
+
+
+def _sfo_entry(base_sfo: bytes, key: str) -> Optional[bytes]:
+    """Return the raw data of an SFO entry, or ``None`` if absent."""
+    if len(base_sfo) < 20 or base_sfo[:4] != b"\x00PSF":
+        return None
+    _, _, key_table_off, data_table_off, nitems = struct.unpack("<IIIII", base_sfo[:20])
+    for i in range(nitems):
+        entry_off = 20 + i * 16
+        koff, _, _, max_len, doff = struct.unpack("<HHIII", base_sfo[entry_off:entry_off + 16])
+        kend = base_sfo.find(b"\x00", key_table_off + koff)
+        k = bytes(base_sfo[key_table_off + koff:kend]).decode("ascii", "replace")
+        if k == key:
+            return base_sfo[data_table_off + doff: data_table_off + doff + max_len]
+    return None
+
+
+def _warn_old_base(base_pbp: Path) -> None:
+    """Warn when the base PBP carries a very old POPS loader.
+
+    PS1 classics from the 3.xx firmware era ship a POPS loader that often
+    fails to link on 6.60/6.61 CFW (the PSP reports error 80020148), so
+    newer base PBPs are strongly preferred.
+    """
+    try:
+        raw = _sfo_entry(extract_sfo(base_pbp), "PSP_SYSTEM_VER")
+        if not raw:
+            return
+        version = raw.rstrip(b"\x00").decode("ascii", "replace")
+        major = version.split(".")[0]
+        if major.isdigit() and int(major) < 6:
+            import sys
+            print(
+                f"warning: base PBP is a {version}-era PS1 classic; its POPS loader "
+                "may fail to load on 6.60/6.61 CFW (error 80020148). "
+                "Prefer a newer base.pbp.",
+                file=sys.stderr,
+            )
+    except Exception:
+        pass
+
+
+def patch_sfo(base_sfo: bytes, title: str, game_id: str) -> bytes:
+    """Patch ``TITLE`` and ``DISC_ID`` in a base PARAM.SFO in place.
+
+    The reference writer reuses the base PBP's SFO verbatim (preserving the
+    original field types, sizes and values) and only rewrites the title and
+    game id. This keeps the output byte-compatible with official PSN classics
+    (e.g. integer fields stay ``0x0404`` rather than ``0x0004``).
+    """
+    if len(base_sfo) < 20 or base_sfo[:4] != b"\x00PSF":
+        raise ConversionError("base PBP has no valid PARAM.SFO")
+    _, _, key_table_off, data_table_off, nitems = struct.unpack("<IIIII", base_sfo[:20])
+    sfo = bytearray(base_sfo)
+
+    patches = {"TITLE": title.encode("utf-8"), "DISC_ID": game_id.encode("ascii")}
+    for i in range(nitems):
+        entry_off = 20 + i * 16
+        koff, _, _, max_len, doff = struct.unpack("<HHIII", sfo[entry_off:entry_off + 16])
+        key_end = sfo.find(b"\x00", key_table_off + koff)
+        key = bytes(sfo[key_table_off + koff:key_end]).decode("ascii", "replace")
+        if key not in patches:
+            continue
+        value = patches[key]
+        # strncpy semantics: value + zero padding, capped at the entry's size.
+        raw = value[:max_len].ljust(max_len, b"\x00")
+        sfo[data_table_off + doff: data_table_off + doff + max_len] = raw
+        # SFO string length includes the NUL terminator (matches the base SFO
+        # and the reference SetSFOTitle implementation).
+        new_len = min(len(value) + 1, max_len)
+        struct.pack_into("<I", sfo, entry_off + 4, new_len)
+
+    return bytes(sfo)
+
+
 # --------------------------------------------------------------------------
 # Minimal PNG generator (no external image deps)
 # --------------------------------------------------------------------------
@@ -397,28 +507,61 @@ def read_pbp_section(path: Path, header: List[int], index: int) -> bytes:
 
 
 def extract_data_psp(base_pbp: Path) -> bytes:
-    """Extract the encrypted POPS loader (DATA.PSP section) from a base PBP."""
+    """Extract the encrypted POPS loader (DATA.PSP) from a base PBP.
+
+    The reference writer reads the ``~PSP`` header's total-size field
+    (u32 at DATA.PSP + 0x2C, the 12th u32) and copies exactly that many
+    bytes. The section in the source PBP is padded with zeros up to the PSAR
+    offset; copying that padding would bloat the DATA.PSP section for no
+    reason, so we copy only ``prx_size`` bytes, matching popstation.
+    """
     header = read_pbp_header(base_pbp)
     idx = SECTION_OFFSET["data_psp"] // 4  # header array index (2 + section index)
-    data = read_pbp_section(base_pbp, header, idx)
+    section = read_pbp_section(base_pbp, header, idx)
-    if not data:
+    if len(section) < 0x30:
         raise ConversionError(f"{base_pbp} has no DATA.PSP section")
-    return data
+    if section[:4] != b"~PSP":
+        raise ConversionError(f"{base_pbp} DATA.PSP has no ~PSP header")
+    (prx_size,) = struct.unpack("<I", section[0x2C:0x30])
+    if prx_size == 0 or prx_size > len(section):
+        prx_size = len(section)
+    return section[:prx_size]
+
+
+def load_data_psp(base_pbp: Path) -> bytes:
+    """Return the POPS loader (DATA.PSP) to embed.
+
+    Prefers the bundled plain-ELF loader (MIT, from iPoPS), which modern CFW
+    can load without ~PSP decryption. Falls back to the base PBP's encrypted
+    ~PSP loader when the bundle is absent.
+    """
+    if BUNDLED_DATA_PSP.is_file():
+        return BUNDLED_DATA_PSP.read_bytes()
+    return extract_data_psp(base_pbp)
+
+
+def load_keys_bin() -> Optional[bytes]:
+    """Return the bundled NPDRM version key (KEYS.BIN), if available."""
+    if BUNDLED_KEYS_BIN.is_file():
+        return BUNDLED_KEYS_BIN.read_bytes()
+    return None
 
 
 def extract_startdat(base_pbp: Path) -> bytes:
     """Extract the trailing STARTDAT block (boot.png + PGD) from a base PBP.
 
-    The STARTDAT offset is discovered the same way the reference writer does:
+    STARTDAT sits immediately after the disc data: at ``PSAR + p1`` where
-    read the u32 at ``psar + 0x0C``, add ``0x50000``, and expect b"STARTDAT"
+    ``p1`` is the u32 at ``PSAR + 0x0C``. (The original popstation computes
-    there. Everything from that marker to EOF is copied verbatim.
+    ``p1 + 0x50000`` as an absolute offset, which only works for bases whose
+    PSAR happens to be aligned to 0x50000; using ``PSAR + p1`` works for any
+    PS1 classic.)
     """
     header = read_pbp_header(base_pbp)
     psar_offset = header[SECTION_OFFSET["psar"] // 4]  # header array index
     with open(base_pbp, "rb") as fh:
         fh.seek(psar_offset + 0x0C)
         (x,) = struct.unpack("<I", fh.read(4))
-        marker_off = x + 0x50000
+        marker_off = psar_offset + x
         fh.seek(marker_off)
         if fh.read(8) != b"STARTDAT":
             raise ConversionError(f"cannot find STARTDAT in {base_pbp} (not a valid PS1 base PBP?)")
@@ -474,22 +617,25 @@ def build_toc(disc: "DiscImage") -> bytes:
     total = disc.total_sectors()
     entries = [bytearray(_TOC_ENTRY) for _ in range(len(tracks) + 3)]
 
-    first_num, first_data, _ = tracks[0]
+    first_num, _, _ = tracks[0]
-    last_num, last_data, _ = tracks[-1]
+    last_num, _, _ = tracks[-1]
 
+    # A-point entries use control byte 0x41 (data) regardless of track type,
+    # matching popstation and official PSN classics. A0's byte 8 is the raw
+    # Red Book disc-type bitfield 0x20 (CD-ROM XA), not a BCD value.
     e = entries[0]
-    e[0] = _TOC_DATA if first_data else _TOC_AUDIO
+    e[0] = _TOC_DATA
     e[2] = _TOC_A0
     e[7] = _to_bcd(first_num)
-    e[8] = _to_bcd(0x20)
+    e[8] = 0x20
 
     e = entries[1]
-    e[0] = _TOC_DATA if last_data else _TOC_AUDIO
+    e[0] = _TOC_DATA
     e[2] = _TOC_A1
     e[7] = _to_bcd(last_num)
 
     e = entries[2]
-    e[0] = _TOC_AUDIO
+    e[0] = _TOC_DATA
     e[2] = _TOC_A2
     mm, ss, ff = _sectors_to_msf(total)
     e[7], e[8], e[9] = _to_bcd(mm), _to_bcd(ss), _to_bcd(ff)
@@ -540,49 +686,104 @@ def _build_psar_header(disc: "DiscImage", game_id: str, title: str) -> bytes:
     return bytes(block)
 
 
+def _write_disc_block(
+    out: io.BufferedWriter,
+    block: bytes,
+    complevel: int,
+    data_offset: int,
+    indexes: bytearray,
+) -> int:
+    """Compress and write one 0x9300 disc block, appending its index entry.
+
+    Blocks are deflated with raw DEFLATE (the zlib header and Adler-32
+    trailer stripped). If a block does not shrink, the raw bytes are stored
+    instead. The index entry follows pop-fe: offset (u32), length (u16),
+    flags (u8), then the first 16 bytes of the block's SHA-1.
+    """
+    assert len(block) == BLOCK_SIZE
+    compressed = zlib.compress(block, complevel)[2:-4] if complevel > 0 else block
+
+    entry = bytearray(INDEX_ENTRY_SIZE)
+    struct.pack_into("<I", entry, 0, data_offset)
+    if complevel == 0:
+        entry[6] = 0x01  # uncompressed block
+    entry[8:24] = hashlib.sha1(block).digest()[:16]
+
+    if len(compressed) >= BLOCK_SIZE:
+        struct.pack_into("<H", entry, 4, BLOCK_SIZE)
+        out.write(block)
+        data_offset += BLOCK_SIZE
+    else:
+        struct.pack_into("<H", entry, 4, len(compressed))
+        out.write(compressed)
+        data_offset += len(compressed)
+
+    indexes.extend(entry)
+    return data_offset
+
+
 def _write_psar(
     out: io.BufferedWriter,
     disc: "DiscImage",
     game_id: str,
     title: str,
+    complevel: int = 1,
     progress_cb=None,
     cancel_event=None,
 ) -> None:
-    """Write the DATA.PSAR section (uncompressed, single-disc).
+    """Write the DATA.PSAR section (single-disc).
+
+    Disc data is split into 0x9300-byte blocks and deflated (``complevel`` >
+    0). The PSAR header's p1/p2 fields and the index table are patched after
+    the compressed data is written, since the compressed size is not known
+    up front.
 
     All offsets are relative to the start of the PSAR section.
     """
     base = out.tell()
     iso_size = disc.padded_size()
+    block_count = iso_size // BLOCK_SIZE
 
-    # 0x000: PSAR signature; 0x00C: disc size hint.
+    # 0x000: PSAR signature; 0x00C: p1 (disc data end, patched below).
     out.write(PSAR_MAGIC)
     out.write(struct.pack("<I", iso_size + PSAR_ISO_OFFSET))
     _pad_to(out, base + PSAR_GAMEID_OFFSET)
 
+    # 0x400..0x4000: game id / TOC / title (p2 at 0x1220 is patched below).
     out.write(_build_psar_header(disc, game_id, title))
     _pad_to(out, base + PSAR_INDEX_OFFSET)
 
-    # ISO block index: (offset, length) per 0x9300 block. Uncompressed =>
+    # Reserve the index table; filled in once the disc data is compressed.
-    # length is always BLOCK_SIZE and offsets increase linearly.
-    block_count = iso_size // BLOCK_SIZE
-    offset = 0
     for _ in range(block_count):
-        out.write(struct.pack("<II", offset, BLOCK_SIZE))
+        out.write(b"\x00" * INDEX_ENTRY_SIZE)
-        out.write(b"\x00" * (INDEX_ENTRY_SIZE - 8))
-        offset += BLOCK_SIZE
 
     _pad_to(out, base + PSAR_ISO_OFFSET)
 
-    # Disc data, block by block.
+    # Compress + write disc data, one 0x9300 block at a time.
+    indexes = bytearray()
+    data_offset = 0  # relative to PSAR_ISO_OFFSET
     buf = bytearray()
     for sector in disc.iter_raw(progress_cb=progress_cb, cancel_event=cancel_event):
         buf.extend(sector)
-        if len(buf) == BLOCK_SIZE:
+        if len(buf) < BLOCK_SIZE:
-            out.write(buf)
+            continue
-            buf = bytearray()
+        data_offset = _write_disc_block(out, buf, complevel, data_offset, indexes)
+        buf = bytearray()
     if buf:
-        out.write(buf.ljust(BLOCK_SIZE, b"\x00"))
+        data_offset = _write_disc_block(out, buf.ljust(BLOCK_SIZE, b"\x00"), complevel, data_offset, indexes)
+
+    # Align the end of the disc data to 0x10 before STARTDAT (which the
+    # caller appends), then patch p1/p2 and the index table.
+    _pad_to(out, (out.tell() + 0xF) & ~0xF)
+    end_offset = out.tell() - base
+    end = out.tell()
+    out.seek(base + 0x0C)
+    out.write(struct.pack("<I", end_offset))
+    out.seek(base + 0x1220)
+    out.write(struct.pack("<I", end_offset + 0x2D31))
+    out.seek(base + PSAR_INDEX_OFFSET)
+    out.write(bytes(indexes))
+    out.seek(end)
 
 
 class DiscImage:
@@ -603,7 +804,7 @@ class DiscImage:
         for track in self.cue.tracks:
             fpath = base_dir / track.file
             if fpath.is_file():
-                total += track.pregap_sectors + (fpath.stat().st_size // track.sector_size)
+                total += track.external_pregap + (fpath.stat().st_size // track.sector_size)
         return total
 
     def padded_size(self) -> int:
@@ -627,10 +828,10 @@ class DiscImage:
         cursor = 0
         for track in self.cue.tracks:
             idx01 = dict(track.indexes).get(1, 0)
-            out.append((track.number, track.mode != "AUDIO", cursor + idx01))
+            out.append((track.number, track.mode != "AUDIO", cursor + track.external_pregap + idx01))
             fpath = base_dir / track.file
             data_sectors = fpath.stat().st_size // track.sector_size if fpath.is_file() else 0
-            cursor += track.pregap_sectors + data_sectors
+            cursor += track.external_pregap + data_sectors
         return out
 
     def iter_raw(self, progress_cb=None, cancel_event=None) -> Iterator[bytes]:
@@ -651,7 +852,7 @@ class DiscImage:
             yield from iter_raw_sectors(
                 fpath,
                 track.sector_size,
-                pregap_sectors=track.pregap_sectors,
+                pregap_sectors=track.external_pregap,
                 progress_cb=progress_cb,
                 cancel_event=cancel_event,
             )
@@ -669,11 +870,14 @@ def convert(
     out_path: Path,
     progress_cb=None,
     cancel_event=None,
+    complevel: int = 1,
 ) -> Path:
     """Convert a PS1 disc image to ``out_path`` (an ``EBOOT.PBP``).
 
     ``source`` may be a ``.cue`` (multi-track supported) or a single
     ``.iso``/``.img``/``.bin``. ``base_pbp`` must be a PSN PS1 classic EBOOT.
+    ``complevel`` is the DEFLATE level for the disc data (1-9; 0 = raw, which
+    does not boot on real hardware and is only useful for diagnostics).
     """
     if not base_pbp.is_file():
         raise ConversionError(f"base.pbp not found: {base_pbp}")
@@ -690,12 +894,18 @@ def convert(
         disc = DiscImage(None, source)
 
     # Resources required from the base PBP.
-    data_psp = extract_data_psp(base_pbp)
+    data_psp = load_data_psp(base_pbp)
     startdat = extract_startdat(base_pbp)
+    _warn_old_base(base_pbp)
 
-    sfo = default_ps1_sfo(title, game_id)
+    # Reuse the base PBP's SFO, patched with our title + game id (the
-    icon0 = default_icon0_png()
+    # popstation approach), rather than rebuilding it from scratch.
-    pic1 = default_pic1_png()
+    sfo = patch_sfo(extract_sfo(base_pbp), title, game_id)
+    # Reuse the base PBP's ICON0 too. The XMB icon parser is picky about PNG
+    # variants, and the base icon is guaranteed to decode; the other image
+    # sections are left empty exactly like popstation when no assets are
+    # supplied alongside the conversion.
+    icon0 = extract_icon0(base_pbp)
 
     # Sections in header order; empty sections keep the previous offset.
     sections = {
@@ -703,7 +913,7 @@ def convert(
         "icon0": icon0,
         "icon1": b"",
         "pic0": b"",
-        "pic1": pic1,
+        "pic1": b"",
         "snd0": b"",
         "data_psp": data_psp,
         # psar written separately, sized after everything else.
@@ -740,7 +950,8 @@ def convert(
             while fh.tell() < psar_offset:
                 fh.write(b"\x00")
 
-            _write_psar(fh, disc, game_id, title, progress_cb=progress_cb, cancel_event=cancel_event)
+            _write_psar(fh, disc, game_id, title, complevel=complevel,
+                        progress_cb=progress_cb, cancel_event=cancel_event)
 
             # STARTDAT goes at the very end.
             fh.write(startdat)
@@ -751,6 +962,13 @@ def convert(
         except OSError:
             pass
         raise
+
+    # Write the NPDRM version key next to the EBOOT (the popstation/iPoPS
+    # convention; ARK and other CFW read it to decrypt the STARTDAT PGD).
+    keys = load_keys_bin()
+    if keys is not None:
+        out_path.with_name("KEYS.BIN").write_bytes(keys)
+
     return out_path
 
 
@@ -762,20 +980,20 @@ def convert_to_folder(
     out_dir: Path,
     progress_cb=None,
     cancel_event=None,
+    complevel: int = 1,
 ) -> Path:
     """Convert a PS1 disc into a game folder ready for /PSP/GAME.
 
-    Writes ``<out_dir>/<GAME_ID>/EBOOT.PBP`` — the same single-file output the
+    Writes ``<out_dir>/<GAME_ID>/EBOOT.PBP`` plus a ``KEYS.BIN`` (the NPDRM
-    popstation tool produces (uncompressed/unencrypted level-0 PSAR). No
+    version key modern CFW uses to decrypt the STARTDAT PGD). No
-    KEYS.BIN or DOCUMENT.DAT is written: popstation doesn't emit them and CFW
+    DOCUMENT.DAT is written; CFW (ARK POPS) doesn't need one.
-    (ARK-4 POPS) doesn't need them.
     """
     folder_name = re.sub(r"[^A-Za-z0-9]", "", game_id).upper() or "PS1GAME"
     folder = Path(out_dir) / folder_name
     folder.mkdir(parents=True, exist_ok=True)
 
     convert(source, title, game_id, base_pbp, folder / "EBOOT.PBP",
-            progress_cb=progress_cb, cancel_event=cancel_event)
+            progress_cb=progress_cb, cancel_event=cancel_event, complevel=complevel)
     return folder
 
 
@@ -805,6 +1023,13 @@ def _self_test() -> None:
     assert cue.tracks[1].pregap_sectors == 150, "expected 2s (150 sector) pregap"
     assert cue.data_track is cue.tracks[0]
 
+    # PREGAP command is a pregap NOT stored in the file.
+    cue_pg = parse_cue(
+        'FILE "t.bin" BINARY\n  TRACK 01 AUDIO\n    PREGAP 00:02:00\n    INDEX 01 00:00:00\n'
+    )
+    assert cue_pg.tracks[0].external_pregap == 150, cue_pg.tracks[0].external_pregap
+    assert cue_pg.tracks[0].pregap_sectors == 0
+
     # TOC generation: 1 data (10 sectors) + 1 audio track (20 sectors + 2s pregap).
     with tempfile.TemporaryDirectory() as td:
         d = Path(td)
@@ -814,11 +1039,16 @@ def _self_test() -> None:
             'FILE "t1.bin" BINARY\n  TRACK 01 MODE2/2352\n    INDEX 01 00:00:00\n'
             'FILE "t2.bin" BINARY\n  TRACK 02 AUDIO\n    INDEX 00 00:00:00\n    INDEX 01 00:02:00\n'
         )
-        toc = build_toc(DiscImage(cue2, d / "t.cue"))
+        disc2 = DiscImage(cue2, d / "t.cue")
+        # INDEX 00 pregap is inside the file, so the disc is just the files
+        # concatenated: 10 + 20 sectors (not 10 + 150 + 20).
+        assert disc2.total_sectors() == 30, disc2.total_sectors()
+        toc = build_toc(disc2)
         assert len(toc) == (2 + 3) * 10, len(toc)
         assert toc[0] == 0x41 and toc[2] == 0xA0 and toc[7] == 0x01  # A0: first track
-        assert toc[10] == 0x01 and toc[12] == 0xA1 and toc[17] == 0x02  # A1: last track
+        assert toc[8] == 0x20  # A0: raw Red Book disc-type byte, not BCD
-        assert toc[22] == 0xA2  # A2: lead-out
+        assert toc[10] == 0x41 and toc[12] == 0xA1 and toc[17] == 0x02  # A1: last track
+        assert toc[20] == 0x41 and toc[22] == 0xA2  # A2: lead-out (control 0x41)
         assert toc[30] == 0x41 and toc[32] == 0x01  # track 1: data, number 1
         assert toc[40] == 0x01 and toc[42] == 0x02  # track 2: audio, number 2
         # track 2 relative address = start(160) + 150 lead-in = 310 = 00:04:10
@@ -834,6 +1064,13 @@ def _self_test() -> None:
     assert parsed.get("CATEGORY") == "ME", parsed
     assert parsed.get("BOOTABLE") == 1, parsed
 
+    # Base-SFO patching preserves everything but TITLE + DISC_ID.
+    patched = patch_sfo(default_ps1_sfo("Hot Shots Golf", "SCUS94476"), "Pepsiman", "SLPS01762")
+    parsed2 = iso_reader.parse_sfo(patched)
+    assert parsed2.get("TITLE") == "Pepsiman", parsed2
+    assert parsed2.get("DISC_ID") == "SLPS01762", parsed2
+    assert parsed2.get("BOOTABLE") == 1, parsed2
+
     # PNG generation produces valid PNG signatures + IHDR dimensions.
     icon = default_icon0_png()
     assert icon[:8] == b"\x89PNG\r\n\x1a\n"
@@ -855,6 +1092,35 @@ def _self_test() -> None:
         assert read_pbp_header(p)[0] == header[0]
         assert read_pbp_section(p, header, 2) == sfo
 
+    # PSAR compression round-trip (level 1): the disc data must survive
+    # raw-DEFLATE and the index table must be readable with the pop-fe
+    # layout (offset u32, length u16, flags u8, SHA-1[16]).
+    with tempfile.TemporaryDirectory() as td:
+        d = Path(td)
+        src = d / "cd.bin"
+        src.write_bytes(bytes(range(256)) * (BLOCK_SIZE // 256 * 3))
+        disc = DiscImage(None, src)
+        buf = io.BytesIO()
+        _write_psar(buf, disc, "SLUS00594", "Round Trip", complevel=1)
+        raw = buf.getvalue()
+        assert raw[:12] == PSAR_MAGIC
+        rebuilt = bytearray()
+        i = 0
+        while True:
+            e = raw[PSAR_INDEX_OFFSET + i * 32: PSAR_INDEX_OFFSET + (i + 1) * 32]
+            off, = struct.unpack("<I", e[0:4])
+            ln, = struct.unpack("<H", e[4:6])
+            if ln == 0:
+                break
+            data = raw[PSAR_ISO_OFFSET + off: PSAR_ISO_OFFSET + off + ln]
+            rebuilt += zlib.decompress(data, wbits=-15) if ln < BLOCK_SIZE else data
+            i += 1
+        assert i == 3, i
+        assert len(rebuilt) == disc.padded_size(), len(rebuilt)
+        assert bytes(rebuilt[: src.stat().st_size]) == src.read_bytes()
+        # first index entry carries the SHA-1 of the first raw block
+        assert raw[PSAR_INDEX_OFFSET + 8: PSAR_INDEX_OFFSET + 24] == hashlib.sha1(bytes(rebuilt[:BLOCK_SIZE])).digest()[:16]
+
     print("ps1_convert self-test OK")
 
 
@@ -866,6 +1132,8 @@ def main(argv: Optional[List[str]] = None) -> int:
     ap.add_argument("--base-pbp", help="path to a PSN PS1 classic EBOOT.PBP (required)")
     ap.add_argument("--out", help="output EBOOT.PBP path (single file)")
     ap.add_argument("--out-dir", help="output folder: writes <dir>/<GAME_ID>/EBOOT.PBP")
+    ap.add_argument("--complevel", type=int, default=1, choices=range(0, 10),
+                    help="DEFLATE compression level 0-9 for the disc data (default 1)")
     ap.add_argument("--test", action="store_true", help="run self-test and exit")
     args = ap.parse_args(argv)
 
@@ -880,13 +1148,13 @@ def main(argv: Optional[List[str]] = None) -> int:
         if args.out_dir:
             folder = convert_to_folder(
                 Path(args.source), args.title, args.game_id,
-                Path(args.base_pbp), Path(args.out_dir),
+                Path(args.base_pbp), Path(args.out_dir), complevel=args.complevel,
             )
             print(f"wrote {folder}")
         else:
             convert(
                 Path(args.source), args.title, args.game_id,
-                Path(args.base_pbp), Path(args.out),
+                Path(args.base_pbp), Path(args.out), complevel=args.complevel,
             )
             print(f"wrote {args.out}")
         return 0

data/loader/ATTRIBUTION.md

@@ -0,0 +1,30 @@
+# Loader assets
+
+The files in this directory are bundled from [iPoPS](https://github.com/julianxhokaxhiu/iPoPS)
+by Julian Xhokaxhiu, licensed under the MIT License:
+
+- `DATA.PSP` — a plain-ELF PSP POPS loader (relocatable PRX, 19002 bytes).
+- `KEYS.BIN` — a 16-byte NPDRM version key used by CFW to decrypt the
+  `STARTDAT` PGD.
+
+MIT License
+
+Copyright (c) 2020 Julian Xhokaxhiu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/loader/KEYS.BIN

@@ -0,0 +1 @@
+MB\Ω:���� �"