dm2-graphics: console-family 4-bit images carry a trailing per-image Palette16 — dropped by the decoder, blocks colour-correct Mac JP / PC-98x1 / IBM PS/V rendering #186

New issue

Closed

opened 2026-07-05 07:47:47 +02:00 by jqueuniet · 0 comments

jqueuniet commented

2026-07-05 07:47:47 +02:00

(Migrated from codeberg.org)

Problem

decode_4bit_body only preserves the trailing 16-byte per-image Palette16 when pc_family is true (crates/dm2-graphics/src/image/decoders.rs ~785, trailing_palette = pc_family). The engine copies the last 16 bytes of every 4-bit dtImage record into the decoded buffer as the per-image palette, unconditionally ([SKPROJECT] v5/bgdat.cpp:1252, the vw_1c == 4 branch of DM2_EXTRACT_GDAT_IMAGE).

Fixture evidence says the engine is right and the crate's console-family assumption is wrong for 4 of the 6 non-PC ports:

RLE consumption probe (reimplementation of the nibble RLE tracking consumed bytes): Mac JP, Amiga EU, PC-9801, PC-9821 4-bit images leave exactly 16 unconsumed bytes as the dominant bucket; Sega CD US leaves 0 on 1,498/1,500 records.
Cross-port tail comparison: for records sharing the same (cls1, cls2, cls4, cls5) tuple and dimensions, the last 16 payload bytes are byte-identical to the dos-en trailing Palette16 on Mac JP (515/623), Amiga (446/452), PC-9821 and IBM PS/V (554 each) — while zero payloads match in full. Bodies differ (different encodings/revisions), palettes match: only a genuine palette explains that.
Clean detection mechanism: bit 0x01 of the (0, 0, 0x0b, 0) inline feature word correlates perfectly across all 9 archives — set on dos-en/dos-de/dos-fr/Mac US (0x7b), Mac JP / PC-9801 / PC-9821 / IBM PS/V (0x0b), Amiga (0x2b); clear on Sega CD / FM-Towns (0x0a), the only two ports without trailing palettes.

Consequences

Mac JP, PC-9821, IBM PS/V carry a 256-entry master palette and per-image palettes → 100 % colour-correct rendering is achievable today. This overturns the claims in Archive::find_image_palette's docstring ("Mac JP … the Mac JP application binary holds the actual palette tables") and the matching sections of docs/formats/graphics-dat.md.
Amiga: the archive lacks a 256-entry master, but the recovered per-image indices are byte-identical to the DOS master's index space — so the pixel→master-index layer is in the archive after all, and pairing with the DOS master RGB is a plausible near-correct stand-in until the engine-side Amiga table is extracted from SKUL.FTL.
Only Sega CD and FM-Towns genuinely lack trailing palettes; their 96-entry-bank stripe-selection question stays open.
Currently the crate returns palette: None for these records and find_image_palette falls back to wrong-colour heuristics; pixels are usually unaffected (the RLE fills the buffer before reaching the palette bytes), but any under-filling stream reads palette bytes as RLE commands.

Fix direction

Gate the trailing palette on pc_family || (inline word bit 0x01) — or simply on bit 0x01, since all four PC-family archives set it — in decode_4bit_body, and plumb the flag through Archive::decode_image_payload / Image::parse. Then let find_image_palette prefer the per-image palette + 256-entry master on Mac JP / PC-9821 / IBM PS/V, and consider a --master-palette style override so Amiga can borrow the DOS master. Update docs/formats/graphics-dat.md and the find_image_palette coverage table accordingly.

Probes: scratch/probe_trailing_pal.py, scratch/probe_tail_match.py (gitignored).

## Problem `decode_4bit_body` only preserves the trailing 16-byte per-image Palette16 when `pc_family` is true (`crates/dm2-graphics/src/image/decoders.rs` ~785, `trailing_palette = pc_family`). The engine copies the last 16 bytes of **every** 4-bit dtImage record into the decoded buffer as the per-image palette, unconditionally (`[SKPROJECT] v5/bgdat.cpp:1252`, the `vw_1c == 4` branch of `DM2_EXTRACT_GDAT_IMAGE`). Fixture evidence says the engine is right and the crate's console-family assumption is wrong for 4 of the 6 non-PC ports: - **RLE consumption probe** (reimplementation of the nibble RLE tracking consumed bytes): Mac JP, Amiga EU, PC-9801, PC-9821 4-bit images leave **exactly 16 unconsumed bytes** as the dominant bucket; Sega CD US leaves 0 on 1,498/1,500 records. - **Cross-port tail comparison**: for records sharing the same `(cls1, cls2, cls4, cls5)` tuple and dimensions, the last 16 payload bytes are **byte-identical to the dos-en trailing Palette16** on Mac JP (515/623), Amiga (446/452), PC-9821 and IBM PS/V (554 each) — while **zero** payloads match in full. Bodies differ (different encodings/revisions), palettes match: only a genuine palette explains that. - **Clean detection mechanism**: bit `0x01` of the `(0, 0, 0x0b, 0)` inline feature word correlates perfectly across all 9 archives — set on dos-en/dos-de/dos-fr/Mac US (`0x7b`), Mac JP / PC-9801 / PC-9821 / IBM PS/V (`0x0b`), Amiga (`0x2b`); clear on Sega CD / FM-Towns (`0x0a`), the only two ports without trailing palettes. ## Consequences - **Mac JP, PC-9821, IBM PS/V carry a 256-entry master palette *and* per-image palettes** → 100 % colour-correct rendering is achievable today. This overturns the claims in `Archive::find_image_palette`'s docstring ("Mac JP … the Mac JP application binary holds the actual palette tables") and the matching sections of `docs/formats/graphics-dat.md`. - **Amiga**: the archive lacks a 256-entry master, but the recovered per-image indices are byte-identical to the DOS master's index space — so the pixel→master-index layer is in the archive after all, and pairing with the DOS master RGB is a plausible near-correct stand-in until the engine-side Amiga table is extracted from `SKUL.FTL`. - Only **Sega CD** and **FM-Towns** genuinely lack trailing palettes; their 96-entry-bank stripe-selection question stays open. - Currently the crate returns `palette: None` for these records and `find_image_palette` falls back to wrong-colour heuristics; pixels are usually unaffected (the RLE fills the buffer before reaching the palette bytes), but any under-filling stream reads palette bytes as RLE commands. ## Fix direction Gate the trailing palette on `pc_family || (inline word bit 0x01)` — or simply on bit 0x01, since all four PC-family archives set it — in `decode_4bit_body`, and plumb the flag through `Archive::decode_image_payload` / `Image::parse`. Then let `find_image_palette` prefer the per-image palette + 256-entry master on Mac JP / PC-9821 / IBM PS/V, and consider a `--master-palette` style override so Amiga can borrow the DOS master. Update `docs/formats/graphics-dat.md` and the `find_image_palette` coverage table accordingly. Probes: `scratch/probe_trailing_pal.py`, `scratch/probe_tail_match.py` (gitignored).

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

jqueuniet/dm2-tools#186

No description provided.

Rows
Columns