jqueuniet/dm2-tools

Fork 0

dm2-saves: retail skull.exe CREATURE_GIDX_ALT_TABLE applied unconditionally to Mac and Beta saves #174

New issue

Closed

opened 2026-07-04 15:15:17 +02:00 by jqueuniet · 2 comments

jqueuniet commented

2026-07-04 15:15:17 +02:00

(Migrated from codeberg.org)

Problem

CREATURE_GIDX_ALT_TABLE (records.rs ~87) is documented as extracted from the retail DOS skull.exe binary, and its own comment notes the Mac build would need its own table. Yet compute_creature_alt_species_mask — which consults it to choose between the 66-bit DEFAULT and 50-bit ALT creature masks — is applied unconditionally to every variant (sksave.rs decode_full ~927 and to_bytes_from_fields ~1142), including Mac retail and DOS Beta v0.9 (Fires.exe is not covered at all).

Impact

Any creature species whose descriptor bit0 differs between the retail skull.exe table and the Mac/Beta engine binaries selects the wrong per-record mask width → full bitstream desync from that record onward on the affected platform. Currently a risk flag rather than a proven divergence (the Mac/Beta tables have not been compared), but the exposure is the same class as the mask-table differences already found between platforms (champion mask, Beta object masks).

Fix direction

Extract the equivalent table from the Mac 68k DATA_0000 resource (same methodology as MAC_CHAMPION_MASK) and from the unpacked Beta Fires.exe, compare against the retail table, and dispatch per variant if they differ (or document byte-identity if they do not).

## Problem `CREATURE_GIDX_ALT_TABLE` (`records.rs` ~87) is documented as extracted from the retail DOS `skull.exe` binary, and its own comment notes the Mac build would need its own table. Yet `compute_creature_alt_species_mask` — which consults it to choose between the 66-bit DEFAULT and 50-bit ALT creature masks — is applied unconditionally to every variant (`sksave.rs` decode_full ~927 and to_bytes_from_fields ~1142), including Mac retail and DOS Beta v0.9 (`Fires.exe` is not covered at all). ## Impact Any creature species whose descriptor bit0 differs between the retail `skull.exe` table and the Mac/Beta engine binaries selects the wrong per-record mask width → full bitstream desync from that record onward on the affected platform. Currently a risk flag rather than a proven divergence (the Mac/Beta tables have not been compared), but the exposure is the same class as the mask-table differences already found between platforms (champion mask, Beta object masks). ## Fix direction Extract the equivalent table from the Mac 68k `DATA_0000` resource (same methodology as `MAC_CHAMPION_MASK`) and from the unpacked Beta `Fires.exe`, compare against the retail table, and dispatch per variant if they differ (or document byte-identity if they do not).

jqueuniet commented

2026-07-06 08:47:23 +02:00

(Migrated from codeberg.org)

Investigated in #209; documented, not code-changed (kept open as a follow-up).

Retail table validated: re-deriving from DOS/USA/skull.exe at file offset 0x72568 (36-byte stride, word[0] & 1, 64 entries, LSB-first) reproduces CREATURE_GIDX_ALT_TABLE byte-for-byte.
Mac/Beta extraction blocked on binary work out of scope: DOS/BETA/Fires.exe is a packed executable (the descriptor table is not a contiguous block in the on-disk image — cf. the FTL/hunk decoder effort), and the Mac 68k build keeps its descriptors in the resource fork in big-endian layout with an unverified stride.
Risk flag, not a proven divergence: the Mac and DOS-Beta save fixtures both decode_full and re-encode byte-for-byte with the retail table applied, so no observed creature diverges; re-selecting per variant without the extracted tables would be a guess that could regress those passing fixtures.

The status is now documented precisely on the constant (commit dca12e2c). Dispatch per variant once the Mac/Beta tables are extracted and shown to differ.

Investigated in #209; documented, not code-changed (kept open as a follow-up). - **Retail table validated**: re-deriving from `DOS/USA/skull.exe` at file offset `0x72568` (36-byte stride, `word[0] & 1`, 64 entries, LSB-first) reproduces `CREATURE_GIDX_ALT_TABLE` byte-for-byte. - **Mac/Beta extraction blocked on binary work out of scope**: `DOS/BETA/Fires.exe` is a packed executable (the descriptor table is not a contiguous block in the on-disk image — cf. the FTL/hunk decoder effort), and the Mac 68k build keeps its descriptors in the resource fork in big-endian layout with an unverified stride. - **Risk flag, not a proven divergence**: the Mac and DOS-Beta save fixtures both `decode_full` and re-encode byte-for-byte with the retail table applied, so no observed creature diverges; re-selecting per variant without the extracted tables would be a guess that could regress those passing fixtures. The status is now documented precisely on the constant (commit dca12e2c). Dispatch per variant once the Mac/Beta tables are extracted and shown to differ.

jqueuniet commented

2026-07-06 23:11:11 +02:00

(Migrated from codeberg.org)

Investigation result: byte-identity proven — no per-variant dispatch needed

Extracted the equivalent tables from both the Mac 68k binary and the unpacked Beta Fires.exe and compared against retail. Both match; this resolves as the "document byte-identity" outcome.

DOS retail (baseline)

Re-deriving from DOS/USA/skull.exe @ 0x72568 (36-byte stride, word[0] & 1, 64 entries, LSB-first) reproduces CREATURE_GIDX_ALT_TABLE = [ff 1f 10 00 e0 20 08 99] byte-for-byte.

Mac retail: identical (all 64 entries)

The engine path mirrors DOS 1:1: the type-4 branch of READ_RECORD_CHECKCODE (CODE_0016 0x1330) reads the 7-bit species, calls jump-table entry A5+0x842 → CODE_0015 0x49ce, which maps species → gidx via the same GDAT find(0x0f, …, 0x0b, 0x05) (plus 0x04 fallback) and indexes a descriptor table at A5-0x3982 with mulu.w #0x24 — same 36-byte stride — testing word[0] bit0.
The below-A5 globals image reconstructs from the resource fork: DATA_0000 is a word stream in which each 0x0000 word appends a zero run sized by the parallel ZERO_0000 resource (decompressor: CODE_0001 0x6e); DREL_0000 holds A5 relocations. Reconstruction consumes both resources exactly to the 0x681e-byte below-A5 size, and validates on three anchors (MAC_CHAMPION_MASK @ A5-0x9d6, SGB mask @ A5-0xa0e, BE-swapped ALT creature mask @ A5-0x89e).
All 64 big-endian word[0] values equal the retail LE values, and no DREL_0000 relocation lands inside the table → identical bit0 table.

DOS Beta v0.9: identical over its real table (56 entries)

scripts/unlzexe.py unpacks Fires.exe (278,128 B load module); the descriptor table is at unpacked offset 0x4332e, same 36-byte layout.
The Beta table has only 56 real entries — its 0xffff terminator sits at entry 56 (retail's at entry 63), and the bytes past it match retail's post-table data. bit0 is identical to retail over entries 0–55.
Reading 56–63 as table entries would diverge at 58–63, but those indices are past the Beta terminator — no valid Beta gidx reaches them.

Consistent with all this, the Mac and Beta fixtures already round-trip byte-for-byte with the retail table applied.

Resolution

Updated the records.rs per-variant status comment to record the proven identity and the extraction recipes. Side note: docs/formats/mac-post-sgb-reads.md quotes the DOS SGB mask bytes for the 0x4640 byte-search — the Mac bytes are u16-swapped and the mask sits at A5-0xa0e in the reconstructed image.

## Investigation result: byte-identity proven — no per-variant dispatch needed Extracted the equivalent tables from both the Mac 68k binary and the unpacked Beta `Fires.exe` and compared against retail. Both match; this resolves as the "document byte-identity" outcome. ### DOS retail (baseline) Re-deriving from `DOS/USA/skull.exe` @ `0x72568` (36-byte stride, `word[0] & 1`, 64 entries, LSB-first) reproduces `CREATURE_GIDX_ALT_TABLE = [ff 1f 10 00 e0 20 08 99]` byte-for-byte. ### Mac retail: identical (all 64 entries) - The engine path mirrors DOS 1:1: the type-4 branch of `READ_RECORD_CHECKCODE` (CODE_0016 `0x1330`) reads the 7-bit species, calls jump-table entry `A5+0x842` → CODE_0015 `0x49ce`, which maps species → gidx via the same GDAT `find(0x0f, …, 0x0b, 0x05)` (plus `0x04` fallback) and indexes a descriptor table at `A5-0x3982` with `mulu.w #0x24` — same 36-byte stride — testing `word[0]` bit0. - The below-A5 globals image reconstructs from the resource fork: `DATA_0000` is a word stream in which each `0x0000` word appends a zero run sized by the parallel `ZERO_0000` resource (decompressor: CODE_0001 `0x6e`); `DREL_0000` holds A5 relocations. Reconstruction consumes both resources exactly to the `0x681e`-byte below-A5 size, and validates on three anchors (`MAC_CHAMPION_MASK` @ `A5-0x9d6`, SGB mask @ `A5-0xa0e`, BE-swapped ALT creature mask @ `A5-0x89e`). - All 64 big-endian `word[0]` values equal the retail LE values, and no `DREL_0000` relocation lands inside the table → identical bit0 table. ### DOS Beta v0.9: identical over its real table (56 entries) - `scripts/unlzexe.py` unpacks `Fires.exe` (278,128 B load module); the descriptor table is at unpacked offset `0x4332e`, same 36-byte layout. - The Beta table has only **56 real entries** — its `0xffff` terminator sits at entry 56 (retail's at entry 63), and the bytes past it match retail's post-table data. `bit0` is identical to retail over entries 0–55. - Reading 56–63 as table entries would diverge at 58–63, but those indices are past the Beta terminator — no valid Beta gidx reaches them. Consistent with all this, the Mac and Beta fixtures already round-trip byte-for-byte with the retail table applied. ### Resolution Updated the `records.rs` per-variant status comment to record the proven identity and the extraction recipes. Side note: `docs/formats/mac-post-sgb-reads.md` quotes the *DOS* SGB mask bytes for the `0x4640` byte-search — the Mac bytes are u16-swapped and the mask sits at `A5-0xa0e` in the reconstructed image.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

jqueuniet/dm2-tools#174

No description provided.

Rows
Columns