jqueuniet/dm2-tools
1
0
Fork 0
Code Issues 2 Pull requests Projects Releases Packages Wiki Activity Actions

dm2-graphics: Ent1::find ignores cls5 language — 545 duplicate tuples on dos-en, last-wins returns arbitrary variant #189

New issue
Closed
opened 2026-07-05 07:48:07 +02:00 by jqueuniet · 1 comment
jqueuniet commented 2026-07-05 07:48:07 +02:00 (Migrated from codeberg.org)
Copy link

Problem

Ent1::find resolves records through a (cls1, cls2, cls3, cls4) → index HashMap with last-wins semantics, and its docstring claims duplicate tuples are "not observed in DOS/Mac fixtures" (crates/dm2-graphics/src/ent1.rs ~191).

They are observed, at scale: dos-en has 545 duplicate 4-tuples (dos-de/dos-fr similar), Amiga EU has 550. Breakdown on dos-en: 540 are dtText language variants (distinguished by cls5), 4 are images — the charsheet records (0x07, 0x00, 0x01, {0x00, 0x06, 0x07, 0x08}) exist in three language variants each (cls5 carries the language tag, e.g. 0x18/0x38/0x48), pointing at different payloads — plus 1 duplicated inline record (same value, harmless).

The engine resolves this by filtering at table-build time: a record is admitted into the runtime lookup table only when cls5 & 0xf0 is 0 or equals the configured language tag ([SKPROJECT] v5/bgdat.cpp:335-342, DM2_dballoc_3e74_2162, used as the admission predicate by DM2_BUILD_GDAT_ENTRY_DATA at bgdat.cpp:675), then binary-searches (cls2, cls4) within (cls1, cls3) buckets (v5/skgdtqdb.cpp:70-103). So the 4-tuple is unique per language view, never globally.

Consequences in the crate:

  • Archive::find / Archive::decode_image on a duplicated tuple silently return whichever record parsed last — e.g. decode_image(0x07, 0x00, cls4) on dos-en returns one arbitrary language's charsheet image.
  • The GraphicFileEntry::cls5 docstring calls the field "vestigial in the engine runtime" for images — wrong: it decides which records exist in the engine's lookup table at all.
  • Iterator-based paths (iter_texts, iter_image_with_palette) are unaffected (they yield every record), and the champion path is safe (its records are cls5 = 0 and unique).

Fix direction

Either add a language-aware lookup (find_in_language(cls1..cls4, lang) mirroring the engine's cls5 & 0xf0 ∈ {0, lang} rule, with find documented as "language-agnostic, ambiguous on 545 dos-en tuples"), or make by_tuple store all matches and let find take/require a language filter. Correct the two docstrings either way.

Probe: scratch/probe_gfx.py (gitignored).

## Problem `Ent1::find` resolves records through a `(cls1, cls2, cls3, cls4) → index` HashMap with last-wins semantics, and its docstring claims duplicate tuples are "not observed in DOS/Mac fixtures" (`crates/dm2-graphics/src/ent1.rs` ~191). They are observed, at scale: **dos-en has 545 duplicate 4-tuples** (dos-de/dos-fr similar), **Amiga EU has 550**. Breakdown on dos-en: 540 are dtText language variants (distinguished by `cls5`), **4 are images** — the charsheet records `(0x07, 0x00, 0x01, {0x00, 0x06, 0x07, 0x08})` exist in three language variants each (cls5 carries the language tag, e.g. 0x18/0x38/0x48), pointing at *different* payloads — plus 1 duplicated inline record (same value, harmless). The engine resolves this by **filtering at table-build time**: a record is admitted into the runtime lookup table only when `cls5 & 0xf0` is 0 or equals the configured language tag (`[SKPROJECT] v5/bgdat.cpp:335-342`, `DM2_dballoc_3e74_2162`, used as the admission predicate by `DM2_BUILD_GDAT_ENTRY_DATA` at `bgdat.cpp:675`), then binary-searches (cls2, cls4) within (cls1, cls3) buckets (`v5/skgdtqdb.cpp:70-103`). So the 4-tuple is unique *per language view*, never globally. Consequences in the crate: - `Archive::find` / `Archive::decode_image` on a duplicated tuple silently return whichever record parsed last — e.g. `decode_image(0x07, 0x00, cls4)` on dos-en returns one arbitrary language's charsheet image. - The `GraphicFileEntry::cls5` docstring calls the field "vestigial in the engine runtime" for images — wrong: it decides which records exist in the engine's lookup table at all. - Iterator-based paths (`iter_texts`, `iter_image_with_palette`) are unaffected (they yield every record), and the champion path is safe (its records are cls5 = 0 and unique). ## Fix direction Either add a language-aware lookup (`find_in_language(cls1..cls4, lang)` mirroring the engine's `cls5 & 0xf0 ∈ {0, lang}` rule, with `find` documented as "language-agnostic, ambiguous on 545 dos-en tuples"), or make `by_tuple` store all matches and let `find` take/require a language filter. Correct the two docstrings either way. Probe: `scratch/probe_gfx.py` (gitignored).
jqueuniet commented 2026-07-06 06:22:43 +02:00 (Migrated from codeberg.org)
Copy link

Fixed by #208 (merged into main).

Fixed by #208 (merged into `main`).
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
pages
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jqueuniet/dm2-tools#189
No description provided.