graphics.dat: Japanese text decodes garbled #438

Closed
opened 2026-10-30 22:19:54 +01:00 by jqueuniet · 2 comments
Owner

Japanese dtText records decode to mojibake: half-width katakana, U+FFFD and stray rune glyphs. Mac JP weapon names (0x10, 0x00, 0x05, 0x18) and (0x10, 0x01, 0x05, 0x18) read チ and �?、ァ肩, both in dm2 extract --text --text-language japanese and in graphicsLabelsJson / graphicsTextJson (which add rune code points, \uE039チ\uE013).

Text::to_display_string decodes Japanese literal runs through encoding_rs::SHIFT_JIS (docs/formats/graphics-dat.md § "Japanese rendering"); the output suggests the bytes are not plain CP932: a game-specific glyph table, kana in a custom range, or the rune bytes overlapping Shift-JIS lead bytes.

Acceptance

  • Mac JP item and champion names decode to readable Japanese, checked against the in-game inventory.
  • The same on the other Japanese ports (FM-Towns, PC-9801/9821, IBM PS/V, Sega CD JP).

https://claude.ai/code/session_01SnENBXnZbhNNrojZT5jbJ7

Japanese `dtText` records decode to mojibake: half-width katakana, U+FFFD and stray rune glyphs. Mac JP weapon names `(0x10, 0x00, 0x05, 0x18)` and `(0x10, 0x01, 0x05, 0x18)` read `チ` and `�?、ァ肩`, both in `dm2 extract --text --text-language japanese` and in `graphicsLabelsJson` / `graphicsTextJson` (which add rune code points, `\uE039チ\uE013`). `Text::to_display_string` decodes Japanese literal runs through `encoding_rs::SHIFT_JIS` (`docs/formats/graphics-dat.md` § "Japanese rendering"); the output suggests the bytes are not plain CP932: a game-specific glyph table, kana in a custom range, or the rune bytes overlapping Shift-JIS lead bytes. ## Acceptance - Mac JP item and champion names decode to readable Japanese, checked against the in-game inventory. - The same on the other Japanese ports (FM-Towns, PC-9801/9821, IBM PS/V, Sega CD JP). https://claude.ai/code/session_01SnENBXnZbhNNrojZT5jbJ7
Author
Owner

Not Shift-JIS: the bytes index the game's own glyph sheets (SKWINSPX DRAW_MBCS_STR, QUERY_CHAR_METRICS, font loader skgdat.cpp:550).

  • Byte 0x20..=0xEF: one glyph. Byte >= 0xF0: lead of a two-byte code. Below 0x20: control; 02 xx switches font.
  • Glyphs are the images (0x1c, font, 0x01, n), grids of cells (cell size from the pict offset). Codes run across the sheets from 0x20, moving to lead 0xF0 once 0xEF is passed; a dtWordValue (0x1c, font, 0x0b, n) can restart a sheet at lead << 8 | first.
  • The sheets hold a hand-picked subset: ASCII, symbols, used katakana and hiragana, a few hundred kanji. Nothing maps them to Unicode.
  • Check: Mac JP "Eye of Time" f0 79 c1 f0 53 = kanji + の + kanji (時の目).

Glyph sets (sheets compared as 1-bit bitmaps):

Set Ports
A Mac JP, PC-9821, IBM PS/V, FM-Towns
B PC-9801 (sheets 04, 08–0c differ)
C Sega CD JP (most sheets differ)

About 380 glyphs per set, ~250 of them kana and kanji.

Plan:

  1. Decode Japanese as glyph codes instead of Shift-JIS; render text through the 0x1C sheets as PNG (exact, no tables).
  2. Unicode tables per glyph set, identified from the sheets, set A first; checked in game.

https://claude.ai/code/session_01SnENBXnZbhNNrojZT5jbJ7

Not Shift-JIS: the bytes index the game's own glyph sheets (SKWINSPX `DRAW_MBCS_STR`, `QUERY_CHAR_METRICS`, font loader `skgdat.cpp:550`). - Byte `0x20..=0xEF`: one glyph. Byte `>= 0xF0`: lead of a two-byte code. Below `0x20`: control; `02 xx` switches font. - Glyphs are the images `(0x1c, font, 0x01, n)`, grids of cells (cell size from the pict offset). Codes run across the sheets from `0x20`, moving to lead `0xF0` once `0xEF` is passed; a `dtWordValue` `(0x1c, font, 0x0b, n)` can restart a sheet at `lead << 8 | first`. - The sheets hold a hand-picked subset: ASCII, symbols, used katakana and hiragana, a few hundred kanji. Nothing maps them to Unicode. - Check: Mac JP "Eye of Time" `f0 79 c1 f0 53` = kanji + の + kanji (時の目). Glyph sets (sheets compared as 1-bit bitmaps): | Set | Ports | | --- | --- | | A | Mac JP, PC-9821, IBM PS/V, FM-Towns | | B | PC-9801 (sheets 04, 08–0c differ) | | C | Sega CD JP (most sheets differ) | About 380 glyphs per set, ~250 of them kana and kanji. Plan: 1. Decode Japanese as glyph codes instead of Shift-JIS; render text through the 0x1C sheets as PNG (exact, no tables). 2. Unicode tables per glyph set, identified from the sheets, set A first; checked in game. https://claude.ai/code/session_01SnENBXnZbhNNrojZT5jbJ7
Author
Owner

Fixed in v0.8.0 (2026-11-01): Japanese text decodes as glyph codes through per-port GlyphSet tables, not Shift-JIS. Mac JP item (0x10, 0x00) reads 時の眼; japanese_text_codes_have_glyphs covers all six Japanese ports with no U+FFFD. Both acceptance criteria met. Closing.

Fixed in v0.8.0 (2026-11-01): Japanese text decodes as glyph codes through per-port `GlyphSet` tables, not Shift-JIS. Mac JP item `(0x10, 0x00)` reads 時の眼; `japanese_text_codes_have_glyphs` covers all six Japanese ports with no U+FFFD. Both acceptance criteria met. Closing.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jqueuniet/dm2-tools#438
No description provided.