Web-facing serialization scheme: graphics.dat write path #307

Open
opened 2026-07-31 22:31:12 +02:00 by jqueuniet · 2 comments
Owner

Question

The DM2 JSON formats designed so far — dm2-dungeon's json::skproject dump and the
planned graphics/asset exports — assume a filesystem layout: a structured JSON
document alongside sidecar binary files (PNG images, WAV/PCM sounds, raw blobs). A
browser has no filesystem to scatter sidecars across. What serialization scheme should
the web path use, and how does it reconcile with the desktop/filesystem scheme?

What we know

  • dm2-dungeon's existing json::skproject dump is one-way and lossy (it re-implements
    the retired C# convertDungeonDatToJson interop format; e.g. start_map is hardcoded
    to 0 with the comment "No header field carries the start map"). Not round-trippable.
  • The parser crates already round-trip at the binary level — byte-exact
    to_bytes / write_to, tested against fixtures.
  • The lossless-JSON work starting with saves uses a self-contained document: raw
    not-yet-decoded sections embedded as base64, so a single JSON round-trips byte-exact
    with no sidecar files. That is inherently web-friendly.
  • Graphics is the hard case: embedding many decoded images as base64 inside one JSON is
    heavy, whereas a filesystem dump would use PNG sidecars.

Options to weigh

  • Single self-contained JSON with embedded/base64 binaries (web-first; heavy for graphics).
  • Filesystem bundle: JSON + sidecar binaries (desktop-first; needs a virtual-FS/zip shim
    on the web).
  • An in-memory container (zip/tar) the web layer unpacks; JSON references entries by name.
  • Two projections over one shared owned document model (self-contained for web, sidecar
    for filesystem).

How we'd confirm

Pick a scheme that (a) round-trips byte-exact for dungeon/saves, (b) keeps graphics
payloads tractable in the browser, and (c) shares one owned document model with the
filesystem tooling. Prototype against a saves fixture and a graphics fixture.

Notes

Gates the web exposure in #301 and the graphics lossless-JSON work. The per-format
lossless serializers (saves, dungeon) can proceed in parallel since their self-contained
form is already web-compatible; this issue settles the cross-cutting policy, especially
for graphics.

Labels

type/research, area/tooling, priority/medium.

## Question The DM2 JSON formats designed so far — `dm2-dungeon`'s `json::skproject` dump and the planned graphics/asset exports — assume a **filesystem layout**: a structured JSON document alongside sidecar binary files (PNG images, WAV/PCM sounds, raw blobs). A browser has no filesystem to scatter sidecars across. What serialization scheme should the web path use, and how does it reconcile with the desktop/filesystem scheme? ## What we know - `dm2-dungeon`'s existing `json::skproject` dump is one-way and lossy (it re-implements the retired C# `convertDungeonDatToJson` interop format; e.g. `start_map` is hardcoded to 0 with the comment "No header field carries the start map"). Not round-trippable. - The parser crates already round-trip at the **binary** level — byte-exact `to_bytes` / `write_to`, tested against fixtures. - The lossless-JSON work starting with saves uses a **self-contained** document: raw not-yet-decoded sections embedded as base64, so a single JSON round-trips byte-exact with no sidecar files. That is inherently web-friendly. - Graphics is the hard case: embedding many decoded images as base64 inside one JSON is heavy, whereas a filesystem dump would use PNG sidecars. ## Options to weigh - Single self-contained JSON with embedded/base64 binaries (web-first; heavy for graphics). - Filesystem bundle: JSON + sidecar binaries (desktop-first; needs a virtual-FS/zip shim on the web). - An in-memory container (zip/tar) the web layer unpacks; JSON references entries by name. - Two projections over one shared owned document model (self-contained for web, sidecar for filesystem). ## How we'd confirm Pick a scheme that (a) round-trips byte-exact for dungeon/saves, (b) keeps graphics payloads tractable in the browser, and (c) shares one owned document model with the filesystem tooling. Prototype against a saves fixture and a graphics fixture. ## Notes Gates the web exposure in #301 and the graphics lossless-JSON work. The per-format lossless serializers (saves, dungeon) can proceed in parallel since their self-contained form is already web-compatible; this issue settles the cross-cutting policy, especially for graphics. ## Labels `type/research`, `area/tooling`, `priority/medium`.
jqueuniet added this to the dm2-tools project 2026-09-03 13:46:53 +02:00
Author
Owner

Dungeon slice tracked in #390: self-contained JSON suffices (no binary payloads); remaining work is the redundant seed / checksum and a decoded projection.

Dungeon slice tracked in #390: self-contained JSON suffices (no binary payloads); remaining work is the redundant seed / checksum and a decoded projection.
Author
Owner

Read path settled by #429, #430 and #431: the graphics.dat bytes are the container. graphicsEntriesJson indexes the records; per-record decoders give PNG, WAV, MIDI or JSON on demand, in formats a browser consumes directly (Blob URLs). No base64 overhead, no up-front decode of 5,676 images / 292 sounds, no second copy. #431 removes the per-call re-parse.

Saves and dungeons are self-contained JSON already (#390).

What remains is the graphics write path: rebuilding graphics.dat from edited assets. That needs encoders back to the record formats (images, dtSnd, HMP) and lossless forms where exports drop bytes (palettes as JSON). The fitting shape is per-record replacement over the archive bytes, mirroring the read API; a zip bundle only as a download format. Narrowing this issue to that.

https://claude.ai/code/session_01SnENBXnZbhNNrojZT5jbJ7

Read path settled by #429, #430 and #431: the `graphics.dat` bytes are the container. `graphicsEntriesJson` indexes the records; per-record decoders give PNG, WAV, MIDI or JSON on demand, in formats a browser consumes directly (Blob URLs). No base64 overhead, no up-front decode of 5,676 images / 292 sounds, no second copy. #431 removes the per-call re-parse. Saves and dungeons are self-contained JSON already (#390). What remains is the graphics **write** path: rebuilding `graphics.dat` from edited assets. That needs encoders back to the record formats (images, `dtSnd`, HMP) and lossless forms where exports drop bytes (palettes as JSON). The fitting shape is per-record replacement over the archive bytes, mirroring the read API; a zip bundle only as a download format. Narrowing this issue to that. https://claude.ai/code/session_01SnENBXnZbhNNrojZT5jbJ7
jqueuniet changed title from Web-facing serialization scheme: self-contained JSON vs filesystem + sidecars to Web-facing serialization scheme: graphics.dat write path 2026-10-30 18:20:58 +01:00
Sign in to join this conversation.
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
jqueuniet/dm2-tools#307
No description provided.