IPX replay: ipx-octets->ascii is not UTF-8 decoding (mojibake for non-ASCII frames) #184

New issue

Open

opened 2026-09-05 04:34:21 +00:00 by mara · 0 comments

mara commented

2026-09-05 04:34:21 +00:00

Collaborator

Reviewed at commit 3b4e2da (filed against the IPX replay introduced in 6f40a44, which advances #26).

Problem
ipx-octets->ascii (source/hackmode-core/capture-replay.lisp, lines 27-28) is (map 'string #'code-char octets). That maps raw octet values 128-255 directly to U+0080..U+00FF code points. It is not UTF-8 decoding.

Impact
Any frame whose JSON contains non-ASCII UTF-8 (IDN hosts, non-ASCII paths, or mitmproxy header values with UTF-8) is decoded as mojibake before jsown:parse. Either the parse fails and a perfectly valid frame is quarantined, or mangled host/path values are persisted into the canonical graph.

Repro
Write a spool frame with a UTF-8 path such as "/sök?x=ö" and replay: the persisted :path is the corrupted code-char sequence, not the original path.

Suggested fix
Decode frames with a real UTF-8 decoder, e.g. (flexi-streams:octets-to-string octets :external-format :utf-8) (verify the dep is in flake.nix), or open the spool with a UTF-8 flexi-stream. Keep the octet framing layer as-is since checkpoints are byte offsets. Add a test frame with non-ASCII path/host asserting round-trip equality.

Reviewed at commit 3b4e2da (filed against the IPX replay introduced in 6f40a44, which advances #26). **Problem** `ipx-octets->ascii` (`source/hackmode-core/capture-replay.lisp`, lines 27-28) is `(map 'string #'code-char octets)`. That maps raw octet values 128-255 directly to U+0080..U+00FF code points. It is not UTF-8 decoding. **Impact** Any frame whose JSON contains non-ASCII UTF-8 (IDN hosts, non-ASCII paths, or mitmproxy header values with UTF-8) is decoded as mojibake before `jsown:parse`. Either the parse fails and a perfectly valid frame is quarantined, or mangled host/path values are persisted into the canonical graph. **Repro** Write a spool frame with a UTF-8 path such as "/sök?x=ö" and replay: the persisted `:path` is the corrupted code-char sequence, not the original path. **Suggested fix** Decode frames with a real UTF-8 decoder, e.g. `(flexi-streams:octets-to-string octets :external-format :utf-8)` (verify the dep is in flake.nix), or open the spool with a UTF-8 flexi-stream. Keep the octet framing layer as-is since checkpoints are byte offsets. Add a test frame with non-ASCII path/host asserting round-trip equality.

mara added the

bug

label

2026-09-05 04:34:21 +00:00