[SECURITY][BUG] Enforce one strict JSON parser at the HTTP boundary #133

New issue

Open

opened 2026-09-02 17:26:32 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-02 17:26:32 +00:00

Owner

Defect

Current master at 8fb297d146e7332fae7e38170b5b49d49530ac53 still has a strict-JSON boundary defect. The shared parse-json-octets helper now pre-validates with YASON and then reparses with JSOWN, but upstream YASON's object/array loops are themselves structurally permissive for trailing commas.

This is a data-integrity/security boundary bug, not a feature request.

Preserved RED

Evidence-only PR #126 preserves the smallest deterministic object-trailing-comma regression.

PR head: a2368ba58d0347af8f85efd85aba7b021ded65b5
test: TRAILING-COMMA-JSON-IS-A-400-CLIENT-ERROR
expected: http-input-error, HTTP 400, code malformed_json
observed on the defect baseline: object input ending in ,} is accepted

Preserve the RED. Do not xfail, skip, weaken, swallow, or bypass it.

Current-source finding

Current source/frontends/http-boundary-core.lisp already owns the correct architectural seam:

request octets -> content-type/byte limit -> parser -> route/schema -> application effects

The defect is the parser authority inside that seam, not a need for route-specific checks.

Current starintel-gserver.asd already depends on com.inuoe.jzon. Jzon provides RFC-8259-oriented parsing with explicit controls for comments, trailing commas, multiple top-level values, max depth and max string length, and accepts UTF-8 octets directly. Its streaming parser emits object-key events, allowing deterministic per-object duplicate-key rejection without another syntax parser.

Canonical ARDR authority

Research:
lost-rob0t/starintel-auto-research/roam/research/star-server/STAR-RESEARCH-060-strict-json-http-boundary.org

Design:
lost-rob0t/starintel-auto-research/roam/design/star-server/STAR-SERVER-060-strict-json-http-boundary.org

Research lifecycle is READY_FOR_DESIGN; design lifecycle is DESIGN_READY_FOR_OPERATOR_REVIEW.

The research conclusion, architecture/design approval, and implementation authorization remain separate human-gated states. Implementation authorization is PENDING / AWAITING_OPERATOR_APPROVAL.

Design decision

Use exactly one strict syntax authority on caller bytes:

preserve content-type validation and the existing body-byte limit;
feed bounded request octets to Jzon with comments, trailing commas, and multiple top-level content explicitly disabled;
use explicit depth and string-length bounds;
while consuming the Jzon event stream, maintain a per-object key set and reject duplicate member names at any nesting level;
convert the accepted parsed value/events in memory to the existing JSOWN-facing representation;
do not reparse the original untrusted text through YASON or JSOWN;
only then perform route shape/schema validation and application behavior.

No StarLang, Sento, storage, lease, or unrelated API redesign belongs in this fix.

Proposed implementation slices — ALL PENDING OPERATOR APPROVAL

Slice 1 — strict syntax adapter + RED corpus

Keep the public parse-json-octets seam stable. Preserve #126 and add pure boundary fixtures for:

object and array trailing commas;
comments;
duplicate keys including nested duplicates;
multiple top-level values/trailing tokens;
malformed strings/escapes/numbers;
invalid UTF-8;
max-depth and max-string overflow.

RED-first target remains the #126 object trailing comma and must fail for the expected permissive-baseline behavior before production mutation.

Slice 2 — compatibility conversion

Prove object, array, string, number, true, false, null, empty object and empty array preserve the semantics current JSOWN-consuming handlers require. Do not assume nil has identical meaning across libraries.

Slice 3 — route convergence and side-effect fencing

Audit all source/frontends/http-*.lisp request-body seams. Every caller-controlled JSON body must enter the common boundary exactly once. Malformed corpus cases must invoke zero downstream mutation/dispatch/publication/lease callbacks.

Slice 4 — full verification

Run focused boundary tests, full deterministic local server gate, Nix/closure verification, container stack, schema lock and security-negative/ZAP checks. Capture the exact resolved Jzon dependency identity because it is present through the ASDF dependency graph but not separately pinned in qlfile.lock.

Acceptance criteria

RFC-8259-invalid trailing commas/comments/multiple content rejected;
duplicate member names rejected deterministically;
invalid UTF-8 and malformed input never normalize into accepted application input;
body/depth/string limits bounded and deterministic;
valid canonical JSON remains wire-compatible;
parser failures retain client-safe 400 malformed_json + correlation behavior;
malformed requests reach zero CouchDB/outbox/Rabbit/actor/user/credential/target/lease authoritative effects;
untrusted original JSON is not reparsed through permissive JSOWN/YASON after strict acceptance;
no unrelated architecture broadening;
no implementation until the canonical Approval Table records exact operator authorization.

Contradiction to keep visible

PR #126 recorded [1,] as rejected in its resolved runtime, while the audited upstream YASON parser shape is permissive for both object and array trailing delimiters. Do not infer either behavior from source alone at realization time: the exact resolved local dependency closure must execute the hostile corpus.

Refs #126, #105.

## Defect Current `master` at `8fb297d146e7332fae7e38170b5b49d49530ac53` still has a strict-JSON boundary defect. The shared `parse-json-octets` helper now pre-validates with YASON and then reparses with JSOWN, but upstream YASON's object/array loops are themselves structurally permissive for trailing commas. This is a data-integrity/security boundary bug, not a feature request. ## Preserved RED Evidence-only PR #126 preserves the smallest deterministic object-trailing-comma regression. - PR head: `a2368ba58d0347af8f85efd85aba7b021ded65b5` - test: `TRAILING-COMMA-JSON-IS-A-400-CLIENT-ERROR` - expected: `http-input-error`, HTTP 400, code `malformed_json` - observed on the defect baseline: object input ending in `,}` is accepted Preserve the RED. Do not xfail, skip, weaken, swallow, or bypass it. ## Current-source finding Current `source/frontends/http-boundary-core.lisp` already owns the correct architectural seam: `request octets -> content-type/byte limit -> parser -> route/schema -> application effects` The defect is the parser authority inside that seam, not a need for route-specific checks. Current `starintel-gserver.asd` already depends on `com.inuoe.jzon`. Jzon provides RFC-8259-oriented parsing with explicit controls for comments, trailing commas, multiple top-level values, max depth and max string length, and accepts UTF-8 octets directly. Its streaming parser emits object-key events, allowing deterministic per-object duplicate-key rejection without another syntax parser. ## Canonical ARDR authority Research: `lost-rob0t/starintel-auto-research/roam/research/star-server/STAR-RESEARCH-060-strict-json-http-boundary.org` Design: `lost-rob0t/starintel-auto-research/roam/design/star-server/STAR-SERVER-060-strict-json-http-boundary.org` Research lifecycle is `READY_FOR_DESIGN`; design lifecycle is `DESIGN_READY_FOR_OPERATOR_REVIEW`. The research conclusion, architecture/design approval, and implementation authorization remain separate human-gated states. **Implementation authorization is PENDING / AWAITING_OPERATOR_APPROVAL.** ## Design decision Use exactly one strict syntax authority on caller bytes: 1. preserve content-type validation and the existing body-byte limit; 2. feed bounded request octets to Jzon with comments, trailing commas, and multiple top-level content explicitly disabled; 3. use explicit depth and string-length bounds; 4. while consuming the Jzon event stream, maintain a per-object key set and reject duplicate member names at any nesting level; 5. convert the accepted parsed value/events in memory to the existing JSOWN-facing representation; 6. do **not** reparse the original untrusted text through YASON or JSOWN; 7. only then perform route shape/schema validation and application behavior. No StarLang, Sento, storage, lease, or unrelated API redesign belongs in this fix. ## Proposed implementation slices — ALL PENDING OPERATOR APPROVAL ### Slice 1 — strict syntax adapter + RED corpus Keep the public `parse-json-octets` seam stable. Preserve #126 and add pure boundary fixtures for: - object and array trailing commas; - comments; - duplicate keys including nested duplicates; - multiple top-level values/trailing tokens; - malformed strings/escapes/numbers; - invalid UTF-8; - max-depth and max-string overflow. RED-first target remains the #126 object trailing comma and must fail for the expected permissive-baseline behavior before production mutation. ### Slice 2 — compatibility conversion Prove object, array, string, number, true, false, null, empty object and empty array preserve the semantics current JSOWN-consuming handlers require. Do not assume `nil` has identical meaning across libraries. ### Slice 3 — route convergence and side-effect fencing Audit all `source/frontends/http-*.lisp` request-body seams. Every caller-controlled JSON body must enter the common boundary exactly once. Malformed corpus cases must invoke zero downstream mutation/dispatch/publication/lease callbacks. ### Slice 4 — full verification Run focused boundary tests, full deterministic local server gate, Nix/closure verification, container stack, schema lock and security-negative/ZAP checks. Capture the exact resolved Jzon dependency identity because it is present through the ASDF dependency graph but not separately pinned in `qlfile.lock`. ## Acceptance criteria - RFC-8259-invalid trailing commas/comments/multiple content rejected; - duplicate member names rejected deterministically; - invalid UTF-8 and malformed input never normalize into accepted application input; - body/depth/string limits bounded and deterministic; - valid canonical JSON remains wire-compatible; - parser failures retain client-safe 400 `malformed_json` + correlation behavior; - malformed requests reach zero CouchDB/outbox/Rabbit/actor/user/credential/target/lease authoritative effects; - untrusted original JSON is not reparsed through permissive JSOWN/YASON after strict acceptance; - no unrelated architecture broadening; - no implementation until the canonical Approval Table records exact operator authorization. ## Contradiction to keep visible PR #126 recorded `[1,]` as rejected in its resolved runtime, while the audited upstream YASON parser shape is permissive for both object and array trailing delimiters. Do not infer either behavior from source alone at realization time: the exact resolved local dependency closure must execute the hostile corpus. Refs #126, #105.