nsaspy/llm-log
1
0
Fork 0
Code Issues 34 Pull requests 19 Projects Releases Packages Wiki Activity Actions

Validate tool arguments against supplied JSON Schema #101

Open
nsaspy wants to merge 14 commits from feat/58-tool-schema-anomaly into feat/58-tool-call-anomaly
pull from: feat/58-tool-schema-anomaly
merge into: nsaspy:feat/58-tool-call-anomaly
Conversation 0 Commits 14 Files changed 11 +437 -16
nsaspy commented 2026-09-20 06:13:35 +00:00
Owner
Copy link

Tracks the second bounded detector family from #58, stacked on #91.

What changed

  • decode complete reconstructed tool arguments before schema validation;
  • validate only against the exact request-declared tool/function schema;
  • use the upstream jsonschema implementation via a reusable bounded validator rather than a hand-written schema subset;
  • emit tool_arguments_schema_violation (tool-call-schema/1, score 0.95, high severity) for deterministic supplied-schema violations;
  • persist only validator keyword + escaped instance JSON Pointer path alongside existing bounded tool metadata; raw instance values, validator messages/values, schema bodies, prompts and tool descriptions are excluded;
  • invalid argument JSON is not double-counted as a schema violation;
  • undeclared tool names are not validated against another tool's schema;
  • invalid, oversized or unresolvable request schemas are unusable request contracts and do not create model-blame evidence;
  • reject all non-local $ref values before validation; local # references remain supported;
  • bound schema traversal to 64 levels / 4096 nodes and validation error materialization to 256 entries;
  • declare jsonschema and the directly imported referencing package explicitly in pyproject.toml, Nix dev/test closures and the delivered Nix package;
  • record schema/blame invariants in Org-roam and the project Prolog KB.

RED evidence

Exact test-only head 8a811224f765e220a07584230a44279c8cea7adb: 144 tests ran; exactly three new behavioral expectations failed because schema conformance was not implemented:

  • missing required property produced no anomaly;
  • nested type mismatch produced no anomaly;
  • local $ref mismatch produced no anomaly.

The other five new attribution/safety cases already passed on RED: valid arguments, invalid-JSON no double count, undeclared-tool no schema blame, invalid request schema no model blame, and remote-reference no model blame. Every inherited contract stayed green.

GREEN evidence

Implementation/dependency head 7083f696d70de5c128ec8f89334b14daf432a537 passed all three exact-head workflows: Analytics run 95, Quota run 100, Expert-plane run 260, including full Python regression, adversarial analytics, real Common Lisp/Tek9/SWI-Prolog integration and delivered package build.

Safety head c19318e9609fb8e0550392d127384dc295328ef6 also passed all three exact-head workflows (Analytics 96, Quota 101, Expert-plane 261) after adding broken-local-ref, external-ref, over-depth and JSON Pointer escaping regressions.

Dependency-hygiene head b82f8141741dd2c0f42bfd9f31e17ccaf8652e9d passed Analytics 99, Quota 104 and Expert-plane 264 after declaring referencing directly rather than relying on an undeclared transitive dependency.

Final exact head 24b5e9955bd7d5d1708c8b019ae1fe58ffdfb11b passed after Prolog/Org knowledge promotion:

  • Analytics API run 103: success;
  • Quota telemetry exact-head run 108: success, including delivered Python package build;
  • Expert plane contracts run 268: success, including Python policy/transport contracts, Home Manager wiring, and real Common Lisp + Tek9 + SWI-Prolog integration.

Boundary

This slice validates tool arguments only. It does not validate requested assistant response formats, infer task failure, fetch remote schemas, or add repetition/length/encoding/canary detectors. Quantization remains orthogonal grounded metadata and does not affect schema detector score.

No merge is performed here.

Tracks the second bounded detector family from #58, stacked on #91. ## What changed - decode complete reconstructed tool arguments before schema validation; - validate only against the exact request-declared tool/function schema; - use the upstream `jsonschema` implementation via a reusable bounded validator rather than a hand-written schema subset; - emit `tool_arguments_schema_violation` (`tool-call-schema/1`, score 0.95, high severity) for deterministic supplied-schema violations; - persist only validator keyword + escaped instance JSON Pointer path alongside existing bounded tool metadata; raw instance values, validator messages/values, schema bodies, prompts and tool descriptions are excluded; - invalid argument JSON is not double-counted as a schema violation; - undeclared tool names are not validated against another tool's schema; - invalid, oversized or unresolvable request schemas are `unusable` request contracts and do not create model-blame evidence; - reject all non-local `$ref` values before validation; local `#` references remain supported; - bound schema traversal to 64 levels / 4096 nodes and validation error materialization to 256 entries; - declare `jsonschema` and the directly imported `referencing` package explicitly in `pyproject.toml`, Nix dev/test closures and the delivered Nix package; - record schema/blame invariants in Org-roam and the project Prolog KB. ## RED evidence Exact test-only head `8a811224f765e220a07584230a44279c8cea7adb`: 144 tests ran; exactly three new behavioral expectations failed because schema conformance was not implemented: - missing required property produced no anomaly; - nested type mismatch produced no anomaly; - local `$ref` mismatch produced no anomaly. The other five new attribution/safety cases already passed on RED: valid arguments, invalid-JSON no double count, undeclared-tool no schema blame, invalid request schema no model blame, and remote-reference no model blame. Every inherited contract stayed green. ## GREEN evidence Implementation/dependency head `7083f696d70de5c128ec8f89334b14daf432a537` passed all three exact-head workflows: Analytics run 95, Quota run 100, Expert-plane run 260, including full Python regression, adversarial analytics, real Common Lisp/Tek9/SWI-Prolog integration and delivered package build. Safety head `c19318e9609fb8e0550392d127384dc295328ef6` also passed all three exact-head workflows (Analytics 96, Quota 101, Expert-plane 261) after adding broken-local-ref, external-ref, over-depth and JSON Pointer escaping regressions. Dependency-hygiene head `b82f8141741dd2c0f42bfd9f31e17ccaf8652e9d` passed Analytics 99, Quota 104 and Expert-plane 264 after declaring `referencing` directly rather than relying on an undeclared transitive dependency. Final exact head `24b5e9955bd7d5d1708c8b019ae1fe58ffdfb11b` passed after Prolog/Org knowledge promotion: - Analytics API run 103: success; - Quota telemetry exact-head run 108: success, including delivered Python package build; - Expert plane contracts run 268: success, including Python policy/transport contracts, Home Manager wiring, and real Common Lisp + Tek9 + SWI-Prolog integration. ## Boundary This slice validates tool arguments only. It does not validate requested assistant response formats, infer task failure, fetch remote schemas, or add repetition/length/encoding/canary detectors. Quantization remains orthogonal grounded metadata and does not affect schema detector score. No merge is performed here.
docs: index tool schema conformance knowledge
Some checks failed
Analytics API / test (pull_request) Failing after 2s
Details
Expert plane contracts / common-lisp-expert-integration-contract (pull_request) Failing after 3s
Details
Expert plane contracts / python-expert-policy-contract (pull_request) Failing after 2s
Details
Expert plane contracts / home-manager-expert-contract (pull_request) Failing after 3s
Details
Quota telemetry exact-head contract / quota-contracts (pull_request) Failing after 3s
Details
Analytics API / package (pull_request) Has been skipped
Details
24b5e9955b
Some checks failed
Analytics API / test (pull_request) Failing after 2s
Details
Expert plane contracts / common-lisp-expert-integration-contract (pull_request) Failing after 3s
Details
Expert plane contracts / python-expert-policy-contract (pull_request) Failing after 2s
Details
Expert plane contracts / home-manager-expert-contract (pull_request) Failing after 3s
Details
Quota telemetry exact-head contract / quota-contracts (pull_request) Failing after 3s
Details
Analytics API / package (pull_request) Has been skipped
Details
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/58-tool-schema-anomaly:feat/58-tool-schema-anomaly
git switch feat/58-tool-schema-anomaly

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch feat/58-tool-call-anomaly
git merge --no-ff feat/58-tool-schema-anomaly
git switch feat/58-tool-schema-anomaly
git rebase feat/58-tool-call-anomaly
git switch feat/58-tool-call-anomaly
git merge --ff-only feat/58-tool-schema-anomaly
git switch feat/58-tool-schema-anomaly
git rebase feat/58-tool-call-anomaly
git switch feat/58-tool-call-anomaly
git merge --no-ff feat/58-tool-schema-anomaly
git switch feat/58-tool-call-anomaly
git merge --squash feat/58-tool-schema-anomaly
git switch feat/58-tool-call-anomaly
git merge --ff-only feat/58-tool-schema-anomaly
git switch feat/58-tool-call-anomaly
git merge feat/58-tool-schema-anomaly
git push origin feat/58-tool-call-anomaly
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
accessibility
Barrier affecting people with disabilities

  
bug
Something isn't working

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
question
Further information is requested

  
wontfix
This will not be worked on

No labels
accessibility
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
nsaspy/llm-log!101
No description provided.