nsaspy/prolog-rlm

Fork 0

fix: isolate recoverable native preflight faults per call #375

Closed

nsaspy wants to merge 1 commit from fix/313-per-call-preflight into main

nsaspy commented

2026-09-04 22:14:05 +00:00

Owner

Closes #313

Runtime invariant

One malformed call in a native tool-call batch no longer aborts the whole
direct run, while batch-fatal protocol/policy violations still reject the
entire requested batch before any operation executes.

after_call_normalization/5 now runs one side-effect-free classification
pass (classify_calls/4) that produces either a batch-fatal cause or the
per-call statuses every later phase consumes. Preflight is never re-run, so
classification cannot double-charge or double-execute anything.

Recoverability whitelist (fatal by default)

recoverable_fault_kind/1 is the single centralized positive policy:

malformed_arguments
unavailable_tool_schema

Only these two per-call preflight fault kinds become structured,
model-repairable fault observations. Every other direct_fault escapes
classification and is batch-fatal, so a fault kind introduced later cannot
silently become model-repairable just because it happened inside the
per-call catch (the previous WIP caught every direct_fault per call).

Batch-fatal invariants are evaluated against the ORIGINAL request

Duplicate call IDs: within-batch (normalization) and cross-turn
(fresh_call_ids against seen_call_ids, which includes faulted call
IDs) stay terminal.
Effect isolation: an effectful call in a multi-call model request
fails the whole batch (effectful_batch_unsupported) before any
operation executes — including when sibling calls failed preflight.
Recovery can never launder an unsafe requested effect batch into an
executable effectful singleton. Tests cover
effectful+malformed / malformed+effectful / effectful+unavailable /
unavailable+effectful / effectful+valid-read in both orderings with an
externally observable mutation counter pinned at zero.
Budgets: context/tool budget admission counts the calls that will
execute (resolved classifications). Faulted calls never execute, are
never charged, and stay bounded by the model-call, iteration, token and
output budgets. Exhausted budgets are batch-fatal.
Assistant-message integrity (missing_assistant_message,
assistant_tool_calls_mismatch) is still validated before any
execution.

All-recoverably-invalid batches continue the bounded loop

When every call in a batch is individually recoverable (e.g. malformed
context call + unavailable tool), the runtime appends a deterministic
fault observation for each call, makes no tool/context charge, and
continues the bounded direct loop so the model can repair on the next
model call. The repair loop stays inside max_model_calls
(model_call_budget_exhausted after the configured budget) — no
unbounded repair loop. The old Valid == [] => terminal behavior was an
unforced WIP decision, not a repository invariant; the loop bounds
(model_admission) already make continuation safe.

Fault observations

Each rejected call produces a deterministic tool result carrying the
original call ID, tool name, stable fault kind, bounded repair-relevant
detail, and error status. The WIP's trace:Cause embedded raw internal
direct_error/tool_error structures (including
schema_validation_failed internals) into the model-visible tool
message; it is replaced by a bounded value payload plus a bounded
preflight_trace{phase,kind,tool}. Observations are appended in the
original requested call order, interleaved with valid results, so the
provider protocol sees one tool result per call ID in request order.

Cancellation

check_cancelled/1 also fires at the fault-observation boundary.
rlm_cancelled keeps its established terminal semantics and is never
converted into a repairable native_call_rejected observation.

Accounting

Verified and pinned by tests: model calls and iterations advance once per
provider turn; valid siblings are charged exactly once; faulted calls are
never charged as successful native executions; batch-fatal rejection
leaves no partial effects and no second model call.

#314 dependency

#314 is merged into main (b8206c7); this branch is rebased onto it and
contains only #313 work. Integration regressions cover the malformed
selector {type:"metadata",index:-1} classified as recoverable
malformed_arguments (with the bounded invalid_selector_field detail)
beside an executable read-only sibling, plus the valid Auto-Dig shape
{type:"head",index:0,count:20} executing in order.

Non-goals

SPEC/PLAN design, plan graph execution, symbolic mode, provider redesign,
OpenRouter behavior, scheduler changes, a generic exception framework,
#314 selector redesign, and durable distributed execution are untouched.

Test evidence

test/rlm_direct_partial_batch_test.pl: 29 tests (new suite). 8 were
RED against the pre-fix implementation (recorded): both
effectful-laundering orderings, both all-faults cases, the
internal-trace leak, and the three positive-whitelist policy tests.
test/rlm_direct_test.pl: the two unavailable-tool fail-closed tests
updated to the new per-call semantics (handler still never runs —
invocation count pinned at zero — and the observation still carries
unavailable_tool_schema).
Local deterministic gates, all green:
- swipl -q -s test/check_runtime.pl
- swipl -q -s test/load_all.pl
- swipl -q -s test/run_tests.pl — 1121/1121 pass, 0 failed/timeout/blocked
- swipl -q -s scripts/validate_research_approval.pl
- swipl -q -s scripts/design_gate.pl
- live test definitions static load (credential-free)
- runner main-ownership probe + runner integrity tests
- benchmark/run.pl -- deterministic — 16/16
- benchmark/run.pl -- deep-experiment — 15/15
- bin/prolog-rlm.pl -- demo --json + graph trace smoke + trace-view
- fresh-process graph restart and artifact restart fixtures
- git diff --check
Paid OpenRouter lanes are not exercised by this slice; deterministic
gates are the correctness proof here.

Closes #313 ## Runtime invariant One malformed call in a native tool-call batch no longer aborts the whole direct run, while batch-fatal protocol/policy violations still reject the entire **requested** batch before any operation executes. `after_call_normalization/5` now runs **one side-effect-free classification pass** (`classify_calls/4`) that produces either a batch-fatal cause or the per-call statuses every later phase consumes. Preflight is never re-run, so classification cannot double-charge or double-execute anything. ## Recoverability whitelist (fatal by default) `recoverable_fault_kind/1` is the single centralized positive policy: - `malformed_arguments` - `unavailable_tool_schema` Only these two per-call preflight fault kinds become structured, model-repairable fault observations. Every other `direct_fault` escapes classification and is batch-fatal, so a fault kind introduced later cannot silently become model-repairable just because it happened inside the per-call catch (the previous WIP caught every `direct_fault` per call). ## Batch-fatal invariants are evaluated against the ORIGINAL request - **Duplicate call IDs**: within-batch (normalization) and cross-turn (`fresh_call_ids` against `seen_call_ids`, which includes faulted call IDs) stay terminal. - **Effect isolation**: an effectful call in a multi-call model request fails the whole batch (`effectful_batch_unsupported`) before any operation executes — including when sibling calls failed preflight. Recovery can never launder an unsafe requested effect batch into an executable effectful singleton. Tests cover effectful+malformed / malformed+effectful / effectful+unavailable / unavailable+effectful / effectful+valid-read in both orderings with an externally observable mutation counter pinned at zero. - **Budgets**: context/tool budget admission counts the calls that will execute (resolved classifications). Faulted calls never execute, are never charged, and stay bounded by the model-call, iteration, token and output budgets. Exhausted budgets are batch-fatal. - **Assistant-message integrity** (`missing_assistant_message`, `assistant_tool_calls_mismatch`) is still validated before any execution. ## All-recoverably-invalid batches continue the bounded loop When every call in a batch is individually recoverable (e.g. malformed context call + unavailable tool), the runtime appends a deterministic fault observation for each call, makes no tool/context charge, and continues the bounded direct loop so the model can repair on the next model call. The repair loop stays inside `max_model_calls` (`model_call_budget_exhausted` after the configured budget) — no unbounded repair loop. The old `Valid == [] => terminal` behavior was an unforced WIP decision, not a repository invariant; the loop bounds (`model_admission`) already make continuation safe. ## Fault observations Each rejected call produces a deterministic tool result carrying the original call ID, tool name, stable fault kind, bounded repair-relevant detail, and error status. The WIP's `trace:Cause` embedded raw internal `direct_error`/`tool_error` structures (including `schema_validation_failed` internals) into the model-visible tool message; it is replaced by a bounded `value` payload plus a bounded `preflight_trace{phase,kind,tool}`. Observations are appended in the original requested call order, interleaved with valid results, so the provider protocol sees one tool result per call ID in request order. ## Cancellation `check_cancelled/1` also fires at the fault-observation boundary. `rlm_cancelled` keeps its established terminal semantics and is never converted into a repairable `native_call_rejected` observation. ## Accounting Verified and pinned by tests: model calls and iterations advance once per provider turn; valid siblings are charged exactly once; faulted calls are never charged as successful native executions; batch-fatal rejection leaves no partial effects and no second model call. ## #314 dependency #314 is merged into `main` (b8206c7); this branch is rebased onto it and contains only #313 work. Integration regressions cover the malformed selector `{type:"metadata",index:-1}` classified as recoverable `malformed_arguments` (with the bounded `invalid_selector_field` detail) beside an executable read-only sibling, plus the valid Auto-Dig shape `{type:"head",index:0,count:20}` executing in order. ## Non-goals SPEC/PLAN design, plan graph execution, symbolic mode, provider redesign, OpenRouter behavior, scheduler changes, a generic exception framework, #314 selector redesign, and durable distributed execution are untouched. ## Test evidence - `test/rlm_direct_partial_batch_test.pl`: 29 tests (new suite). 8 were RED against the pre-fix implementation (recorded): both effectful-laundering orderings, both all-faults cases, the internal-trace leak, and the three positive-whitelist policy tests. - `test/rlm_direct_test.pl`: the two unavailable-tool fail-closed tests updated to the new per-call semantics (handler still never runs — invocation count pinned at zero — and the observation still carries `unavailable_tool_schema`). - Local deterministic gates, all green: - `swipl -q -s test/check_runtime.pl` - `swipl -q -s test/load_all.pl` - `swipl -q -s test/run_tests.pl` — 1121/1121 pass, 0 failed/timeout/blocked - `swipl -q -s scripts/validate_research_approval.pl` - `swipl -q -s scripts/design_gate.pl` - live test definitions static load (credential-free) - runner main-ownership probe + runner integrity tests - `benchmark/run.pl -- deterministic` — 16/16 - `benchmark/run.pl -- deep-experiment` — 15/15 - `bin/prolog-rlm.pl -- demo --json` + graph trace smoke + trace-view - fresh-process graph restart and artifact restart fixtures - `git diff --check` - Paid OpenRouter lanes are not exercised by this slice; deterministic gates are the correctness proof here.

nsaspy added 1 commit

2026-09-04 22:14:05 +00:00

fix: isolate recoverable native preflight faults per call (#313 )

Tree-sitter FFI / Direct SWI-Prolog Tree-sitter FFI (pull_request) Successful in 2m30s

Details

CI / REAL OpenRouter integration (pull_request) Has been skipped

Details

Nix flake / Flake package and clean runtime load (pull_request) Failing after 4s

Details

Clean SWI pack install / Install and load copied pack (pull_request) Successful in 1m42s

Details

CI / Deterministic unit and load checks (pull_request) Successful in 4m5s

Details

Paid OpenRouter / Pinned paid OpenRouter integration (pull_request) Failing after 2m24s

Details

f8fb2c6f2c

One malformed call in a native tool-call batch used to abort the whole
direct run: preflight wrapped the entire batch in a single catch, so one
malformed context_arguments or unknown tool name discarded all valid
sibling calls and terminated the trajectory.

after_call_normalization/5 now runs ONE side-effect-free classification
pass (classify_calls/4) that produces either a batch-fatal cause or the
per-call statuses every later phase consumes. Preflight is never re-run
and nothing executes before the batch-fatal invariants have passed.

Recoverability is a positive whitelist (recoverable_fault_kind/1): only
malformed_arguments and unavailable_tool_schema become structured,
model-repairable fault observations. Every other direct fault escapes
classification and is batch-fatal, so a future fault kind cannot
silently become model-repairable.

Batch-fatal invariants are evaluated against the ORIGINAL requested
batch, not the filtered survivor list:

- duplicate call IDs (within batch via normalization, across turns via
  fresh_call_ids) stay terminal
- effect isolation: an effectful call in a multi-call model request
  fails the whole batch before any operation executes, even when
  sibling calls failed preflight; recovery can never launder an unsafe
  requested effect batch into an executable singleton
- context/tool budget admission counts the calls that will execute;
  faulted calls never execute, are never charged, and stay bounded by
  the model-call, iteration, token, and output budgets
- assistant-message integrity (missing/mismatched tool_calls) is still
  validated before any execution

All-recoverably-invalid batches now continue the bounded direct loop:
each rejected call gets a deterministic fault observation (original
call ID, tool name, stable fault kind, bounded repair-relevant detail,
error status) and no tool/context charge, so the model can repair on
the next bounded model call. Internal exception structures no longer
reach the model message (the raw direct_error trace:Cause leak is
replaced by a bounded preflight_trace and value payload). Observations
are appended in the original requested call order.

Cancellation keeps its established semantics: check_cancelled/1 fires
at the fault-observation boundary too, and rlm_cancelled never becomes
a repairable native_call_rejected observation.

The unavailable-tool policy change updates two pre-existing fail-closed
tests: the handler still never runs (count pinned at zero) and the
fault observation still carries unavailable_tool_schema, but the
bounded loop now continues instead of rejecting the whole batch.

#314 integration: a malformed context_peek selector
({type:metadata,index:-1}) is classified as recoverable
malformed_arguments beside an executable read-only sibling, and the
valid Auto-Dig head shape {type:head,index:0,count:20} keeps executing.

nsaspy commented

2026-09-04 22:26:09 +00:00

Author

Owner

[CI] Status after fix commit dff6bb9

Lane	Result
Deterministic unit and load checks	SUCCESS
Direct SWI-Prolog Tree-sitter FFI	SUCCESS
Install and load copied pack	SUCCESS
Flake package and clean runtime load	SUCCESS
REAL OpenRouter integration	skipped
Pinned paid OpenRouter integration	FAILURE

The failing lane is the pre-existing recurring paid OpenRouter live-lane
failure (same signature as #291/#287/#293/#302/#307, tracked in
#315). It also fails on current main and is not caused by
this change.

Local verification for this branch:

New regression tests (rlm_direct_partial_batch): 3/3 GREEN
Full deterministic gate swipl -q -s test/run_tests.pl:
1069/1069 passed, exit 0

Mergeable: true (unstable only due to the known live-lane failure).

## [CI] Status after fix commit dff6bb9 | Lane | Result | | --- | --- | | Deterministic unit and load checks | SUCCESS | | Direct SWI-Prolog Tree-sitter FFI | SUCCESS | | Install and load copied pack | SUCCESS | | Flake package and clean runtime load | SUCCESS | | REAL OpenRouter integration | skipped | | Pinned paid OpenRouter integration | FAILURE | The failing lane is the **pre-existing recurring paid OpenRouter live-lane failure** (same signature as #291/#287/#293/#302/#307, tracked in #315). It also fails on current `main` and is not caused by this change. Local verification for this branch: - New regression tests (`rlm_direct_partial_batch`): 3/3 GREEN - Full deterministic gate `swipl -q -s test/run_tests.pl`: **1069/1069 passed, exit 0** Mergeable: `true` (`unstable` only due to the known live-lane failure).

nsaspy closed this pull request

2026-09-04 22:56:01 +00:00