nsaspy/prolog-rlm

Fork 0

Planner retries parse failures but not structurally invalid typed plans #245

New issue

Closed

opened 2026-08-26 13:58:38 +00:00 by lost-rob0t · 6 comments

lost-rob0t commented

2026-08-26 13:58:38 +00:00

(Migrated from github.com)

Live regression evidence

Discovered while running the approved #183 live RLM operator behavior acceptance on PR #242.

At exact candidate cefa11974cb94c65eb7e5eef5c6a887653e54350, pinned paid OpenRouter (openai/gpt-oss-120b) produced a parseable typed plan on the first planner model call, but plan_run/5 rejected it as:

plan_error{phase:validate,kind:invalid_plan,detail:final_must_be_unique_and_last,...}

Despite planner_attempts(2), usage shows model_calls:1; no second planner attempt occurs.

Current architecture explains why: planner_loop retries only plan_parse/2 failures. Once parsing succeeds, completion_after_planner/12 proceeds through recursive validation/budget shaping and then plan_run/5; structural plan validation therefore happens after the retry loop.

Required contract

TDD first from canonical main 267697bef10a3fffff7c093e1435ece770e7444b:

A planner handler returns a parseable but structurally invalid typed plan on attempt 1 (for example duplicate/non-final final), then a valid plan on attempt 2.
With planner_attempts(2), completion must recover through the existing bounded planner retry budget and execute only the valid second plan.
Two structurally invalid attempts must return an explicit structured planner/validation failure while the test suite stays green by asserting that expected failure.
Invalid plans must never execute any step/effect/tool/model operation before retry/rejection.
Usage/model-call accounting must include every attempted planner call and remain bounded by the global completion budget.
Cancellation, capability, authority, effect, and recursive validation invariants remain unchanged.

Design boundary

This is planner repair/retry orchestration, not validation weakening. Do not normalize away structural errors, auto-delete duplicate finals, execute a partially valid plan, or add provider-specific special cases. Reuse the existing planner attempt loop and canonical rlm_plan validation semantics.

Cross-link: blocks behavioral proof in #183 / PR #242, where live-provider evidence exposed the gap.

## Live regression evidence Discovered while running the approved #183 live RLM operator behavior acceptance on PR #242. At exact candidate `cefa11974cb94c65eb7e5eef5c6a887653e54350`, pinned paid OpenRouter (`openai/gpt-oss-120b`) produced a parseable typed plan on the first planner model call, but `plan_run/5` rejected it as: `plan_error{phase:validate,kind:invalid_plan,detail:final_must_be_unique_and_last,...}` Despite `planner_attempts(2)`, usage shows `model_calls:1`; no second planner attempt occurs. Current architecture explains why: `planner_loop` retries only `plan_parse/2` failures. Once parsing succeeds, `completion_after_planner/12` proceeds through recursive validation/budget shaping and then `plan_run/5`; structural plan validation therefore happens after the retry loop. ## Required contract TDD first from canonical `main` `267697bef10a3fffff7c093e1435ece770e7444b`: 1. A planner handler returns a parseable but structurally invalid typed plan on attempt 1 (for example duplicate/non-final `final`), then a valid plan on attempt 2. 2. With `planner_attempts(2)`, completion must recover through the existing bounded planner retry budget and execute only the valid second plan. 3. Two structurally invalid attempts must return an explicit structured planner/validation failure while the test suite stays green by asserting that expected failure. 4. Invalid plans must never execute any step/effect/tool/model operation before retry/rejection. 5. Usage/model-call accounting must include every attempted planner call and remain bounded by the global completion budget. 6. Cancellation, capability, authority, effect, and recursive validation invariants remain unchanged. ## Design boundary This is planner repair/retry orchestration, not validation weakening. Do not normalize away structural errors, auto-delete duplicate finals, execute a partially valid plan, or add provider-specific special cases. Reuse the existing planner attempt loop and canonical `rlm_plan` validation semantics. Cross-link: blocks behavioral proof in #183 / PR #242, where live-provider evidence exposed the gap.

lost-rob0t commented

2026-08-26 13:59:44 +00:00

(Migrated from github.com)

RAGE analyze / design / adversarial gate

Analyze

Canonical main remains 267697bef10a3fffff7c093e1435ece770e7444b. Source confirms the live failure mechanism: planner_loop retries only plan_parse/2 errors. A parseable candidate exits as ok(planner_result{...}); completion_after_planner then performs recursive checks and ultimately calls canonical plan_run/5, where structural validation such as final_must_be_unique_and_last can fail. That later failure has no path back into the bounded planner attempt loop.

Design

Keep canonical plan validation unchanged. Introduce a pre-execution candidate-validation step inside the planner-attempt lifecycle that uses the same structural validation contract as rlm_plan, without executing the candidate. A structurally invalid candidate consumes its planner usage, is rejected before any plan operation runs, and may trigger the next configured planner attempt. After attempts are exhausted, return an explicit structured planner validation error carrying aggregate attempt usage and the last validation cause.

The smallest implementation should reuse existing validation/public API if available rather than duplicate the validator. If no non-executing validation entrypoint exists, add the smallest reusable one in rlm_plan and keep plan_run/5 validating again at execution as defense in depth.

TDD convention

First add a deterministic classifier that proves current main sees a parseable structurally-invalid first candidate, returns the structural error, and calls the injected planner only once even with planner_attempts(2). This expected-failure regression remains green. Then invert it with the implementation to assert recovery on attempt 2 plus exhaustion semantics on two invalid candidates.

Adversarial review

Never execute any invalid candidate before retry/rejection.
Never repair by deleting/reordering model steps or finals.
Do not catch arbitrary execution errors and call them planner errors; only pre-execution structural validation belongs in this retry path.
Preserve aggregate token/cost/model-call accounting across attempts.
Respect global model-call/token/cost budgets before every retry.
Cancellation still interrupts immediately.
Recursive capability/depth checks remain separate and authoritative; this slice is structural plan validity only unless executable evidence requires broader unification.
No provider-specific special case.

Decision: GO for a focused #245 branch from exact current main. #242 remains HOLD until #245 lands and its live behavior candidate is rebased/reverified.

## RAGE analyze / design / adversarial gate ### Analyze Canonical `main` remains `267697bef10a3fffff7c093e1435ece770e7444b`. Source confirms the live failure mechanism: `planner_loop` retries only `plan_parse/2` errors. A parseable candidate exits as `ok(planner_result{...})`; `completion_after_planner` then performs recursive checks and ultimately calls canonical `plan_run/5`, where structural validation such as `final_must_be_unique_and_last` can fail. That later failure has no path back into the bounded planner attempt loop. ### Design Keep canonical plan validation unchanged. Introduce a pre-execution candidate-validation step inside the planner-attempt lifecycle that uses the same structural validation contract as `rlm_plan`, without executing the candidate. A structurally invalid candidate consumes its planner usage, is rejected before any plan operation runs, and may trigger the next configured planner attempt. After attempts are exhausted, return an explicit structured planner validation error carrying aggregate attempt usage and the last validation cause. The smallest implementation should reuse existing validation/public API if available rather than duplicate the validator. If no non-executing validation entrypoint exists, add the smallest reusable one in `rlm_plan` and keep `plan_run/5` validating again at execution as defense in depth. ### TDD convention First add a deterministic classifier that proves current `main` sees a parseable structurally-invalid first candidate, returns the structural error, and calls the injected planner only once even with `planner_attempts(2)`. This expected-failure regression remains green. Then invert it with the implementation to assert recovery on attempt 2 plus exhaustion semantics on two invalid candidates. ### Adversarial review - Never execute any invalid candidate before retry/rejection. - Never repair by deleting/reordering model steps or finals. - Do not catch arbitrary execution errors and call them planner errors; only pre-execution structural validation belongs in this retry path. - Preserve aggregate token/cost/model-call accounting across attempts. - Respect global model-call/token/cost budgets before every retry. - Cancellation still interrupts immediately. - Recursive capability/depth checks remain separate and authoritative; this slice is structural plan validity only unless executable evidence requires broader unification. - No provider-specific special case. Decision: **GO** for a focused #245 branch from exact current `main`. #242 remains HOLD until #245 lands and its live behavior candidate is rebased/reverified.

lost-rob0t commented

2026-08-26 14:11:05 +00:00

(Migrated from github.com)

TDD baseline verified — exact head `37637cab5b2be4f04cf4458744e9037fe915d9a7`

The deterministic regression classifier is now correctly registered in the canonical corpus and passes without intentionally redlining CI.

It proves the current defect from exact canonical base 267697bef10a3fffff7c093e1435ece770e7444b:

planner_attempts(2) is configured;
attempt 1 returns a parseable typed plan with a model step followed by duplicate final steps;
canonical completion returns phase:validate / kind:invalid_plan / detail:final_must_be_unique_and_last;
the injected planner is called exactly once;
the invalid candidate executes zero injected model operations.

The earlier classifier run exposed a test bug (assertion/1 binding was not preserved); that was fixed rather than treated as runtime evidence. Current PlUnit has passed this classifier on 37637cab….

This satisfies the BUG/TDD gate. Realization remains the approved narrow design: bring existing non-executing rlm_plan:plan_validate/4 into the bounded planner-attempt lifecycle, retain plan_run/5 validation as defense in depth, retry only pre-execution invalid candidates, aggregate planner usage across attempts, and never execute/auto-repair an invalid plan.

## TDD baseline verified — exact head `37637cab5b2be4f04cf4458744e9037fe915d9a7` The deterministic regression classifier is now correctly registered in the canonical corpus and passes without intentionally redlining CI. It proves the current defect from exact canonical base `267697bef10a3fffff7c093e1435ece770e7444b`: - `planner_attempts(2)` is configured; - attempt 1 returns a parseable typed plan with a model step followed by duplicate `final` steps; - canonical completion returns `phase:validate / kind:invalid_plan / detail:final_must_be_unique_and_last`; - the injected planner is called exactly **once**; - the invalid candidate executes **zero** injected model operations. The earlier classifier run exposed a test bug (`assertion/1` binding was not preserved); that was fixed rather than treated as runtime evidence. Current PlUnit has passed this classifier on `37637cab…`. This satisfies the BUG/TDD gate. Realization remains the approved narrow design: bring existing non-executing `rlm_plan:plan_validate/4` into the bounded planner-attempt lifecycle, retain `plan_run/5` validation as defense in depth, retry only pre-execution invalid candidates, aggregate planner usage across attempts, and never execute/auto-repair an invalid plan.

lost-rob0t commented

2026-08-26 14:14:18 +00:00

(Migrated from github.com)

Adversarial correction — do not retry capability/budget rejection as planner repair

Fresh source audit against exact main 267697bef10a3fffff7c093e1435ece770e7444b found an important boundary in the realization design.

rlm_plan:plan_validate/4 is non-executing, but it is not structural-only. It currently performs, in one call:

plan normalization;
capability validation (require_capability/2 throughout validate_plan_structure/...);
structural/scope validation such as unique-last final, bindings, expressions, retry/parallel shape;
plan estimate and plan-budget validation.

Therefore wiring full plan_validate/4 directly into planner_parse_result(ok(Plan),...) and retrying every error(...) would broaden planner_attempts/1 into retries for capability denial and plan-budget rejection. That conflicts with the existing adversarial contract: planner repair is for malformed/structurally invalid candidates, not authority/capability policy or budget failures.

Revised realization boundary

GO remains, but HOLD the naive plan_validate/4 -> retry on any error implementation.

Use/reuse the smallest canonical structural preflight that validates the closed typed plan before execution without consuming authority semantics. If current public API cannot express that boundary, add the smallest reusable rlm_plan entrypoint/refactor so:

parse/normalize
-> structural + binding/scope validity
-> retry if invalid
-> capability / recursive policy / global budgets remain authoritative failures
-> plan_run validates again before execution

The regression matrix should include at least one explicit guard proving a parseable candidate denied by capability is not converted into a planner-repair retry, plus the existing invalid-candidate zero-execution proof.

This preserves the original security invariant and avoids accidentally teaching the model that host policy denial is something it may retry around.

## Adversarial correction — do not retry capability/budget rejection as planner repair Fresh source audit against exact `main` `267697bef10a3fffff7c093e1435ece770e7444b` found an important boundary in the realization design. `rlm_plan:plan_validate/4` is non-executing, but it is **not structural-only**. It currently performs, in one call: - plan normalization; - capability validation (`require_capability/2` throughout `validate_plan_structure/...`); - structural/scope validation such as unique-last `final`, bindings, expressions, retry/parallel shape; - plan estimate and plan-budget validation. Therefore wiring full `plan_validate/4` directly into `planner_parse_result(ok(Plan),...)` and retrying every `error(...)` would broaden `planner_attempts/1` into retries for capability denial and plan-budget rejection. That conflicts with the existing adversarial contract: planner repair is for malformed/structurally invalid candidates, not authority/capability policy or budget failures. ### Revised realization boundary **GO remains**, but HOLD the naive `plan_validate/4 -> retry on any error` implementation. Use/reuse the smallest canonical **structural preflight** that validates the closed typed plan before execution without consuming authority semantics. If current public API cannot express that boundary, add the smallest reusable `rlm_plan` entrypoint/refactor so: ```text parse/normalize -> structural + binding/scope validity -> retry if invalid -> capability / recursive policy / global budgets remain authoritative failures -> plan_run validates again before execution ``` The regression matrix should include at least one explicit guard proving a parseable candidate denied by capability is **not** converted into a planner-repair retry, plus the existing invalid-candidate zero-execution proof. This preserves the original security invariant and avoids accidentally teaching the model that host policy denial is something it may retry around.

lost-rob0t commented

2026-08-26 15:23:36 +00:00

(Migrated from github.com)

RAGE realization update on exact branch head d65a233cf098b8951db9cc6cfd9c102ac113900c (base remains canonical main 267697bef10a3fffff7c093e1435ece770e7444b).

The adversarial boundary is now realized without adding a second validator:

parsed planner candidates are passed through canonical non-executing rlm_plan:plan_validate/4 before leaving the planner attempt loop;
only canonical phase:validate / kind:invalid_plan is classified as a repairable planner candidate and consumes another configured planner attempt;
canonical capability_denied and budget_exceeded are explicitly not repair signals: the candidate exits the planner loop unchanged and the existing authoritative downstream validation rejects/allows it exactly as before;
unexpected validation failures are terminal rather than being retried or converted to success;
planner usage remains accumulated through the existing planner_loop, so retries consume the same global model-call/token/cost budget;
plan_run/5 still performs full validation immediately before execution, preserving defense in depth.

The deterministic acceptance contract now proves:

structurally invalid attempt 1 + valid attempt 2 recovers and executes only the valid candidate;
two structurally invalid candidates return phase:planner / kind:plan_validation_failed with aggregate planner usage;
invalid candidates execute zero model operations;
a structurally valid but capability-denied candidate gets exactly one planner call and zero execution calls — host policy denial is not repaired around.

The branch was advanced as one atomic two-file commit after inspecting the synthetic commit diff. Fresh exact-head repository-native gates are queued; no prior-head green result is being reused.

RAGE realization update on exact branch head `d65a233cf098b8951db9cc6cfd9c102ac113900c` (base remains canonical `main` `267697bef10a3fffff7c093e1435ece770e7444b`). The adversarial boundary is now realized without adding a second validator: - parsed planner candidates are passed through canonical non-executing `rlm_plan:plan_validate/4` before leaving the planner attempt loop; - **only** canonical `phase:validate / kind:invalid_plan` is classified as a repairable planner candidate and consumes another configured planner attempt; - canonical `capability_denied` and `budget_exceeded` are explicitly *not* repair signals: the candidate exits the planner loop unchanged and the existing authoritative downstream validation rejects/allows it exactly as before; - unexpected validation failures are terminal rather than being retried or converted to success; - planner usage remains accumulated through the existing `planner_loop`, so retries consume the same global model-call/token/cost budget; - `plan_run/5` still performs full validation immediately before execution, preserving defense in depth. The deterministic acceptance contract now proves: 1. structurally invalid attempt 1 + valid attempt 2 recovers and executes only the valid candidate; 2. two structurally invalid candidates return `phase:planner / kind:plan_validation_failed` with aggregate planner usage; 3. invalid candidates execute zero model operations; 4. a structurally valid but capability-denied candidate gets exactly one planner call and zero execution calls — host policy denial is not repaired around. The branch was advanced as one atomic two-file commit after inspecting the synthetic commit diff. Fresh exact-head repository-native gates are queued; no prior-head green result is being reused.

lost-rob0t commented

2026-08-26 15:34:52 +00:00

(Migrated from github.com)

RAGE realization/adversarial reconciliation — current implementation shape is acceptable, exact-head gate still HOLD

Re-read against live PR #246 head 39d8e1ff030e011e8e0c37619c0c2ba053d83c47 (implementation tree from d65a233cf098b8951db9cc6cfd9c102ac113900c) and unchanged canonical main 267697bef10a3fffff7c093e1435ece770e7444b.

The current realization does call canonical plan_validate/4, but it does not implement the rejected naive plan_validate -> retry on any error design. It classifies the canonical structured validation outcome:

phase:validate, kind:invalid_plan => planner-repair candidate;
kind:capability_denied => deferred to the existing authoritative validation/execution path, no planner retry;
kind:budget_exceeded => likewise deferred, no planner retry;
anything else remains an explicit error.

That preserves the adversarial boundary without introducing a second structural validator or a new public rlm_plan API. The deterministic matrix on the branch also explicitly asserts that a parseable capability-denied plan gets exactly one planner call and executes zero model operations.

The realization tests now cover recovery on attempt 2, two-invalid-attempt exhaustion with aggregate usage, zero execution of invalid candidates, and capability denial not becoming a repair signal. This is the smallest coherent reuse of the canonical validator unless exact-head evidence exposes another validation class that needs separation.

Decision: implementation design GO; exact-head verification HOLD. Current head is not promotable yet: the fresh GitHub Actions runs attached to 39d8e1ff... are in a bad startup/queue state (CI and Clean SWI pack report startup_failure; Paid OpenRouter is failed/queued inconsistently; Nix and Tree-sitter are queued). The previous 37637cab... green evidence is TDD-baseline evidence only and cannot verify the realization.

Do not weaken or bypass those gates. Once GitHub executes the current exact-head workflows normally, use that evidence; if a real test/provider failure appears, preserve it and repair the root cause.

## RAGE realization/adversarial reconciliation — current implementation shape is acceptable, exact-head gate still HOLD Re-read against live PR #246 head `39d8e1ff030e011e8e0c37619c0c2ba053d83c47` (implementation tree from `d65a233cf098b8951db9cc6cfd9c102ac113900c`) and unchanged canonical `main` `267697bef10a3fffff7c093e1435ece770e7444b`. The current realization does call canonical `plan_validate/4`, but it does **not** implement the rejected naive `plan_validate -> retry on any error` design. It classifies the canonical structured validation outcome: - `phase:validate, kind:invalid_plan` => planner-repair candidate; - `kind:capability_denied` => deferred to the existing authoritative validation/execution path, no planner retry; - `kind:budget_exceeded` => likewise deferred, no planner retry; - anything else remains an explicit error. That preserves the adversarial boundary without introducing a second structural validator or a new public `rlm_plan` API. The deterministic matrix on the branch also explicitly asserts that a parseable capability-denied plan gets exactly one planner call and executes zero model operations. The realization tests now cover recovery on attempt 2, two-invalid-attempt exhaustion with aggregate usage, zero execution of invalid candidates, and capability denial not becoming a repair signal. This is the smallest coherent reuse of the canonical validator unless exact-head evidence exposes another validation class that needs separation. **Decision: implementation design GO; exact-head verification HOLD.** Current head is not promotable yet: the fresh GitHub Actions runs attached to `39d8e1ff...` are in a bad startup/queue state (CI and Clean SWI pack report `startup_failure`; Paid OpenRouter is failed/queued inconsistently; Nix and Tree-sitter are queued). The previous `37637cab...` green evidence is TDD-baseline evidence only and cannot verify the realization. Do not weaken or bypass those gates. Once GitHub executes the current exact-head workflows normally, use that evidence; if a real test/provider failure appears, preserve it and repair the root cause.

lost-rob0t commented

2026-08-27 03:07:17 +00:00

(Migrated from github.com)

RAGE reconciliation after canonical-main merge burst

Canonical main has advanced from the original #245 baseline 267697bef10a3fffff7c093e1435ece770e7444b to exact head 340dd0aec4d2e1b555aac7b108beed9c29706a65 after merging tool-result projection, provider tool-choice normalization, numeric schema bounds, subagent deadline policy, bounded parent resume, and proof-carrying child-result acceptance.

Fresh source inspection on that exact head shows the #245 defect still exists: planner_call_result/12 calls plan_parse/2, and planner_parse_result(ok(Plan), ...) immediately returns ok(planner_result{...}). Structural validation is still later in completion/plan execution, outside the configured planner retry loop. So #245 is not obsolete and still blocks #183 / PR #242.

PR #246 is now a recovery transaction rather than a merge-ready candidate. Its head 39d8e1ff030e011e8e0c37619c0c2ba053d83c47 is 7 commits ahead / 15 behind current main, GitHub reports it non-mergeable, and its merge base remains the old 267697be... head. The branch still owns exactly the intended three-file slice: prolog/rlm_completion.pl, deterministic-corpus registration, and the focused planner-validation retry test.

Adversarial comparison matters here: the 15 new canonical commits do not modify prolog/rlm_completion.pl; they do modify test/deterministic_corpus.pl by registering the newly merged deadline/result-acceptance suites. Therefore recovery must preserve both sides of that corpus inventory rather than choosing one and silently dropping tests.

Decision

GO to recover the existing #246 transaction onto exact 340dd0ae...; HOLD promotion/merge until fresh changed-head evidence.

Recovery requirements:

preserve the existing #245 TDD matrix, including zero execution of invalid candidates and no retry around capability/budget denial;
retain all newly merged canonical deterministic-corpus entries;
do not reintroduce a second validator or weaken plan_validate/4/plan_run/5;
rerun the complete repository-native gate on the recovered immutable head, including credential-backed REAL/Paid OpenRouter, Nix, clean pack, and Tree-sitter;
only after #245 is green may #242 be rebased/reverified as behavioral acceptance evidence.

No old exact-head CI result is reusable after this recovery.

## RAGE reconciliation after canonical-main merge burst Canonical `main` has advanced from the original #245 baseline `267697bef10a3fffff7c093e1435ece770e7444b` to exact head `340dd0aec4d2e1b555aac7b108beed9c29706a65` after merging tool-result projection, provider tool-choice normalization, numeric schema bounds, subagent deadline policy, bounded parent resume, and proof-carrying child-result acceptance. Fresh source inspection on that exact head shows the #245 defect still exists: `planner_call_result/12` calls `plan_parse/2`, and `planner_parse_result(ok(Plan), ...)` immediately returns `ok(planner_result{...})`. Structural validation is still later in completion/plan execution, outside the configured planner retry loop. So #245 is **not obsolete** and still blocks #183 / PR #242. PR #246 is now a recovery transaction rather than a merge-ready candidate. Its head `39d8e1ff030e011e8e0c37619c0c2ba053d83c47` is 7 commits ahead / 15 behind current `main`, GitHub reports it non-mergeable, and its merge base remains the old `267697be...` head. The branch still owns exactly the intended three-file slice: `prolog/rlm_completion.pl`, deterministic-corpus registration, and the focused planner-validation retry test. Adversarial comparison matters here: the 15 new canonical commits do **not** modify `prolog/rlm_completion.pl`; they do modify `test/deterministic_corpus.pl` by registering the newly merged deadline/result-acceptance suites. Therefore recovery must preserve both sides of that corpus inventory rather than choosing one and silently dropping tests. ### Decision **GO to recover the existing #246 transaction onto exact `340dd0ae...`; HOLD promotion/merge until fresh changed-head evidence.** Recovery requirements: 1. preserve the existing #245 TDD matrix, including zero execution of invalid candidates and no retry around capability/budget denial; 2. retain all newly merged canonical deterministic-corpus entries; 3. do not reintroduce a second validator or weaken `plan_validate/4`/`plan_run/5`; 4. rerun the complete repository-native gate on the recovered immutable head, including credential-backed REAL/Paid OpenRouter, Nix, clean pack, and Tree-sitter; 5. only after #245 is green may #242 be rebased/reverified as behavioral acceptance evidence. No old exact-head CI result is reusable after this recovery.