Add live RLM operator behavior acceptance

lost-rob0t commented

2026-08-26 12:33:09 +00:00

(Migrated from github.com)

Advances #183.

RAGE slice

TDD-first from exact canonical main 267697bef10a3fffff7c093e1435ece770e7444b.

The remaining non-overlapping #183 gap is behavioral provider evidence. Existing live_completion_openrouter_test.pl supplies exact plan JSON through planner_instruction/1, so it cannot prove the model independently learned how to operate the RLM runtime from the default permanent skills.

Initial falsifiable contract

This draft adds a credential-backed OpenRouter behavior suite on the ordinary rlm_completion/4 path with no exact planner instruction:

trivial task must complete with zero recursive calls;
unknown-information task must observably invoke an available typed behavior_lookup tool rather than guess;
decomposable opaque-context task must observably choose bounded rlm recursion and use at least two model calls.

Assertions use canonical runtime transition/usage/recursion evidence, not model self-report.

Adversarial boundary

no spoon-fed JSON plan or step sequence;
no second planner/selector/policy layer;
tool visibility does not grant capability/authority;
provider/malformed-plan failures remain hard failures;
trivial recursion is explicitly rejected;
tool use requires a real successful tool transition;
recursion requires a real successful rlm transition/depth evidence;
#225 retains ownership of the solver-exposed finite-CSP benchmark.

Gate

This PR is intentionally draft until the same-repository PR workflow runs the credential-backed REAL OpenRouter suite. If the behavioral contract fails, preserve the failure as evidence and refine the approved #183 implementation rather than adding an exact plan hint or weakening the test.

Advances #183. ## RAGE slice TDD-first from exact canonical `main` `267697bef10a3fffff7c093e1435ece770e7444b`. The remaining non-overlapping #183 gap is behavioral provider evidence. Existing `live_completion_openrouter_test.pl` supplies exact plan JSON through `planner_instruction/1`, so it cannot prove the model independently learned how to operate the RLM runtime from the default permanent skills. ## Initial falsifiable contract This draft adds a credential-backed OpenRouter behavior suite on the ordinary `rlm_completion/4` path with **no exact planner instruction**: - trivial task must complete with zero recursive calls; - unknown-information task must observably invoke an available typed `behavior_lookup` tool rather than guess; - decomposable opaque-context task must observably choose bounded `rlm` recursion and use at least two model calls. Assertions use canonical runtime transition/usage/recursion evidence, not model self-report. ## Adversarial boundary - no spoon-fed JSON plan or step sequence; - no second planner/selector/policy layer; - tool visibility does not grant capability/authority; - provider/malformed-plan failures remain hard failures; - trivial recursion is explicitly rejected; - tool use requires a real successful tool transition; - recursion requires a real successful `rlm` transition/depth evidence; - #225 retains ownership of the solver-exposed finite-CSP benchmark. ## Gate This PR is intentionally draft until the same-repository PR workflow runs the credential-backed REAL OpenRouter suite. If the behavioral contract fails, preserve the failure as evidence and refine the approved #183 implementation rather than adding an exact plan hint or weakening the test.

lost-rob0t commented

2026-08-27 04:28:23 +00:00

(Migrated from github.com)

Live gate evidence is useful and should remain a hard failure for this candidate. The decomposable fixture exposed rlm at the root ([rlm, context(slice), model(openrouter)]) and the branch strengthens rlm-recurse to treat independently investigable evidence as a strong recursion signal. The provider nevertheless selected two context slices + two direct model calls and synthesized successfully with recursive_calls:0 / max_depth:0. This is not a missing capability or malformed-plan failure, and the trivial/no-recursion + typed-tool cases both passed. Keep the recursion assertion; do not convert successful direct decomposition into acceptance for #183's explicit recursive-behavior case. Next design/debug target is why the permanent recursion guidance is insufficiently behavior-shaping for this model/provider, without spoon-feeding an exact plan.

Live gate evidence is useful and should remain a hard failure for this candidate. The decomposable fixture exposed `rlm` at the root (`[rlm, context(slice), model(openrouter)]`) and the branch strengthens `rlm-recurse` to treat independently investigable evidence as a strong recursion signal. The provider nevertheless selected two context slices + two direct model calls and synthesized successfully with `recursive_calls:0` / `max_depth:0`. This is not a missing capability or malformed-plan failure, and the trivial/no-recursion + typed-tool cases both passed. Keep the recursion assertion; do not convert successful direct decomposition into acceptance for #183's explicit recursive-behavior case. Next design/debug target is why the permanent recursion guidance is insufficiently behavior-shaping for this model/provider, without spoon-feeding an exact plan.

lost-rob0t commented

2026-08-27 04:47:59 +00:00

(Migrated from github.com)

Second live attempt on exact head f0e09b9ddfe59d91772ae4b98a2c3b8a5ba08ab4 is still a useful hard failure.

The strengthened generic rlm-recurse rule closed the previous wording loophole (multiple direct root model calls no longer count as satisfying an explicit recursion signal), but openai/gpt-oss-120b still selected the same basic root strategy: two context(slice) operations followed by two direct root model(...) calls and no rlm transition (recursive_calls:0, max_depth:0). Trivial/no-recursion and typed-tool behavior still pass.

More importantly, the direct fallback is semantically bad rather than merely non-recursive: the first leaf received a truncated raw evidence chunk as its entire prompt and asked what task it was supposed to perform; the second leaf received the other raw chunk and emitted another typed-plan JSON instead of evidence. The final result therefore also failed to preserve ALPHA-17/BETA-42.

This points at the next boundary to inspect: how planner-selected leaf model calls and nested rlm plans carry task framing + retrieved evidence. Do not weaken the recursion assertion, and do not add an exact plan hint. Before changing skill prose again, inspect the model-step expression/prompt composition surface and the nested-RLM execution contract to determine whether the planner actually has a clean domain-neutral way to delegate analyze this evidence for this subgoal rather than feeding raw context bytes as a standalone prompt.

Second live attempt on exact head `f0e09b9ddfe59d91772ae4b98a2c3b8a5ba08ab4` is still a useful hard failure. The strengthened generic `rlm-recurse` rule closed the previous wording loophole (multiple direct root `model` calls no longer count as satisfying an explicit recursion signal), but `openai/gpt-oss-120b` still selected the same basic root strategy: two `context(slice)` operations followed by two direct root `model(...)` calls and no `rlm` transition (`recursive_calls:0`, `max_depth:0`). Trivial/no-recursion and typed-tool behavior still pass. More importantly, the direct fallback is semantically bad rather than merely non-recursive: the first leaf received a truncated raw evidence chunk as its entire prompt and asked what task it was supposed to perform; the second leaf received the other raw chunk and emitted another typed-plan JSON instead of evidence. The final result therefore also failed to preserve ALPHA-17/BETA-42. This points at the next boundary to inspect: how planner-selected leaf model calls and nested `rlm` plans carry **task framing + retrieved evidence**. Do not weaken the recursion assertion, and do not add an exact plan hint. Before changing skill prose again, inspect the model-step expression/prompt composition surface and the nested-RLM execution contract to determine whether the planner actually has a clean domain-neutral way to delegate `analyze this evidence for this subgoal` rather than feeding raw context bytes as a standalone prompt.

lost-rob0t commented

2026-08-27 05:34:59 +00:00

(Migrated from github.com)

RAGE exact-head gate split for 3e7f04b4787be408158f97f92376236514bad1ca:

The #183 behavior contract itself has now crossed an important line: in both the normal CI REAL OpenRouter job and the pinned Paid OpenRouter job, Run REAL OpenRouter core suite is green. This branch wires live_rlm_operator_behavior_openrouter into that exact core suite, so the repaired terms/context(peek) fixture plus current rlm-operate/rlm-recurse skills now satisfy the three live no-spoon-feed cases on this immutable head: trivial stays non-recursive, unknown information uses the typed tool, and the decomposable two-record task uses the required bounded recursion/evidence path.

Do not promote the PR yet. Both live jobs then fail at the separate pre-existing Run REAL depth 0/1/2 recursion experiment step; deterministic CI is green. That is now the bug-first blocker. Treat it as a potential regression until isolated—do not waive it just because the new #183 acceptance finally passes, and do not weaken/remove the depth gate.

Next decision gate: compare the failing depth experiment on this head against a clean baseline containing the same current-main runtime, then inspect whether the new permanent skill text changes fixed-plan child model behavior or whether this is an already-red baseline/provider behavior. If branch-caused, preserve the #183 live success while repairing the smallest generic skill/runtime incompatibility. If baseline-red, keep that failure owned by its canonical benchmark transaction rather than falsely attributing it to #183. Exact-head full-gate promotion remains HOLD.

RAGE exact-head gate split for `3e7f04b4787be408158f97f92376236514bad1ca`: The #183 behavior contract itself has now crossed an important line: in both the normal CI REAL OpenRouter job and the pinned Paid OpenRouter job, `Run REAL OpenRouter core suite` is **green**. This branch wires `live_rlm_operator_behavior_openrouter` into that exact core suite, so the repaired terms/`context(peek)` fixture plus current `rlm-operate`/`rlm-recurse` skills now satisfy the three live no-spoon-feed cases on this immutable head: trivial stays non-recursive, unknown information uses the typed tool, and the decomposable two-record task uses the required bounded recursion/evidence path. Do **not** promote the PR yet. Both live jobs then fail at the separate pre-existing `Run REAL depth 0/1/2 recursion experiment` step; deterministic CI is green. That is now the bug-first blocker. Treat it as a potential regression until isolated—do not waive it just because the new #183 acceptance finally passes, and do not weaken/remove the depth gate. Next decision gate: compare the failing depth experiment on this head against a clean baseline containing the same current-main runtime, then inspect whether the new permanent skill text changes fixed-plan child model behavior or whether this is an already-red baseline/provider behavior. If branch-caused, preserve the #183 live success while repairing the smallest generic skill/runtime incompatibility. If baseline-red, keep that failure owned by its canonical benchmark transaction rather than falsely attributing it to #183. Exact-head full-gate promotion remains HOLD.

lost-rob0t commented

2026-08-27 06:04:45 +00:00

(Migrated from github.com)

RAGE exact-head failure isolation on 3e7f04b4787be408158f97f92376236514bad1ca:

The new #183 behavioral acceptance itself is green in the REAL OpenRouter core suite: trivial/no-recursion, typed unknown-info tool use, and decomposable bounded recursion all passed. Deterministic CI, Nix, clean-pack, and Tree-sitter are also green.

The remaining CI failure is the older injected-plan deep_openrouter_experiment, and the logs make the cause deterministic enough to classify: depth 0 and depth 1 pass; depth 2 reaches the requested recursion depth but is rejected by the benchmark's fixed max_total_tokens:3000 after accumulating 3795 tokens (prompt_tokens:3636, completion_tokens:159, model_calls:4 in the structured error). The benchmark source still gives every depth the same 3000-token ceiling even though the fixed depth-2 plan necessarily performs three nested provider calls, and the now-permanent RLM operating context materially increases each provider-visible prompt (~1212 prompt tokens/call in this run).

Adversarial decision: this is not evidence that depth-2 recursion execution regressed, and it is not a reason to weaken/remove the depth gate. It is a stale benchmark-budget contract exposed by the larger required permanent context. The repair should be TDD-first and benchmark-local: make the live depth fixture's token ceiling explicitly sufficient for its expected fixed number of provider calls (prefer a depth-derived/tested ceiling or another deterministic contract), while retaining a finite hard token budget and all depth/provider-call assertions. Do not change production runtime budgeting, skip the live lane, or borrow the successful #183 behavior result as proof for this separate gate.

PR remains HOLD/draft until a changed exact head passes the full gate.

RAGE exact-head failure isolation on `3e7f04b4787be408158f97f92376236514bad1ca`: The new #183 behavioral acceptance itself is green in the REAL OpenRouter core suite: trivial/no-recursion, typed unknown-info tool use, and decomposable bounded recursion all passed. Deterministic CI, Nix, clean-pack, and Tree-sitter are also green. The remaining CI failure is the older injected-plan `deep_openrouter_experiment`, and the logs make the cause deterministic enough to classify: depth 0 and depth 1 pass; depth 2 reaches the requested recursion depth but is rejected by the benchmark's fixed `max_total_tokens:3000` after accumulating `3795` tokens (`prompt_tokens:3636`, `completion_tokens:159`, `model_calls:4` in the structured error). The benchmark source still gives every depth the same 3000-token ceiling even though the fixed depth-2 plan necessarily performs three nested provider calls, and the now-permanent RLM operating context materially increases each provider-visible prompt (~1212 prompt tokens/call in this run). Adversarial decision: this is **not evidence that depth-2 recursion execution regressed**, and it is not a reason to weaken/remove the depth gate. It is a stale benchmark-budget contract exposed by the larger required permanent context. The repair should be TDD-first and benchmark-local: make the live depth fixture's token ceiling explicitly sufficient for its expected fixed number of provider calls (prefer a depth-derived/tested ceiling or another deterministic contract), while retaining a finite hard token budget and all depth/provider-call assertions. Do not change production runtime budgeting, skip the live lane, or borrow the successful #183 behavior result as proof for this separate gate. PR remains HOLD/draft until a changed exact head passes the full gate.

lost-rob0t commented

2026-08-27 17:09:58 +00:00

(Migrated from github.com)

RAGE reconciliation on exact head cff9f518c609286509c49e6be2877ce6936b75cd:

Re-read current canonical main at 87189f7128bacfe97cd09874e958dd07e03322d3; the old #242 branch had diverged by 34 main commits and still carried benchmark/runtime patches whose ownership/implementation has since moved on main.
Recovered the existing transaction without history rewrite by creating a real merge commit with current main as the second parent.
Reconciled the candidate tree against current main rather than reviving stale files. The PR now differs from main only in the #183 live behavioral acceptance, wiring that suite into the REAL OpenRouter core gate, and small current-compatible rlm-operate/rlm-recurse guidance.
Preserved the hard behavioral assertions: trivial work must stay non-recursive; unknown information must use the typed tool; explicitly separate evidence investigations must use bounded rlm recursion and preserve requested evidence/provenance identifiers.
Kept current main's newer typed-plan/query-reference guidance, including nested access to the original query; ordinary model steps are now explicitly told not to emit another planner object unless asked to plan.
Dropped the stale branch-local live-depth budget/CLI-budget patches from the candidate tree because current main has superseded those benchmark/runtime surfaces. No production runtime budget was weakened.

Fresh-head evidence so far: deterministic unit/load gate SUCCESS; Nix flake SUCCESS; Clean SWI pack SUCCESS; Tree-sitter FFI SUCCESS. Credential-backed REAL OpenRouter core and Paid OpenRouter lanes are still running. PR remains draft/HOLD until those exact-head live gates finish; no old-head success is being reused.

RAGE reconciliation on exact head `cff9f518c609286509c49e6be2877ce6936b75cd`: - Re-read current canonical `main` at `87189f7128bacfe97cd09874e958dd07e03322d3`; the old #242 branch had diverged by 34 main commits and still carried benchmark/runtime patches whose ownership/implementation has since moved on main. - Recovered the existing transaction without history rewrite by creating a real merge commit with current main as the second parent. - Reconciled the candidate tree against current main rather than reviving stale files. The PR now differs from main only in the #183 live behavioral acceptance, wiring that suite into the REAL OpenRouter core gate, and small current-compatible `rlm-operate`/`rlm-recurse` guidance. - Preserved the hard behavioral assertions: trivial work must stay non-recursive; unknown information must use the typed tool; explicitly separate evidence investigations must use bounded `rlm` recursion and preserve requested evidence/provenance identifiers. - Kept current main's newer typed-plan/query-reference guidance, including nested access to the original `query`; ordinary model steps are now explicitly told not to emit another planner object unless asked to plan. - Dropped the stale branch-local live-depth budget/CLI-budget patches from the candidate tree because current main has superseded those benchmark/runtime surfaces. No production runtime budget was weakened. Fresh-head evidence so far: deterministic unit/load gate SUCCESS; Nix flake SUCCESS; Clean SWI pack SUCCESS; Tree-sitter FFI SUCCESS. Credential-backed REAL OpenRouter core and Paid OpenRouter lanes are still running. PR remains draft/HOLD until those exact-head live gates finish; no old-head success is being reused.