feat(recursion): add controlled depth >1 experiments #43

Merged

lost-rob0t merged 23 commits from feature/issue-20-deep-recursion-experiments into main

2026-08-13 07:10:36 +00:00

lost-rob0t commented

2026-08-13 06:45:50 +00:00

(Migrated from github.com)

Summary

add an explicit public-facade opt-in for recursion depth >1: increasing max_recursion_depth alone is no longer enough; callers must also pass experimental_deep_recursion(true)
keep the production default at depth 1 and preserve the independent adaptive-recursion deep-policy gates
add rlm_deep_experiment without introducing a second executor; it reuses the production typed-plan, agent, artifact, and benchmark runtimes
add deterministic depth 0/1/2 fixtures plus delegated-subagent and fresh-root-artifact comparisons
prove whole-tree step/model-call budgets, recursive capability narrowing, and root-token cancellation reaching already-running grandchild work
deliberately classify deeper recursion as helps, hurts, or neutral instead of assuming depth is beneficial
encode a promotion rule requiring live trials, multiple fixtures, positive quality delta, bounded cost/latency ratios, and zero budget/capability/cancellation failures
add benchmark/run.pl -- deep-experiment for deterministic structural evidence
add benchmark/run.pl -- deep-integration for REAL OpenRouter depth 0/1/2 measurements using a trusted injected planner and real provider calls inside the measured tree
gate the deterministic deep experiment in every CI run and the real depth experiment in the credentialed OpenRouter lane
fix nested model usage accounting discovered by the live depth experiment: rlm_plan now records an append-only model_responses ledger that survives nested lexical scope restoration, and rlm_completion aggregates that ledger instead of inferring all calls from top-level vars
retain the legacy top-level-vars usage fallback for older plan-result shapes
add deterministic regression coverage proving three nested responses produce three calls/all tokens/all cost and can trip the combined completion token budget
document the experiment boundary, composition tradeoffs, benchmark semantics, promotion criteria, and research evidence

Safety boundary

experimental_deep_recursion(true) is an opt-in marker only. It grants no model, tool, context, agent, parallel, retry, graph, or filesystem capability and does not widen any runtime budget.

The supported public facade rejects requested depth >1 before planner execution unless that explicit marker is present.

Nested typed plans continue to execute through the existing shared rlm_plan state, so recursive children do not receive fresh step/model/tool/context budgets.

Accounting defect found during live validation

The first REAL depth-2 run executed successfully, but reported only 2 provider calls where the tree had actually made 3. The root cause was a legacy assumption in plan_usage/2: it scanned only final top-level variables. Nested rlm(...) scopes intentionally restore child-local variables, so the intermediate depth-2 model response disappeared from the final lexical scope even though its provider call, tokens, and cost were real.

The fix is an append-only model-response ledger in the plan execution state. Every successful provider response is recorded before lexical binding. Nested scope restoration does not remove ledger entries. Successful plan results expose the ledger, and completion usage now sums it exactly once per recorded provider response. Error plan results also retain the ledger for follow-up error-path accounting work (#44).

Regression tests verify:

ledger order survives nested scope finalization;
three recorded responses account as exactly three model calls;
prompt/completion/total tokens and cost all aggregate correctly;
hidden nested usage can trigger the completion token ceiling;
legacy plan-result shapes without the ledger retain the previous visible-var fallback.

Deterministic evidence

Exact head 158716e07196cb94f4dad5ff7192d02ea948f845 passes:

supported SWI runtime check
all production/static loads
complete PlUnit, including adversarial depth-2 tests and nested-usage regressions
normal benchmark/conformance
the 15-case deterministic deep-recursion experiment
credential-free CLI/trace smoke
graph and artifact fresh-process restart checks
whitespace

The deterministic experiment intentionally labels token/call/cost numbers as modeled fixture data; it does not claim provider economics. Its promotion evidence sets live_trials:0, so deterministic success cannot promote depth >1.

REAL OpenRouter evidence

The exact-head credentialed lane is fully green: core, structured repair, normal provider benchmark, depth 0/1/2 experiment, and one-command RLM CLI smoke.

The fixed live experiment planner is injected locally and makes zero network calls. The recursive tree itself produced the expected exact provider-call cardinality:

Depth	Provider calls	Tokens	Latency	Exact token	Status
0	1	91	1,853 ms	yes	pass
1	2	205	22,470 ms	yes	pass
2	3	171	12,081 ms	yes	pass

All three returned LIVE_DEEP_OK, had the requested recursion depth, HTTP-successful provider responses, and quality 1.0. Free-route provider-reported cost was $0 for this run. Total experiment usage was 6 provider calls and 467 tokens.

The latency result is intentionally not treated as evidence that depth 2 is faster than depth 1: openrouter/free dynamically selected different models and one run is noisy. The encoded promotion rule still requires at least 20 live trials across at least 3 independent fixtures, positive quality delta, bounded cost/latency ratios, and zero safety failures.

Follow-ups discovered during this work

#42 — canonicalize recursive-plan fingerprints before cycle detection; anonymous-tag SWI dicts can otherwise produce a false recursive-cycle result in the term API
#44 — preserve executed provider usage on completion error paths; plan errors now carry the response ledger but completion currently returns the execution error before aggregating it
#45 — preserve every nested model event and true depth in completion trajectories; usage is authoritative now, but trajectory reconstruction still relies too much on final lexical vars

These are explicit follow-up correctness items. They do not weaken #20's execution, global-budget, capability-narrowing, cancellation, or successful-call accounting acceptance evidence.

Closes #20

## Summary - add an explicit public-facade opt-in for recursion depth >1: increasing `max_recursion_depth` alone is no longer enough; callers must also pass `experimental_deep_recursion(true)` - keep the production default at depth 1 and preserve the independent adaptive-recursion deep-policy gates - add `rlm_deep_experiment` without introducing a second executor; it reuses the production typed-plan, agent, artifact, and benchmark runtimes - add deterministic depth 0/1/2 fixtures plus delegated-subagent and fresh-root-artifact comparisons - prove whole-tree step/model-call budgets, recursive capability narrowing, and root-token cancellation reaching already-running grandchild work - deliberately classify deeper recursion as `helps`, `hurts`, or `neutral` instead of assuming depth is beneficial - encode a promotion rule requiring live trials, multiple fixtures, positive quality delta, bounded cost/latency ratios, and zero budget/capability/cancellation failures - add `benchmark/run.pl -- deep-experiment` for deterministic structural evidence - add `benchmark/run.pl -- deep-integration` for REAL OpenRouter depth 0/1/2 measurements using a trusted injected planner and real provider calls inside the measured tree - gate the deterministic deep experiment in every CI run and the real depth experiment in the credentialed OpenRouter lane - fix nested model usage accounting discovered by the live depth experiment: `rlm_plan` now records an append-only `model_responses` ledger that survives nested lexical scope restoration, and `rlm_completion` aggregates that ledger instead of inferring all calls from top-level vars - retain the legacy top-level-vars usage fallback for older plan-result shapes - add deterministic regression coverage proving three nested responses produce three calls/all tokens/all cost and can trip the combined completion token budget - document the experiment boundary, composition tradeoffs, benchmark semantics, promotion criteria, and research evidence ## Safety boundary `experimental_deep_recursion(true)` is an opt-in marker only. It grants no model, tool, context, agent, parallel, retry, graph, or filesystem capability and does not widen any runtime budget. The supported public facade rejects requested depth >1 before planner execution unless that explicit marker is present. Nested typed plans continue to execute through the existing shared `rlm_plan` state, so recursive children do not receive fresh step/model/tool/context budgets. ## Accounting defect found during live validation The first REAL depth-2 run executed successfully, but reported only 2 provider calls where the tree had actually made 3. The root cause was a legacy assumption in `plan_usage/2`: it scanned only final top-level variables. Nested `rlm(...)` scopes intentionally restore child-local variables, so the intermediate depth-2 model response disappeared from the final lexical scope even though its provider call, tokens, and cost were real. The fix is an append-only model-response ledger in the plan execution state. Every successful provider response is recorded before lexical binding. Nested scope restoration does not remove ledger entries. Successful plan results expose the ledger, and completion usage now sums it exactly once per recorded provider response. Error plan results also retain the ledger for follow-up error-path accounting work (#44). Regression tests verify: - ledger order survives nested scope finalization; - three recorded responses account as exactly three model calls; - prompt/completion/total tokens and cost all aggregate correctly; - hidden nested usage can trigger the completion token ceiling; - legacy plan-result shapes without the ledger retain the previous visible-var fallback. ## Deterministic evidence Exact head `158716e07196cb94f4dad5ff7192d02ea948f845` passes: - supported SWI runtime check - all production/static loads - complete PlUnit, including adversarial depth-2 tests and nested-usage regressions - normal benchmark/conformance - the 15-case deterministic deep-recursion experiment - credential-free CLI/trace smoke - graph and artifact fresh-process restart checks - whitespace The deterministic experiment intentionally labels token/call/cost numbers as modeled fixture data; it does not claim provider economics. Its promotion evidence sets `live_trials:0`, so deterministic success cannot promote depth >1. ## REAL OpenRouter evidence The exact-head credentialed lane is fully green: core, structured repair, normal provider benchmark, depth 0/1/2 experiment, and one-command RLM CLI smoke. The fixed live experiment planner is injected locally and makes zero network calls. The recursive tree itself produced the expected exact provider-call cardinality: | Depth | Provider calls | Tokens | Latency | Exact token | Status | |---:|---:|---:|---:|---|---| | 0 | 1 | 91 | 1,853 ms | yes | pass | | 1 | 2 | 205 | 22,470 ms | yes | pass | | 2 | 3 | 171 | 12,081 ms | yes | pass | All three returned `LIVE_DEEP_OK`, had the requested recursion depth, HTTP-successful provider responses, and quality 1.0. Free-route provider-reported cost was $0 for this run. Total experiment usage was 6 provider calls and 467 tokens. The latency result is intentionally **not** treated as evidence that depth 2 is faster than depth 1: `openrouter/free` dynamically selected different models and one run is noisy. The encoded promotion rule still requires at least 20 live trials across at least 3 independent fixtures, positive quality delta, bounded cost/latency ratios, and zero safety failures. ## Follow-ups discovered during this work - #42 — canonicalize recursive-plan fingerprints before cycle detection; anonymous-tag SWI dicts can otherwise produce a false recursive-cycle result in the term API - #44 — preserve executed provider usage on completion error paths; plan errors now carry the response ledger but completion currently returns the execution error before aggregating it - #45 — preserve every nested model event and true depth in completion trajectories; usage is authoritative now, but trajectory reconstruction still relies too much on final lexical vars These are explicit follow-up correctness items. They do not weaken #20's execution, global-budget, capability-narrowing, cancellation, or successful-call accounting acceptance evidence. Closes #20