Recover real recursive constraint benchmark on current main #264

Merged

lost-rob0t merged 14 commits from rage/223-constraint-benchmark-recovery into main

2026-08-27 11:05:37 +00:00

lost-rob0t commented

2026-08-27 04:25:39 +00:00

(Migrated from github.com)

Recovers #225 / advances #223 on current main without rewriting shared branch history.

Why recover

The original #225 deterministic corpus was green, but its REAL OpenRouter depth gate failed in all six CSP lanes before verification because the root planner returned no parseable typed JSON plan (plan_parse_failed / no_json_object). Since that branch was cut, current main has landed planner-validation retry support and related runtime fixes.

The two pre-existing benchmark files modified by #225 were byte-identical on current main to #225's original base, so replaying the seven benchmark/test files did not overwrite newer changes in those files.

Recovery contents

The candidate preserves the #225 seven-file benchmark slice:

deterministic uniquely-solvable 30-value CSP + trusted CLP(FD) oracle;
core-minimal and harness-guided depth 0/1/2 lanes;
old fixed-plan recursion coverage retained as deep-smoke;
closed JSON assignment parsing and violated-constraint reporting;
Frozen Spec + production verifier acceptance;
deterministic focused benchmark runner and regressions.

It also reconciles the two live CI workflow gates with the new benchmark contract:

the old workflow assertions expected deep_openrouter_experiment with 3 cases, while this slice intentionally emits deep_constraint_openrouter with 6 cases;
both CI and Paid OpenRouter now validate that six-lane report;
Paid OpenRouter now prints the generated report on failure so per-lane structured errors are observable instead of losing the only useful evidence in $RUNNER_TEMP.

No benchmark problem, depth lane, time/token/cost budget, verifier, or acceptance criterion was weakened to make a provider failure green.

Gate

Keep draft until exact-head CI, Paid OpenRouter, Nix, clean-pack and Tree-sitter evidence returns. If the live CSP still fails, preserve the failure and diagnose the planner/provider behavior rather than injecting an exact plan.

Supersedes the stale implementation transaction in #225; #223 remains open.

Recovers #225 / advances #223 on current `main` without rewriting shared branch history. ## Why recover The original #225 deterministic corpus was green, but its REAL OpenRouter depth gate failed in all six CSP lanes before verification because the root planner returned no parseable typed JSON plan (`plan_parse_failed` / `no_json_object`). Since that branch was cut, current main has landed planner-validation retry support and related runtime fixes. The two pre-existing benchmark files modified by #225 were byte-identical on current main to #225's original base, so replaying the seven benchmark/test files did not overwrite newer changes in those files. ## Recovery contents The candidate preserves the #225 seven-file benchmark slice: - deterministic uniquely-solvable 30-value CSP + trusted CLP(FD) oracle; - core-minimal and harness-guided depth 0/1/2 lanes; - old fixed-plan recursion coverage retained as `deep-smoke`; - closed JSON assignment parsing and violated-constraint reporting; - Frozen Spec + production verifier acceptance; - deterministic focused benchmark runner and regressions. It also reconciles the two live CI workflow gates with the new benchmark contract: - the old workflow assertions expected `deep_openrouter_experiment` with 3 cases, while this slice intentionally emits `deep_constraint_openrouter` with 6 cases; - both CI and Paid OpenRouter now validate that six-lane report; - Paid OpenRouter now prints the generated report on failure so per-lane structured errors are observable instead of losing the only useful evidence in `$RUNNER_TEMP`. No benchmark problem, depth lane, time/token/cost budget, verifier, or acceptance criterion was weakened to make a provider failure green. ## Gate Keep draft until exact-head CI, Paid OpenRouter, Nix, clean-pack and Tree-sitter evidence returns. If the live CSP still fails, preserve the failure and diagnose the planner/provider behavior rather than injecting an exact plan. Supersedes the stale implementation transaction in #225; #223 remains open.