WIP: real recursive constraint-solving benchmark #225

Closed

lost-rob0t wants to merge 11 commits from feature/223-real-constraint-benchmark into main

lost-rob0t commented

2026-08-26 08:26:34 +00:00

(Migrated from github.com)

Implements the first vertical slice of #223.

What changed

replaces the live deep-integration token-echo benchmark with a hard 30-value CSP across core_minimal and harness_guided lanes at recursion ceilings 0/1/2;
preserves the previous fixed-plan LIVE_DEEP_OK path as deep-smoke so provider/recursion plumbing coverage is not lost;
adds a deterministic CLP(FD) fixture with an independently proven unique solution;
parses model output as closed JSON data and reports exact violated constraint IDs;
routes final acceptance through a Frozen Spec + production rlm_verify requirement backed by the trusted CSP oracle;
adds a focused deterministic runner at test/run_constraint_benchmark.pl;
explicitly proves the core_minimal lane adds no benchmark-specific planner instruction.

Focused gate

swipl -q -s test/run_constraint_benchmark.pl

Live modes

swipl -q -s benchmark/run.pl -- deep-integration report.json
swipl -q -s benchmark/run.pl -- deep-smoke smoke.json

deep-integration requires OPENROUTER_API_KEY and now measures actual solution correctness rather than token presence.

Still to do in this epic

run/fix focused and aggregate CI;
run the credentialed live benchmark and inspect minimal-vs-guided/depth behavior;
tighten metrics if runtime observability exposes additional planner-attempt data;
then begin Slice 2: durable context mounts with independent lifetime/scope/visibility semantics.

Closes no issue yet; #223 remains the umbrella epic.

Implements the first vertical slice of #223. ## What changed - replaces the live `deep-integration` token-echo benchmark with a hard 30-value CSP across `core_minimal` and `harness_guided` lanes at recursion ceilings 0/1/2; - preserves the previous fixed-plan `LIVE_DEEP_OK` path as `deep-smoke` so provider/recursion plumbing coverage is not lost; - adds a deterministic CLP(FD) fixture with an independently proven unique solution; - parses model output as closed JSON data and reports exact violated constraint IDs; - routes final acceptance through a Frozen Spec + production `rlm_verify` requirement backed by the trusted CSP oracle; - adds a focused deterministic runner at `test/run_constraint_benchmark.pl`; - explicitly proves the `core_minimal` lane adds no benchmark-specific planner instruction. ## Focused gate ```sh swipl -q -s test/run_constraint_benchmark.pl ``` ## Live modes ```sh swipl -q -s benchmark/run.pl -- deep-integration report.json swipl -q -s benchmark/run.pl -- deep-smoke smoke.json ``` `deep-integration` requires `OPENROUTER_API_KEY` and now measures actual solution correctness rather than token presence. ## Still to do in this epic - run/fix focused and aggregate CI; - run the credentialed live benchmark and inspect minimal-vs-guided/depth behavior; - tighten metrics if runtime observability exposes additional planner-attempt data; - then begin Slice 2: durable context mounts with independent lifetime/scope/visibility semantics. Closes no issue yet; #223 remains the umbrella epic.

lost-rob0t commented

2026-08-27 04:26:19 +00:00

(Migrated from github.com)

Superseded by #264. The original branch preserved useful deterministic benchmark work, but its exact live gate failed before verification because all six CSP lanes hit planner no_json_object / plan_parse_failed. #264 replays exactly this seven-file benchmark slice onto current main, including the newer planner-validation retry/runtime fixes, without rewriting this branch's shared history. Continue the #223 Slice 1 gate there.

Superseded by #264. The original branch preserved useful deterministic benchmark work, but its exact live gate failed before verification because all six CSP lanes hit planner `no_json_object` / `plan_parse_failed`. #264 replays exactly this seven-file benchmark slice onto current main, including the newer planner-validation retry/runtime fixes, without rewriting this branch's shared history. Continue the #223 Slice 1 gate there.