feat(benchmark): add deterministic conformance harness #40

Merged

lost-rob0t merged 20 commits from feature/issue-18-benchmark-conformance into main

2026-08-13 05:17:34 +00:00

lost-rob0t commented

2026-08-13 05:07:00 +00:00

(Migrated from github.com)

Summary

add a canonical benchmark metric/report layer that normalizes quality, model/tool/context calls, token usage, provider cost, latency, recursion depth, and context inspection bytes/items
add JSON and human-readable report output plus fixed-budget regression detection for calls, tokens, cost, latency, depth, and inspected context
add a one-command deterministic conformance runner: swipl -q -s benchmark/run.pl -- deterministic
cover 16 reportable production-subsystem cases: opaque context peek/search/partition/map/reduce; the same task at direct/depth-0, RLM depth-1, and explicitly gated depth-2; malformed structured-plan rejection; agent backpressure, parent/child cancellation, and 20 logical agents over a bounded worker pool; graph checkpoint/resume; MCP 2025, MCP 2026, and dual-facade anchors
preserve detailed PlUnit protocol matrices as the source of fine-grained MCP/agent/graph behavior while adding comparable cross-subsystem benchmark output instead of duplicating telemetry
expose benchmark primitives from the public rlm entrypoint and include them in rlm_ready/0
add an optional real-provider mode: swipl -q -s benchmark/run.pl -- integration, using the production OpenRouter provider path and reporting real latency, token usage, provider-reported cost, selected model/HTTP evidence, provider health, and instruction quality
separate live integration status from quality: usable assistant output proves the provider/runtime path is healthy, while exact benchmark-token compliance scores 1.0; usable output that misses the token scores 0.5 and remains visible as quality evidence instead of making openrouter/free routing a flaky health gate
gate deterministic conformance independently from PlUnit in CI and add the real-provider benchmark to the existing REAL OpenRouter job
document schemas, commands, deterministic-vs-live measurement semantics, and regression budgets

Deterministic comparison

The direct/depth-0, RLM depth-1, and RLM depth-2 cases execute the same ground task with the same remaining_calls=4 and remaining_tokens=8000 budget. Depth 2 is admitted only with the production allow_deep_recursion(true) + deep_recursion_capability(true) gates. Deterministic call/token/cost numbers are fixed conformance fixtures, not claims about provider economics.

Validation

Deterministic CI passes production/static load, the complete PlUnit suite, the standalone 16-case deterministic benchmark CLI, fresh-process graph restart, durable artifact restart, and whitespace checks. The PR live job exercises the existing REAL OpenRouter core/structured-repair gates plus the machine-readable integration benchmark. A first live run exposed a healthy HTTP/usage/assistant-output response that missed the requested token under openrouter/free; the benchmark contract now records that distinction explicitly rather than conflating instruction quality with provider health.

Closes #18

## Summary - add a canonical benchmark metric/report layer that normalizes quality, model/tool/context calls, token usage, provider cost, latency, recursion depth, and context inspection bytes/items - add JSON and human-readable report output plus fixed-budget regression detection for calls, tokens, cost, latency, depth, and inspected context - add a one-command deterministic conformance runner: `swipl -q -s benchmark/run.pl -- deterministic` - cover 16 reportable production-subsystem cases: opaque context peek/search/partition/map/reduce; the same task at direct/depth-0, RLM depth-1, and explicitly gated depth-2; malformed structured-plan rejection; agent backpressure, parent/child cancellation, and 20 logical agents over a bounded worker pool; graph checkpoint/resume; MCP 2025, MCP 2026, and dual-facade anchors - preserve detailed PlUnit protocol matrices as the source of fine-grained MCP/agent/graph behavior while adding comparable cross-subsystem benchmark output instead of duplicating telemetry - expose benchmark primitives from the public `rlm` entrypoint and include them in `rlm_ready/0` - add an optional real-provider mode: `swipl -q -s benchmark/run.pl -- integration`, using the production OpenRouter provider path and reporting real latency, token usage, provider-reported cost, selected model/HTTP evidence, provider health, and instruction quality - separate live integration status from quality: usable assistant output proves the provider/runtime path is healthy, while exact benchmark-token compliance scores `1.0`; usable output that misses the token scores `0.5` and remains visible as quality evidence instead of making `openrouter/free` routing a flaky health gate - gate deterministic conformance independently from PlUnit in CI and add the real-provider benchmark to the existing REAL OpenRouter job - document schemas, commands, deterministic-vs-live measurement semantics, and regression budgets ## Deterministic comparison The direct/depth-0, RLM depth-1, and RLM depth-2 cases execute the same ground task with the same `remaining_calls=4` and `remaining_tokens=8000` budget. Depth 2 is admitted only with the production `allow_deep_recursion(true)` + `deep_recursion_capability(true)` gates. Deterministic call/token/cost numbers are fixed conformance fixtures, not claims about provider economics. ## Validation Deterministic CI passes production/static load, the complete PlUnit suite, the standalone 16-case deterministic benchmark CLI, fresh-process graph restart, durable artifact restart, and whitespace checks. The PR live job exercises the existing REAL OpenRouter core/structured-repair gates plus the machine-readable integration benchmark. A first live run exposed a healthy HTTP/usage/assistant-output response that missed the requested token under `openrouter/free`; the benchmark contract now records that distinction explicitly rather than conflating instruction quality with provider health. Closes #18