[P1] Build benchmark and conformance suite for RLM, agents, and MCP #18

New issue

Closed

opened 2026-08-12 01:20:01 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-12 01:20:01 +00:00

(Migrated from github.com)

Goal

Make architectural claims measurable and prevent regressions across recursion, providers, tools, graph execution, and MCP versions.

Scope

add long-context fixtures for peek/search/partition/map/reduce;
add direct-LM vs RLM comparisons;
add depth 0/1/2 comparisons under fixed budgets;
measure quality, calls, tokens, latency, cost, recursion depth, and context bytes inspected;
add structured-plan validation/failure fixtures;
add agent supervision/cancellation/backpressure fixtures;
add graph checkpoint/resume fixtures;
add MCP 2025-only, 2026-only, and dual-version conformance fixtures;
produce machine-readable benchmark output plus concise human summary.

Acceptance criteria

one command runs deterministic conformance tests without paid APIs;
optional integration mode runs real-provider benchmarks when credentials exist;
benchmark output can compare direct LM and RLM under the same task/budget;
MCP compatibility matrix is automated;
CI runs the deterministic subset;
regressions in token/call/latency budgets can be detected.

Dependencies

#9
#10
#11
#12
#15
#17
parent #3

Research

See the evaluation sections across RLM-RESEARCH-002 through RLM-RESEARCH-009.

## Goal Make architectural claims measurable and prevent regressions across recursion, providers, tools, graph execution, and MCP versions. ## Scope - add long-context fixtures for peek/search/partition/map/reduce; - add direct-LM vs RLM comparisons; - add depth 0/1/2 comparisons under fixed budgets; - measure quality, calls, tokens, latency, cost, recursion depth, and context bytes inspected; - add structured-plan validation/failure fixtures; - add agent supervision/cancellation/backpressure fixtures; - add graph checkpoint/resume fixtures; - add MCP 2025-only, 2026-only, and dual-version conformance fixtures; - produce machine-readable benchmark output plus concise human summary. ## Acceptance criteria - one command runs deterministic conformance tests without paid APIs; - optional integration mode runs real-provider benchmarks when credentials exist; - benchmark output can compare direct LM and RLM under the same task/budget; - MCP compatibility matrix is automated; - CI runs the deterministic subset; - regressions in token/call/latency budgets can be detected. ## Dependencies - #9 - #10 - #11 - #12 - #15 - #17 - parent #3 ## Research See the evaluation sections across `RLM-RESEARCH-002` through `RLM-RESEARCH-009`.