[P1] Add baseline-vs-skill outcome evaluation with verifier-backed evidence #444

New issue

Closed

opened 2026-09-10 21:18:17 +00:00 by nsaspy · 1 comment

nsaspy commented

2026-09-10 21:18:17 +00:00

Owner

Parent: #167
Related: #56, #68-#71, #101, #168

Goal

Measure whether loading a skill actually improves task outcomes, not merely whether the prompt compiler selected it.

Contract

Run equivalent task fixtures in paired modes:

baseline: skill disabled/absent
variant:  candidate skill active

Collect comparable structured results and objective evidence where available.

Required measurements

verifier/result state;
model calls;
input/output/total tokens and cost where available;
latency;
tool calls;
repair/replan iterations;
subagent calls;
provider-visible context tokens/bytes;
failure/regression class;
trace/evidence references.

Requirements

reuse existing completion/graph/agent/trace/usage and Spec/Verify primitives;
deterministic fixtures must work without live credentials;
optional live-provider lane must be clearly separate;
task success should use objective verifier/evidence when one exists, not model self-report;
paired runs preserve equivalent task/spec/budget conditions except for the skill variable under test;
emit machine-readable comparison records suitable for later evolution fitness;
record uncertainty when evidence cannot determine a winner.

Acceptance

deterministic baseline-vs-skill fixture produces one comparable result record;
verifier-backed task success/failure is preserved distinctly from worker claims;
token/cost/latency/tool/subagent/repair metrics are captured where available;
skill regression can be represented even when trigger selection itself was correct;
equivalent run conditions are fingerprinted;
output can feed #167 evolution/lifecycle work without another adapter-specific format;
live-provider evaluation remains optional and cannot make deterministic CI depend on credentials.

Non-goals

no lifecycle promotion policy in this issue;
no mutation/crossover orchestration;
no product-specific coding benchmark corpus.

Inspect current merged truth before implementation and reuse existing evaluator/verifier/runtime APIs. Do as much coherent work as possible per cycle.

Parent: #167 Related: #56, #68-#71, #101, #168 ## Goal Measure whether loading a skill actually improves task outcomes, not merely whether the prompt compiler selected it. ## Contract Run equivalent task fixtures in paired modes: ```text baseline: skill disabled/absent variant: candidate skill active ``` Collect comparable structured results and objective evidence where available. ## Required measurements - verifier/result state; - model calls; - input/output/total tokens and cost where available; - latency; - tool calls; - repair/replan iterations; - subagent calls; - provider-visible context tokens/bytes; - failure/regression class; - trace/evidence references. ## Requirements - reuse existing completion/graph/agent/trace/usage and Spec/Verify primitives; - deterministic fixtures must work without live credentials; - optional live-provider lane must be clearly separate; - task success should use objective verifier/evidence when one exists, not model self-report; - paired runs preserve equivalent task/spec/budget conditions except for the skill variable under test; - emit machine-readable comparison records suitable for later evolution fitness; - record uncertainty when evidence cannot determine a winner. ## Acceptance - [ ] deterministic baseline-vs-skill fixture produces one comparable result record; - [ ] verifier-backed task success/failure is preserved distinctly from worker claims; - [ ] token/cost/latency/tool/subagent/repair metrics are captured where available; - [ ] skill regression can be represented even when trigger selection itself was correct; - [ ] equivalent run conditions are fingerprinted; - [ ] output can feed #167 evolution/lifecycle work without another adapter-specific format; - [ ] live-provider evaluation remains optional and cannot make deterministic CI depend on credentials. ## Non-goals - no lifecycle promotion policy in this issue; - no mutation/crossover orchestration; - no product-specific coding benchmark corpus. Inspect current merged truth before implementation and reuse existing evaluator/verifier/runtime APIs. Do as much coherent work as possible per cycle.

~~nsaspy referenced this issue 2026-09-10 21:20:45 +00:00~~

[P1] Add baseline-vs-skill outcome evaluation with verifier-backed evidence #169

nsaspy commented

2026-09-10 21:21:30 +00:00

Author

Owner

Duplicate of #169 (pre-existing Forgejo mirror). Closing this accidental duplicate created by today's open-state sync; #169 stays canonical on Forgejo.

nsaspy closed this issue

2026-09-10 21:21:30 +00:00