[RAGE-6][P0] Orchestrate candidate RAGE evaluation through upstream selection, outcome, and reviewer contracts #42

New issue

Open

opened 2026-09-20 01:40:01 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-20 01:40:01 +00:00

Owner

Parent: #6
Depends on: RAGE-5
Consumes:

Goal

Build the AgentProlog product orchestration that evaluates a candidate against real/product fixtures while reusing upstream scoring, task-eval, verifier, trace, usage, and bounded reviewer/subagent semantics.

Flow

candidate
-> selection corpus eval
-> baseline vs candidate task eval
-> adversarial/reviewer pass
-> aggregate evidence references
-> downstream recommendation record

AgentProlog chooses which product/workflow fixtures to run and how to present the evidence. It must not implement a second generic scoring engine.

Measurements surfaced

trigger precision/recall and FP/FN;
verifier-backed task result;
regressions;
model/tool/subagent calls;
tokens/cost/latency;
repair/replan count;
trace/evidence references;
uncertainty/inconclusive state.

Acceptance

one candidate runs through upstream-compatible selection + outcome records;
reviewer role uses canonical bounded delegation, not a local agent loop;
baseline and candidate runs preserve equivalent task/spec/budget conditions;
inconclusive evidence cannot masquerade as promotion-ready;
all results link back to candidate + source evidence + eval fixtures;
deterministic CI lane works without network credentials.

Parent: #6 Depends on: RAGE-5 Consumes: - lost-rob0t/prolog-rlm#168 - lost-rob0t/prolog-rlm#169 - lost-rob0t/prolog-rlm#172 ## Goal Build the AgentProlog product orchestration that evaluates a candidate against real/product fixtures while reusing upstream scoring, task-eval, verifier, trace, usage, and bounded reviewer/subagent semantics. ## Flow ```text candidate -> selection corpus eval -> baseline vs candidate task eval -> adversarial/reviewer pass -> aggregate evidence references -> downstream recommendation record ``` AgentProlog chooses which product/workflow fixtures to run and how to present the evidence. It must not implement a second generic scoring engine. ## Measurements surfaced - trigger precision/recall and FP/FN; - verifier-backed task result; - regressions; - model/tool/subagent calls; - tokens/cost/latency; - repair/replan count; - trace/evidence references; - uncertainty/inconclusive state. ## Acceptance - [ ] one candidate runs through upstream-compatible selection + outcome records; - [ ] reviewer role uses canonical bounded delegation, not a local agent loop; - [ ] baseline and candidate runs preserve equivalent task/spec/budget conditions; - [ ] inconclusive evidence cannot masquerade as promotion-ready; - [ ] all results link back to candidate + source evidence + eval fixtures; - [ ] deterministic CI lane works without network credentials.

nsaspy referenced this issue

2026-09-20 01:40:38 +00:00

[EPIC] Personal skill refinery: observe workflows, synthesize candidates, RAGE-evaluate, and import external skills #6