[RAGE-10][P0] Add deterministic end-to-end RAGE golden corpus and CI regression gate #38

New issue

Open

opened 2026-09-20 01:40:01 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-20 01:40:01 +00:00

Owner

Parent: #6
Depends on: RAGE-1 through RAGE-6; extend as RAGE-7/RAGE-8 land.

Goal

Make the refinery impossible to fake-green. Check the full downstream contract with deterministic golden evidence and separately gated live-provider experiments.

Golden scenarios

Include at minimum:

repeated operator correction becomes a project-scoped skill candidate;
one-off behavior is ignored/insufficient evidence;
repeated deterministic procedure is classified as script/tool, not skill;
secret-bearing source is redacted before candidate synthesis;
positive trigger fires and adversarial near-miss does not;
candidate improves verifier-backed outcome;
regression causes rejection and active set remains unchanged;
malformed/duplicate imported evidence is handled deterministically;
cancellation/budget exhaustion does not yield false success.

CI lanes

offline deterministic mandatory gate;
optional live-provider evaluation lane with explicit credentials/budgets;
schema/golden compatibility gate against pinned prolog-rlm;
failure artifacts retain evidence references without leaking secrets.

Acceptance

mandatory lane requires no model/network credentials;
golden outputs are stable and diffable;
intentional regression in classifier/synthesis/eval wiring fails CI;
candidate success requires verifier/eval evidence, not model self-report;
upstream contract drift fails clearly instead of silently falling back to local behavior.

Parent: #6 Depends on: RAGE-1 through RAGE-6; extend as RAGE-7/RAGE-8 land. ## Goal Make the refinery impossible to fake-green. Check the full downstream contract with deterministic golden evidence and separately gated live-provider experiments. ## Golden scenarios Include at minimum: 1. repeated operator correction becomes a project-scoped skill candidate; 2. one-off behavior is ignored/insufficient evidence; 3. repeated deterministic procedure is classified as script/tool, not skill; 4. secret-bearing source is redacted before candidate synthesis; 5. positive trigger fires and adversarial near-miss does not; 6. candidate improves verifier-backed outcome; 7. regression causes rejection and active set remains unchanged; 8. malformed/duplicate imported evidence is handled deterministically; 9. cancellation/budget exhaustion does not yield false success. ## CI lanes - offline deterministic mandatory gate; - optional live-provider evaluation lane with explicit credentials/budgets; - schema/golden compatibility gate against pinned prolog-rlm; - failure artifacts retain evidence references without leaking secrets. ## Acceptance - [ ] mandatory lane requires no model/network credentials; - [ ] golden outputs are stable and diffable; - [ ] intentional regression in classifier/synthesis/eval wiring fails CI; - [ ] candidate success requires verifier/eval evidence, not model self-report; - [ ] upstream contract drift fails clearly instead of silently falling back to local behavior.

nsaspy referenced this issue

2026-09-20 01:40:38 +00:00

[EPIC] Personal skill refinery: observe workflows, synthesize candidates, RAGE-evaluate, and import external skills #6