[P1] Add deterministic skill-selection eval corpus and trigger metrics #443

New issue

Closed

opened 2026-09-10 21:18:17 +00:00 by nsaspy · 1 comment

nsaspy commented

2026-09-10 21:18:17 +00:00

Owner

Parent: #167
Related: #117, #101

Goal

Make rlm_prompt_compiler skill activation quality measurable instead of inferred from a handful of hand-written examples.

This issue evaluates selection, not downstream task success.

Runtime contract

Given a deterministic fixture containing:

request + trusted signals + available skill catalog
-> expected applicable skills
-> expected forbidden/non-applicable skills
-> compiler selection/rejection explanation

produce structured metrics without requiring a live model provider.

Required metrics

trigger precision;
trigger recall;
false-positive rate;
false-negative rate;
explicit-only correctness;
negation correctness;
dependency closure correctness;
conflict/supersession correctness;
selected provider-visible token cost;
compile latency;
deterministic fingerprint stability.

Requirements

reuse rlm_prompt_compiler; no second router;
corpus format is closed declarative data and suitable for checked-in deterministic fixtures;
expected selections and forbidden selections are explicit;
explanations preserve enough provenance to diagnose why a case fired or did not fire;
support multiple equivalent phrasings and adversarial near-misses;
lexical/explicit deterministic selection remains the baseline;
any future semantic/learned ranking must be benchmarked against the same corpus rather than silently replacing the baseline;
provider-visible token accounting uses the normal context ledger.

Acceptance

checked-in corpus can score a skill catalog without network credentials;
precision/recall and FP/FN counts are produced deterministically;
adversarial near-match cases prove unrelated skills remain inactive;
explicit-only and negated skills cannot count as successful accidental activations;
dependency/conflict/supersession cases are scored separately from lexical triggering;
compiler explanations identify the evidence used for every selected/rejected skill;
repeated runs over identical inputs produce identical metrics/fingerprints;
aggregate repository tests remain authoritative.

Non-goals

no task outcome grading;
no live-provider requirement;
no embeddings/LLM router required for this slice;
no skill mutation/promotion.

Work autonomously and inspect current main, #167/#117, tests, compiler code, and CI before each slice. Do as much coherent work as possible rather than stopping at one trivial fixture.

Parent: #167 Related: #117, #101 ## Goal Make `rlm_prompt_compiler` skill activation quality measurable instead of inferred from a handful of hand-written examples. This issue evaluates **selection**, not downstream task success. ## Runtime contract Given a deterministic fixture containing: ```text request + trusted signals + available skill catalog -> expected applicable skills -> expected forbidden/non-applicable skills -> compiler selection/rejection explanation ``` produce structured metrics without requiring a live model provider. ## Required metrics - trigger precision; - trigger recall; - false-positive rate; - false-negative rate; - explicit-only correctness; - negation correctness; - dependency closure correctness; - conflict/supersession correctness; - selected provider-visible token cost; - compile latency; - deterministic fingerprint stability. ## Requirements - reuse `rlm_prompt_compiler`; no second router; - corpus format is closed declarative data and suitable for checked-in deterministic fixtures; - expected selections and forbidden selections are explicit; - explanations preserve enough provenance to diagnose why a case fired or did not fire; - support multiple equivalent phrasings and adversarial near-misses; - lexical/explicit deterministic selection remains the baseline; - any future semantic/learned ranking must be benchmarked against the same corpus rather than silently replacing the baseline; - provider-visible token accounting uses the normal context ledger. ## Acceptance - [ ] checked-in corpus can score a skill catalog without network credentials; - [ ] precision/recall and FP/FN counts are produced deterministically; - [ ] adversarial near-match cases prove unrelated skills remain inactive; - [ ] explicit-only and negated skills cannot count as successful accidental activations; - [ ] dependency/conflict/supersession cases are scored separately from lexical triggering; - [ ] compiler explanations identify the evidence used for every selected/rejected skill; - [ ] repeated runs over identical inputs produce identical metrics/fingerprints; - [ ] aggregate repository tests remain authoritative. ## Non-goals - no task outcome grading; - no live-provider requirement; - no embeddings/LLM router required for this slice; - no skill mutation/promotion. Work autonomously and inspect current `main`, #167/#117, tests, compiler code, and CI before each slice. Do as much coherent work as possible rather than stopping at one trivial fixture.

~~nsaspy referenced this issue 2026-09-10 21:20:44 +00:00~~

[P1] Add deterministic skill-selection eval corpus and trigger metrics #168

nsaspy commented

2026-09-10 21:21:29 +00:00

Author

Owner

Duplicate of #168 (pre-existing Forgejo mirror). Closing this accidental duplicate created by today's open-state sync; #168 stays canonical on Forgejo.

nsaspy closed this issue

2026-09-10 21:21:29 +00:00