[P1] Add deterministic skill-selection eval corpus and trigger metrics #168

New issue

Open

opened 2026-08-22 01:52:16 +00:00 by lost-rob0t · 1 comment

lost-rob0t commented

2026-08-22 01:52:16 +00:00

(Migrated from github.com)

Parent: #167
Related: #117, #101

Goal

Make rlm_prompt_compiler skill activation quality measurable instead of inferred from a handful of hand-written examples.

This issue evaluates selection, not downstream task success.

Runtime contract

Given a deterministic fixture containing:

request + trusted signals + available skill catalog
-> expected applicable skills
-> expected forbidden/non-applicable skills
-> compiler selection/rejection explanation

produce structured metrics without requiring a live model provider.

Required metrics

trigger precision;
trigger recall;
false-positive rate;
false-negative rate;
explicit-only correctness;
negation correctness;
dependency closure correctness;
conflict/supersession correctness;
selected provider-visible token cost;
compile latency;
deterministic fingerprint stability.

Requirements

reuse rlm_prompt_compiler; no second router;
corpus format is closed declarative data and suitable for checked-in deterministic fixtures;
expected selections and forbidden selections are explicit;
explanations preserve enough provenance to diagnose why a case fired or did not fire;
support multiple equivalent phrasings and adversarial near-misses;
lexical/explicit deterministic selection remains the baseline;
any future semantic/learned ranking must be benchmarked against the same corpus rather than silently replacing the baseline;
provider-visible token accounting uses the normal context ledger.

Acceptance

checked-in corpus can score a skill catalog without network credentials;
precision/recall and FP/FN counts are produced deterministically;
adversarial near-match cases prove unrelated skills remain inactive;
explicit-only and negated skills cannot count as successful accidental activations;
dependency/conflict/supersession cases are scored separately from lexical triggering;
compiler explanations identify the evidence used for every selected/rejected skill;
repeated runs over identical inputs produce identical metrics/fingerprints;
aggregate repository tests remain authoritative.

Non-goals

no task outcome grading;
no live-provider requirement;
no embeddings/LLM router required for this slice;
no skill mutation/promotion.

Work autonomously and inspect current main, #167/#117, tests, compiler code, and CI before each slice. Do as much coherent work as possible rather than stopping at one trivial fixture.

Parent: #167 Related: #117, #101 ## Goal Make `rlm_prompt_compiler` skill activation quality measurable instead of inferred from a handful of hand-written examples. This issue evaluates **selection**, not downstream task success. ## Runtime contract Given a deterministic fixture containing: ```text request + trusted signals + available skill catalog -> expected applicable skills -> expected forbidden/non-applicable skills -> compiler selection/rejection explanation ``` produce structured metrics without requiring a live model provider. ## Required metrics - trigger precision; - trigger recall; - false-positive rate; - false-negative rate; - explicit-only correctness; - negation correctness; - dependency closure correctness; - conflict/supersession correctness; - selected provider-visible token cost; - compile latency; - deterministic fingerprint stability. ## Requirements - reuse `rlm_prompt_compiler`; no second router; - corpus format is closed declarative data and suitable for checked-in deterministic fixtures; - expected selections and forbidden selections are explicit; - explanations preserve enough provenance to diagnose why a case fired or did not fire; - support multiple equivalent phrasings and adversarial near-misses; - lexical/explicit deterministic selection remains the baseline; - any future semantic/learned ranking must be benchmarked against the same corpus rather than silently replacing the baseline; - provider-visible token accounting uses the normal context ledger. ## Acceptance - [ ] checked-in corpus can score a skill catalog without network credentials; - [ ] precision/recall and FP/FN counts are produced deterministically; - [ ] adversarial near-match cases prove unrelated skills remain inactive; - [ ] explicit-only and negated skills cannot count as successful accidental activations; - [ ] dependency/conflict/supersession cases are scored separately from lexical triggering; - [ ] compiler explanations identify the evidence used for every selected/rejected skill; - [ ] repeated runs over identical inputs produce identical metrics/fingerprints; - [ ] aggregate repository tests remain authoritative. ## Non-goals - no task outcome grading; - no live-provider requirement; - no embeddings/LLM router required for this slice; - no skill mutation/promotion. Work autonomously and inspect current `main`, #167/#117, tests, compiler code, and CI before each slice. Do as much coherent work as possible rather than stopping at one trivial fixture.

lost-rob0t commented

2026-08-26 17:41:40 +00:00

(Migrated from github.com)

RAGE realization/verification update for #168.

Exact starting main was 267697bef10a3fffff7c093e1435ece770e7444b. I excluded fresh active #250 (rage/250-skill-catalog-graph) and found no competing downstream implementation; AgentProlog #6 explicitly expects to consume upstream #168.

TDD first exposed two evaluator defects on exact head 51e8ae90b215f56559aef11d734b2a75799f57e0: aggregate PlUnit discovered 79 suites / 892 tests and failed only the new evaluator assertions. Provider-visible token cost was incorrectly filtering on hashed context-unit IDs, so it reported zero; the default anonymous context-policy dict also made fingerprint material unsuitable for the demanded repeatability contract. The fingerprint representation assertion additionally expected a string even though the repository's crypto_data_hash/3 convention returns an atom. These failures were preserved and fixed rather than weakened.

Realization on current exact head 10cdd81032e452093c1b294537433f478c1262c3:

adds rlm_skill_eval:skill_selection_evaluate/4 as a thin measurement layer around canonical rlm_prompt_compiler; no second selector/router;
scores explicit expected/forbidden skills with TP/FP/TN/FN, precision/recall, FP/FN rates and per-dimension correctness for lexical, explicit-only, negation, dependency, conflict and supersession cases;
captures compiler explanations for every expected/forbidden unit;
charges selected skill tokens from the canonical context_pack.selected representation by inspecting the selected value's unit, rather than guessing from hashed context IDs;
records compile latency as observational data but excludes timing from the stable eval fingerprint;
uses a ground tagged default context_policy{...} and rejects nonground eval/compile options, so both the aggregate eval fingerprint and the underlying per-case compiler fingerprints are required to repeat;
keeps fixtures as inert ground data and never calls, activates, authorizes or executes corpus content;
registers the suite in the authoritative deterministic corpus.

Adversarial review: GO. The evaluator only invokes prompt_compile/4 and prompt_explain/3; it neither mutates the catalog nor grants capability/authority/effects. Token accounting comes from the normal packed provider-visible selection. Negative fixture validation is structured failure while CI stays green. No downstream product policy is introduced.

Exact-head verification for 10cdd81032e452093c1b294537433f478c1262c3 is green: canonical deterministic CI, benchmark/conformance, deep recursion, CLI/trace, graph/artifact restart and whitespace; REAL OpenRouter; Paid OpenRouter; Nix flake; clean SWI pack install; Tree-sitter FFI. Aggregate PlUnit: 79 suites / 893 discovered / 893 passed / 0 failed / 0 timeout / 0 blocked / 0 fixme.

This slice satisfies the #168 selection-evaluation contract and is ready for review at PR #251. Merge remains subject to the repository's explicit merge-on-green authorization gate.

RAGE realization/verification update for #168. Exact starting main was `267697bef10a3fffff7c093e1435ece770e7444b`. I excluded fresh active #250 (`rage/250-skill-catalog-graph`) and found no competing downstream implementation; AgentProlog #6 explicitly expects to consume upstream #168. TDD first exposed two evaluator defects on exact head `51e8ae90b215f56559aef11d734b2a75799f57e0`: aggregate PlUnit discovered 79 suites / 892 tests and failed only the new evaluator assertions. Provider-visible token cost was incorrectly filtering on hashed context-unit IDs, so it reported zero; the default anonymous context-policy dict also made fingerprint material unsuitable for the demanded repeatability contract. The fingerprint representation assertion additionally expected a string even though the repository's `crypto_data_hash/3` convention returns an atom. These failures were preserved and fixed rather than weakened. Realization on current exact head `10cdd81032e452093c1b294537433f478c1262c3`: - adds `rlm_skill_eval:skill_selection_evaluate/4` as a thin measurement layer around canonical `rlm_prompt_compiler`; no second selector/router; - scores explicit expected/forbidden skills with TP/FP/TN/FN, precision/recall, FP/FN rates and per-dimension correctness for lexical, explicit-only, negation, dependency, conflict and supersession cases; - captures compiler explanations for every expected/forbidden unit; - charges selected skill tokens from the canonical `context_pack.selected` representation by inspecting the selected value's `unit`, rather than guessing from hashed context IDs; - records compile latency as observational data but excludes timing from the stable eval fingerprint; - uses a ground tagged default `context_policy{...}` and rejects nonground eval/compile options, so both the aggregate eval fingerprint and the underlying per-case compiler fingerprints are required to repeat; - keeps fixtures as inert ground data and never calls, activates, authorizes or executes corpus content; - registers the suite in the authoritative deterministic corpus. Adversarial review: GO. The evaluator only invokes `prompt_compile/4` and `prompt_explain/3`; it neither mutates the catalog nor grants capability/authority/effects. Token accounting comes from the normal packed provider-visible selection. Negative fixture validation is structured failure while CI stays green. No downstream product policy is introduced. Exact-head verification for `10cdd81032e452093c1b294537433f478c1262c3` is green: canonical deterministic CI, benchmark/conformance, deep recursion, CLI/trace, graph/artifact restart and whitespace; REAL OpenRouter; Paid OpenRouter; Nix flake; clean SWI pack install; Tree-sitter FFI. Aggregate PlUnit: **79 suites / 893 discovered / 893 passed / 0 failed / 0 timeout / 0 blocked / 0 fixme**. This slice satisfies the #168 selection-evaluation contract and is ready for review at PR #251. Merge remains subject to the repository's explicit merge-on-green authorization gate.