Add deterministic skill-selection evaluation metrics #251

Merged

lost-rob0t merged 2 commits from rage/168-skill-selection-eval into main

2026-08-27 03:47:33 +00:00

lost-rob0t commented

2026-08-26 17:31:52 +00:00

(Migrated from github.com)

Advances #168.

RAGE slice

Starts from exact canonical main 267697bef10a3fffff7c093e1435ece770e7444b after complete live backlog/transaction audit. #250 has a fresh active branch and was explicitly excluded; no matching downstream implementation ownership was found in a0-symbolics or agentProlog (AgentProlog #6 explicitly expects to consume upstream #168).

Current exact candidate head: 10cdd81032e452093c1b294537433f478c1262c3.

TDD evidence

The first authoritative deterministic run on 51e8ae90b215f56559aef11d734b2a75799f57e0 discovered 79 suites / 892 tests and failed only the new evaluator assertions. That evidence exposed real evaluator mistakes rather than desired-red CI:

selected provider-token accounting inspected hashed context-unit IDs rather than the packed selection value's skill unit, producing zero;
the evaluator's anonymous default policy introduced unstable material into the fingerprint contract;
the test expected a string fingerprint even though repository crypto_data_hash/3 fingerprints are atoms.

The tests were not weakened or skipped. The implementation now uses canonical packed-selection accounting, a ground tagged context policy, rejects nonground options, and additionally asserts underlying compiler fingerprints repeat across identical runs.

Realization

rlm_skill_eval:skill_selection_evaluate/4 is a thin deterministic measurement layer around the existing rlm_prompt_compiler; it does not add another selector/router or execute skill data.

It provides:

explicit expected/forbidden skill scoring;
TP/FP/TN/FN, trigger precision/recall, false-positive/false-negative rates;
separate correctness dimensions for lexical, explicit-only, negation, dependency, conflict and supersession behavior;
canonical compiler explanations for every expected/forbidden unit;
selected provider-visible skill token cost from the normal context_pack.selected representation;
compile latency as observational data;
stable material fingerprinting that excludes timing noise while requiring per-case compiler fingerprints to remain stable;
structured failures for contradictory fixtures, nonground case input and nonground evaluator/compiler options.

The suite is registered in the authoritative deterministic corpus. Corpus terms remain inert data: no case input is called, granted capability/authority, or executed.

Adversarial review

GO. The evaluator calls canonical prompt_compile/4 and prompt_explain/3 only. It neither mutates selection semantics nor introduces execution/authority/effect paths. Provider-visible token accounting comes from the existing bounded pack. Activation remains distinct from authority. No Agent Zero, AgentProlog, or DeepSeek-Harness product logic is introduced.

Exact-head verification

Exact head 10cdd81032e452093c1b294537433f478c1262c3 is green for:

canonical deterministic PlUnit: 79 suites / 893 discovered / 893 passed / 0 failed / 0 timeout / 0 blocked / 0 fixme;
deterministic benchmark/conformance;
deterministic deep-recursion experiment;
credential-free CLI/trace smoke;
fresh-process graph and artifact restart checks;
whitespace checks;
credential-backed REAL OpenRouter core, structured repair, benchmark, depth 0/1/2 and CLI smoke;
pinned Paid OpenRouter equivalents;
Nix flake;
clean SWI pack install;
Tree-sitter FFI.

Reviews, conversation comments and unresolved review threads were empty during the exact-head audit.

Merge gate

Engineering decision: GO. Merge remains subject to the repository's stronger explicit merge-on-green authorization requirement.

Advances #168. ## RAGE slice Starts from exact canonical `main` `267697bef10a3fffff7c093e1435ece770e7444b` after complete live backlog/transaction audit. #250 has a fresh active branch and was explicitly excluded; no matching downstream implementation ownership was found in `a0-symbolics` or `agentProlog` (AgentProlog #6 explicitly expects to consume upstream #168). Current exact candidate head: `10cdd81032e452093c1b294537433f478c1262c3`. ## TDD evidence The first authoritative deterministic run on `51e8ae90b215f56559aef11d734b2a75799f57e0` discovered 79 suites / 892 tests and failed only the new evaluator assertions. That evidence exposed real evaluator mistakes rather than desired-red CI: - selected provider-token accounting inspected hashed context-unit IDs rather than the packed selection value's skill unit, producing zero; - the evaluator's anonymous default policy introduced unstable material into the fingerprint contract; - the test expected a string fingerprint even though repository `crypto_data_hash/3` fingerprints are atoms. The tests were not weakened or skipped. The implementation now uses canonical packed-selection accounting, a ground tagged context policy, rejects nonground options, and additionally asserts underlying compiler fingerprints repeat across identical runs. ## Realization `rlm_skill_eval:skill_selection_evaluate/4` is a thin deterministic measurement layer around the existing `rlm_prompt_compiler`; it does not add another selector/router or execute skill data. It provides: - explicit expected/forbidden skill scoring; - TP/FP/TN/FN, trigger precision/recall, false-positive/false-negative rates; - separate correctness dimensions for lexical, explicit-only, negation, dependency, conflict and supersession behavior; - canonical compiler explanations for every expected/forbidden unit; - selected provider-visible skill token cost from the normal `context_pack.selected` representation; - compile latency as observational data; - stable material fingerprinting that excludes timing noise while requiring per-case compiler fingerprints to remain stable; - structured failures for contradictory fixtures, nonground case input and nonground evaluator/compiler options. The suite is registered in the authoritative deterministic corpus. Corpus terms remain inert data: no case input is called, granted capability/authority, or executed. ## Adversarial review GO. The evaluator calls canonical `prompt_compile/4` and `prompt_explain/3` only. It neither mutates selection semantics nor introduces execution/authority/effect paths. Provider-visible token accounting comes from the existing bounded pack. Activation remains distinct from authority. No Agent Zero, AgentProlog, or DeepSeek-Harness product logic is introduced. ## Exact-head verification Exact head `10cdd81032e452093c1b294537433f478c1262c3` is green for: - canonical deterministic PlUnit: **79 suites / 893 discovered / 893 passed / 0 failed / 0 timeout / 0 blocked / 0 fixme**; - deterministic benchmark/conformance; - deterministic deep-recursion experiment; - credential-free CLI/trace smoke; - fresh-process graph and artifact restart checks; - whitespace checks; - credential-backed REAL OpenRouter core, structured repair, benchmark, depth 0/1/2 and CLI smoke; - pinned Paid OpenRouter equivalents; - Nix flake; - clean SWI pack install; - Tree-sitter FFI. Reviews, conversation comments and unresolved review threads were empty during the exact-head audit. ## Merge gate Engineering decision: GO. Merge remains subject to the repository's stronger explicit merge-on-green authorization requirement.