nsaspy/prolog-rlm

Fork 0

[EPIC] SKILLS revamp: first-class Agent Skills, evals, evolution, and skill-aware delegation #167

New issue

Open

opened 2026-08-22 01:51:26 +00:00 by lost-rob0t · 1 comment

lost-rob0t commented

2026-08-22 01:51:26 +00:00

(Migrated from github.com)

Mission

Turn the existing prolog-rlm prompt compiler, context budgeting, subagent, Spec/Verify, artifact, tracing, and evolution primitives into a first-class Agent Skills runtime.

Do not invent a parallel skill engine. Reuse the current compiler/runtime contracts and make skills a coherent, testable layer over them.

The target shape is:

Agent Skills packages / trusted host skill sources
                |
                v
        safe skill loader
                |
                v
      rlm_prompt_compiler
   select / reject / explain
                |
        +-------+--------+
        |                |
        v                v
 provider context    skill-aware
 progressive load    delegation
        |                |
        +-------+--------+
                v
          execution
                |
                v
        traces + evidence
                |
                v
          skill evals
                |
                v
       rlm_evolution
                |
                v
     candidate lifecycle

This epic is about the generic runtime. Product-specific observation/mining of a user's work across Claude/OpenCode/Agent Zero/AgentProlog belongs downstream in AgentProlog and should consume the public contracts created here.

Existing foundations — reuse them

Current core already has substantial relevant machinery:

rlm_prompt_compiler with bounded catalog units, triggers/signals, dependencies, suggestions, conflicts, supersession, capability filtering, provenance, explanations, recompilation and context-budget packing;
first-class skills grouping in compiled prompt/context projections;
rlm_context_budget for provider-visible hard budgeting;
rlm_subagent for bounded supervised RLM child delegation through normal tool/capability boundaries;
rlm_agent supervision, capability narrowing, cancellation and typed outcomes;
Spec/Verify and evidence/provenance primitives;
durable artifacts and traces;
merged pure generic rlm_evolution candidate/mutation/crossover/Pareto-selection kernel from PR #148.

The revamp must extend those pieces rather than clone them.

#117 — Prolog-owned automatic SKILL.md activation. Treat this as the loader + deterministic activation child slice, not a competing architecture.
#144 / merged subagent work — bounded RLM subagent fallback and prompt command binding.
#101 — managed conversation/context budgeting.
#126 — programmable trusted configuration/hooks/tools/detectors.
#68-#71 — compiler-style verified workflow/context pipeline.
#141 — standalone AgentProlog consumes prolog-rlm as the reusable core runtime.
PR #148 — pure generic evolution kernel. Evolution execution/evaluation/promotion remains deliberately outside that merged kernel.

Core invariants

Prolog owns selection and execution semantics. The model does not receive a flat catalog and guess which skills to load when deterministic/runtime selection can decide first.
Skill activation is not authority. A skill cannot grant tool capability, widen host authority, install an MCP server, or bypass effect policy.
Skill bodies/resources are inert data unless they refer to already registered trusted runtime operations through typed/allow-listed contracts.
No arbitrary skill metadata/body becomes call/1 or trusted executable Prolog.
Provider-visible skill material participates in the existing hard context/token budget.
Selection, rejection, evaluation, evolution and promotion decisions are explainable and traceable.
Generic runtime remains domain-neutral. Coding/product/user-observation policy belongs downstream.

Track A — Agent Skills package compatibility

Complete/reconcile #117 into a real package boundary.

Required capabilities:

load trusted SKILL.md package roots safely;
parse supported front matter/discovery metadata;
preserve Markdown body as inert instruction content;
support explicit-user-only / disable-model-invocation-style semantics where representable;
stable package identity, version/fingerprint and provenance;
confined package-relative resources/references/scripts metadata with path traversal rejection;
lazy/progressive loading: metadata first, body after selection, referenced resources only when needed;
deterministic duplicate/conflict/supersession handling;
compatible empty-catalog path for existing callers.

Do not bind core permanently to one vendor's private extensions. Define a small canonical internal skill representation with adapters where formats differ.

Track B — Skill selection quality

The current prompt compiler can represent skill selection, but selection quality must become measurable rather than assumed.

Add a domain-neutral evaluation contract for:

request/signals/context
-> expected applicable skills / forbidden skills
-> compiler selection
-> selected/rejected explanation
-> metrics

Measure at least:

trigger precision;
trigger recall;
false positive activation;
false negative activation;
dependency closure correctness;
negation/explicit-only correctness;
provider-visible tokens added;
compile latency;
stable deterministic fingerprinting.

Keep lexical/explicit deterministic selection available. More advanced ranking may be added only behind explicit bounded contracts and must not silently replace explainable host selection.

Track C — Skill outcome evaluation

Selection accuracy alone is insufficient. Add a reusable skill-eval substrate capable of comparing equivalent tasks:

baseline: task without candidate skill
variant:  task with candidate skill

Capture structured evidence for:

task success / verifier state;
model calls;
tokens and cost;
latency;
tool calls;
repair iterations;
subagent use;
context bytes/tokens;
regression/failure class.

Where an objective verifier exists, use Spec/Verify/evidence primitives rather than worker self-report.

The eval API must work with deterministic fixtures and optionally live providers without making live provider credentials a unit-test requirement.

Track D — Skill lifecycle

Add a first-class generic lifecycle for skill candidates without turning Markdown files into mutable magic state.

Target states should cover semantics equivalent to:

draft/candidate
-> validated
-> evaluated
-> active/promoted
-> superseded/rejected/disabled

Requirements:

immutable candidate identity/fingerprint;
provenance and parent lineage;
evaluation evidence references;
explicit promotion policy;
rollback/supersession without losing history;
no model self-promotion without trusted policy;
deterministic active-version resolution.

Reuse artifacts/scoped state/effect boundaries where appropriate instead of creating another persistence subsystem.

Track E — Evolution harness around `rlm_evolution`

rlm_evolution is intentionally only a pure closed-data kernel. Keep it that way.

Add a separate orchestration/evaluation layer that can:

construct a trusted genotype/schema for a skill experiment;
generate/mutate/crossover candidate configurations through rlm_evolution;
materialize candidate skill variants through a bounded trusted renderer;
run deterministic/live eval suites;
collect vector fitness;
select candidates with existing Pareto selection;
retain lineage/evidence;
propose promotion through trusted lifecycle policy.

Do not add arbitrary model-weight evolution or unrestricted generated code execution to core.

A model may propose instruction text/variant material through a bounded provider call, but Prolog owns candidate identity, allowed mutation surface, evaluator invocation, budgets, evidence and promotion.

Track F — Skill-aware subagent delegation

Build on the existing rlm_subagent tool instead of another child runtime.

Add a generic way for an unresolved task to identify that a selected skill/role is better handled by a bounded child, for example semantics equivalent to:

unresolved need
-> compiler resolves applicable skill/role
-> canonical rlm_subagent invocation
-> narrowed child capabilities/authority
-> selected skill context
-> structured result/evidence/usage/trace
-> parent resume/replan

Requirements:

skill/role selection cannot widen capability or authority;
child receives only the bounded selected context it needs;
parent/child correlation is explicit;
evidence/usage propagate through the result envelope;
recursive delegation remains globally budgeted;
critic/reviewer roles remain generic, not hardcoded personas.

Track G — Progressive disclosure and context economy

Prove that the skill system reduces token churn instead of merely adding another prompt layer.

Support and benchmark:

catalog metadata
  -> selected skill metadata
  -> selected skill body
  -> referenced resources/examples only on demand

All provider-visible stages must be charged through the normal context ledger. Mandatory skill overflow must fail structurally rather than silently exceed the provider/model hard cap.

Downstream boundary: AgentProlog

Do not put personal workflow mining or harness-specific session importers in core.

Standalone AgentProlog should later consume these APIs to:

observe real work across AgentProlog/OpenCode/Claude/Agent Zero adapters;
detect repeated workflows and failures;
synthesize candidate skills;
build representative eval corpora;
invoke the generic skill eval/evolution runtime;
export/install compatible skill packages.

Runtime/library flow is:

prolog-rlm -> AgentProlog

In dependency terms, AgentProlog depends on and consumes prolog-rlm.

Acceptance criteria

#117 is reconciled as the canonical safe Agent Skills loader/activation slice.
A trusted SKILL.md package can be loaded, identified, selected and lazily injected without granting authority.
Skill resources cannot escape the trusted package root.
Compiler explanations record why skills were selected/rejected and remain fingerprint-stable.
Explicit-only/negated/conflicting/dependent skill cases have deterministic tests.
Selection precision/recall can be measured over a fixture corpus.
Equivalent baseline-vs-skill task runs produce structured comparable metrics/evidence.
Objective verification is used where available instead of self-reported success.
Skill candidates have immutable identity, provenance, lifecycle state and rollback/supersession semantics.
Evolution orchestration composes with rlm_evolution without adding scheduler/effect/provider logic to the pure kernel.
At least one deterministic mutation/evaluate/select/promote-or-reject cycle is demonstrated end-to-end.
Skill-aware delegation uses the existing supervised RLM subagent boundary with narrowed authority/capabilities.
Provider-visible skill material is fully charged to the existing context budget.
Existing no-skill callers remain compatible.
No arbitrary skill content becomes executable Prolog or bypasses normal tool/authority/effect boundaries.

Non-goals

No second prompt compiler.
No second agent/subagent runtime.
No second verifier/evidence stack.
No universal model-generated plugin ABI.
No arbitrary call/1 from skill content.
No capability or authority grants from skill activation.
No product-specific coding skills in core.
No personal activity/session mining inside prolog-rlm.
No automatic promotion based solely on model preference.

Work contract

Before each implementation slice, inspect current main, related issues/PRs/commits, source, tests, docs/research and CI. Merged executable truth outranks stale issue prose. Reuse existing runtime contracts first; create focused child issues for coherent missing generic surfaces. Use TDD and keep the aggregate repository test gate authoritative. Do as much coherent work as possible per cycle rather than stopping after one trivial checkbox.

## Mission Turn the existing `prolog-rlm` prompt compiler, context budgeting, subagent, Spec/Verify, artifact, tracing, and evolution primitives into a **first-class Agent Skills runtime**. Do **not** invent a parallel skill engine. Reuse the current compiler/runtime contracts and make skills a coherent, testable layer over them. The target shape is: ```text Agent Skills packages / trusted host skill sources | v safe skill loader | v rlm_prompt_compiler select / reject / explain | +-------+--------+ | | v v provider context skill-aware progressive load delegation | | +-------+--------+ v execution | v traces + evidence | v skill evals | v rlm_evolution | v candidate lifecycle ``` This epic is about the **generic runtime**. Product-specific observation/mining of a user's work across Claude/OpenCode/Agent Zero/AgentProlog belongs downstream in AgentProlog and should consume the public contracts created here. ## Existing foundations — reuse them Current core already has substantial relevant machinery: - `rlm_prompt_compiler` with bounded catalog units, triggers/signals, dependencies, suggestions, conflicts, supersession, capability filtering, provenance, explanations, recompilation and context-budget packing; - first-class `skills` grouping in compiled prompt/context projections; - `rlm_context_budget` for provider-visible hard budgeting; - `rlm_subagent` for bounded supervised RLM child delegation through normal tool/capability boundaries; - `rlm_agent` supervision, capability narrowing, cancellation and typed outcomes; - Spec/Verify and evidence/provenance primitives; - durable artifacts and traces; - merged pure generic `rlm_evolution` candidate/mutation/crossover/Pareto-selection kernel from PR #148. The revamp must extend those pieces rather than clone them. ## Related existing work - #117 — Prolog-owned automatic `SKILL.md` activation. Treat this as the **loader + deterministic activation child slice**, not a competing architecture. - #144 / merged subagent work — bounded RLM subagent fallback and prompt command binding. - #101 — managed conversation/context budgeting. - #126 — programmable trusted configuration/hooks/tools/detectors. - #68-#71 — compiler-style verified workflow/context pipeline. - #141 — standalone AgentProlog consumes `prolog-rlm` as the reusable core runtime. - PR #148 — pure generic evolution kernel. Evolution execution/evaluation/promotion remains deliberately outside that merged kernel. ## Core invariants 1. **Prolog owns selection and execution semantics.** The model does not receive a flat catalog and guess which skills to load when deterministic/runtime selection can decide first. 2. Skill activation is **not authority**. A skill cannot grant tool capability, widen host authority, install an MCP server, or bypass effect policy. 3. Skill bodies/resources are inert data unless they refer to already registered trusted runtime operations through typed/allow-listed contracts. 4. No arbitrary skill metadata/body becomes `call/1` or trusted executable Prolog. 5. Provider-visible skill material participates in the existing hard context/token budget. 6. Selection, rejection, evaluation, evolution and promotion decisions are explainable and traceable. 7. Generic runtime remains domain-neutral. Coding/product/user-observation policy belongs downstream. ## Track A — Agent Skills package compatibility Complete/reconcile #117 into a real package boundary. Required capabilities: - load trusted `SKILL.md` package roots safely; - parse supported front matter/discovery metadata; - preserve Markdown body as inert instruction content; - support explicit-user-only / disable-model-invocation-style semantics where representable; - stable package identity, version/fingerprint and provenance; - confined package-relative resources/references/scripts metadata with path traversal rejection; - lazy/progressive loading: metadata first, body after selection, referenced resources only when needed; - deterministic duplicate/conflict/supersession handling; - compatible empty-catalog path for existing callers. Do not bind core permanently to one vendor's private extensions. Define a small canonical internal skill representation with adapters where formats differ. ## Track B — Skill selection quality The current prompt compiler can represent skill selection, but selection quality must become measurable rather than assumed. Add a domain-neutral evaluation contract for: ```text request/signals/context -> expected applicable skills / forbidden skills -> compiler selection -> selected/rejected explanation -> metrics ``` Measure at least: - trigger precision; - trigger recall; - false positive activation; - false negative activation; - dependency closure correctness; - negation/explicit-only correctness; - provider-visible tokens added; - compile latency; - stable deterministic fingerprinting. Keep lexical/explicit deterministic selection available. More advanced ranking may be added only behind explicit bounded contracts and must not silently replace explainable host selection. ## Track C — Skill outcome evaluation Selection accuracy alone is insufficient. Add a reusable skill-eval substrate capable of comparing equivalent tasks: ```text baseline: task without candidate skill variant: task with candidate skill ``` Capture structured evidence for: - task success / verifier state; - model calls; - tokens and cost; - latency; - tool calls; - repair iterations; - subagent use; - context bytes/tokens; - regression/failure class. Where an objective verifier exists, use Spec/Verify/evidence primitives rather than worker self-report. The eval API must work with deterministic fixtures and optionally live providers without making live provider credentials a unit-test requirement. ## Track D — Skill lifecycle Add a first-class generic lifecycle for skill candidates without turning Markdown files into mutable magic state. Target states should cover semantics equivalent to: ```text draft/candidate -> validated -> evaluated -> active/promoted -> superseded/rejected/disabled ``` Requirements: - immutable candidate identity/fingerprint; - provenance and parent lineage; - evaluation evidence references; - explicit promotion policy; - rollback/supersession without losing history; - no model self-promotion without trusted policy; - deterministic active-version resolution. Reuse artifacts/scoped state/effect boundaries where appropriate instead of creating another persistence subsystem. ## Track E — Evolution harness around `rlm_evolution` `rlm_evolution` is intentionally only a pure closed-data kernel. Keep it that way. Add a separate orchestration/evaluation layer that can: 1. construct a trusted genotype/schema for a skill experiment; 2. generate/mutate/crossover candidate configurations through `rlm_evolution`; 3. materialize candidate skill variants through a bounded trusted renderer; 4. run deterministic/live eval suites; 5. collect vector fitness; 6. select candidates with existing Pareto selection; 7. retain lineage/evidence; 8. propose promotion through trusted lifecycle policy. Do not add arbitrary model-weight evolution or unrestricted generated code execution to core. A model may propose instruction text/variant material through a bounded provider call, but Prolog owns candidate identity, allowed mutation surface, evaluator invocation, budgets, evidence and promotion. ## Track F — Skill-aware subagent delegation Build on the existing `rlm_subagent` tool instead of another child runtime. Add a generic way for an unresolved task to identify that a selected skill/role is better handled by a bounded child, for example semantics equivalent to: ```text unresolved need -> compiler resolves applicable skill/role -> canonical rlm_subagent invocation -> narrowed child capabilities/authority -> selected skill context -> structured result/evidence/usage/trace -> parent resume/replan ``` Requirements: - skill/role selection cannot widen capability or authority; - child receives only the bounded selected context it needs; - parent/child correlation is explicit; - evidence/usage propagate through the result envelope; - recursive delegation remains globally budgeted; - critic/reviewer roles remain generic, not hardcoded personas. ## Track G — Progressive disclosure and context economy Prove that the skill system reduces token churn instead of merely adding another prompt layer. Support and benchmark: ```text catalog metadata -> selected skill metadata -> selected skill body -> referenced resources/examples only on demand ``` All provider-visible stages must be charged through the normal context ledger. Mandatory skill overflow must fail structurally rather than silently exceed the provider/model hard cap. ## Downstream boundary: AgentProlog Do not put personal workflow mining or harness-specific session importers in core. Standalone AgentProlog should later consume these APIs to: - observe real work across AgentProlog/OpenCode/Claude/Agent Zero adapters; - detect repeated workflows and failures; - synthesize candidate skills; - build representative eval corpora; - invoke the generic skill eval/evolution runtime; - export/install compatible skill packages. Runtime/library flow is: ```text prolog-rlm -> AgentProlog ``` In dependency terms, AgentProlog depends on and consumes `prolog-rlm`. ## Acceptance criteria - [ ] #117 is reconciled as the canonical safe Agent Skills loader/activation slice. - [ ] A trusted `SKILL.md` package can be loaded, identified, selected and lazily injected without granting authority. - [ ] Skill resources cannot escape the trusted package root. - [ ] Compiler explanations record why skills were selected/rejected and remain fingerprint-stable. - [ ] Explicit-only/negated/conflicting/dependent skill cases have deterministic tests. - [ ] Selection precision/recall can be measured over a fixture corpus. - [ ] Equivalent baseline-vs-skill task runs produce structured comparable metrics/evidence. - [ ] Objective verification is used where available instead of self-reported success. - [ ] Skill candidates have immutable identity, provenance, lifecycle state and rollback/supersession semantics. - [ ] Evolution orchestration composes with `rlm_evolution` without adding scheduler/effect/provider logic to the pure kernel. - [ ] At least one deterministic mutation/evaluate/select/promote-or-reject cycle is demonstrated end-to-end. - [ ] Skill-aware delegation uses the existing supervised RLM subagent boundary with narrowed authority/capabilities. - [ ] Provider-visible skill material is fully charged to the existing context budget. - [ ] Existing no-skill callers remain compatible. - [ ] No arbitrary skill content becomes executable Prolog or bypasses normal tool/authority/effect boundaries. ## Non-goals - No second prompt compiler. - No second agent/subagent runtime. - No second verifier/evidence stack. - No universal model-generated plugin ABI. - No arbitrary `call/1` from skill content. - No capability or authority grants from skill activation. - No product-specific coding skills in core. - No personal activity/session mining inside `prolog-rlm`. - No automatic promotion based solely on model preference. ## Work contract Before each implementation slice, inspect current `main`, related issues/PRs/commits, source, tests, docs/research and CI. Merged executable truth outranks stale issue prose. Reuse existing runtime contracts first; create focused child issues for coherent missing generic surfaces. Use TDD and keep the aggregate repository test gate authoritative. Do as much coherent work as possible per cycle rather than stopping after one trivial checkbox.

lost-rob0t commented

2026-08-22 01:53:34 +00:00

(Migrated from github.com)

Child implementation slices

Existing/reused:

#117 — safe Agent Skills SKILL.md loader + deterministic activation/progressive disclosure
#142 — generic evolutionary-search research/background; merged PR #148 supplies the pure rlm_evolution kernel
#144 — generic bounded RLM subagent plumbing/KB command binding background

New children for this epic:

#168 — deterministic skill-selection eval corpus + trigger precision/recall
#169 — baseline-vs-skill outcome evaluation with verifier-backed evidence
#170 — immutable skill lifecycle, promotion, supersession and rollback
#171 — skill evolution orchestration over rlm_evolution + eval evidence
#172 — skill-aware bounded subagent delegation + generic reviewer/critic roles

Downstream companion:

lost-rob0t/agentProlog#6 — observe real operator workflows across Claude/OpenCode/Agent Zero/AgentProlog, synthesize candidates, run RAGE evals through upstream APIs, and export/install canonical skills.

Dependency direction remains AgentProlog -> prolog-rlm. Avoid duplicate implementations between these slices.

## Child implementation slices Existing/reused: - #117 — safe Agent Skills `SKILL.md` loader + deterministic activation/progressive disclosure - #142 — generic evolutionary-search research/background; merged PR #148 supplies the pure `rlm_evolution` kernel - #144 — generic bounded RLM subagent plumbing/KB command binding background New children for this epic: - #168 — deterministic skill-selection eval corpus + trigger precision/recall - #169 — baseline-vs-skill outcome evaluation with verifier-backed evidence - #170 — immutable skill lifecycle, promotion, supersession and rollback - #171 — skill evolution orchestration over `rlm_evolution` + eval evidence - #172 — skill-aware bounded subagent delegation + generic reviewer/critic roles Downstream companion: - `lost-rob0t/agentProlog#6` — observe real operator workflows across Claude/OpenCode/Agent Zero/AgentProlog, synthesize candidates, run RAGE evals through upstream APIs, and export/install canonical skills. Dependency direction remains `AgentProlog -> prolog-rlm`. Avoid duplicate implementations between these slices.