Add provider-neutral symbolic prompt compiler core #116

Merged

lost-rob0t merged 18 commits from agent/prompt-context-compiler into main

2026-08-21 01:13:48 +00:00

lost-rob0t commented

2026-08-19 04:25:54 +00:00

(Migrated from github.com)

Scope

This PR deliberately lands the compiler core only from RLM-RESEARCH-010/011. It does not claim completion or managed-conversation integration; that executable convergence remains open under #101.

Implemented here:

one provider-neutral compilation-unit catalog for tools, skills, MCP metadata, resources, and instructions;
deterministic contextual activation/deactivation without runtime unregistering;
evidence from text, verbs/objects, explicit selection, negation, and model-authored needs(...);
transitive dependency closure, conflicts, supersession, availability, capability filtering, and host-controlled discovery-scope narrowing;
bounded sanitized prompt_catalog_search/4 that returns declarative metadata only;
tool-registry schema import without importing handlers or granting authority;
compiler context units compatible with the canonical rlm_context_budget algebra;
token-accounted representation variants and deterministic fingerprints;
all_tools compatibility mode for regression/benchmark comparison;
deterministic 5,000-unit narrowing coverage plus adversarial authority/scope tests.

Authority invariant

registered(Unit) != available(Unit) != active(Unit) != authorized(Unit).

Compiler selection controls only the model-visible projection. It never executes a tool, installs or starts MCP, stores trusted callables, or widens runtime capabilities. Existing tool/MCP/effect/authority runtimes remain authoritative.

Explicit non-goals for this PR

The following are intentionally not claimed here and remain work for #101 / subsequent prompt-runtime slices:

wiring selected schemas into rlm_completion/4 so the planner stops receiving every discovered registry schema by default;
merging compiler context units with managed hot/warm/cold conversation units before the single final rlm_context_budget pack;
rendering the exact selected representation into provider requests and charging that exact rendering in the final request ledger;
project-instruction and complete MCP runtime metadata integration.

Keeping those out of this PR satisfies the prior reconciliation direction to either implement the whole executable integration or narrow the transaction to the compiler core. It also preserves the authority boundary: activation is model-visible projection only.

Reconciliation

The branch was merged forward onto current main without rewriting history. Final head 7dafce18d3e50f3b5e6be06220ddb19263da765d is zero commits behind main and the final delta is six intentional files. The reconciliation preserves the fail-closed production static loader, project-source coverage, UI fixture codec regression, and recursive fingerprint fixes already present on main.

Research

research/RLM-RESEARCH-011-managed-context-tool-discovery.org records the broader convergence model: bounded candidate generation -> closure/policy -> compiler context units -> one shared context pack. Research remains broader than this implementation slice.

Exact-head validation

Final head: 7dafce18d3e50f3b5e6be06220ddb19263da765d

PASS:

Tree-sitter FFI;
supported SWI-Prolog runtime check;
fail-closed static load of all production modules;
static load of live integration definitions;
deterministic PlUnit suite, including prompt-compiler tests and current-main regressions;
deterministic benchmark/conformance suite;
deterministic deep-recursion experiment;
credential-free CLI demo/trace smoke;
persistent graph restart verification;
durable artifact handoff verification;
whitespace checks;
REAL OpenRouter core suite;
REAL OpenRouter structured repair suite;
REAL OpenRouter benchmark suite;
REAL depth 0/1/2 recursion experiment;
one-command REAL RLM CLI smoke.

The earlier live failure was provider HTTP 429 quota exhaustion; the fresh reconciled exact head subsequently passed the entire REAL OpenRouter job.

Remaining architecture work

Issue #101 remains open and authoritative for executable prompt-compiler integration with managed conversation and final provider-visible packing. This PR does not close or weaken that requirement.

## Scope This PR deliberately lands the **compiler core only** from RLM-RESEARCH-010/011. It does **not** claim completion or managed-conversation integration; that executable convergence remains open under #101. Implemented here: - one provider-neutral compilation-unit catalog for tools, skills, MCP metadata, resources, and instructions; - deterministic contextual activation/deactivation without runtime unregistering; - evidence from text, verbs/objects, explicit selection, negation, and model-authored `needs(...)`; - transitive dependency closure, conflicts, supersession, availability, capability filtering, and host-controlled discovery-scope narrowing; - bounded sanitized `prompt_catalog_search/4` that returns declarative metadata only; - tool-registry schema import without importing handlers or granting authority; - compiler context units compatible with the canonical `rlm_context_budget` algebra; - token-accounted representation variants and deterministic fingerprints; - `all_tools` compatibility mode for regression/benchmark comparison; - deterministic 5,000-unit narrowing coverage plus adversarial authority/scope tests. ## Authority invariant `registered(Unit) != available(Unit) != active(Unit) != authorized(Unit)`. Compiler selection controls only the model-visible projection. It never executes a tool, installs or starts MCP, stores trusted callables, or widens runtime capabilities. Existing tool/MCP/effect/authority runtimes remain authoritative. ## Explicit non-goals for this PR The following are intentionally **not** claimed here and remain work for #101 / subsequent prompt-runtime slices: - wiring selected schemas into `rlm_completion/4` so the planner stops receiving every discovered registry schema by default; - merging compiler context units with managed hot/warm/cold conversation units before the single final `rlm_context_budget` pack; - rendering the exact selected representation into provider requests and charging that exact rendering in the final request ledger; - project-instruction and complete MCP runtime metadata integration. Keeping those out of this PR satisfies the prior reconciliation direction to either implement the whole executable integration or narrow the transaction to the compiler core. It also preserves the authority boundary: activation is model-visible projection only. ## Reconciliation The branch was merged forward onto current `main` without rewriting history. Final head `7dafce18d3e50f3b5e6be06220ddb19263da765d` is zero commits behind `main` and the final delta is six intentional files. The reconciliation preserves the fail-closed production static loader, project-source coverage, UI fixture codec regression, and recursive fingerprint fixes already present on `main`. ## Research `research/RLM-RESEARCH-011-managed-context-tool-discovery.org` records the broader convergence model: bounded candidate generation -> closure/policy -> compiler context units -> one shared context pack. Research remains broader than this implementation slice. ## Exact-head validation Final head: `7dafce18d3e50f3b5e6be06220ddb19263da765d` PASS: - Tree-sitter FFI; - supported SWI-Prolog runtime check; - fail-closed static load of all production modules; - static load of live integration definitions; - deterministic PlUnit suite, including prompt-compiler tests and current-main regressions; - deterministic benchmark/conformance suite; - deterministic deep-recursion experiment; - credential-free CLI demo/trace smoke; - persistent graph restart verification; - durable artifact handoff verification; - whitespace checks; - REAL OpenRouter core suite; - REAL OpenRouter structured repair suite; - REAL OpenRouter benchmark suite; - REAL depth 0/1/2 recursion experiment; - one-command REAL RLM CLI smoke. The earlier live failure was provider HTTP 429 quota exhaustion; the fresh reconciled exact head subsequently passed the entire REAL OpenRouter job. ## Remaining architecture work Issue #101 remains open and authoritative for executable prompt-compiler integration with managed conversation and final provider-visible packing. This PR does not close or weaken that requirement.

lost-rob0t commented

2026-08-19 04:39:54 +00:00

(Migrated from github.com)

Backlog reconciliation note: the compiler core/research/tests are substantial, but the current six-file delta does not yet wire selected schemas into rlm_completion or the managed conversation runtime even though the PR scope claims that integration. Research 011 explicitly says completion should stop exposing every discovered registry schema by default and managed turns should merge compiler units with hot/warm/cold units before the single rlm_context_budget pack. Keep this draft until that executable integration is present and tested (or narrow the PR/body to compiler-core only). Do not satisfy this by moving execution/authority into the compiler: activation remains model-visible projection only, while the full trusted registry and existing authority/tool runtime stay authoritative.

Backlog reconciliation note: the compiler core/research/tests are substantial, but the current six-file delta does not yet wire selected schemas into `rlm_completion` or the managed conversation runtime even though the PR scope claims that integration. Research 011 explicitly says completion should stop exposing every discovered registry schema by default and managed turns should merge compiler units with hot/warm/cold units before the single `rlm_context_budget` pack. Keep this draft until that executable integration is present and tested (or narrow the PR/body to compiler-core only). Do not satisfy this by moving execution/authority into the compiler: activation remains model-visible projection only, while the full trusted registry and existing authority/tool runtime stay authoritative.