nsaspy/prolog-rlm

Fork 0

[AP0] Retrieval intelligence, embeddings, source-aware KB, and RLM integration #219

New issue

Open

opened 2026-08-26 06:36:58 +00:00 by lost-rob0t · 1 comment

lost-rob0t commented

2026-08-26 06:36:58 +00:00

(Migrated from github.com)

Priority

AP0 — next coding move.

This epic captures the complete design discussed for making prolog-rlm smarter as a reusable library for building LLM harnesses, while preserving the existing default/reference RLM runtime.

This work is ADARD, human-gated. Do not silently advance through the full implementation/merge loop.

Human-gated ADARD contract

Use distinct fresh reasoning phases and preserve evidence/decisions between them:

A — Analyze / research the current code, existing issues, tests, provider/context/tool/agent/source APIs, and relevant external semantics.
D — Design the smallest coherent architecture and API changes.
A — Adversarial review the design for regressions, hidden coupling, stale-KB failures, concurrency races, provider leaks, over-opinionated library behavior, and daemon/process assumptions.
R/D — Decision gate: present the design + adversarial findings and STOP FOR HUMAN APPROVAL before realization.
Realize TDD-first only after explicit human approval.
Verify exact head with deterministic tests/integration gates and report evidence.
STOP FOR HUMAN APPROVAL BEFORE MERGE. No auto-merge.

If implementation uncovers a material architecture change, return to the human decision gate.

0. Architectural thesis / non-negotiables

prolog-rlm is a reusable Prolog library/runtime substrate for writing harnesses. It may ship a default/reference RLM harness, CLI, agent runtime, graph runtime, prompt compiler, etc., but new work must remain composable and reusable by other harnesses.

Think:

prolog-rlm library
  ├─ providers / chains
  ├─ context
  ├─ tools
  ├─ prompt compiler / skills
  ├─ agents
  ├─ graphs
  ├─ MCP / traces
  ├─ source knowledge
  ├─ embeddings              <-- this epic
  ├─ retrieval               <-- this epic
  └─ retrieval expert        <-- this epic
       |
       +--> coding harness
       +--> research harness
       +--> ADARD/RAGE harness
       +--> editor integrations
       +--> other applications

Do not turn the library into a required daemon. Do not require a filesystem watcher. Do not bake one application persona/workflow into core primitives.

The model may request standard structured actions/tool calls, but Prolog/runtime code owns dispatch, capabilities, budgets, validation, state, and execution. Final model prose does not need to be structured.

1. Reuse existing provider layer

Do not duplicate model/provider plumbing in downstream harnesses.

Reuse and extend rlm_chain conventions:

OpenRouter provider support;
generic OpenAI-compatible endpoints;
local OpenAI-compatible servers;
sync/async surfaces;
streaming;
retries/backoff;
structured output/tool schemas;
usage/cost accounting;
structured errors.

Credentials remain referenced indirectly (for example env('OPENROUTER_API_KEY')) and are resolved at execution time. A host/editor may set that environment variable from its own config file, but raw secrets must not leak into provider terms, traces, errors, fixtures, or logs.

2. Provider-neutral embeddings

Add a first-class embedding abstraction separate from chat-completion models.

API direction (exact naming/arity should follow repo conventions):

embed(+Provider, +Text, +Options, -Outcome).
embed_batch(+Provider, +Texts, +Options, -Outcome).
embedding_provider_capability(+Provider, ?Capability).

Requirements:

OpenAI-compatible embeddings first;
local OpenAI-compatible embedding endpoints;
batch support;
model + dimensions + provider provenance;
structured transport/provider errors;
usage where available;
secret redaction;
embeddings are not assumed to share the same provider/model as chat completion.

3. Generic `rlm_retrieval` library

Add backend-neutral retrieval infrastructure. Keep it deliberately dumb and reusable.

Conceptual surface:

retriever_create(...).
retrieval_search(+Retriever, +Query, +Options, -Outcome).
retrieval_fetch(+Retriever, +Ids, +Options, -Outcome).
retrieval_upsert(+Retriever, +Documents, +Options, -Outcome).
retrieval_delete(+Retriever, +Ids, +Options, -Outcome).

Support normalized capabilities/strategies for:

exact lookup;
lexical/BM25-style retrieval where provided by backend;
vector similarity;
metadata filtering;
hybrid retrieval;
project/source structural retrieval;
graph/relationship retrieval;
normalized score/provenance/result envelopes;
bounded result count/output size/cancellation/timeouts.

Do not make Chroma semantics the generic retrieval API.

4. Basic Chroma adapter

Add rlm_chroma (or equivalent) on top of rlm_retrieval.

First useful slice:

connect via Chroma HTTP API;
collection create/get/list where required;
add/upsert documents, embeddings, metadata;
query by embedding/text as supported;
fetch/delete;
bounded result limits;
structured backend/transport errors;
deterministic HTTP fixture tests where practical;
optional real integration lane, separately gated.

Chroma is one adapter, not the architecture.

5. `retrieval_expert`: symbolic/non-LLM retrieval intelligence

Build a separate reusable expert-system layer on top of rlm_retrieval and rlm_embedding.

This is intentionally classic/symbolic AI driving modern generative AI. It must be able to make useful retrieval decisions without calling an LLM.

Conceptual API:

retrieval_plan(+Expert, +Query, +Context, -Plan).
retrieval_execute(+Expert, +Plan, -Evidence).
retrieval_explain(+Expert, +PlanOrEvidence, -Explanation).

Initial intelligence should support extensible rules/facts for:

query classification;
retrieval-source selection;
strategy selection: exact, lexical, vector, graph, structural, hybrid;
source/provenance filtering;
confidence/certainty handling;
evidence-gap detection;
ontology/concept expansion hooks;
truth-maintenance / invalidation when supporting evidence changes;
case-based reasoning / reuse of successful retrieval strategies;
tabling/memoization of repeated semantic subproblems;
constraint-based filtering (project, file, time, entity, source type, provenance, etc.);
cost-aware ordering (cheap exact/structural checks before expensive retrieval where appropriate);
optional non-LLM learned/ranking/bandit policy later behind a stable policy interface.

Embeddings are a sensor, not the decision-maker.

Example intent:

strategy(identifier, exact).
strategy(relationship, graph).
strategy(conceptual, vector).
strategy(broad_research, hybrid).

The expert should be inspectable/explainable and reusable by arbitrary harnesses.

6. Existing source parsing / project-KB plan is part of this epic

Integrate with the existing Project/source work rather than creating a parallel code-index system.

Existing dependency chain:

#94 direct SWI-Prolog <-> Tree-sitter C FFI          DONE
  ↓
#95 Project/File/Language/grammar registry           DONE
  ↓
#96 CST -> versioned Prolog syntax facts
  ↓
#97 Tree-sitter query/capture API
  ↓
#98 symbols / references / calls / imports / exports
  ↓
#99 incremental reparse + changed ranges + freshness

Preserve the existing layering:

source bytes
  -> parser/CST
  -> query captures
  -> normalized common semantic facts
  -> richer language-specific facts
  -> project KB
  -> retrieval
  -> RLM/context consumers

For Prolog source, keep SWI-native semantic analysis where it is stronger than Tree-sitter.

The project KB must remain useful independently of any coding agent or LLM.

7. Source-KB freshness is a hard correctness invariant

A served source-derived KB fact must never silently appear current when it was derived from different source bytes.

Every material source-derived fact must resolve to provenance including at least:

Project;
File identity;
exact source content hash;
file/source generation;
parse generation;
parser backend;
grammar identity/version where applicable;
query/extractor identity/version where applicable;
normalization/schema version where applicable;
source range supporting the fact.

Current-vs-stale state must be explicit.

Do not rely on mtimes alone. Exact bytes/hash are authoritative.

8. No required watcher: lazy freshness is baked into access/loop boundaries

A filesystem watcher may exist only as an optional host hint interface. Correctness must not require a running background process.

Authoritative model:

consumer asks for project knowledge / RLM needs context
        ↓
acquire coherent project snapshot
        ↓
validate relevant current source hashes
        ↓
unchanged -> reuse indexed generation
changed   -> refresh affected source facts
        ↓
return coherent snapshot/evidence

Possible abstraction:

with_project_snapshot(+Project, -Snapshot, :Goal).
ensure_project_fresh(+Project, +Scope, -Outcome).

Do not re-hash the entire repository on every tiny predicate call. Validate at meaningful acquisition/retrieval/RLM-loop boundaries, cache within a coherent snapshot, and narrow work to relevant/dirty files where safely known.

Optional host hint:

source_change_hint(Project, File).

A watcher, Emacs, IDE, Git integration, etc. may call that. It is an optimization only.

9. Harness-owned writes update the KB directly

When a Prolog-RLM tool/harness writes or patches source, the runtime already knows what changed. Exploit that immediately rather than waiting for later rediscovery.

Conceptual flow:

write/apply-patch effect
  -> file + expected old hash
  -> exact edit/range information when available
  -> new bytes/hash
  -> mark/update source generation
  -> incremental Tree-sitter edit/reparse when safe
  -> invalidate/re-extract affected facts
  -> stage new KB generation
  -> atomic publish

Conceptual API direction:

source_commit_edit(+Project, +File, +Edit, +EffectResult, -Outcome).

Do not couple this to one particular file-writing tool. Define a reusable source-update boundary that canonical effectful write tools can call.

10. Multiple writes must be coalesced

Do not thrash parsing/indexing for every intermediate write when several writes occur in one logical tool/agent/RLM step.

Maintain a bounded dirty-set/edit journal:

mark_source_dirty(+Project, +File, +EditOrHint, -Outcome).
flush_source_updates(+Project, +Options, -Outcome).

Conceptual behavior:

write #1 ─┐
write #2 ─┼─> dirty file + bounded edit journal
write #3 ─┘
            ↓
logical flush/loop boundary
            ↓
read authoritative final bytes once
            ↓
hash / incremental-or-full reparse
            ↓
re-extract once
            ↓
atomic new KB generation

Intermediate half-written source must not become a supposedly complete current KB generation.

11. Multiple programs/editors and concurrent writers

A file being open in multiple programs is normal. No open handle implies ownership.

Use optimistic concurrency and exact-byte validation.

For indexing/publication:

read exact bytes
  -> hash H
  -> parse/extract those bytes
  -> before publishing, verify authoritative file bytes/hash still equal H
  -> same: publish
  -> changed: discard stale candidate and retry/rebase according to bounded policy

For harness writes, use expected-hash/generation semantics where possible:

apply_patch(File, ExpectedHash, Patch, Outcome).

If another program changed the file between read and write, return a structured conflict rather than clobbering or publishing a KB based on stale assumptions.

Concurrent source publication must serialize/linearize per relevant Project/File generation and never expose mixed generations as one complete current snapshot.

12. Coherent snapshot semantics

RLM/retrieval consumers should query a coherent project/source snapshot rather than observe a mixture of:

old generation for file A;
new generation for file B;
half-reindexed file C.

Provide a snapshot identity/reference and make current queries bind to it where appropriate.

Conceptual direction:

project_snapshot(+Project, -Snapshot).
project_kb_query(+Snapshot, +Query, -Outcome).

Publishing a new generation must be atomic at the defined visibility boundary. Reindexing/error/partial states must never masquerade as complete current evidence.

13. RLM integration: two entry points

Retrieval/source knowledge must plug into the existing RLM as a library capability, not become a second orchestration system.

A. Initial context acquisition

Before the planner/model receives project evidence:

Query
  -> acquire fresh coherent Project snapshot
  -> retrieval_expert chooses retrieval plan
  -> rlm_retrieval executes
  -> evidence becomes bounded rlm_context/provider projection
  -> RLM planner/model

This should compose with the existing context-budget/prompt-compiler machinery.

B. Retrieval during recursive execution

A running RLM/subagent must be able to request more evidence through the normal typed/tool execution boundary.

Conceptual operation/tool:

retrieve(Query, Options, Evidence)

The model expresses an information need; retrieval_expert decides how to satisfy it using exact/lexical/vector/graph/source retrieval. Model output does not bypass retrieval policy or source-freshness guarantees.

Do not invent an incompatible proprietary model protocol if standard tool/function-calling + internal Prolog terms already suffice.

14. Prompt/tool boundary

Keep the existing distinction clear:

ordinary final LLM prose may remain text;
machine actions/tool calls/subagent calls/context operations use structured requests so the runtime can reliably dispatch them;
structured requests do not themselves grant authority;
prompt/skill/compiler projection decides what is visible/relevant;
capabilities/authority/effect boundaries decide what may execute.

The library supplies prompting/selection/runtime mechanics; individual harnesses own their domain persona, goals, project workflow, and application-level instructions.

15. Testing / acceptance

At minimum, the eventual implementation path must prove:

Embeddings/retrieval

provider-neutral embedding works against deterministic fixture and one explicitly gated real/local-compatible endpoint;
Chroma adapter round-trips documents/embeddings/metadata/query/fetch/delete;
retrieval-expert can choose different strategies without an LLM and explain the choice;
hybrid/project structural retrieval can be composed without backend-specific leakage.

Source knowledge

current facts are bound to exact source hash/generation;
external edit is detected lazily on next relevant snapshot/access;
a harness-owned write can update/dirty the KB immediately;
multiple writes coalesce into one coherent publication;
concurrent external modification between parse and publish causes candidate rejection/retry, not stale publication;
expected-hash write conflict is structured;
stale/partial/error generations cannot satisfy a query requiring current complete evidence;
RLM context sees a coherent snapshot;
no watcher/background daemon is required for correctness.

Regression

existing rlm_chain, rlm_context, rlm_tool, rlm_agent, rlm_graph, prompt compiler, MCP, trace, and source APIs remain usable independently;
no new mandatory Python/Node runtime;
no raw secrets in traces/errors;
no source parsing grants execution authority;
no model-provided grammar/library path gains ambient dlopen authority;
no arbitrary source text becomes executable Prolog.

16. First coding move after human design approval

Do not try to land the entire epic in one PR.

During ADARD design, identify the smallest dependency-correct vertical slice. Strong candidate ordering to evaluate:

provider-neutral embedding contract + deterministic OpenAI-compatible fixture;
generic retrieval result/backend contract;
minimal Chroma adapter;
retrieval-expert rule/plan skeleton with deterministic exact-vs-vector choice;
snapshot/freshness boundary wired to existing Project/source registry;
write dirty/coalescing + optimistic hash validation;
integration into rlm_context/RLM retrieval path;
continue #96-#99 source semantic pipeline as required by structural retrieval.

The ADARD design/adversarial phases may reorder these if repository dependencies prove a different smallest safe slice.

Human approval is required before realization and again before merge.

Refs: #93 #94 #95 #96 #97 #98 #99 and existing provider/context/tool/agent/graph/prompt compiler work.

## Priority **AP0 — next coding move.** This epic captures the complete design discussed for making `prolog-rlm` smarter as a reusable **library for building LLM harnesses**, while preserving the existing default/reference RLM runtime. This work is **ADARD, human-gated**. Do not silently advance through the full implementation/merge loop. ### Human-gated ADARD contract Use distinct fresh reasoning phases and preserve evidence/decisions between them: 1. **A — Analyze / research** the current code, existing issues, tests, provider/context/tool/agent/source APIs, and relevant external semantics. 2. **D — Design** the smallest coherent architecture and API changes. 3. **A — Adversarial review** the design for regressions, hidden coupling, stale-KB failures, concurrency races, provider leaks, over-opinionated library behavior, and daemon/process assumptions. 4. **R/D — Decision gate**: present the design + adversarial findings and **STOP FOR HUMAN APPROVAL** before realization. 5. **Realize TDD-first** only after explicit human approval. 6. **Verify exact head** with deterministic tests/integration gates and report evidence. 7. **STOP FOR HUMAN APPROVAL BEFORE MERGE**. No auto-merge. If implementation uncovers a material architecture change, return to the human decision gate. --- # 0. Architectural thesis / non-negotiables `prolog-rlm` is a reusable Prolog library/runtime substrate for **writing harnesses**. It may ship a default/reference RLM harness, CLI, agent runtime, graph runtime, prompt compiler, etc., but new work must remain composable and reusable by other harnesses. Think: ```text prolog-rlm library ├─ providers / chains ├─ context ├─ tools ├─ prompt compiler / skills ├─ agents ├─ graphs ├─ MCP / traces ├─ source knowledge ├─ embeddings <-- this epic ├─ retrieval <-- this epic └─ retrieval expert <-- this epic | +--> coding harness +--> research harness +--> ADARD/RAGE harness +--> editor integrations +--> other applications ``` Do **not** turn the library into a required daemon. Do **not** require a filesystem watcher. Do **not** bake one application persona/workflow into core primitives. The model may request standard structured actions/tool calls, but Prolog/runtime code owns dispatch, capabilities, budgets, validation, state, and execution. Final model prose does not need to be structured. --- # 1. Reuse existing provider layer Do not duplicate model/provider plumbing in downstream harnesses. Reuse and extend `rlm_chain` conventions: - OpenRouter provider support; - generic OpenAI-compatible endpoints; - local OpenAI-compatible servers; - sync/async surfaces; - streaming; - retries/backoff; - structured output/tool schemas; - usage/cost accounting; - structured errors. Credentials remain referenced indirectly (for example `env('OPENROUTER_API_KEY')`) and are resolved at execution time. A host/editor may set that environment variable from its own config file, but raw secrets must not leak into provider terms, traces, errors, fixtures, or logs. --- # 2. Provider-neutral embeddings Add a first-class embedding abstraction separate from chat-completion models. API direction (exact naming/arity should follow repo conventions): ```prolog embed(+Provider, +Text, +Options, -Outcome). embed_batch(+Provider, +Texts, +Options, -Outcome). embedding_provider_capability(+Provider, ?Capability). ``` Requirements: - OpenAI-compatible embeddings first; - local OpenAI-compatible embedding endpoints; - batch support; - model + dimensions + provider provenance; - structured transport/provider errors; - usage where available; - secret redaction; - embeddings are not assumed to share the same provider/model as chat completion. --- # 3. Generic `rlm_retrieval` library Add backend-neutral retrieval infrastructure. Keep it deliberately **dumb** and reusable. Conceptual surface: ```prolog retriever_create(...). retrieval_search(+Retriever, +Query, +Options, -Outcome). retrieval_fetch(+Retriever, +Ids, +Options, -Outcome). retrieval_upsert(+Retriever, +Documents, +Options, -Outcome). retrieval_delete(+Retriever, +Ids, +Options, -Outcome). ``` Support normalized capabilities/strategies for: - exact lookup; - lexical/BM25-style retrieval where provided by backend; - vector similarity; - metadata filtering; - hybrid retrieval; - project/source structural retrieval; - graph/relationship retrieval; - normalized score/provenance/result envelopes; - bounded result count/output size/cancellation/timeouts. Do not make Chroma semantics the generic retrieval API. --- # 4. Basic Chroma adapter Add `rlm_chroma` (or equivalent) on top of `rlm_retrieval`. First useful slice: - connect via Chroma HTTP API; - collection create/get/list where required; - add/upsert documents, embeddings, metadata; - query by embedding/text as supported; - fetch/delete; - bounded result limits; - structured backend/transport errors; - deterministic HTTP fixture tests where practical; - optional real integration lane, separately gated. Chroma is one adapter, not the architecture. --- # 5. `retrieval_expert`: symbolic/non-LLM retrieval intelligence Build a separate reusable expert-system layer **on top of** `rlm_retrieval` and `rlm_embedding`. This is intentionally classic/symbolic AI driving modern generative AI. It must be able to make useful retrieval decisions **without calling an LLM**. Conceptual API: ```prolog retrieval_plan(+Expert, +Query, +Context, -Plan). retrieval_execute(+Expert, +Plan, -Evidence). retrieval_explain(+Expert, +PlanOrEvidence, -Explanation). ``` Initial intelligence should support extensible rules/facts for: - query classification; - retrieval-source selection; - strategy selection: `exact`, `lexical`, `vector`, `graph`, `structural`, `hybrid`; - source/provenance filtering; - confidence/certainty handling; - evidence-gap detection; - ontology/concept expansion hooks; - truth-maintenance / invalidation when supporting evidence changes; - case-based reasoning / reuse of successful retrieval strategies; - tabling/memoization of repeated semantic subproblems; - constraint-based filtering (project, file, time, entity, source type, provenance, etc.); - cost-aware ordering (cheap exact/structural checks before expensive retrieval where appropriate); - optional non-LLM learned/ranking/bandit policy later behind a stable policy interface. Embeddings are a **sensor**, not the decision-maker. Example intent: ```prolog strategy(identifier, exact). strategy(relationship, graph). strategy(conceptual, vector). strategy(broad_research, hybrid). ``` The expert should be inspectable/explainable and reusable by arbitrary harnesses. --- # 6. Existing source parsing / project-KB plan is part of this epic Integrate with the existing Project/source work rather than creating a parallel code-index system. Existing dependency chain: ```text #94 direct SWI-Prolog <-> Tree-sitter C FFI DONE ↓ #95 Project/File/Language/grammar registry DONE ↓ #96 CST -> versioned Prolog syntax facts ↓ #97 Tree-sitter query/capture API ↓ #98 symbols / references / calls / imports / exports ↓ #99 incremental reparse + changed ranges + freshness ``` Preserve the existing layering: ```text source bytes -> parser/CST -> query captures -> normalized common semantic facts -> richer language-specific facts -> project KB -> retrieval -> RLM/context consumers ``` For Prolog source, keep SWI-native semantic analysis where it is stronger than Tree-sitter. The project KB must remain useful independently of any coding agent or LLM. --- # 7. Source-KB freshness is a hard correctness invariant A served source-derived KB fact must never silently appear current when it was derived from different source bytes. Every material source-derived fact must resolve to provenance including at least: - Project; - File identity; - exact source content hash; - file/source generation; - parse generation; - parser backend; - grammar identity/version where applicable; - query/extractor identity/version where applicable; - normalization/schema version where applicable; - source range supporting the fact. Current-vs-stale state must be explicit. **Do not rely on mtimes alone. Exact bytes/hash are authoritative.** --- # 8. No required watcher: lazy freshness is baked into access/loop boundaries A filesystem watcher may exist only as an **optional host hint interface**. Correctness must not require a running background process. Authoritative model: ```text consumer asks for project knowledge / RLM needs context ↓ acquire coherent project snapshot ↓ validate relevant current source hashes ↓ unchanged -> reuse indexed generation changed -> refresh affected source facts ↓ return coherent snapshot/evidence ``` Possible abstraction: ```prolog with_project_snapshot(+Project, -Snapshot, :Goal). ensure_project_fresh(+Project, +Scope, -Outcome). ``` Do not re-hash the entire repository on every tiny predicate call. Validate at meaningful acquisition/retrieval/RLM-loop boundaries, cache within a coherent snapshot, and narrow work to relevant/dirty files where safely known. Optional host hint: ```prolog source_change_hint(Project, File). ``` A watcher, Emacs, IDE, Git integration, etc. may call that. It is an optimization only. --- # 9. Harness-owned writes update the KB directly When a Prolog-RLM tool/harness writes or patches source, the runtime already knows what changed. Exploit that immediately rather than waiting for later rediscovery. Conceptual flow: ```text write/apply-patch effect -> file + expected old hash -> exact edit/range information when available -> new bytes/hash -> mark/update source generation -> incremental Tree-sitter edit/reparse when safe -> invalidate/re-extract affected facts -> stage new KB generation -> atomic publish ``` Conceptual API direction: ```prolog source_commit_edit(+Project, +File, +Edit, +EffectResult, -Outcome). ``` Do not couple this to one particular file-writing tool. Define a reusable source-update boundary that canonical effectful write tools can call. --- # 10. Multiple writes must be coalesced Do not thrash parsing/indexing for every intermediate write when several writes occur in one logical tool/agent/RLM step. Maintain a bounded dirty-set/edit journal: ```prolog mark_source_dirty(+Project, +File, +EditOrHint, -Outcome). flush_source_updates(+Project, +Options, -Outcome). ``` Conceptual behavior: ```text write #1 ─┐ write #2 ─┼─> dirty file + bounded edit journal write #3 ─┘ ↓ logical flush/loop boundary ↓ read authoritative final bytes once ↓ hash / incremental-or-full reparse ↓ re-extract once ↓ atomic new KB generation ``` Intermediate half-written source must not become a supposedly complete current KB generation. --- # 11. Multiple programs/editors and concurrent writers A file being open in multiple programs is normal. No open handle implies ownership. Use optimistic concurrency and exact-byte validation. For indexing/publication: ```text read exact bytes -> hash H -> parse/extract those bytes -> before publishing, verify authoritative file bytes/hash still equal H -> same: publish -> changed: discard stale candidate and retry/rebase according to bounded policy ``` For harness writes, use expected-hash/generation semantics where possible: ```prolog apply_patch(File, ExpectedHash, Patch, Outcome). ``` If another program changed the file between read and write, return a structured conflict rather than clobbering or publishing a KB based on stale assumptions. Concurrent source publication must serialize/linearize per relevant Project/File generation and never expose mixed generations as one complete current snapshot. --- # 12. Coherent snapshot semantics RLM/retrieval consumers should query a coherent project/source snapshot rather than observe a mixture of: - old generation for file A; - new generation for file B; - half-reindexed file C. Provide a snapshot identity/reference and make current queries bind to it where appropriate. Conceptual direction: ```prolog project_snapshot(+Project, -Snapshot). project_kb_query(+Snapshot, +Query, -Outcome). ``` Publishing a new generation must be atomic at the defined visibility boundary. Reindexing/error/partial states must never masquerade as complete current evidence. --- # 13. RLM integration: two entry points Retrieval/source knowledge must plug into the existing RLM as a library capability, not become a second orchestration system. ## A. Initial context acquisition Before the planner/model receives project evidence: ```text Query -> acquire fresh coherent Project snapshot -> retrieval_expert chooses retrieval plan -> rlm_retrieval executes -> evidence becomes bounded rlm_context/provider projection -> RLM planner/model ``` This should compose with the existing context-budget/prompt-compiler machinery. ## B. Retrieval during recursive execution A running RLM/subagent must be able to request more evidence through the normal typed/tool execution boundary. Conceptual operation/tool: ```prolog retrieve(Query, Options, Evidence) ``` The model expresses an information need; `retrieval_expert` decides how to satisfy it using exact/lexical/vector/graph/source retrieval. Model output does not bypass retrieval policy or source-freshness guarantees. Do not invent an incompatible proprietary model protocol if standard tool/function-calling + internal Prolog terms already suffice. --- # 14. Prompt/tool boundary Keep the existing distinction clear: - ordinary final LLM prose may remain text; - machine actions/tool calls/subagent calls/context operations use structured requests so the runtime can reliably dispatch them; - structured requests do not themselves grant authority; - prompt/skill/compiler projection decides what is visible/relevant; - capabilities/authority/effect boundaries decide what may execute. The library supplies prompting/selection/runtime mechanics; individual harnesses own their domain persona, goals, project workflow, and application-level instructions. --- # 15. Testing / acceptance At minimum, the eventual implementation path must prove: ### Embeddings/retrieval - provider-neutral embedding works against deterministic fixture and one explicitly gated real/local-compatible endpoint; - Chroma adapter round-trips documents/embeddings/metadata/query/fetch/delete; - retrieval-expert can choose different strategies without an LLM and explain the choice; - hybrid/project structural retrieval can be composed without backend-specific leakage. ### Source knowledge - current facts are bound to exact source hash/generation; - external edit is detected lazily on next relevant snapshot/access; - a harness-owned write can update/dirty the KB immediately; - multiple writes coalesce into one coherent publication; - concurrent external modification between parse and publish causes candidate rejection/retry, not stale publication; - expected-hash write conflict is structured; - stale/partial/error generations cannot satisfy a query requiring current complete evidence; - RLM context sees a coherent snapshot; - no watcher/background daemon is required for correctness. ### Regression - existing `rlm_chain`, `rlm_context`, `rlm_tool`, `rlm_agent`, `rlm_graph`, prompt compiler, MCP, trace, and source APIs remain usable independently; - no new mandatory Python/Node runtime; - no raw secrets in traces/errors; - no source parsing grants execution authority; - no model-provided grammar/library path gains ambient `dlopen` authority; - no arbitrary source text becomes executable Prolog. --- # 16. First coding move after human design approval Do **not** try to land the entire epic in one PR. During ADARD design, identify the smallest dependency-correct vertical slice. Strong candidate ordering to evaluate: 1. provider-neutral embedding contract + deterministic OpenAI-compatible fixture; 2. generic retrieval result/backend contract; 3. minimal Chroma adapter; 4. retrieval-expert rule/plan skeleton with deterministic exact-vs-vector choice; 5. snapshot/freshness boundary wired to existing Project/source registry; 6. write dirty/coalescing + optimistic hash validation; 7. integration into `rlm_context`/RLM retrieval path; 8. continue #96-#99 source semantic pipeline as required by structural retrieval. The ADARD design/adversarial phases may reorder these if repository dependencies prove a different smallest safe slice. **Human approval is required before realization and again before merge.** Refs: #93 #94 #95 #96 #97 #98 #99 and existing provider/context/tool/agent/graph/prompt compiler work.

lost-rob0t commented

2026-08-27 04:36:12 +00:00

(Migrated from github.com)

Fresh ADARD decision gate — current-main reconciliation

A — Analyze / research

Current executable truth changes the safest first slice:

There is no rlm_embedding or rlm_retrieval implementation on current main.
rlm_chain already owns the canonical async-first/sync-await provider execution direction and closed provider(Name, Config) shape.
rlm_openai_compatible already owns endpoint/model/env-credential validation, timeout/address-family handling, secret-safe exception normalization, and the concrete OpenAI-compatible HTTP boundary. Do not create a second credential/HTTP policy stack for embeddings.
rlm_context already establishes the correct trusted-adapter pattern: executable callbacks are process-local trusted registry state; model-visible/serializable state stays closed data; registration is not authority.
rlm_project_source is the merged #95 registry layer only. It explicitly does not parse source or materialize syntax/semantic facts.
#96 remains open for versioned CST facts; #97/#98/#99 remain the dependency chain for captures, semantic source facts, and incremental freshness. Therefore project-structural retrieval is not dependency-ready as the first #219 implementation slice.
#99 already owns current-vs-stale source-generation semantics and atomic publication. #219 should extend that path with lazy hash validation/snapshots/write coalescing rather than creating another source index.

D — Design

Slice E1 — provider-neutral embeddings first

Add a dedicated rlm_embedding public runtime, separate from chat completion semantics but following the same provider/async conventions.

Proposed public surface (names may be adjusted only to repository naming conventions):

embed(+Provider, +Text, +Options, -Outcome).
embed_async(+Provider, +Text, +Options, -Future).
embed_batch(+Provider, +Texts, +Options, -Outcome).
embed_batch_async(+Provider, +Texts, +Options, -Future).
embedding_provider_capability(?ProviderName, ?Capability).

Canonical direction remains:

embed_execute / embed_batch_execute
        -> async Future
        -> sync facade awaits same Future

Use the existing ground provider(Name, Config) convention. Chat and embeddings may use entirely different provider terms/endpoints/models; no implicit inheritance from the active chat model.

Extend the existing OpenAI-compatible transport module (or factor only the minimum genuinely shared private helper) so embedding execution reuses its current provider-config, env-credential, timeout/address-family, redaction and transport-exception semantics. Do not copy those policies into a new transport stack.

First provider capability:

openai_compatible -> embeddings

The configured endpoint is the embeddings endpoint. No special OpenRouter dependency is required for E1; a local OpenAI-compatible embedding server remains a normal provider configuration.

Normalized result should be closed ground data equivalent to:

embedding_result{
    provider: ProviderName,
    model: Model,
    dimensions: Dimensions,
    embeddings: [embedding{index:0, vector:[...]}],
    usage: chain_usage{
        prompt_tokens: InputTokens,
        completion_tokens: 0,
        total_tokens: TotalTokens,
        cost: Cost
    }
}.

Preserve current chain_usage field semantics so embeddings can later participate in one accounting surface. Unknown/missing provider cost remains the same conservative zero/default convention already used by chain normalization; provenance must say which provider/model produced the vector.

E1 options should stay narrow: optional requested dimensions plus bounded input/batch limits. Do not expose arbitrary wire JSON as options. Use float vectors in the first slice; no base64 vector format.

E1 tests

TDD-first deterministic acceptance:

single text -> exact normalized vector;
batch -> exact count/order and common dimension;
sync/async outcomes equivalent and use one canonical execution path;
chat provider and embedding provider can be different endpoint/model terms;
optional dimensions is transmitted and reflected/validated;
empty/oversized input and batch cardinality fail structurally;
missing/duplicate/out-of-range response indexes fail structurally;
inconsistent vector dimensions fail structurally;
nonnumeric/non-finite vector entries fail structurally;
malformed/non-JSON/provider HTTP errors stay structured;
missing env credential is reported by reference name only;
resolved credential bytes never appear in result/error/trace fixtures;
timeout/cancellation remain control outcomes, not provider success;
usage normalizes into the existing usage vocabulary;
static load + aggregate deterministic gate include the module.

Use a tiny SWI local HTTP fixture for deterministic OpenAI-compatible wire behavior; no Python/Node service. Add an optional explicitly gated real/local-compatible smoke configured by environment/provider terms. Missing external endpoint must be reported as unavailable, never counted as correctness evidence.

Slice E2 — generic dumb retrieval contract

Only after E1. Add rlm_retrieval with the same trusted-adapter architecture style as rlm_context:

trusted backend adapter callback registry (process-local)
        +
closed retriever/backend configuration
        -> search/fetch/upsert/delete

Latency-bearing retrieval gets canonical async APIs plus sync-await wrappers. Registration/availability never grants model capability or network/write authority.

Do not force all scores into a fake universal similarity number. Canonical hits must preserve score semantics/direction and provenance, e.g. distance vs similarity vs rank, with an optional normalized relevance only when the backend can justify it. This prevents invalid hybrid comparisons.

Slice E3 — Chroma adapter

Implement Chroma solely as an rlm_retrieval backend. Chroma is an optional service selected by the host, not a required daemon for prolog-rlm. Store/return source provenance metadata but never treat the Chroma index as source-of-truth freshness authority.

Slice E4 — symbolic retrieval expert

Build retrieval_expert above E1+E2. Initial policy is non-LLM and explainable: cheap exact/structural routes before vector where appropriate; deterministic exact-vs-vector/hybrid selection; rule IDs/evidence in explanations. Embeddings are a sensor, not authority.

Do not add a second scheduler or agent runtime. Execution delegates to rlm_retrieval.

Source integration ordering

Source-aware structural/graph retrieval must consume the existing #93 chain:

#96 CST facts
 -> #97 query/captures
 -> #98 semantic facts
 -> #99 freshness/publication
 -> #219 snapshot/retrieval adapter

Do not create a parallel filename/symbol index just to make #219 appear complete sooner.

When that chain is ready, #219 adds the access-boundary semantics already specified in the epic:

acquire snapshot
 -> lazily validate authoritative source hashes for relevant scope
 -> refresh/stage changed observations
 -> verify bytes/hash again before publish
 -> atomically publish coherent generation
 -> retrieval/query binds to snapshot identity

Watcher/editor notifications remain optional dirty hints only. Harness-owned writes feed the same dirty/edit journal directly and coalesce until a logical flush boundary.

A — Adversarial review

The design must reject these failure modes:

Credential fork — embedding transport independently resolves/stores secrets: reject; reuse current env-reference/redaction semantics.
Chat-model coupling — embedding silently inherits active chat provider/model: reject; explicit provider term per call/runtime policy.
Malformed vectors — NaN/Inf/nonnumeric/mixed dimensions reach retrieval index: structural error before publication.
Batch ambiguity — provider omits/duplicates/reorders indexes and runtime guesses: reject; require deterministic index reconciliation.
Unbounded payloads — arbitrary huge texts/batches: host bounds before network dispatch.
Retry duplication — hidden retry/backtracking causes unexpected paid embedding calls: E1 adds no independent retry engine; later retry/effect adoption must compose with #57/#79 rather than inventing a ledger.
Backend authority laundering — registering Chroma/custom backend grants network/write/tool authority: reject; registration is availability only.
Score laundering — distance and cosine similarity merged as if numerically equivalent: reject; retain score kind/direction and explicit normalization provenance.
Stale-vector truth — Chroma hit from source hash H is presented as current after file is H2: reject at source/snapshot freshness boundary; index is cache/evidence, not authoritative source state.
Mixed project generation — retrieval combines independently-current files into a supposedly coherent snapshot: reject; source-aware retrieval binds to one published snapshot identity.
Required watcher/daemon — correctness depends on background file monitor: reject. Chroma/local embedding servers are optional selected adapters, never core runtime requirements.
Half-written source publication — each harness write immediately reindexes intermediate bytes: reject; dirty journal + bounded coalesced flush.
Concurrent external editor race — parse H, external write H2, publish facts for H as current: reject via exact-byte/hash revalidation immediately before publication.
Source code becomes authority — parsed/query metadata becomes callable Prolog or executable adapter path: reject; source observations remain inert closed evidence.
Premature structural retrieval — #219 duplicates #96-#99 because they are unfinished: reject.

R/D — Decision gate

Recommended first coding transaction after explicit human approval: E1 only — rlm_embedding + OpenAI-compatible embedding transport + deterministic fixture + optional gated real/local smoke.

Do not include rlm_retrieval, Chroma, source freshness, or RLM tool integration in that first PR. This keeps the dependency graph small and gives later retrieval code a stable, provider-neutral embedding sensor.

Per #219's human-gated ADARD contract: STOP before realization until explicit human approval.

## Fresh ADARD decision gate — current-main reconciliation ### A — Analyze / research Current executable truth changes the safest first slice: - There is **no** `rlm_embedding` or `rlm_retrieval` implementation on current main. - `rlm_chain` already owns the canonical async-first/sync-await provider execution direction and closed `provider(Name, Config)` shape. - `rlm_openai_compatible` already owns endpoint/model/env-credential validation, timeout/address-family handling, secret-safe exception normalization, and the concrete OpenAI-compatible HTTP boundary. Do not create a second credential/HTTP policy stack for embeddings. - `rlm_context` already establishes the correct trusted-adapter pattern: executable callbacks are process-local trusted registry state; model-visible/serializable state stays closed data; registration is not authority. - `rlm_project_source` is the merged #95 registry layer only. It explicitly does **not** parse source or materialize syntax/semantic facts. - #96 remains open for versioned CST facts; #97/#98/#99 remain the dependency chain for captures, semantic source facts, and incremental freshness. Therefore project-structural retrieval is **not dependency-ready** as the first #219 implementation slice. - #99 already owns current-vs-stale source-generation semantics and atomic publication. #219 should extend that path with lazy hash validation/snapshots/write coalescing rather than creating another source index. ### D — Design #### Slice E1 — provider-neutral embeddings **first** Add a dedicated `rlm_embedding` public runtime, separate from chat completion semantics but following the same provider/async conventions. Proposed public surface (names may be adjusted only to repository naming conventions): ```prolog embed(+Provider, +Text, +Options, -Outcome). embed_async(+Provider, +Text, +Options, -Future). embed_batch(+Provider, +Texts, +Options, -Outcome). embed_batch_async(+Provider, +Texts, +Options, -Future). embedding_provider_capability(?ProviderName, ?Capability). ``` Canonical direction remains: ```text embed_execute / embed_batch_execute -> async Future -> sync facade awaits same Future ``` Use the existing ground `provider(Name, Config)` convention. Chat and embeddings may use entirely different provider terms/endpoints/models; no implicit inheritance from the active chat model. Extend the existing OpenAI-compatible transport module (or factor only the minimum genuinely shared private helper) so embedding execution reuses its current provider-config, env-credential, timeout/address-family, redaction and transport-exception semantics. Do **not** copy those policies into a new transport stack. First provider capability: ```text openai_compatible -> embeddings ``` The configured endpoint is the embeddings endpoint. No special OpenRouter dependency is required for E1; a local OpenAI-compatible embedding server remains a normal provider configuration. Normalized result should be closed ground data equivalent to: ```prolog embedding_result{ provider: ProviderName, model: Model, dimensions: Dimensions, embeddings: [embedding{index:0, vector:[...]}], usage: chain_usage{ prompt_tokens: InputTokens, completion_tokens: 0, total_tokens: TotalTokens, cost: Cost } }. ``` Preserve current `chain_usage` field semantics so embeddings can later participate in one accounting surface. Unknown/missing provider cost remains the same conservative zero/default convention already used by chain normalization; provenance must say which provider/model produced the vector. E1 options should stay narrow: optional requested dimensions plus bounded input/batch limits. Do not expose arbitrary wire JSON as options. Use float vectors in the first slice; no base64 vector format. #### E1 tests TDD-first deterministic acceptance: 1. single text -> exact normalized vector; 2. batch -> exact count/order and common dimension; 3. sync/async outcomes equivalent and use one canonical execution path; 4. chat provider and embedding provider can be different endpoint/model terms; 5. optional `dimensions` is transmitted and reflected/validated; 6. empty/oversized input and batch cardinality fail structurally; 7. missing/duplicate/out-of-range response indexes fail structurally; 8. inconsistent vector dimensions fail structurally; 9. nonnumeric/non-finite vector entries fail structurally; 10. malformed/non-JSON/provider HTTP errors stay structured; 11. missing env credential is reported by reference name only; 12. resolved credential bytes never appear in result/error/trace fixtures; 13. timeout/cancellation remain control outcomes, not provider success; 14. usage normalizes into the existing usage vocabulary; 15. static load + aggregate deterministic gate include the module. Use a tiny SWI local HTTP fixture for deterministic OpenAI-compatible wire behavior; no Python/Node service. Add an **optional explicitly gated** real/local-compatible smoke configured by environment/provider terms. Missing external endpoint must be reported as unavailable, never counted as correctness evidence. #### Slice E2 — generic dumb retrieval contract Only after E1. Add `rlm_retrieval` with the same trusted-adapter architecture style as `rlm_context`: ```text trusted backend adapter callback registry (process-local) + closed retriever/backend configuration -> search/fetch/upsert/delete ``` Latency-bearing retrieval gets canonical async APIs plus sync-await wrappers. Registration/availability never grants model capability or network/write authority. Do not force all scores into a fake universal similarity number. Canonical hits must preserve score semantics/direction and provenance, e.g. distance vs similarity vs rank, with an optional normalized relevance only when the backend can justify it. This prevents invalid hybrid comparisons. #### Slice E3 — Chroma adapter Implement Chroma solely as an `rlm_retrieval` backend. Chroma is an optional service selected by the host, **not a required daemon** for prolog-rlm. Store/return source provenance metadata but never treat the Chroma index as source-of-truth freshness authority. #### Slice E4 — symbolic retrieval expert Build `retrieval_expert` above E1+E2. Initial policy is non-LLM and explainable: cheap exact/structural routes before vector where appropriate; deterministic exact-vs-vector/hybrid selection; rule IDs/evidence in explanations. Embeddings are a sensor, not authority. Do not add a second scheduler or agent runtime. Execution delegates to `rlm_retrieval`. #### Source integration ordering Source-aware structural/graph retrieval must consume the existing #93 chain: ```text #96 CST facts -> #97 query/captures -> #98 semantic facts -> #99 freshness/publication -> #219 snapshot/retrieval adapter ``` Do not create a parallel filename/symbol index just to make #219 appear complete sooner. When that chain is ready, #219 adds the access-boundary semantics already specified in the epic: ```text acquire snapshot -> lazily validate authoritative source hashes for relevant scope -> refresh/stage changed observations -> verify bytes/hash again before publish -> atomically publish coherent generation -> retrieval/query binds to snapshot identity ``` Watcher/editor notifications remain optional dirty hints only. Harness-owned writes feed the same dirty/edit journal directly and coalesce until a logical flush boundary. ### A — Adversarial review The design must reject these failure modes: 1. **Credential fork** — embedding transport independently resolves/stores secrets: reject; reuse current env-reference/redaction semantics. 2. **Chat-model coupling** — embedding silently inherits active chat provider/model: reject; explicit provider term per call/runtime policy. 3. **Malformed vectors** — NaN/Inf/nonnumeric/mixed dimensions reach retrieval index: structural error before publication. 4. **Batch ambiguity** — provider omits/duplicates/reorders indexes and runtime guesses: reject; require deterministic index reconciliation. 5. **Unbounded payloads** — arbitrary huge texts/batches: host bounds before network dispatch. 6. **Retry duplication** — hidden retry/backtracking causes unexpected paid embedding calls: E1 adds no independent retry engine; later retry/effect adoption must compose with #57/#79 rather than inventing a ledger. 7. **Backend authority laundering** — registering Chroma/custom backend grants network/write/tool authority: reject; registration is availability only. 8. **Score laundering** — distance and cosine similarity merged as if numerically equivalent: reject; retain score kind/direction and explicit normalization provenance. 9. **Stale-vector truth** — Chroma hit from source hash H is presented as current after file is H2: reject at source/snapshot freshness boundary; index is cache/evidence, not authoritative source state. 10. **Mixed project generation** — retrieval combines independently-current files into a supposedly coherent snapshot: reject; source-aware retrieval binds to one published snapshot identity. 11. **Required watcher/daemon** — correctness depends on background file monitor: reject. Chroma/local embedding servers are optional selected adapters, never core runtime requirements. 12. **Half-written source publication** — each harness write immediately reindexes intermediate bytes: reject; dirty journal + bounded coalesced flush. 13. **Concurrent external editor race** — parse H, external write H2, publish facts for H as current: reject via exact-byte/hash revalidation immediately before publication. 14. **Source code becomes authority** — parsed/query metadata becomes callable Prolog or executable adapter path: reject; source observations remain inert closed evidence. 15. **Premature structural retrieval** — #219 duplicates #96-#99 because they are unfinished: reject. ### R/D — Decision gate **Recommended first coding transaction after explicit human approval: E1 only — `rlm_embedding` + OpenAI-compatible embedding transport + deterministic fixture + optional gated real/local smoke.** Do **not** include `rlm_retrieval`, Chroma, source freshness, or RLM tool integration in that first PR. This keeps the dependency graph small and gives later retrieval code a stable, provider-neutral embedding sensor. Per #219's human-gated ADARD contract: **STOP before realization until explicit human approval.**