nsaspy/prolog-rlm

Fork 0

[AP0] Retrieval intelligence, embeddings, source-aware KB, and RLM integration #452

New issue

Closed

opened 2026-09-10 21:18:24 +00:00 by nsaspy · 1 comment

nsaspy commented

2026-09-10 21:18:24 +00:00

Owner

Priority

AP0 — next coding move.

This epic captures the complete design discussed for making prolog-rlm smarter as a reusable library for building LLM harnesses, while preserving the existing default/reference RLM runtime.

This work is ADARD, human-gated. Do not silently advance through the full implementation/merge loop.

Human-gated ADARD contract

Use distinct fresh reasoning phases and preserve evidence/decisions between them:

A — Analyze / research the current code, existing issues, tests, provider/context/tool/agent/source APIs, and relevant external semantics.
D — Design the smallest coherent architecture and API changes.
A — Adversarial review the design for regressions, hidden coupling, stale-KB failures, concurrency races, provider leaks, over-opinionated library behavior, and daemon/process assumptions.
R/D — Decision gate: present the design + adversarial findings and STOP FOR HUMAN APPROVAL before realization.
Realize TDD-first only after explicit human approval.
Verify exact head with deterministic tests/integration gates and report evidence.
STOP FOR HUMAN APPROVAL BEFORE MERGE. No auto-merge.

If implementation uncovers a material architecture change, return to the human decision gate.

0. Architectural thesis / non-negotiables

prolog-rlm is a reusable Prolog library/runtime substrate for writing harnesses. It may ship a default/reference RLM harness, CLI, agent runtime, graph runtime, prompt compiler, etc., but new work must remain composable and reusable by other harnesses.

Think:

prolog-rlm library
  ├─ providers / chains
  ├─ context
  ├─ tools
  ├─ prompt compiler / skills
  ├─ agents
  ├─ graphs
  ├─ MCP / traces
  ├─ source knowledge
  ├─ embeddings              <-- this epic
  ├─ retrieval               <-- this epic
  └─ retrieval expert        <-- this epic
       |
       +--> coding harness
       +--> research harness
       +--> ADARD/RAGE harness
       +--> editor integrations
       +--> other applications

Do not turn the library into a required daemon. Do not require a filesystem watcher. Do not bake one application persona/workflow into core primitives.

The model may request standard structured actions/tool calls, but Prolog/runtime code owns dispatch, capabilities, budgets, validation, state, and execution. Final model prose does not need to be structured.

1. Reuse existing provider layer

Do not duplicate model/provider plumbing in downstream harnesses.

Reuse and extend rlm_chain conventions:

OpenRouter provider support;
generic OpenAI-compatible endpoints;
local OpenAI-compatible servers;
sync/async surfaces;
streaming;
retries/backoff;
structured output/tool schemas;
usage/cost accounting;
structured errors.

Credentials remain referenced indirectly (for example env('OPENROUTER_API_KEY')) and are resolved at execution time. A host/editor may set that environment variable from its own config file, but raw secrets must not leak into provider terms, traces, errors, fixtures, or logs.

2. Provider-neutral embeddings

Add a first-class embedding abstraction separate from chat-completion models.

API direction (exact naming/arity should follow repo conventions):

embed(+Provider, +Text, +Options, -Outcome).
embed_batch(+Provider, +Texts, +Options, -Outcome).
embedding_provider_capability(+Provider, ?Capability).

Requirements:

OpenAI-compatible embeddings first;
local OpenAI-compatible embedding endpoints;
batch support;
model + dimensions + provider provenance;
structured transport/provider errors;
usage where available;
secret redaction;
embeddings are not assumed to share the same provider/model as chat completion.

3. Generic `rlm_retrieval` library

Add backend-neutral retrieval infrastructure. Keep it deliberately dumb and reusable.

Conceptual surface:

retriever_create(...).
retrieval_search(+Retriever, +Query, +Options, -Outcome).
retrieval_fetch(+Retriever, +Ids, +Options, -Outcome).
retrieval_upsert(+Retriever, +Documents, +Options, -Outcome).
retrieval_delete(+Retriever, +Ids, +Options, -Outcome).

Support normalized capabilities/strategies for:

exact lookup;
lexical/BM25-style retrieval where provided by backend;
vector similarity;
metadata filtering;
hybrid retrieval;
project/source structural retrieval;
graph/relationship retrieval;
normalized score/provenance/result envelopes;
bounded result count/output size/cancellation/timeouts.

Do not make Chroma semantics the generic retrieval API.

4. Basic Chroma adapter

Add rlm_chroma (or equivalent) on top of rlm_retrieval.

First useful slice:

connect via Chroma HTTP API;
collection create/get/list where required;
add/upsert documents, embeddings, metadata;
query by embedding/text as supported;
fetch/delete;
bounded result limits;
structured backend/transport errors;
deterministic HTTP fixture tests where practical;
optional real integration lane, separately gated.

Chroma is one adapter, not the architecture.

5. `retrieval_expert`: symbolic/non-LLM retrieval intelligence

Build a separate reusable expert-system layer on top of rlm_retrieval and rlm_embedding.

This is intentionally classic/symbolic AI driving modern generative AI. It must be able to make useful retrieval decisions without calling an LLM.

Conceptual API:

retrieval_plan(+Expert, +Query, +Context, -Plan).
retrieval_execute(+Expert, +Plan, -Evidence).
retrieval_explain(+Expert, +PlanOrEvidence, -Explanation).

Initial intelligence should support extensible rules/facts for:

query classification;
retrieval-source selection;
strategy selection: exact, lexical, vector, graph, structural, hybrid;
source/provenance filtering;
confidence/certainty handling;
evidence-gap detection;
ontology/concept expansion hooks;
truth-maintenance / invalidation when supporting evidence changes;
case-based reasoning / reuse of successful retrieval strategies;
tabling/memoization of repeated semantic subproblems;
constraint-based filtering (project, file, time, entity, source type, provenance, etc.);
cost-aware ordering (cheap exact/structural checks before expensive retrieval where appropriate);
optional non-LLM learned/ranking/bandit policy later behind a stable policy interface.

Embeddings are a sensor, not the decision-maker.

Example intent:

strategy(identifier, exact).
strategy(relationship, graph).
strategy(conceptual, vector).
strategy(broad_research, hybrid).

The expert should be inspectable/explainable and reusable by arbitrary harnesses.

6. Existing source parsing / project-KB plan is part of this epic

Integrate with the existing Project/source work rather than creating a parallel code-index system.

Existing dependency chain:

#94 direct SWI-Prolog <-> Tree-sitter C FFI          DONE
  ↓
#95 Project/File/Language/grammar registry           DONE
  ↓
#96 CST -> versioned Prolog syntax facts
  ↓
#97 Tree-sitter query/capture API
  ↓
#98 symbols / references / calls / imports / exports
  ↓
#99 incremental reparse + changed ranges + freshness

Preserve the existing layering:

source bytes
  -> parser/CST
  -> query captures
  -> normalized common semantic facts
  -> richer language-specific facts
  -> project KB
  -> retrieval
  -> RLM/context consumers

For Prolog source, keep SWI-native semantic analysis where it is stronger than Tree-sitter.

The project KB must remain useful independently of any coding agent or LLM.

7. Source-KB freshness is a hard correctness invariant

A served source-derived KB fact must never silently appear current when it was derived from different source bytes.

Every material source-derived fact must resolve to provenance including at least:

Project;
File identity;
exact source content hash;
file/source generation;
parse generation;
parser backend;
grammar identity/version where applicable;
query/extractor identity/version where applicable;
normalization/schema version where applicable;
source range supporting the fact.

Current-vs-stale state must be explicit.

Do not rely on mtimes alone. Exact bytes/hash are authoritative.

8. No required watcher: lazy freshness is baked into access/loop boundaries

A filesystem watcher may exist only as an optional host hint interface. Correctness must not require a running background process.

Authoritative model:

consumer asks for project knowledge / RLM needs context
        ↓
acquire coherent project snapshot
        ↓
validate relevant current source hashes
        ↓
unchanged -> reuse indexed generation
changed   -> refresh affected source facts
        ↓
return coherent snapshot/evidence

Possible abstraction:

with_project_snapshot(+Project, -Snapshot, :Goal).
ensure_project_fresh(+Project, +Scope, -Outcome).

Do not re-hash the entire repository on every tiny predicate call. Validate at meaningful acquisition/retrieval/RLM-loop boundaries, cache within a coherent snapshot, and narrow work to relevant/dirty files where safely known.

Optional host hint:

source_change_hint(Project, File).

A watcher, Emacs, IDE, Git integration, etc. may call that. It is an optimization only.

9. Harness-owned writes update the KB directly

When a Prolog-RLM tool/harness writes or patches source, the runtime already knows what changed. Exploit that immediately rather than waiting for later rediscovery.

Conceptual flow:

write/apply-patch effect
  -> file + expected old hash
  -> exact edit/range information when available
  -> new bytes/hash
  -> mark/update source generation
  -> incremental Tree-sitter edit/reparse when safe
  -> invalidate/re-extract affected facts
  -> stage new KB generation
  -> atomic publish

Conceptual API direction:

source_commit_edit(+Project, +File, +Edit, +EffectResult, -Outcome).

Do not couple this to one particular file-writing tool. Define a reusable source-update boundary that canonical effectful write tools can call.

10. Multiple writes must be coalesced

Do not thrash parsing/indexing for every intermediate write when several writes occur in one logical tool/agent/RLM step.

Maintain a bounded dirty-set/edit journal:

mark_source_dirty(+Project, +File, +EditOrHint, -Outcome).
flush_source_updates(+Project, +Options, -Outcome).

Conceptual behavior:

write #1 ─┐
write #2 ─┼─> dirty file + bounded edit journal
write #3 ─┘
            ↓
logical flush/loop boundary
            ↓
read authoritative final bytes once
            ↓
hash / incremental-or-full reparse
            ↓
re-extract once
            ↓
atomic new KB generation

Intermediate half-written source must not become a supposedly complete current KB generation.

11. Multiple programs/editors and concurrent writers

A file being open in multiple programs is normal. No open handle implies ownership.

Use optimistic concurrency and exact-byte validation.

For indexing/publication:

read exact bytes
  -> hash H
  -> parse/extract those bytes
  -> before publishing, verify authoritative file bytes/hash still equal H
  -> same: publish
  -> changed: discard stale candidate and retry/rebase according to bounded policy

For harness writes, use expected-hash/generation semantics where possible:

apply_patch(File, ExpectedHash, Patch, Outcome).

If another program changed the file between read and write, return a structured conflict rather than clobbering or publishing a KB based on stale assumptions.

Concurrent source publication must serialize/linearize per relevant Project/File generation and never expose mixed generations as one complete current snapshot.

12. Coherent snapshot semantics

RLM/retrieval consumers should query a coherent project/source snapshot rather than observe a mixture of:

old generation for file A;
new generation for file B;
half-reindexed file C.

Provide a snapshot identity/reference and make current queries bind to it where appropriate.

Conceptual direction:

project_snapshot(+Project, -Snapshot).
project_kb_query(+Snapshot, +Query, -Outcome).

Publishing a new generation must be atomic at the defined visibility boundary. Reindexing/error/partial states must never masquerade as complete current evidence.

13. RLM integration: two entry points

Retrieval/source knowledge must plug into the existing RLM as a library capability, not become a second orchestration system.

A. Initial context acquisition

Before the planner/model receives project evidence:

Query
  -> acquire fresh coherent Project snapshot
  -> retrieval_expert chooses retrieval plan
  -> rlm_retrieval executes
  -> evidence becomes bounded rlm_context/provider projection
  -> RLM planner/model

This should compose with the existing context-budget/prompt-compiler machinery.

B. Retrieval during recursive execution

A running RLM/subagent must be able to request more evidence through the normal typed/tool execution boundary.

Conceptual operation/tool:

retrieve(Query, Options, Evidence)

The model expresses an information need; retrieval_expert decides how to satisfy it using exact/lexical/vector/graph/source retrieval. Model output does not bypass retrieval policy or source-freshness guarantees.

Do not invent an incompatible proprietary model protocol if standard tool/function-calling + internal Prolog terms already suffice.

14. Prompt/tool boundary

Keep the existing distinction clear:

ordinary final LLM prose may remain text;
machine actions/tool calls/subagent calls/context operations use structured requests so the runtime can reliably dispatch them;
structured requests do not themselves grant authority;
prompt/skill/compiler projection decides what is visible/relevant;
capabilities/authority/effect boundaries decide what may execute.

The library supplies prompting/selection/runtime mechanics; individual harnesses own their domain persona, goals, project workflow, and application-level instructions.

15. Testing / acceptance

At minimum, the eventual implementation path must prove:

Embeddings/retrieval

provider-neutral embedding works against deterministic fixture and one explicitly gated real/local-compatible endpoint;
Chroma adapter round-trips documents/embeddings/metadata/query/fetch/delete;
retrieval-expert can choose different strategies without an LLM and explain the choice;
hybrid/project structural retrieval can be composed without backend-specific leakage.

Source knowledge

current facts are bound to exact source hash/generation;
external edit is detected lazily on next relevant snapshot/access;
a harness-owned write can update/dirty the KB immediately;
multiple writes coalesce into one coherent publication;
concurrent external modification between parse and publish causes candidate rejection/retry, not stale publication;
expected-hash write conflict is structured;
stale/partial/error generations cannot satisfy a query requiring current complete evidence;
RLM context sees a coherent snapshot;
no watcher/background daemon is required for correctness.

Regression

existing rlm_chain, rlm_context, rlm_tool, rlm_agent, rlm_graph, prompt compiler, MCP, trace, and source APIs remain usable independently;
no new mandatory Python/Node runtime;
no raw secrets in traces/errors;
no source parsing grants execution authority;
no model-provided grammar/library path gains ambient dlopen authority;
no arbitrary source text becomes executable Prolog.

16. First coding move after human design approval

Do not try to land the entire epic in one PR.

During ADARD design, identify the smallest dependency-correct vertical slice. Strong candidate ordering to evaluate:

provider-neutral embedding contract + deterministic OpenAI-compatible fixture;
generic retrieval result/backend contract;
minimal Chroma adapter;
retrieval-expert rule/plan skeleton with deterministic exact-vs-vector choice;
snapshot/freshness boundary wired to existing Project/source registry;
write dirty/coalescing + optimistic hash validation;
integration into rlm_context/RLM retrieval path;
continue #96-#99 source semantic pipeline as required by structural retrieval.

The ADARD design/adversarial phases may reorder these if repository dependencies prove a different smallest safe slice.

Human approval is required before realization and again before merge.

Refs: #93 #94 #95 #96 #97 #98 #99 and existing provider/context/tool/agent/graph/prompt compiler work.

## Priority **AP0 — next coding move.** This epic captures the complete design discussed for making `prolog-rlm` smarter as a reusable **library for building LLM harnesses**, while preserving the existing default/reference RLM runtime. This work is **ADARD, human-gated**. Do not silently advance through the full implementation/merge loop. ### Human-gated ADARD contract Use distinct fresh reasoning phases and preserve evidence/decisions between them: 1. **A — Analyze / research** the current code, existing issues, tests, provider/context/tool/agent/source APIs, and relevant external semantics. 2. **D — Design** the smallest coherent architecture and API changes. 3. **A — Adversarial review** the design for regressions, hidden coupling, stale-KB failures, concurrency races, provider leaks, over-opinionated library behavior, and daemon/process assumptions. 4. **R/D — Decision gate**: present the design + adversarial findings and **STOP FOR HUMAN APPROVAL** before realization. 5. **Realize TDD-first** only after explicit human approval. 6. **Verify exact head** with deterministic tests/integration gates and report evidence. 7. **STOP FOR HUMAN APPROVAL BEFORE MERGE**. No auto-merge. If implementation uncovers a material architecture change, return to the human decision gate. --- # 0. Architectural thesis / non-negotiables `prolog-rlm` is a reusable Prolog library/runtime substrate for **writing harnesses**. It may ship a default/reference RLM harness, CLI, agent runtime, graph runtime, prompt compiler, etc., but new work must remain composable and reusable by other harnesses. Think: ```text prolog-rlm library ├─ providers / chains ├─ context ├─ tools ├─ prompt compiler / skills ├─ agents ├─ graphs ├─ MCP / traces ├─ source knowledge ├─ embeddings <-- this epic ├─ retrieval <-- this epic └─ retrieval expert <-- this epic | +--> coding harness +--> research harness +--> ADARD/RAGE harness +--> editor integrations +--> other applications ``` Do **not** turn the library into a required daemon. Do **not** require a filesystem watcher. Do **not** bake one application persona/workflow into core primitives. The model may request standard structured actions/tool calls, but Prolog/runtime code owns dispatch, capabilities, budgets, validation, state, and execution. Final model prose does not need to be structured. --- # 1. Reuse existing provider layer Do not duplicate model/provider plumbing in downstream harnesses. Reuse and extend `rlm_chain` conventions: - OpenRouter provider support; - generic OpenAI-compatible endpoints; - local OpenAI-compatible servers; - sync/async surfaces; - streaming; - retries/backoff; - structured output/tool schemas; - usage/cost accounting; - structured errors. Credentials remain referenced indirectly (for example `env('OPENROUTER_API_KEY')`) and are resolved at execution time. A host/editor may set that environment variable from its own config file, but raw secrets must not leak into provider terms, traces, errors, fixtures, or logs. --- # 2. Provider-neutral embeddings Add a first-class embedding abstraction separate from chat-completion models. API direction (exact naming/arity should follow repo conventions): ```prolog embed(+Provider, +Text, +Options, -Outcome). embed_batch(+Provider, +Texts, +Options, -Outcome). embedding_provider_capability(+Provider, ?Capability). ``` Requirements: - OpenAI-compatible embeddings first; - local OpenAI-compatible embedding endpoints; - batch support; - model + dimensions + provider provenance; - structured transport/provider errors; - usage where available; - secret redaction; - embeddings are not assumed to share the same provider/model as chat completion. --- # 3. Generic `rlm_retrieval` library Add backend-neutral retrieval infrastructure. Keep it deliberately **dumb** and reusable. Conceptual surface: ```prolog retriever_create(...). retrieval_search(+Retriever, +Query, +Options, -Outcome). retrieval_fetch(+Retriever, +Ids, +Options, -Outcome). retrieval_upsert(+Retriever, +Documents, +Options, -Outcome). retrieval_delete(+Retriever, +Ids, +Options, -Outcome). ``` Support normalized capabilities/strategies for: - exact lookup; - lexical/BM25-style retrieval where provided by backend; - vector similarity; - metadata filtering; - hybrid retrieval; - project/source structural retrieval; - graph/relationship retrieval; - normalized score/provenance/result envelopes; - bounded result count/output size/cancellation/timeouts. Do not make Chroma semantics the generic retrieval API. --- # 4. Basic Chroma adapter Add `rlm_chroma` (or equivalent) on top of `rlm_retrieval`. First useful slice: - connect via Chroma HTTP API; - collection create/get/list where required; - add/upsert documents, embeddings, metadata; - query by embedding/text as supported; - fetch/delete; - bounded result limits; - structured backend/transport errors; - deterministic HTTP fixture tests where practical; - optional real integration lane, separately gated. Chroma is one adapter, not the architecture. --- # 5. `retrieval_expert`: symbolic/non-LLM retrieval intelligence Build a separate reusable expert-system layer **on top of** `rlm_retrieval` and `rlm_embedding`. This is intentionally classic/symbolic AI driving modern generative AI. It must be able to make useful retrieval decisions **without calling an LLM**. Conceptual API: ```prolog retrieval_plan(+Expert, +Query, +Context, -Plan). retrieval_execute(+Expert, +Plan, -Evidence). retrieval_explain(+Expert, +PlanOrEvidence, -Explanation). ``` Initial intelligence should support extensible rules/facts for: - query classification; - retrieval-source selection; - strategy selection: `exact`, `lexical`, `vector`, `graph`, `structural`, `hybrid`; - source/provenance filtering; - confidence/certainty handling; - evidence-gap detection; - ontology/concept expansion hooks; - truth-maintenance / invalidation when supporting evidence changes; - case-based reasoning / reuse of successful retrieval strategies; - tabling/memoization of repeated semantic subproblems; - constraint-based filtering (project, file, time, entity, source type, provenance, etc.); - cost-aware ordering (cheap exact/structural checks before expensive retrieval where appropriate); - optional non-LLM learned/ranking/bandit policy later behind a stable policy interface. Embeddings are a **sensor**, not the decision-maker. Example intent: ```prolog strategy(identifier, exact). strategy(relationship, graph). strategy(conceptual, vector). strategy(broad_research, hybrid). ``` The expert should be inspectable/explainable and reusable by arbitrary harnesses. --- # 6. Existing source parsing / project-KB plan is part of this epic Integrate with the existing Project/source work rather than creating a parallel code-index system. Existing dependency chain: ```text #94 direct SWI-Prolog <-> Tree-sitter C FFI DONE ↓ #95 Project/File/Language/grammar registry DONE ↓ #96 CST -> versioned Prolog syntax facts ↓ #97 Tree-sitter query/capture API ↓ #98 symbols / references / calls / imports / exports ↓ #99 incremental reparse + changed ranges + freshness ``` Preserve the existing layering: ```text source bytes -> parser/CST -> query captures -> normalized common semantic facts -> richer language-specific facts -> project KB -> retrieval -> RLM/context consumers ``` For Prolog source, keep SWI-native semantic analysis where it is stronger than Tree-sitter. The project KB must remain useful independently of any coding agent or LLM. --- # 7. Source-KB freshness is a hard correctness invariant A served source-derived KB fact must never silently appear current when it was derived from different source bytes. Every material source-derived fact must resolve to provenance including at least: - Project; - File identity; - exact source content hash; - file/source generation; - parse generation; - parser backend; - grammar identity/version where applicable; - query/extractor identity/version where applicable; - normalization/schema version where applicable; - source range supporting the fact. Current-vs-stale state must be explicit. **Do not rely on mtimes alone. Exact bytes/hash are authoritative.** --- # 8. No required watcher: lazy freshness is baked into access/loop boundaries A filesystem watcher may exist only as an **optional host hint interface**. Correctness must not require a running background process. Authoritative model: ```text consumer asks for project knowledge / RLM needs context ↓ acquire coherent project snapshot ↓ validate relevant current source hashes ↓ unchanged -> reuse indexed generation changed -> refresh affected source facts ↓ return coherent snapshot/evidence ``` Possible abstraction: ```prolog with_project_snapshot(+Project, -Snapshot, :Goal). ensure_project_fresh(+Project, +Scope, -Outcome). ``` Do not re-hash the entire repository on every tiny predicate call. Validate at meaningful acquisition/retrieval/RLM-loop boundaries, cache within a coherent snapshot, and narrow work to relevant/dirty files where safely known. Optional host hint: ```prolog source_change_hint(Project, File). ``` A watcher, Emacs, IDE, Git integration, etc. may call that. It is an optimization only. --- # 9. Harness-owned writes update the KB directly When a Prolog-RLM tool/harness writes or patches source, the runtime already knows what changed. Exploit that immediately rather than waiting for later rediscovery. Conceptual flow: ```text write/apply-patch effect -> file + expected old hash -> exact edit/range information when available -> new bytes/hash -> mark/update source generation -> incremental Tree-sitter edit/reparse when safe -> invalidate/re-extract affected facts -> stage new KB generation -> atomic publish ``` Conceptual API direction: ```prolog source_commit_edit(+Project, +File, +Edit, +EffectResult, -Outcome). ``` Do not couple this to one particular file-writing tool. Define a reusable source-update boundary that canonical effectful write tools can call. --- # 10. Multiple writes must be coalesced Do not thrash parsing/indexing for every intermediate write when several writes occur in one logical tool/agent/RLM step. Maintain a bounded dirty-set/edit journal: ```prolog mark_source_dirty(+Project, +File, +EditOrHint, -Outcome). flush_source_updates(+Project, +Options, -Outcome). ``` Conceptual behavior: ```text write #1 ─┐ write #2 ─┼─> dirty file + bounded edit journal write #3 ─┘ ↓ logical flush/loop boundary ↓ read authoritative final bytes once ↓ hash / incremental-or-full reparse ↓ re-extract once ↓ atomic new KB generation ``` Intermediate half-written source must not become a supposedly complete current KB generation. --- # 11. Multiple programs/editors and concurrent writers A file being open in multiple programs is normal. No open handle implies ownership. Use optimistic concurrency and exact-byte validation. For indexing/publication: ```text read exact bytes -> hash H -> parse/extract those bytes -> before publishing, verify authoritative file bytes/hash still equal H -> same: publish -> changed: discard stale candidate and retry/rebase according to bounded policy ``` For harness writes, use expected-hash/generation semantics where possible: ```prolog apply_patch(File, ExpectedHash, Patch, Outcome). ``` If another program changed the file between read and write, return a structured conflict rather than clobbering or publishing a KB based on stale assumptions. Concurrent source publication must serialize/linearize per relevant Project/File generation and never expose mixed generations as one complete current snapshot. --- # 12. Coherent snapshot semantics RLM/retrieval consumers should query a coherent project/source snapshot rather than observe a mixture of: - old generation for file A; - new generation for file B; - half-reindexed file C. Provide a snapshot identity/reference and make current queries bind to it where appropriate. Conceptual direction: ```prolog project_snapshot(+Project, -Snapshot). project_kb_query(+Snapshot, +Query, -Outcome). ``` Publishing a new generation must be atomic at the defined visibility boundary. Reindexing/error/partial states must never masquerade as complete current evidence. --- # 13. RLM integration: two entry points Retrieval/source knowledge must plug into the existing RLM as a library capability, not become a second orchestration system. ## A. Initial context acquisition Before the planner/model receives project evidence: ```text Query -> acquire fresh coherent Project snapshot -> retrieval_expert chooses retrieval plan -> rlm_retrieval executes -> evidence becomes bounded rlm_context/provider projection -> RLM planner/model ``` This should compose with the existing context-budget/prompt-compiler machinery. ## B. Retrieval during recursive execution A running RLM/subagent must be able to request more evidence through the normal typed/tool execution boundary. Conceptual operation/tool: ```prolog retrieve(Query, Options, Evidence) ``` The model expresses an information need; `retrieval_expert` decides how to satisfy it using exact/lexical/vector/graph/source retrieval. Model output does not bypass retrieval policy or source-freshness guarantees. Do not invent an incompatible proprietary model protocol if standard tool/function-calling + internal Prolog terms already suffice. --- # 14. Prompt/tool boundary Keep the existing distinction clear: - ordinary final LLM prose may remain text; - machine actions/tool calls/subagent calls/context operations use structured requests so the runtime can reliably dispatch them; - structured requests do not themselves grant authority; - prompt/skill/compiler projection decides what is visible/relevant; - capabilities/authority/effect boundaries decide what may execute. The library supplies prompting/selection/runtime mechanics; individual harnesses own their domain persona, goals, project workflow, and application-level instructions. --- # 15. Testing / acceptance At minimum, the eventual implementation path must prove: ### Embeddings/retrieval - provider-neutral embedding works against deterministic fixture and one explicitly gated real/local-compatible endpoint; - Chroma adapter round-trips documents/embeddings/metadata/query/fetch/delete; - retrieval-expert can choose different strategies without an LLM and explain the choice; - hybrid/project structural retrieval can be composed without backend-specific leakage. ### Source knowledge - current facts are bound to exact source hash/generation; - external edit is detected lazily on next relevant snapshot/access; - a harness-owned write can update/dirty the KB immediately; - multiple writes coalesce into one coherent publication; - concurrent external modification between parse and publish causes candidate rejection/retry, not stale publication; - expected-hash write conflict is structured; - stale/partial/error generations cannot satisfy a query requiring current complete evidence; - RLM context sees a coherent snapshot; - no watcher/background daemon is required for correctness. ### Regression - existing `rlm_chain`, `rlm_context`, `rlm_tool`, `rlm_agent`, `rlm_graph`, prompt compiler, MCP, trace, and source APIs remain usable independently; - no new mandatory Python/Node runtime; - no raw secrets in traces/errors; - no source parsing grants execution authority; - no model-provided grammar/library path gains ambient `dlopen` authority; - no arbitrary source text becomes executable Prolog. --- # 16. First coding move after human design approval Do **not** try to land the entire epic in one PR. During ADARD design, identify the smallest dependency-correct vertical slice. Strong candidate ordering to evaluate: 1. provider-neutral embedding contract + deterministic OpenAI-compatible fixture; 2. generic retrieval result/backend contract; 3. minimal Chroma adapter; 4. retrieval-expert rule/plan skeleton with deterministic exact-vs-vector choice; 5. snapshot/freshness boundary wired to existing Project/source registry; 6. write dirty/coalescing + optimistic hash validation; 7. integration into `rlm_context`/RLM retrieval path; 8. continue #96-#99 source semantic pipeline as required by structural retrieval. The ADARD design/adversarial phases may reorder these if repository dependencies prove a different smallest safe slice. **Human approval is required before realization and again before merge.** Refs: #93 #94 #95 #96 #97 #98 #99 and existing provider/context/tool/agent/graph/prompt compiler work.

~~nsaspy referenced this issue 2026-09-10 21:20:52 +00:00~~

[AP0] Retrieval intelligence, embeddings, source-aware KB, and RLM integration #219

nsaspy commented

2026-09-10 21:21:39 +00:00

Author

Owner

Duplicate of #219 (pre-existing Forgejo mirror). Closing this accidental duplicate created by today's open-state sync; #219 stays canonical on Forgejo.

nsaspy closed this issue

2026-09-10 21:21:40 +00:00