nsaspy/prolog-rlm

Fork 0

[EPIC][expert] Retrieval / evidence expert: symbolic source selection, evidence gaps, and explainable retrieval strategy #417

New issue

Open

opened 2026-09-08 01:29:22 +00:00 by nsaspy · 5 comments

nsaspy commented

2026-09-08 01:29:22 +00:00

Owner

Parent: #376
Depends on: #377
Consumes/integrates with: #219, #380

Goal

Extract the expert-system portion of #219 into a first-class Retrieval / Evidence Expert that decides how to satisfy an information need using symbolic rules and current evidence, without requiring an LLM.

Embeddings and retrieval backends are sensors/capabilities. They do not decide the research strategy.

Responsibilities

Given a normalized information need, determine:

whether the answer is already present in current symbolic/project knowledge;
which source classes are appropriate;
which retrieval strategy should run: exact, structural, graph, lexical, vector, hybrid, external source, or combinations;
ordering by expected value/cost/work;
filters by project/file/entity/time/provenance/source type;
whether current evidence is sufficient, contradictory, stale or incomplete;
which evidence gap to pursue next;
when additional retrieval has stopped producing useful evidence.

Symbolic policy

Initial policy should support rules equivalent to:

strategy(identifier, exact).
strategy(symbol_relation, structural).
strategy(dependency, graph).
strategy(conceptual, vector).
strategy(broad_question, hybrid).

Real policy should consider observed state, available backends, provenance requirements, freshness, prior attempts and ambiguity.

Use tabling/case reuse for repeated semantic subproblems where safe. Persist strategy statistics separately from authoritative facts.

Evidence contract

Return structured evidence with source/provenance, not just chunks of text.

Distinguish:

retrieved observation
derived relation
candidate fact
corroborated fact
conflicting evidence
stale evidence
missing evidence

The expert should be able to explain why it selected a source/strategy and what gap remains.

Interaction with #219

#219 remains the infrastructure/source-retrieval umbrella for embeddings, backend-neutral retrieval, Chroma/adapters and freshness integration.

This epic owns the expert reasoning layer over those capabilities. Do not duplicate the backend API, embedding abstraction or source freshness implementation.

LLM policy

Normal retrieval planning uses zero model calls.

An LLM may be an explicitly configured semantic fallback or optional summarizer, but:

only the real fallback call is metered;
model output does not become evidence by default;
retrieved primary/observable data retains source authority;
any model-derived query expansion is candidate data and traceable.

Cooperation

Project Knowledge Expert is checked before expensive retrieval.
Planner delegates unresolved evidence needs here.
Verify Expert requests missing evidence classes here when verification is unknown.
Repair Expert may ask for evidence needed to discriminate failure hypotheses.

Acceptance

Chooses exact vs structural vs vector/hybrid strategy deterministically on fixtures with zero model calls.
Avoids external/vector work when Project Knowledge already satisfies the need.
Detects one weak source and requests independent corroboration under configured rules.
Preserves conflicting evidence rather than selecting a convenient answer.
Stale evidence cannot satisfy a current-evidence requirement.
Repeated equivalent failed retrieval reaches a bounded exhausted/no-progress state.
Source/strategy selection is explainable from rules and observed state.
Optional model query expansion is separately traced/metered and cannot create authoritative evidence.
Integrates with backend-neutral retrieval from #219 without backend-specific logic leaking into the expert contract.

Non-goals

No second retrieval backend API.
No assumption that embeddings are authoritative reasoning.
No model-required retrieval policy.

Refs #376 #377 #219 #380 #93

Parent: #376 Depends on: #377 Consumes/integrates with: #219, #380 ## Goal Extract the expert-system portion of #219 into a first-class **Retrieval / Evidence Expert** that decides how to satisfy an information need using symbolic rules and current evidence, without requiring an LLM. Embeddings and retrieval backends are sensors/capabilities. They do not decide the research strategy. ## Responsibilities Given a normalized information need, determine: - whether the answer is already present in current symbolic/project knowledge; - which source classes are appropriate; - which retrieval strategy should run: exact, structural, graph, lexical, vector, hybrid, external source, or combinations; - ordering by expected value/cost/work; - filters by project/file/entity/time/provenance/source type; - whether current evidence is sufficient, contradictory, stale or incomplete; - which evidence gap to pursue next; - when additional retrieval has stopped producing useful evidence. ## Symbolic policy Initial policy should support rules equivalent to: ```prolog strategy(identifier, exact). strategy(symbol_relation, structural). strategy(dependency, graph). strategy(conceptual, vector). strategy(broad_question, hybrid). ``` Real policy should consider observed state, available backends, provenance requirements, freshness, prior attempts and ambiguity. Use tabling/case reuse for repeated semantic subproblems where safe. Persist strategy statistics separately from authoritative facts. ## Evidence contract Return structured evidence with source/provenance, not just chunks of text. Distinguish: ```text retrieved observation derived relation candidate fact corroborated fact conflicting evidence stale evidence missing evidence ``` The expert should be able to explain why it selected a source/strategy and what gap remains. ## Interaction with #219 #219 remains the infrastructure/source-retrieval umbrella for embeddings, backend-neutral retrieval, Chroma/adapters and freshness integration. This epic owns the **expert reasoning layer** over those capabilities. Do not duplicate the backend API, embedding abstraction or source freshness implementation. ## LLM policy Normal retrieval planning uses zero model calls. An LLM may be an explicitly configured semantic fallback or optional summarizer, but: - only the real fallback call is metered; - model output does not become evidence by default; - retrieved primary/observable data retains source authority; - any model-derived query expansion is candidate data and traceable. ## Cooperation - Project Knowledge Expert is checked before expensive retrieval. - Planner delegates unresolved evidence needs here. - Verify Expert requests missing evidence classes here when verification is unknown. - Repair Expert may ask for evidence needed to discriminate failure hypotheses. ## Acceptance - [ ] Chooses exact vs structural vs vector/hybrid strategy deterministically on fixtures with zero model calls. - [ ] Avoids external/vector work when Project Knowledge already satisfies the need. - [ ] Detects one weak source and requests independent corroboration under configured rules. - [ ] Preserves conflicting evidence rather than selecting a convenient answer. - [ ] Stale evidence cannot satisfy a current-evidence requirement. - [ ] Repeated equivalent failed retrieval reaches a bounded exhausted/no-progress state. - [ ] Source/strategy selection is explainable from rules and observed state. - [ ] Optional model query expansion is separately traced/metered and cannot create authoritative evidence. - [ ] Integrates with backend-neutral retrieval from #219 without backend-specific logic leaking into the expert contract. ## Non-goals - No second retrieval backend API. - No assumption that embeddings are authoritative reasoning. - No model-required retrieval policy. Refs #376 #377 #219 #380 #93

nsaspy commented

2026-09-08 01:46:38 +00:00

Author

Owner

Machine Spirit #403A handoff — TAPS retrieval contract

#403A establishes a required distinction for this expert:

hard semantic admission + mandatory dependency/support closure
    !=
soft relevance/candidate ranking

The Retrieval/Evidence Expert should continue to own strategy choice and evidence-gap pursuit, but its result should carry a TAPS-style projection/receipt with at least:

exact world-model/package cut + freshness/profile/namespace context;
seeds and discovery strategy (exact, structural, lexical, case, graph_activation, vector, optional model expansion);
hard admissibility result before ranking;
mandatory rule/premise/guard/justification/counterevidence dependencies;
support_closure = closed|partial|resource_limited|stale;
search_coverage = exhaustive_for_declared_fragment|bounded_structural|heuristic|partial_backend|unknown;
unresolved frontier/evidence gaps;
inspectable rank features and deterministic tie-break receipt.

A support-closed returned result may still have heuristic search coverage. Do not report it as exhaustive merely because its included proof is self-contained.

Hard invariants:

retrieval score != truth/warrant
ranking != trust
similarity != identity
omitted != false
vector/model expansion != evidence

Exact/structural symbolic routes precede vector/model work when they can satisfy the need. MMR/diversity/source-dependence may shape soft periphery but cannot drop mandatory counterevidence or semantic dependencies.

This is a design handoff only; implementation remains open. Refs #403 #400 #402 #219.

## Machine Spirit #403A handoff — TAPS retrieval contract #403A establishes a required distinction for this expert: ```text hard semantic admission + mandatory dependency/support closure != soft relevance/candidate ranking ``` The Retrieval/Evidence Expert should continue to own **strategy choice and evidence-gap pursuit**, but its result should carry a TAPS-style projection/receipt with at least: - exact world-model/package cut + freshness/profile/namespace context; - seeds and discovery strategy (`exact`, `structural`, `lexical`, `case`, `graph_activation`, `vector`, optional model expansion); - hard admissibility result before ranking; - mandatory rule/premise/guard/justification/counterevidence dependencies; - `support_closure = closed|partial|resource_limited|stale`; - `search_coverage = exhaustive_for_declared_fragment|bounded_structural|heuristic|partial_backend|unknown`; - unresolved frontier/evidence gaps; - inspectable rank features and deterministic tie-break receipt. A support-closed returned result may still have heuristic search coverage. Do not report it as exhaustive merely because its included proof is self-contained. Hard invariants: ```text retrieval score != truth/warrant ranking != trust similarity != identity omitted != false vector/model expansion != evidence ``` Exact/structural symbolic routes precede vector/model work when they can satisfy the need. MMR/diversity/source-dependence may shape soft periphery but cannot drop mandatory counterevidence or semantic dependencies. This is a design handoff only; implementation remains open. Refs #403 #400 #402 #219.

nsaspy commented

2026-09-08 01:46:38 +00:00

Author

Owner

Machine Spirit #403B handoff — adversarial retrieval obligations

#403B is complete and refines TAPS into OATH-TAPS. Retrieval/Evidence Expert should not treat one ranked candidate set as the evidence universe.

Canonical handoff:

represent typed retrieval obligations such as seek_support, seek_refutation, seek_exception_or_defeater, seek_fresher_state, seek_identity_counterexample, and seek_independent_origin;
trigger stronger obligation sets for contested/current/VERIFY/action-sensitive evidence profiles;
canonical-revalidate lexical/vector/graph hits against the exact admitted world-model cut before using them as evidence;
collapse source-dependence/origin components before diversity/corroboration reasoning;
preserve backend/sensor disagreement rather than averaging it into one confidence;
report support_closure, search_coverage, opposition_coverage, selection integrity, and unresolved exposure gaps;
never infer no defeater exists merely because top-k contained none;
source/query text cannot self-set pin/mandatory/trust/attention-policy metadata;
poisoning suspicion is not semantic false; retriever agreement is not independent corroboration;
backend exhaustion/disagreement is explicit degraded/partial evidence, never an implicit LLM bailout.

The substantive design, sources, APIs and 34 fixtures are on #403B. Refs #403 #397 #400 #402.

## Machine Spirit #403B handoff — adversarial retrieval obligations #403B is complete and refines TAPS into **OATH-TAPS**. Retrieval/Evidence Expert should not treat one ranked candidate set as the evidence universe. Canonical handoff: - represent typed retrieval obligations such as `seek_support`, `seek_refutation`, `seek_exception_or_defeater`, `seek_fresher_state`, `seek_identity_counterexample`, and `seek_independent_origin`; - trigger stronger obligation sets for contested/current/VERIFY/action-sensitive evidence profiles; - canonical-revalidate lexical/vector/graph hits against the exact admitted world-model cut before using them as evidence; - collapse source-dependence/origin components before diversity/corroboration reasoning; - preserve backend/sensor disagreement rather than averaging it into one confidence; - report `support_closure`, `search_coverage`, `opposition_coverage`, selection integrity, and unresolved exposure gaps; - never infer `no defeater exists` merely because top-k contained none; - source/query text cannot self-set pin/mandatory/trust/attention-policy metadata; - poisoning suspicion is not semantic false; retriever agreement is not independent corroboration; - backend exhaustion/disagreement is explicit degraded/partial evidence, never an implicit LLM bailout. The substantive design, sources, APIs and 34 fixtures are on #403B. Refs #403 #397 #400 #402.

nsaspy commented

2026-09-08 01:46:39 +00:00

Author

Owner

MACHINE-SPIRIT #403C handoff — MOSAIC-TAPS

Depth 6C establishes the interoperability contract for this expert. #381 should own semantic retrieval orchestration, not backend mechanics:

select retrievers by versioned Retrieval Semantic Contracts (RSCs): query classes, identity/source domain, result/ordering semantics, coverage, freshness/snapshot, filters, provenance, threat claims, egress class;
compile typed attention obligations into a bounded DAG and backend-native requests through loss-aware translators;
compose with typed operators (union, cascade, fallback, intersect, join, refine, corroborate, oppose, freshen);
preserve backend-local BM25/vector/graph/rank values rather than inventing one universal confidence;
use RRF only for soft candidate rank fusion when score calibration is absent; fusion score is selection provenance, not warrant;
aggregate coverage/freshness/robustness conservatively and expose unsatisfied obligations;
keep backend independence separate from source independence and use #402/#symbolic-memory identity/dependence lineage;
capability-check remote query egress independently from local read authority.

#219 remains responsible for primitive retrieval adapters/RSC metadata. Exact implementation/scaling algorithms are reserved for #403D.

Refs #397 #403 #219 #396 and symbolic-memory#7/#9/#10.

### MACHINE-SPIRIT #403C handoff — MOSAIC-TAPS Depth 6C establishes the interoperability contract for this expert. #381 should own **semantic retrieval orchestration**, not backend mechanics: - select retrievers by versioned Retrieval Semantic Contracts (RSCs): query classes, identity/source domain, result/ordering semantics, coverage, freshness/snapshot, filters, provenance, threat claims, egress class; - compile typed attention obligations into a bounded DAG and backend-native requests through loss-aware translators; - compose with typed operators (`union`, `cascade`, `fallback`, `intersect`, `join`, `refine`, `corroborate`, `oppose`, `freshen`); - preserve backend-local BM25/vector/graph/rank values rather than inventing one universal confidence; - use RRF only for soft candidate rank fusion when score calibration is absent; fusion score is selection provenance, not warrant; - aggregate coverage/freshness/robustness conservatively and expose unsatisfied obligations; - keep backend independence separate from source independence and use #402/#symbolic-memory identity/dependence lineage; - capability-check remote query egress independently from local read authority. #219 remains responsible for primitive retrieval adapters/RSC metadata. Exact implementation/scaling algorithms are reserved for #403D. Refs #397 #403 #219 #396 and symbolic-memory#7/#9/#10.

nsaspy commented

2026-09-08 01:46:39 +00:00

Author

Owner

Machine Spirit #403D handoff — RACE-TAPS physical execution

Depth 6D is design-complete. #381 remains the owner of logical retrieval obligations, source/strategy selection, typed composition and gap policy. Add/implement a physical-plan boundary rather than another expert:

#381 logical obligation/composition plan
  -> RSC capability matching
  -> bounded memoized physical alternatives
  -> deterministic async execution
  -> optional conformance-preserving physical revisions
  -> normalized evidence + coverage/gaps receipt

Required D invariants:

logical plan != physical plan;
cost/cardinality/latency/ANN-recall statistics are operational, never epistemic warrant;
physical adaptation may substitute/reorder compatible implementations but may not drop/weakly satisfy an obligation silently;
hard filters require explicit RSC support class (native_prefilter_exact, native_filtered_ann(profile,recall), postfilter_only, unsupported); ANN top-k then postfilter cannot claim exhaustive filtered search;
async completion order cannot affect normalized results/fusion;
snapshot/index/translator/RSC/identity/source-dependence/freshness/authority generations belong in plan/cache fingerprints;
remote timeout/cancel becomes an explicit coverage gap, not absence;
exact replay of mutable remote results requires retained response artifacts; live requery is new lineage;
Fagin/TA-style early stop only when its monotone aggregation/access assumptions hold;
known symbolic paths must execute with model_calls = 0.

The full mechanisms, typed records, rejected alternatives and 44 D fixtures are on #403D. This does not authorize implementation outside #381's existing human-gated rules.

## Machine Spirit #403D handoff — RACE-TAPS physical execution Depth 6D is design-complete. #381 remains the owner of **logical retrieval obligations, source/strategy selection, typed composition and gap policy**. Add/implement a physical-plan boundary rather than another expert: ```text #381 logical obligation/composition plan -> RSC capability matching -> bounded memoized physical alternatives -> deterministic async execution -> optional conformance-preserving physical revisions -> normalized evidence + coverage/gaps receipt ``` Required D invariants: - logical plan != physical plan; - cost/cardinality/latency/ANN-recall statistics are operational, never epistemic warrant; - physical adaptation may substitute/reorder compatible implementations but may not drop/weakly satisfy an obligation silently; - hard filters require explicit RSC support class (`native_prefilter_exact`, `native_filtered_ann(profile,recall)`, `postfilter_only`, `unsupported`); ANN top-k then postfilter cannot claim exhaustive filtered search; - async completion order cannot affect normalized results/fusion; - snapshot/index/translator/RSC/identity/source-dependence/freshness/authority generations belong in plan/cache fingerprints; - remote timeout/cancel becomes an explicit coverage gap, not absence; - exact replay of mutable remote results requires retained response artifacts; live requery is new lineage; - Fagin/TA-style early stop only when its monotone aggregation/access assumptions hold; - known symbolic paths must execute with `model_calls = 0`. The full mechanisms, typed records, rejected alternatives and 44 D fixtures are on #403D. This does not authorize implementation outside #381's existing human-gated rules.

nsaspy commented

2026-09-08 01:46:39 +00:00

Author

Owner

Machine Spirit #404B / CITADEL-EPOCH handoff

If retrieval-strategy statistics are learned, keep them advisory and preserve enough eligibility/selection lineage to detect selective-evidence/self-confirmation attacks.

A learned retrieval strategy may optimize physical/operational choice only inside the existing OATH/MOSAIC/RACE obligation contract. It may not suppress opposition/defeater/freshness/independent-origin obligations, reinterpret absence as evidence, or mark its own retrieved corpus independent. If a candidate influences which evidence is gathered, that influence is part of its exposure closure and may contaminate later promotion evidence.

Refs #404B CITADEL B19-B20, B30, plus #403 OATH-TAPS.

## Machine Spirit #404B / CITADEL-EPOCH handoff If retrieval-strategy statistics are learned, keep them advisory and preserve enough eligibility/selection lineage to detect selective-evidence/self-confirmation attacks. A learned retrieval strategy may optimize physical/operational choice only inside the existing OATH/MOSAIC/RACE obligation contract. It may not suppress opposition/defeater/freshness/independent-origin obligations, reinterpret absence as evidence, or mark its own retrieved corpus independent. If a candidate influences which evidence is gathered, that influence is part of its exposure closure and may contaminate later promotion evidence. Refs #404B CITADEL B19-B20, B30, plus #403 OATH-TAPS.