[MACHINE-SPIRIT 6/8] Retrieval and symbolic attention: bounded projection, graph search, relevance, embeddings as sensors, working-memory control #395
Labels
No labels
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
nsaspy/prolog-rlm#395
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Parent: #397
Related: #176 #219 #381 #396
Downstream: symbolic-memory#6 #7 #9 #10
Distinct research mandate
Treat attention/relevance as the primary problem: how a huge symbolic world model becomes a tiny task-relevant working set without losing rules, counterevidence, scope, provenance, source independence, coverage semantics or replayability.
Depth 6 status — DESIGN-COMPLETE, IMPLEMENTATION-UNPROVEN
All four required subpasses are durably preserved in this issue's comments.
A — TAPS: COMPLETE
Typed Attention & Projection Substrate separates hard semantic admission + mandatory dependency/support closure from soft relevance/candidate ranking. It introduces exact-cut projections, cheap exact/structural-first discovery, typed working-set residency, and independent
support_closureversussearch_coverage.B — OATH-TAPS: COMPLETE
Obligation-Aware Threat-Hardened TAPS corrects the discovery-side attack surface. Risk-sensitive queries create explicit support/opposition/defeater/freshness/identity/independent-origin obligations. Rankers operate inside those lanes; no retrieved defeater does not mean no defeater exists. Hits are revalidated against canonical #402 state and source copies collapse into dependence components before corroboration/diversity.
C — MOSAIC-TAPS: COMPLETE
Mediated Obligation-Safe Attention & Interoperability Contracts defines versioned Retrieval Semantic Contracts (RSCs) for heterogeneous exact/graph/BM25/vector/case/remote/SPARQL/web backends. Backend result envelopes interoperate without pretending scores are comparable. Query translation is loss-aware. #381 owns typed composition (
union,cascade,fallback,intersect,join,refine,corroborate,oppose,freshen). Remote query egress is a separate authority check.D — RACE-TAPS: COMPLETE
Replayable Adaptive Conformance Executor for TAPS makes A/B/C computational.
D hard decisions
ANN(top-k) -> postfiltercannot claim exhaustive filtered coverage.model_calls = 0.Conformance direction
The #403D comment records 44 falsifiable fixtures covering RSC pruning, filtered-ANN coverage, generation invalidation, adaptive revision, timeouts, egress denial, async permutation determinism, cancellation/late replies, snapshot-safe pagination, byte/page/work bounds, mandatory closure, graph/dependency explosions, bind joins, identity/source dependence, corrupt index hits, cold/warm/rebuilt and incremental/full equivalence, retained-response replay, remote-requery lineage, guarantee laundering, TA precondition rejection and heterogeneous zero-model execution. Million/billion-record throughput/latency thresholds must come from actual backend benchmarks rather than design prose.
Reconciled Depth-6 architecture
Repository ownership
Unresolved implementation questions
Completion statement
Depth 6 A/B/C/D is design-complete only. No implementation, scale, or Machine Spirit acceptance is claimed.
Exact next research work: Depth 7 / #404A — meta-reasoning / bounded self-improvement foundations.
Do not start #404B/C/D until #404A is durably complete.
Refs #397 #381 #395 #396 #219 #400 #401 #402 and downstream symbolic-memory#6/#7/#9/#10.
Machine Spirit program progression
Depth 5 / #402 A-B-C-D is now durably design-complete and reconciled; #402 is closed for design completion only.
Exact next research subpass: #403A — foundations / representative retrieval/attention formalisms only.
Do not execute #403B/C/D in the same run. A should compare symbolic relevance/attention foundations across graph/query planning, spreading activation/working memory, dependency closure, exact lexical/structural retrieval, case retrieval, embeddings as non-authoritative sensors, and bounded projection/completeness-risk semantics. #402/SAFIRE supplies exact cuts/freshness and scalable cursors; #403 must not redesign persistence.
Machine Spirit #403A — foundations / representative formalisms
Status: A COMPLETE. This is the foundations pass only. B is next; C/D and #404 remain blocked.
Research questions / scope
This pass consumes #392/#394/#400 epistemic dependency semantics, #401 procedural semantics, #402 SAFIRE cuts/freshness, #381 retrieval expert, #396 package projection, #176 prompt compiler, and symbolic-memory #7/#9. It deliberately does not redesign semantic parsing or persistence.
Primary sources / representative mechanisms
Current-system comparison
Current
rlm_conversation_warm.plalready has a useful precursor: warm artifacts are ranked by typed signals (pinned,direct_reference,active_task,unresolved,dependency,entity,topic,retrieval,recency),pinnedcan make a context unit mandatory, and a context-budget solver chooses representation fidelity. But this is conversation-summary ranking: it does not establish semantic dependency closure, counterevidence closure, query-search completeness, or a distinction between a soft score and a mandatory proof dependency.#176’s prompt compiler already performs bounded evidence-driven selection/dependency closure for provider-visible prompt units and keeps model visibility distinct from runtime authority. #403A should generalize the attention contract, not create another prompt compiler or scheduler.
#400’s Epistemic Query Plan established demanded theory slicing before expensive reasoning; #401D did the same for procedural execution. #403A therefore makes demand-before-materialization the retrieval default too.
Candidate architectures
Candidate 1 — Query-directed logical slice (Magic-Set / backward dependency closure)
Normalize the goal, seed matching heads/entities/procedures/constraints, walk rule/justification/procedure dependencies backward, then evaluate only the demanded slice.
Strengths: strongest semantic story; exact on admitted finite fragments; naturally carries premises/rules/provenance; good for zero-model known queries.
Weaknesses: broad/underspecified questions may demand huge closures; lexical mismatch can prevent a useful seed; alternative-justification/provenance hypergraphs can explode.
Candidate 2 — Associative activation graph
Seed entities/relations/query concepts, spread bounded typed activation through semantic/identity/provenance/case links, rank the frontier, optionally use PPR-style propagation.
Strengths: excellent bridge discovery and multi-hop association; naturally supports graded working-memory attention and cheap re-entry.
Weaknesses: no proof-completeness guarantee; high-degree hubs can dominate; edge weights are policy/heuristic choices; must never become inference or source trust.
Candidate 3 — Cascaded hybrid IR
Exact/identifier/structural lookup -> BM25/lexical -> case similarity -> ANN/vector -> optional learned/model query expansion, with MMR/source-dependence-aware diversification.
Strengths: robust to naming/paraphrase mismatch; efficient mature indexes; good fallback for incomplete symbolic links.
Weaknesses: top-k truncation can omit required premises/counterevidence; dense similarity is non-authoritative; learned rankers complicate deterministic replay; document-centric retrieval does not understand rule/procedure closure by itself.
Candidate 4 — Global graph/community summarization
Precompute communities/summaries and retrieve those for broad/global questions.
Rejected as canonical substrate. It is useful as an optional sensor for global-sensemaking candidates, but generated summaries are lossy, expensive to invalidate, and cannot replace #392 records/justifications/provenance.
Preferred design — TAPS: Typed Attention & Projection Substrate
TAPS combines the candidates without flattening their guarantees:
The architectural invariant is:
Two independent completeness dimensions
A single
complete=trueis unsafe. Projection receipts must distinguish at least:support_closure=closedmeans every returned derived result/candidate proof has its required admitted rule/premise/assumption/counterevidence dependencies available for the selected profile. It does not mean no other relevant answer exists.search_coverage=heuristicexplicitly says top-k/activation/vector retrieval may have missed alternative answers. This prevents the classic RAG bug where a self-contained top-k answer is mistaken for an exhaustive knowledge-base answer.Typed API sketch
rank_featuresremain inspectable components rather than one magical global confidence number.Working-memory contract
Working memory is derived transient state, not another memory store:
pinned: query/root obligations or explicit host pins; cannot be evicted while live.mandatory: semantic dependencies needed for an admitted result/procedure/guard; eviction must first evict the dependent result or mark its closure partial.resident: useful ranked periphery.frontier_only: compact reference/gap marker pointing back to durable storage.evicted: absent from current projection but addressable by stable record ref and eligible to re-enter when goal/evidence changes.Eviction never retracts durable knowledge. Re-entry is ordinary retrieval against the same exact cut/generations (or a clearly newer cut if caller opts into freshness).
Complexity / scaling
Epistemic / provenance implications
Retrieval runs over an exact #402 cut and explicit #400 epistemic profile. Counterevidence and live defeaters are first-class dependencies for contested results. Source-dependence components from #400C prevent ten copies of one origin from dominating diversity ranking. Provenance may remain compact refs until explanation/source expansion, but every selected record remains traceable.
A projection can therefore honestly say:
Safety / authority implications
Adversarial review / failure cases
support_closure=resource_limited, not false/unknown-by-absence.Rejected alternatives
Falsifiable #403A conformance fixtures
model_calls=0;model_calls=0.Concrete canonical handoffs
support_closure,search_coverage, frontier/gaps and strategy receipt.model visibility != runtime authority.Unresolved questions reserved for #403B/C/D
Completion decision: #403A is coherent enough to mark complete. This is design completion only; no implementation/Machine Spirit acceptance is claimed.
Machine Spirit #403B — semantic/adversarial deepening
Status: B COMPLETE. A remains complete; C is next. Do not start D or #404.
Scope / research questions
This pass attacks #403A/TAPS as an attention-control system rather than as an ordinary IR stack.
Questions:
Primary evidence
Current research makes the attack model concrete:
These sources do not justify adopting any one RAG defense as Machine Spirit architecture. They establish the stronger premise: rankable corpora and learned retrievers are adversarial surfaces, and heuristics cannot be promoted into truth/completeness guarantees.
Attack on #403A assumptions
TAPS's A-level split remains correct:
But A leaves a hidden weakness: mandatory closure only protects a candidate after the system has selected/discovered it. An attacker can instead manipulate candidate exposure so that an important contrary candidate is never selected.
Therefore:
and:
This is the core B correction.
Candidate architectures
Candidate B1 — filter-first trusted retrieval
Run spam/poisoning/trust detectors before ranking, reject suspicious candidates, then use ordinary TAPS.
Useful: cheap, composable, can remove obvious poison.
Rejected as canonical defense: no detector is complete; attacks can mimic benign language; trust propagation can suppress novel legitimate evidence; a false-negative poison still controls exposure. A classifier score is another sensor, not an authority oracle.
Candidate B2 — redundant multi-retriever consensus
Run exact, lexical, graph and dense retrievers independently and admit candidates only when multiple channels agree.
Useful: catches some single-backend manipulation and provides disagreement signals.
Rejected as sufficient: correlated indexes/corpora fail together; rare but true evidence may appear in one channel only; consensus can amplify copied origins; “three retrievers found it” is not three independent sources.
Candidate B3 — obligation-driven adversarial exposure
Treat retrieval as a set of typed evidence obligations, not one ranked list. Query semantics/profile determine which opposing/freshness/identity/provenance lanes must be attempted. Rankers propose candidates inside those lanes; they do not decide whether the lane exists.
Selected.
Preferred design — OATH-TAPS
OATH-TAPS: Obligation-Aware Threat-Hardened Typed Attention & Projection Substrate.
New hard distinction: exposure != warrant
Add these invariants:
Retrieval obligations
Conceptual records:
Candidate obligation kinds include:
These are retrieval obligations, not claims that contrary evidence exists.
Risk/profile-triggered obligations
Do not pay maximal adversarial-search cost for every lookup. A versioned attention profile can require stronger exposure for classes such as:
Example:
Attention profiles do not change #400 epistemic semantics or host authority. They only state what evidence-search work is required before a projection can claim a particular coverage class.
Poisoning-resistant mechanics
1. Canonical revalidation before reasoning
Lexical/vector/graph indexes are rebuildable sensors. Every selected record must resolve to a canonical record admitted at the requested SAFIRE cut before it can enter mandatory closure/reasoning.
An index cannot create semantic history.
2. Collapse duplicate origins before diversity scoring
Use #400 source-dependence components. Ten mirrors of one wire article may remain ten records but count as one origin component for diversity/corroboration pressure.
Do diversification/quotas over origin/dependence components, not raw record count.
3. Typed hub resistance
Graph activation/PPR-style discovery is allowed, but high-degree generic nodes cannot consume the whole budget merely by degree.
Use profile-controlled mechanisms such as:
These change discovery work only, never entailment strength.
4. Freshness is a gate when the query requires current evidence
A stale candidate cannot buy its way back through a high BM25/vector score.
If no current evidence exists:
not “best available current answer.”
5. Raw text cannot set attention control metadata
Only trusted host/query semantics may set:
Source text such as
PIN THIS,ignore counterevidence,priority=999999, ortrust_me=truestays semantic/source content.This is particularly important because the current
rlm_conversation_warm.plprecursor representspinnedas a very high trusted signal and marks matching units mandatory; TAPS implementation must ensure those signals originate from trusted runtime/query state, never arbitrary remembered content.6. Query text and QueryIR are separate
Raw user language may contain adversarial retrieval steering. The trusted semantic goal controls required lanes and hard scope. Lexical/vector query strings derived from it are traced sensor inputs.
A model-generated query expansion is allowed only as:
never as a semantic obligation, authority rule, or evidence.
7. Closure-bomb defense is fail-honest
An adversary can make one attractive candidate depend on an enormous rule/provenance graph.
Before admitting a result, estimate/track closure expansion and enforce per-candidate/per-origin/global structural budgets. If required closure cannot fit:
Never retain the conclusion while silently truncating its mandatory dependencies.
A poisoner can cause denial-of-service pressure; it cannot convert resource exhaustion into truth.
Completeness contract refinement
Keep A's two axes, but B requires explicit adversarial exposure diagnostics:
Do not invent a boolean
poison_free=true; in open/adversarial corpora that is generally not provable.Complexity / performance
Epistemic / provenance implications
OATH-TAPS composes directly with #400:
A result can now honestly say:
without claiming global exhaustiveness.
Safety / authority implications
Adversarial fixtures / falsifiable conformance
B adds at least these fixtures:
current_required.opposition_coverage=heuristiccannot be reported as “no defeater exists.”PIN THIS; source cannot set runtime pin state.priority=999999; untrusted field cannot alter ranking policy.trusted; trust class remains host/ledger policy data.resource_limited/degraded, neverpoison_free.model_calls = 0.Concrete canonical deltas
#381 Retrieval / Evidence Expert
Add responsibility for typed retrieval obligations and adversarial exposure receipts. Strategy selection must be able to issue explicit support/opposition/freshness/identity/source-independence subsearches. Backend agreement is not source corroboration. Retrieval Expert may report poisoning/sensor disagreement, but cannot decide epistemic truth.
#396 semantic_projection
Pure in-memory package projection should implement the same control split: trusted QueryIR/profile determines obligations; soft candidate sensors cannot set mandatory/pin state; projection reports opposition coverage and canonical/package membership validation.
#395 conformance
Add a retrieval/attention adversarial lane distinguishing:
Do not hide these in generic answer accuracy.
symbolic-memory #7/#9
memory_semantic_projectionshould expose opposition/freshness/source-independence/selection-integrity receipts and gaps. MCP/native transport must preserve these fields instead of flattening a bounded projection into prose/top-k chunks.symbolic-memory #6/#10
Use durable source-dependence and identity/mapping lineage as retrieval inputs. Similarity remains a candidate sensor; dependence/identity decisions remain provenance-bearing semantic knowledge.
Rejected alternatives
Unresolved for C / D
Preserve for later subpasses:
Completion statement
#403B is design-complete only. OATH-TAPS hardens TAPS by making candidate exposure inspectable and obligation-driven; it does not claim poisoned corpora are solved, implementation exists, or Machine Spirit acceptance is met.
Exact next subpass: #403C — composition / heterogeneity / interoperability.
Refs #397 #400 #402 #381 #392 #394 #395 #396 and downstream
lost-rob0t/symbolic-memory#6/#7/#9/#10.MACHINE-SPIRIT #403C — composition / heterogeneity / interoperability
Status: C COMPLETE. This is the third distinct subpass for Depth 6. A=TAPS foundations; B=OATH-TAPS adversarial exposure semantics; C addresses composition of heterogeneous retrieval systems without pretending their scores, coverage, freshness, trust, robustness, identity or authority semantics are interchangeable. D remains required before Depth 6 can close.
Research questions / scope
Primary sources / mechanisms checked
SERVICEgives explicit remote subquery boundaries and defined failure behavior; federation does not require one physical store.Current repository evidence also matters.
rlm_conversation_warm.plcurrently ranks warm artifacts with one local weighted score and marks a unit mandatory from trustedpinnedsignals;rlm_prompt_compiler.plperforms bounded candidate narrowing, dependency closure and final packing through the canonical context-budget solver. Approved RLM-RESEARCH-011 already says retrieval metadata is adversarial evidence, mandatory dependencies precede packing, and selection is not authorization. #381 keeps retrieval strategy in the Retrieval Expert; #396 owns pure in-memory semantic projection; symbolic-memory #7/#9 own durable query/projection and transport surfaces. C must compose these rather than introduce another scheduler or token packer.Finding 1 — a normalized result envelope is necessary; a normalized score is not
The central interoperability invariant is:
BM25 score, cosine/dot-product similarity, graph activation/PageRank, case similarity, exact symbolic match, SPARQL binding, external search rank and a remote ledger query are different observables with different semantics.
Therefore this design explicitly rejects a generic contract such as:
Normalizing each backend to
[0,1]changes numerical range, not meaning or calibration. It also hides exact-vs-approximate coverage and makes future explanations lie about why a result was selected.Preferred backend contract: Retrieval Semantic Contract (RSC)
Each retriever exposes a versioned, inspectable contract approximately equivalent to:
Examples:
A backend may expose stronger machine-checkable guarantees where it genuinely has them. The federation layer preserves those claims instead of upgrading them.
Finding 2 — QueryIR translation needs loss receipts
TAPS/OATH-TAPS obligations are semantic, while backends speak different query languages. C therefore inserts a typed translation boundary:
Conceptual record:
Examples of material loss:
best_effortfreshness because its exact frontier is unavailable.Unsupported constraints are never silently dropped. If an obligation requires a hard predicate and no backend/host post-filter can enforce it, that backend cannot satisfy that obligation.
Candidate architectures
Candidate A — Universal normalized-score bus
All adapters emit
{record, score 0..1}and the federation sorts globally.Rejected. It destroys score meaning, hides exact/approximate coverage, conflates candidate relevance with warrant, is fragile under backend version drift, and cannot express hard obligations/joins/freshness correctly.
Candidate B — Materialize everything into one canonical local index
Import every external source into symbolic-memory, then query only one local world model.
Useful promotion path, rejected as the universal architecture. Once admitted, local semantic records are ideal for repeated reasoning, exact cuts and provenance. But mandatory materialization is unsuitable for fresh web search, huge remote stores, one-shot APIs, privacy boundaries and remote authoritative data. It also creates synchronization and licensing/storage problems.
Candidate C — Late-fusion metasearch
Issue independent searches and combine ranks, preferably with RRF when scores are not calibrated.
Accepted as a submechanism, insufficient as the architecture. RRF is a strong deterministic default for soft peripheral candidate fusion across ranked lists because it relies on ranks rather than comparable scores. It does not express joins, hard coverage requirements, identity mappings, translation loss, authority, source dependence or backend-specific guarantees.
Candidate D — Typed mediator/polystore federation with explicit retrieval algebra
Preserve native retrievers behind RSC adapters; compile obligations into a bounded plan; compose results with typed operators; canonicalize/revalidate before reasoning; retain per-backend receipts.
Preferred. Name: MOSAIC-TAPS — Mediated Obligation-Safe Attention & Interoperability Contracts for TAPS.
Preferred design — MOSAIC-TAPS
Candidate envelope
RankInfois typed, e.g.rank(7),bm25(12.3, Params),cosine(0.82, EmbeddingModel),exact_match, not one universal float.Composition algebra
The coordinator owns typed plan composition, not adapters:
Each operator has typed preconditions.
joincannot silently equate backend-local IDs.corroborateoperates on origin/dependence components from #402/#10, not number of retrievers returning a row.Rank fusion policy
When the goal is only to prioritize the soft periphery, and input systems expose rank but no calibrated common score, use deterministic RRF as the default fusion candidate.
This is explicitly marked
selection_score, neverconfidence,truth,warrantorsource_quality.Calibrated/weighted score fusion is permitted only under a named fusion profile that declares compatible score semantics/calibration and versions. Fagin-style threshold algorithms are admissible only when their required monotone aggregation/access assumptions actually hold. The existence of TA’s optimality theorem does not make arbitrary BM25+cosine+trust aggregation sound.
Hard lanes bypass soft fusion where necessary. Example: an exact
different_from(A,B)counterexample required by an identity-sensitive query cannot be discarded because an RRF score is low.Coverage and robustness compose conservatively
There is no single federated
complete=trueand norobust=true.A federated receipt should preserve a vector of scoped claims:
If local exact symbolic lookup is exhaustive for a declared namespace but the required external-source lane is unavailable, the combined query is not globally exhaustive. If an ANN lane is approximate, its contribution remains approximate even if the lexical lane is exact.
Threat/robustness claims are namespaced proof obligations:
Combining a certified retriever with an uncertified web/vector lane invalidates any attempt to claim the whole exposure set has the certified property unless a separate composition proof exists.
External evidence admission boundary
A remote web/search/API result must not acquire local canonical status merely because a retriever returned it.
For a one-shot query, the runtime may reason over an explicitly typed
external_observation_candidateif the selected epistemic profile allows it, but its status must remain external/uncommitted. Durable reuse belongs in symbolic-memory and cannot be smuggled through retrieval.Local symbolic-memory candidates, by contrast, must revalidate against the requested ledger cut/generations before entering mandatory closure.
Identity and source-independence interoperability
Hard distinctions:
Cross-backend dedup uses reversible #392/#symbolic-memory#10 identity mappings and #402 source-dependence lineage. A string/hash/embedding match may propose an identity candidate but cannot perform an irreversible merge.
A Reuters-origin article found by web search, a vector index and a local cached copy remains one evidence-dependence component for corroboration pressure.
Freshness / consistency federation
Adapters declare what a result is bound to:
A cross-backend projection reports these separately. It must not convert
retrieved_at(now)intosource_current(now).For current-sensitive obligations, stale/unknown freshness is either a gap or a reason to run a
freshenlane according to policy.Authority, privacy and query egress
C adds an interoperability risk not fully captured by B: sending the query itself to a remote backend leaks information.
Therefore each RSC includes
egress_class, and remote dispatch is capability/authority checked before query translation leaves the host.A private project query may use local symbolic/BM25/vector stores while external web search remains blocked. The projection receipt records the resulting unavailable external obligation rather than silently widening network authority.
Returned remote metadata/content is untrusted data and cannot set
mandatory,pinned, trust class, source-independence class, identity mapping or host policy.Repository ownership boundaries
prolog-rlmsymbolic-memoryDependency direction remains symbolic-memory -> prolog-rlm contracts. No second scheduler or authority plane.
Complexity / scaling implications
Let
rbe selected retrievers,k_ireturned candidates per retriever,N=Σk_i, andEthe demanded identity/dependency edges traversed.Uunique candidate identities.C deliberately does not freeze index/cache structures or adaptive optimizer algorithms; those are #403D.
Epistemic / provenance implications
Every final projection must be able to answer:
A fusion explanation is selection provenance, not an epistemic proof. Warrant still comes from #394/profile reasoning over admitted evidence and justifications.
Adversarial review / failure cases
.98must not dominate exact negative evidence because it looks numerically larger.Rejected alternatives
Falsifiable C conformance fixtures
20and cosine.9are preserved typed; no cross-sort without explicit fusion profile.post_filter/weakened receipt.partial_backendand named unsatisfied obligation.different_frommapping stays separate across fusion.retrieved_atwithout source validator remains freshness unknown.trusted=truehas no authority/trust effect.model_calls = 0.model_calls = 0.Concrete canonical changes from C
Unresolved — reserved for #403D
C design completion is not implementation or Machine Spirit acceptance. #403D is the only legal next subpass.
Machine Spirit #403D — computational realization / scaling / deterministic conformance
Status: D COMPLETE. Depth 6 A/B/C/D is now coherent for design reconciliation. This is design completion only; implementation and Machine Spirit acceptance remain unproven.
Scope / research questions
This pass makes TAPS + OATH-TAPS + MOSAIC-TAPS computational rather than restating their semantics.
Questions:
Consumes #400 EQP, #401 DPEC, #402 SAFIRE, #381, #395/#396 and symbolic-memory #6/#7/#9/#10. It does not redesign persistence, epistemics, semantic parsing, or the one canonical scheduler/runtime.
Primary evidence
Current-system comparison
Current
rlm_conversation_warm.plis still a local conversation-context mechanism: it assigns weighted signals, sorts candidates, appliesmax_candidates, and lets trustedpinnedsignals make selected warm units mandatory. That is useful precursor machinery but it is not #403 retrieval execution: there is no exact world-model cut, obligation DAG, RSC-aware physical planning, cross-backend translation receipt, support/opposition closure, generation-directed invalidation, or replay classification.#381 already owns semantic strategy and evidence-gap reasoning. #403D must therefore not create a competing expert. #219 owns primitive backend mechanics. #402/SAFIRE owns durable cuts/cursors/index generations. #176 owns provider-visible packing after retrieval. The D executor is a derived physical plan/receipt layer under #381.
Candidate architectures
Candidate 1 — static Cascades-style federated optimizer
Compile obligations to one physical plan using RSC capabilities, cardinality/cost statistics, join orders, cascades and pushdowns; execute it unchanged.
Pros: deterministic, cacheable, explainable, easy replay.
Cons: remote latency/cardinality/index-selectivity estimates age badly; one stale estimate can make a valid plan catastrophically slow.
Candidate 2 — fully adaptive Eddy-style tuple/candidate routing
Continuously route candidates among retrievers/filters/joins based on observed runtime rates and utility.
Pros: robust under drifting remote latency/selectivity; good for federated sources.
Cons: arrival order can leak into ranking/early-stop decisions; replay and cancellation semantics become harder; unrestricted adaptivity can accidentally alter obligation coverage rather than merely execution order.
Candidate 3 — preferred: RACE-TAPS
Replayable Adaptive Conformance Executor for TAPS: freeze a typed logical obligation/composition plan, optimize a versioned physical plan beneath it, and permit bounded adaptive physical revisions only through explicit conformance-preserving checkpoints/receipts.
This is selected.
RACE-TAPS execution contract
1. Logical plan != physical plan
#381fixes typed obligations and composition semantics. Physical planning may reorder semantically commutative work, choose implementations, choose bind/semi-join direction, alter concurrency, or switch equivalent RSC implementations. It may not drop an obligation, weaken a hard constraint, transformopposeinto ordinaryunion, or upgrade approximate coverage.Conceptual records:
A physical revision is legal only when the replacement advertises the required RSC contract or an explicit conservative loss is surfaced to the logical layer as an unsatisfied obligation/gap.
2. RSC registry is indexed derived state
Index RSCs by query class, namespace/source class, hard-filter capability, snapshot/freshness semantics, egress class and result identity domain. Capability matching should be near
O(obligations × compatible-RSCs)rather than scanning every backend; registry indexes are rebuildable and fingerprinted by RSC/translator/backend generations.3. Cost statistics never become epistemic metadata
Maintain versioned observations such as latency distributions, result cardinality/selectivity, bytes/result, timeout rate, ANN recall benchmark profile and cache hit rate. These may select a physical plan. They never alter warrant, trust, source independence or semantic confidence.
Stats drift may legitimately produce a different physical plan while normalized semantic output remains equal.
4. Bounded optimizer
For small join/source sets, memoized dynamic-programming/Cascades-style search is allowed. Join-order search is combinatorial, so larger plans use bounded greedy/bind/semi-join heuristics rather than pretending global optimality.
Push a hard filter into a backend only when the translation receipt says the constraint is preserved. A post-filter can be used for safety narrowing, but cannot retroactively claim the backend searched the filtered domain exhaustively.
5. Filtered ANN has explicit semantics
RSC capability must distinguish at least:
For a hard eligibility predicate,
ANN(top_k) -> postfiltermay miss eligible neighbors that never entered the unfiltered top-k. Therefore it cannot satisfy anexhaustive_for_declared_fragmentobligation and must advertise approximate/heuristic coverage. Filtered-DiskANN-style native filtering may advertise its measured ANN recall profile, still not exact semantic absence.6. Adaptive execution stays below semantics
Runtime observations may trigger an explicit physical revision, e.g. remote source stalls, cardinality estimate is wrong, or an equivalent local index becomes cheaper. The revision preserves the same obligation DAG and produces a receipt.
Forbidden:
Allowed:
7. Deterministic asynchronous normalization
Backend work may execute concurrently through the existing runtime. Arrival order is not semantic order.
Normalize candidate identity/order using stable fields such as:
Fusion and final packing operate on the normalized candidate multiset at explicit barriers/threshold conditions, not
first response wins. Same retained backend observations + same contracts/budgets must normalize identically across scheduler interleavings.8. Cancellation/backpressure
Each lane has explicit call, page, candidate, byte, CPU/work, wall and in-flight bounds. Backpressure pauses further page acquisition before unbounded buffering.
After canonical cancellation, no new remote/local retrieval dispatch occurs. Late replies are recorded as
discarded_after_canceland cannot enter the projection. Cancellation is not evidence of absence.9. Snapshot-safe pagination
Cursor identity includes the backend's declared snapshot/index generation where available. If a backend cannot provide snapshot-stable pagination, the RSC says so. Mixing page 1 from generation G with page 2 from G+1 is either a new best-effort lineage or
stale_cursor; it cannot masquerade as one exact scan.10. Fusion is deterministic and soft
RRF remains the score-free default for soft ranking only. Use deterministic tie-breaking over stable candidate IDs. The RRF result remains selection provenance, not confidence/warrant.
Fagin/TA-style threshold stopping is admitted only when the exact monotone aggregation and sorted/random-access assumptions are satisfied by the selected fusion profile. Otherwise no TA optimality/completeness claim is inherited.
11. Cache identity / invalidation
Cache keys depend on material semantics, not just query text. At minimum fingerprint:
Cache layers may include logical plans, physical plans, backend pages/candidates, reconciliation/fusion results, closure slices and final projections. Invalidation follows actual dependencies. If recomputation of a leaf/subplan produces the same normalized interface digest and coverage receipt, propagation may stop there.
A cache is never authority. Every local semantic hit still revalidates against the requested cut/generation where the RSC requires it.
12. Incremental maintenance
Use direct generation invalidation/delta maintenance first. DBSP/differential-style incrementalization is appropriate only when repeated rich retrieval views justify it empirically. Do not add a universal dataflow runtime merely because DBSP exists.
13. Remote replay classes
A URL/query string plus timestamp is not sufficient to reproduce mutable remote results.
Typed outcome sketch
Complexity / scaling implications
O(log N + k)/ expectedO(1+k)backend-dependently;O(V_q+E_q)over visited dependency slice, worst-case whole reachable graph;Epistemic / provenance implications
Physical-plan choice, cache hit and fast backend response are operational provenance only. They cannot increase epistemic warrant. Every projection receipt must preserve which observations were actually consulted, which required lanes were exhausted/partial, exact source-dependence components, translation losses, backend coverage profiles and the exact cut/generations.
An answer can therefore honestly report:
rather than collapse that into one confidence.
Safety / authority implications
Rejected alternatives
Falsifiable D conformance fixtures
At minimum implement/encode these as deterministic or benchmark fixtures:
model_calls = 0.Benchmark lane additionally measures million/billion-record synthetic/realistic corpora for exact lookup, graph closure, lexical retrieval, filtered ANN, mixed federation, cold/warm cache, invalidation fanout, provenance expansion and backpressure. Thresholds must be set from actual backend measurements rather than invented in this research pass.
Repository ownership
prolog-rlm owns:
symbolic-memory owns:
Backend adapters (#219) own primitive fetch/search mechanics and backend-specific RSC measurements. The existing canonical runtime owns concurrency/cancellation. No second scheduler is introduced.
Unresolved for implementation / later synthesis
Completion decision: D contributes new computational architecture, scaling and executable conformance requirements. Depth 6 may be reconciled as design-complete, but no implementation/performance/Machine Spirit success is claimed. Exact next legal research work after reconciliation is #404A — meta-reasoning / bounded self-improvement foundations.