nsaspy/prolog-rlm

Fork 0

[EPIC] Prolog-first deep recursive autonomous research with LLM fallback only #423

New issue

Closed

opened 2026-09-08 01:29:22 +00:00 by nsaspy · 1 comment

nsaspy commented

2026-09-08 01:29:22 +00:00

Owner

Mission

Build a Prolog-first autonomous research engine inside prolog-rlm that can investigate open-ended questions deeply and recursively without requiring an LLM for planning, search, reasoning, evidence evaluation, or control flow.

LLMs are optional fallback components, not the intelligence substrate.

Primary target:

question / goal
  -> deterministic NL normalization where possible
  -> Prolog research goal
  -> recursive symbolic decomposition
  -> source/tool discovery
  -> evidence acquisition
  -> claim + provenance graph
  -> contradiction / uncertainty analysis
  -> recursive gap filling
  -> convergence / stop decision
  -> structured answer + proof/evidence
  -> optional conversational rendering

An LLM may be used when useful for:

conversational natural-language rendering;
clarification of genuinely ambiguous user language;
fallback semantic parsing when deterministic parsing cannot normalize the input;
optional summarization of already-grounded evidence;
explicitly requested model-assisted reasoning.

An LLM must not be required to decide what to research next, what sources to inspect, whether evidence supports a claim, whether a contradiction exists, whether more research is needed, or when a research branch is complete.

Core principle

The system should be as capable as practical using ordinary computation, symbolic AI, search, graph algorithms, expert rules, statistical/heuristic ranking, and external information sources.

LLM unavailable != research unavailable

A machine with SWI-Prolog, configured research adapters, and network/data access should still be able to perform a useful autonomous research run.

Relationship to existing runtime

Reuse current canonical runtime boundaries rather than creating a second agent runtime.

Relevant existing work includes:

#335 reasoning modes (symbolic, symbolic-recursive, auto);
#288 plan dependency graphs / closed plan vocabulary;
#93 Project / source-knowledge / SPEC / VERIFY APIs;
#367 first-class bridge/harness exposure;
#352 recursive-budget correctness;
existing tool registry, MCP, capabilities, authority, tracing, usage, cancellation, async/Futures, graphs, context, and persistence surfaces.

This epic is not permission to add another scheduler, another tool runtime, another authority system, or a model-controlled research loop parallel to the existing runtime.

Desired experience

A user should be able to ask:

Research whether claim X is true and keep digging until the important uncertainty is resolved.

The runtime should be able to proceed roughly as:

research_goal(G0)
  -> decompose(G0, Subgoals)
  -> rank(Subgoals)
  -> select_next(Subgoal)
  -> determine_required_evidence(Subgoal)
  -> discover_sources(Subgoal, Sources)
  -> acquire_evidence(Sources, Evidence)
  -> normalize_evidence(Evidence, Facts)
  -> update_claim_graph(Facts)
  -> evaluate_claims
  -> discover_gaps
  -> recurse_on_gaps
  -> convergence_check
  -> research_result(Result).

The exact predicates may differ. The semantic boundary matters.

1. Research state is symbolic data

Represent research as ordinary inspectable Prolog data with stable identities.

Conceptual minimum:

research_run(Run, Goal).
research_question(Run, Question).
research_subgoal(Run, Parent, Goal).
research_status(Run, Status).

claim(Claim, Proposition).
claim_status(Claim, supported | contradicted | mixed | unknown).
claim_support(Claim, Evidence).
claim_conflict(Claim, Evidence).

evidence(Evidence, Source, Observation).
source(Source, Kind, Locator).
source_quality(Source, ScoreOrClass).

needs_evidence(Claim, Requirement).
research_gap(Run, Gap).

Do not flatten the research process into chat history or an opaque model transcript.

The complete current research state must be inspectable/queryable without asking a model to reconstruct it.

2. Deterministic natural-language front door

Natural language should be usable without forcing every utterance through an LLM.

Build a bounded deterministic normalization layer using appropriate Prolog-native techniques such as:

DCGs;
tokenization and morphology;
configurable lexicons and aliases;
entity/relationship dictionaries;
semantic templates;
typed slot filling;
date/number/unit normalization;
question/goal classification;
explicit ambiguity sets rather than guessed meanings.

Example:

"Did company X acquire company Y before 2024?"

may normalize to data equivalent to:

research_question(
    acquisition_before(company_x, company_y, date(2024,1,1))
).

If deterministic parsing cannot produce one defensible interpretation, return an inspectable ambiguity/unknown outcome.

Only then may conversation mode optionally invoke an LLM parser or ask the operator for clarification.

Model-generated Prolog must never be blindly consulted or executed as trusted code.

3. Deep recursive research planner

Implement a non-LLM research planner over symbolic goals.

It should support recursive decomposition from reusable expert rules such as:

identity question
  -> authoritative identity sources
  -> aliases
  -> conflicting identity records

timeline question
  -> event extraction
  -> date normalization
  -> ordering
  -> missing intervals

causal claim
  -> candidate causes
  -> mechanism evidence
  -> temporal evidence
  -> counterevidence
  -> alternative explanations

entity relationship
  -> direct records
  -> common identifiers
  -> temporal consistency
  -> independent corroboration

Research strategies should be registered/extensible Prolog rules, not hard-coded into a model prompt.

The planner may use classical search/AI techniques where appropriate:

best-first search;
beam search;
iterative deepening;
branch-and-bound;
dependency graphs;
rule priorities;
novelty/information-gain heuristics;
source diversity scoring;
confidence/uncertainty propagation.

Use the existing graph/plan/async infrastructure where semantics align.

4. Autonomous research agenda

Each run maintains a bounded agenda/frontier of unresolved work.

Conceptual states:

unexpanded
ready
searching
awaiting_evidence
supported
contradicted
mixed
blocked
unknown
superseded
complete

The engine chooses the next branch from symbolic state and deterministic/heuristic scoring.

Priority should account for factors such as:

importance to the root question;
unresolved uncertainty;
dependency centrality;
evidence availability;
expected information gain;
source independence;
cost/time/tool budgets;
prior failed attempts;
novelty vs already-known evidence.

No model call should be necessary to maintain or select the agenda.

5. Research-tool abstraction

Research sources are normal runtime tools/adapters.

Potential adapters include:

web search;
URL fetch/readable-text extraction;
APIs;
MCP research tools;
local files;
databases;
Git repositories;
project/source KB;
scholarly indexes;
operator-provided corpora.

The research engine reasons over a generic source/evidence contract rather than being coupled to one search provider.

Availability is not authority. Existing capability/authority/network/effect boundaries remain canonical.

6. Query generation without an LLM

Research must be able to generate useful searches from symbolic goals deterministically.

Support query construction from:

entity names + aliases;
predicates/relationship vocabulary;
dates/time windows;
domain keywords;
source-type hints;
previous failed queries;
discovered terminology;
citation/reference expansion.

Example:

search_terms(
    acquisition_before(A, B, Date),
    [AName, BName, acquisition, acquired, merger, Date]
).

Use configurable synonym/alias/ontology knowledge to broaden or specialize queries.

The engine should be able to recursively learn in-run vocabulary from acquired sources without permanently trusting it as host code.

7. Evidence normalization and provenance

Every material claim needs evidence lineage.

Preserve at minimum:

source identity / locator
retrieval time
content/version/hash where possible
extractor/parser identity
relevant span/record
research run
claim(s) derived from it
confidence/quality metadata

Separate:

raw observation
normalized fact candidate
derived fact
inference
claim assessment

A derived Prolog conclusion must not masquerade as a directly observed fact.

8. Claim graph / knowledge graph

Maintain an explicit graph connecting:

questions
subgoals
entities
claims
evidence
sources
inferences
contradictions
unknowns

Support traversal such as:

why(Claim, Proof).
what_supports(Claim, Evidence).
what_conflicts(Claim, Evidence).
what_is_missing(Claim, Requirements).
source_dependencies(Claim, Sources).

Recursive research should operate over this graph rather than repeatedly rereading an ever-growing prose context.

9. Contradiction and uncertainty handling

Do not collapse absence of evidence into falsehood.

Research outcomes must distinguish at least:

supported
contradicted
mixed
unknown
blocked
error

When sources conflict, preserve both branches and investigate the conflict.

Useful recursive follow-up strategies include:

source independence checks;
primary-vs-secondary source resolution;
timestamp/version differences;
entity disambiguation;
scope/definition mismatch;
newer superseding evidence;
explicit counterexample search.

10. Source quality / corroboration

Implement configurable non-LLM scoring/rules for evidence quality.

Possible inputs:

primary vs secondary source;
provenance completeness;
publication/update time;
directness of evidence;
independent corroboration;
duplicate/syndicated-source detection;
internal consistency;
known source/domain policy;
specificity to the claim;
extraction confidence.

Do not turn a numeric score into fake certainty. Preserve the underlying evidence and reasons.

11. Recursive gap discovery

After each evidence update, derive what remains unknown.

Examples:

claim has one weak source
  -> seek independent corroboration

claim has conflicting dates
  -> research event timeline

entity alias uncertain
  -> resolve identity before downstream inference

source references another document
  -> inspect referenced primary document

causal claim lacks mechanism evidence
  -> create mechanism subgoal

This is the heart of autonomous research: the next questions are generated from the current symbolic knowledge state.

12. Convergence and stop conditions

The engine needs explicit rules for knowing when to stop.

Stop/reduce expansion based on combinations of:

root claims sufficiently supported/contradicted;
required evidence classes satisfied;
remaining gaps are low-impact;
no novel evidence after N strategies;
branch exhausted;
repeated equivalent queries/results;
confidence/quality threshold;
depth/node/tool/time/byte budgets;
operator policy.

A run must never recurse forever merely because more web pages exist.

Return the stop reason.

13. Learning without an LLM

Support safe accumulation of useful knowledge from successful research runs.

Distinguish:

run-local observations
candidate learned facts
verified durable facts
research strategy statistics
source/domain metadata
operator-authored expert rules

Do not auto-promote arbitrary web text into trusted executable Prolog.

Useful non-LLM adaptation may include:

source success/failure statistics;
query-template performance;
alias/entity resolution caches;
evidence-pattern statistics;
discovered source relationships;
strategy ordering based on measured utility.

Persistence must retain provenance and generation/version semantics.

14. Optional LLM fallback contract

The fallback boundary must be explicit and observable.

Conceptual API:

fallback_policy(Session, Policy).

fallback_allowed(nl_parse_ambiguity).
fallback_allowed(conversation_render).
fallback_allowed(optional_summary).

Default autonomous research policy should be equivalent to:

planner:             no LLM
agenda selection:    no LLM
query generation:    no LLM
source selection:    no LLM
claim evaluation:    no LLM
contradiction check: no LLM
convergence:         no LLM
conversation:        LLM allowed
NL parse fallback:   LLM allowed when deterministic parser fails

Every fallback event must be traceable with reason and usage.

A model response is evidence only when the operator explicitly configures a model itself as a source; otherwise model output is never treated as factual evidence.

15. Conversation mode

Conversation is a projection over the symbolic state, not the state itself.

Preferred flow:

user text
  -> deterministic parser
  -> symbolic intent/question
  -> research / query / explanation
  -> structured result
  -> deterministic renderer when sufficient
  -> optional LLM renderer for natural conversation

The LLM renderer receives bounded structured facts/proofs/evidence and may improve phrasing, but must not silently add unsupported factual claims.

The user should be able to ask conversationally:

Why do you think that?
What evidence contradicts it?
What are you least sure about?
Keep digging on the weak part.
Where did that fact come from?
Try another source.

Those utterances should map onto operations over the existing research graph.

16. Explainability

A completed run should be able to expose:

answer
claim statuses
proof/inference paths
supporting evidence
conflicting evidence
source provenance
unresolved gaps
branches explored
branches abandoned + reason
stop reason
resource usage
LLM fallback calls, if any

A useful answer should never require replaying an opaque hidden reasoning transcript.

17. Determinism and replay

Given a fixed corpus/tool-result fixture, config, strategy registry, and budgets, the symbolic research controller should be reproducible enough for deterministic tests.

Persist stable event/decision records sufficient to explain:

why a branch was selected;
why a query was generated;
why evidence changed a claim status;
why recursion continued/stopped.

Live network results may naturally vary; control-flow semantics must still be testable with fixtures.

18. Safety / authority invariants

Web/source text is untrusted data.
Never auto-consult retrieved Prolog/source text.
Research tools retain ordinary capabilities/authority.
Read-only research does not imply mutation authority.
LLM fallback cannot widen capabilities.
Learned facts cannot become executable trusted configuration without an explicit trust/promotion boundary.
Prompt injection in retrieved content is data, not an instruction to the runtime.
Recursive children inherit/narrow authority and budgets through existing runtime contracts.

Proposed implementation slices

S1 — Research state + claim/evidence graph

Add generic symbolic research-run, goal, claim, evidence, source, gap, and provenance contracts.

S2 — Deterministic research agenda

Add frontier selection, dependency handling, budgets, progress/novelty accounting, and stop reasons.

S3 — Research strategy registry

Add extensible Prolog rule packs for decomposition/evidence requirements and strategy selection.

S4 — Generic search/fetch evidence adapters

Compose existing tool/MCP surfaces into a generic research-source contract.

S5 — Deterministic query compiler

Symbolic goal -> search/query candidates using aliases, ontology/lexicon, dates, source classes, and previous observations.

S6 — Evidence normalization + contradiction engine

Normalize observations, track provenance, derive support/conflict/mixed/unknown states, and generate evidence gaps.

S7 — Recursive gap-filling loop

Drive autonomous continuation entirely from research graph state.

S8 — Source quality + corroboration

Add configurable evidence-quality and independence rules.

S9 — Deterministic NL front door

DCG/template/entity/alias based intent + question normalization with explicit ambiguity.

S10 — Optional LLM fallback

Add strictly bounded fallback policy and trace events for parsing/rendering/optional summarization.

S11 — Conversation projection

Natural conversational inspection and continuation over the symbolic research state.

S12 — Durable learning / strategy metrics

Persist safe, provenance-carrying learned facts and research-strategy statistics without promoting untrusted executable code.

Child issues should be created as each slice reaches a concrete design boundary. Do not implement the entire epic as one PR.

Acceptance criteria

A complete autonomous research run can execute with zero model/provider calls.
With model providers disabled, the engine can recursively decompose a nontrivial fixture question, search fixture sources, acquire evidence, detect an evidence gap, recurse, resolve or preserve uncertainty, and terminate with a structured result.
Research planning/agenda/query generation/evidence evaluation/convergence are implemented in Prolog or deterministic host algorithms, not prompts.
Research state is inspectable as ordinary symbolic data.
Claims retain evidence/provenance and distinguish observed vs derived facts.
Contradictory evidence is preserved and can trigger recursive investigation.
Source duplication/independence can affect corroboration decisions.
Repeated searches/non-progress are detected and bounded.
Depth, node, source, tool, time, and byte budgets are enforced.
Cancellation terminates recursive work through the canonical runtime.
Fixed-source fixtures provide deterministic recursive-research tests.
Natural-language questions can be handled through deterministic normalization for a useful bounded grammar.
Ambiguous/unhandled language returns a structured outcome instead of fabricated semantics.
LLM fallback is optional, policy-controlled, usage-traced, and cannot widen authority.
Conversation mode can use an LLM renderer without making model output authoritative evidence.
why, support, conflict, uncertainty, provenance, branch history, and stop reason are inspectable.
Existing canonical tool/capability/authority/effect/async/graph/trace systems are reused rather than duplicated.
Full deterministic runtime gate remains green.

North-star test

The strongest integration test should deliberately provide no LLM credentials/provider at all.

Fixture question requires multiple hops and conflicting evidence:

Q
 -> discover entity alias
 -> find source A
 -> derive intermediate claim
 -> notice missing corroboration
 -> search independent source B
 -> encounter conflicting source C
 -> resolve conflict using date/version/provenance evidence
 -> answer with support + remaining uncertainty

The run passes only if the runtime performs the complete recursive investigation and produces a structured evidence-backed conclusion with model_calls = 0.

Then run the same research session with conversation fallback enabled and prove that the optional model changes presentation only, not the underlying claim/evidence result.

Non-goals

No requirement to eliminate LLM support from Prolog-RLM generally.
No attempt to build a human-level unrestricted English parser in the first slice.
No opaque model-generated research plans as the default engine.
No model-generated arbitrary executable Prolog.
No second scheduler/agent runtime/tool system.
No fake certainty from source scores.
No assumption that every open-ended question is decidable.

End state

The intended architecture is:

                 optional
                 LLM conversation
                       ^
                       |
user -> NL -> symbolic research engine -> evidence/proofs -> answer
              |        ^
              v        |
         tools/sources |
              |        |
              +-> claim/evidence graph
                       |
                       +-> gaps -> recursive research

Prolog owns the research intelligence.

The LLM is a useful language adapter when desired, not the thing keeping the system alive.

## Mission Build a **Prolog-first autonomous research engine** inside `prolog-rlm` that can investigate open-ended questions deeply and recursively **without requiring an LLM for planning, search, reasoning, evidence evaluation, or control flow**. LLMs are optional fallback components, not the intelligence substrate. Primary target: ```text question / goal -> deterministic NL normalization where possible -> Prolog research goal -> recursive symbolic decomposition -> source/tool discovery -> evidence acquisition -> claim + provenance graph -> contradiction / uncertainty analysis -> recursive gap filling -> convergence / stop decision -> structured answer + proof/evidence -> optional conversational rendering ``` An LLM may be used when useful for: - conversational natural-language rendering; - clarification of genuinely ambiguous user language; - fallback semantic parsing when deterministic parsing cannot normalize the input; - optional summarization of already-grounded evidence; - explicitly requested model-assisted reasoning. An LLM must **not** be required to decide what to research next, what sources to inspect, whether evidence supports a claim, whether a contradiction exists, whether more research is needed, or when a research branch is complete. ## Core principle The system should be as capable as practical using ordinary computation, symbolic AI, search, graph algorithms, expert rules, statistical/heuristic ranking, and external information sources. ```text LLM unavailable != research unavailable ``` A machine with SWI-Prolog, configured research adapters, and network/data access should still be able to perform a useful autonomous research run. ## Relationship to existing runtime Reuse current canonical runtime boundaries rather than creating a second agent runtime. Relevant existing work includes: - #335 reasoning modes (`symbolic`, `symbolic-recursive`, `auto`); - #288 plan dependency graphs / closed plan vocabulary; - #93 Project / source-knowledge / SPEC / VERIFY APIs; - #367 first-class bridge/harness exposure; - #352 recursive-budget correctness; - existing tool registry, MCP, capabilities, authority, tracing, usage, cancellation, async/Futures, graphs, context, and persistence surfaces. This epic is **not** permission to add another scheduler, another tool runtime, another authority system, or a model-controlled research loop parallel to the existing runtime. ## Desired experience A user should be able to ask: ```text Research whether claim X is true and keep digging until the important uncertainty is resolved. ``` The runtime should be able to proceed roughly as: ```prolog research_goal(G0) -> decompose(G0, Subgoals) -> rank(Subgoals) -> select_next(Subgoal) -> determine_required_evidence(Subgoal) -> discover_sources(Subgoal, Sources) -> acquire_evidence(Sources, Evidence) -> normalize_evidence(Evidence, Facts) -> update_claim_graph(Facts) -> evaluate_claims -> discover_gaps -> recurse_on_gaps -> convergence_check -> research_result(Result). ``` The exact predicates may differ. The semantic boundary matters. # 1. Research state is symbolic data Represent research as ordinary inspectable Prolog data with stable identities. Conceptual minimum: ```prolog research_run(Run, Goal). research_question(Run, Question). research_subgoal(Run, Parent, Goal). research_status(Run, Status). claim(Claim, Proposition). claim_status(Claim, supported | contradicted | mixed | unknown). claim_support(Claim, Evidence). claim_conflict(Claim, Evidence). evidence(Evidence, Source, Observation). source(Source, Kind, Locator). source_quality(Source, ScoreOrClass). needs_evidence(Claim, Requirement). research_gap(Run, Gap). ``` Do not flatten the research process into chat history or an opaque model transcript. The complete current research state must be inspectable/queryable without asking a model to reconstruct it. # 2. Deterministic natural-language front door Natural language should be usable without forcing every utterance through an LLM. Build a bounded deterministic normalization layer using appropriate Prolog-native techniques such as: - DCGs; - tokenization and morphology; - configurable lexicons and aliases; - entity/relationship dictionaries; - semantic templates; - typed slot filling; - date/number/unit normalization; - question/goal classification; - explicit ambiguity sets rather than guessed meanings. Example: ```text "Did company X acquire company Y before 2024?" ``` may normalize to data equivalent to: ```prolog research_question( acquisition_before(company_x, company_y, date(2024,1,1)) ). ``` If deterministic parsing cannot produce one defensible interpretation, return an inspectable ambiguity/unknown outcome. Only then may conversation mode optionally invoke an LLM parser or ask the operator for clarification. Model-generated Prolog must never be blindly `consult`ed or executed as trusted code. # 3. Deep recursive research planner Implement a non-LLM research planner over symbolic goals. It should support recursive decomposition from reusable expert rules such as: ```text identity question -> authoritative identity sources -> aliases -> conflicting identity records timeline question -> event extraction -> date normalization -> ordering -> missing intervals causal claim -> candidate causes -> mechanism evidence -> temporal evidence -> counterevidence -> alternative explanations entity relationship -> direct records -> common identifiers -> temporal consistency -> independent corroboration ``` Research strategies should be registered/extensible Prolog rules, not hard-coded into a model prompt. The planner may use classical search/AI techniques where appropriate: - best-first search; - beam search; - iterative deepening; - branch-and-bound; - dependency graphs; - rule priorities; - novelty/information-gain heuristics; - source diversity scoring; - confidence/uncertainty propagation. Use the existing graph/plan/async infrastructure where semantics align. # 4. Autonomous research agenda Each run maintains a bounded agenda/frontier of unresolved work. Conceptual states: ```text unexpanded ready searching awaiting_evidence supported contradicted mixed blocked unknown superseded complete ``` The engine chooses the next branch from symbolic state and deterministic/heuristic scoring. Priority should account for factors such as: - importance to the root question; - unresolved uncertainty; - dependency centrality; - evidence availability; - expected information gain; - source independence; - cost/time/tool budgets; - prior failed attempts; - novelty vs already-known evidence. No model call should be necessary to maintain or select the agenda. # 5. Research-tool abstraction Research sources are normal runtime tools/adapters. Potential adapters include: - web search; - URL fetch/readable-text extraction; - APIs; - MCP research tools; - local files; - databases; - Git repositories; - project/source KB; - scholarly indexes; - operator-provided corpora. The research engine reasons over a generic source/evidence contract rather than being coupled to one search provider. Availability is not authority. Existing capability/authority/network/effect boundaries remain canonical. # 6. Query generation without an LLM Research must be able to generate useful searches from symbolic goals deterministically. Support query construction from: - entity names + aliases; - predicates/relationship vocabulary; - dates/time windows; - domain keywords; - source-type hints; - previous failed queries; - discovered terminology; - citation/reference expansion. Example: ```prolog search_terms( acquisition_before(A, B, Date), [AName, BName, acquisition, acquired, merger, Date] ). ``` Use configurable synonym/alias/ontology knowledge to broaden or specialize queries. The engine should be able to recursively learn **in-run vocabulary** from acquired sources without permanently trusting it as host code. # 7. Evidence normalization and provenance Every material claim needs evidence lineage. Preserve at minimum: ```text source identity / locator retrieval time content/version/hash where possible extractor/parser identity relevant span/record research run claim(s) derived from it confidence/quality metadata ``` Separate: ```text raw observation normalized fact candidate derived fact inference claim assessment ``` A derived Prolog conclusion must not masquerade as a directly observed fact. # 8. Claim graph / knowledge graph Maintain an explicit graph connecting: ```text questions subgoals entities claims evidence sources inferences contradictions unknowns ``` Support traversal such as: ```prolog why(Claim, Proof). what_supports(Claim, Evidence). what_conflicts(Claim, Evidence). what_is_missing(Claim, Requirements). source_dependencies(Claim, Sources). ``` Recursive research should operate over this graph rather than repeatedly rereading an ever-growing prose context. # 9. Contradiction and uncertainty handling Do not collapse absence of evidence into falsehood. Research outcomes must distinguish at least: ```text supported contradicted mixed unknown blocked error ``` When sources conflict, preserve both branches and investigate the conflict. Useful recursive follow-up strategies include: - source independence checks; - primary-vs-secondary source resolution; - timestamp/version differences; - entity disambiguation; - scope/definition mismatch; - newer superseding evidence; - explicit counterexample search. # 10. Source quality / corroboration Implement configurable non-LLM scoring/rules for evidence quality. Possible inputs: - primary vs secondary source; - provenance completeness; - publication/update time; - directness of evidence; - independent corroboration; - duplicate/syndicated-source detection; - internal consistency; - known source/domain policy; - specificity to the claim; - extraction confidence. Do not turn a numeric score into fake certainty. Preserve the underlying evidence and reasons. # 11. Recursive gap discovery After each evidence update, derive what remains unknown. Examples: ```text claim has one weak source -> seek independent corroboration claim has conflicting dates -> research event timeline entity alias uncertain -> resolve identity before downstream inference source references another document -> inspect referenced primary document causal claim lacks mechanism evidence -> create mechanism subgoal ``` This is the heart of autonomous research: the next questions are generated from the current symbolic knowledge state. # 12. Convergence and stop conditions The engine needs explicit rules for knowing when to stop. Stop/reduce expansion based on combinations of: - root claims sufficiently supported/contradicted; - required evidence classes satisfied; - remaining gaps are low-impact; - no novel evidence after N strategies; - branch exhausted; - repeated equivalent queries/results; - confidence/quality threshold; - depth/node/tool/time/byte budgets; - operator policy. A run must never recurse forever merely because more web pages exist. Return the stop reason. # 13. Learning without an LLM Support safe accumulation of useful knowledge from successful research runs. Distinguish: ```text run-local observations candidate learned facts verified durable facts research strategy statistics source/domain metadata operator-authored expert rules ``` Do not auto-promote arbitrary web text into trusted executable Prolog. Useful non-LLM adaptation may include: - source success/failure statistics; - query-template performance; - alias/entity resolution caches; - evidence-pattern statistics; - discovered source relationships; - strategy ordering based on measured utility. Persistence must retain provenance and generation/version semantics. # 14. Optional LLM fallback contract The fallback boundary must be explicit and observable. Conceptual API: ```prolog fallback_policy(Session, Policy). fallback_allowed(nl_parse_ambiguity). fallback_allowed(conversation_render). fallback_allowed(optional_summary). ``` Default autonomous research policy should be equivalent to: ```text planner: no LLM agenda selection: no LLM query generation: no LLM source selection: no LLM claim evaluation: no LLM contradiction check: no LLM convergence: no LLM conversation: LLM allowed NL parse fallback: LLM allowed when deterministic parser fails ``` Every fallback event must be traceable with reason and usage. A model response is evidence only when the operator explicitly configures a model itself as a source; otherwise model output is never treated as factual evidence. # 15. Conversation mode Conversation is a projection over the symbolic state, not the state itself. Preferred flow: ```text user text -> deterministic parser -> symbolic intent/question -> research / query / explanation -> structured result -> deterministic renderer when sufficient -> optional LLM renderer for natural conversation ``` The LLM renderer receives bounded structured facts/proofs/evidence and may improve phrasing, but must not silently add unsupported factual claims. The user should be able to ask conversationally: ```text Why do you think that? What evidence contradicts it? What are you least sure about? Keep digging on the weak part. Where did that fact come from? Try another source. ``` Those utterances should map onto operations over the existing research graph. # 16. Explainability A completed run should be able to expose: ```text answer claim statuses proof/inference paths supporting evidence conflicting evidence source provenance unresolved gaps branches explored branches abandoned + reason stop reason resource usage LLM fallback calls, if any ``` A useful answer should never require replaying an opaque hidden reasoning transcript. # 17. Determinism and replay Given a fixed corpus/tool-result fixture, config, strategy registry, and budgets, the symbolic research controller should be reproducible enough for deterministic tests. Persist stable event/decision records sufficient to explain: - why a branch was selected; - why a query was generated; - why evidence changed a claim status; - why recursion continued/stopped. Live network results may naturally vary; control-flow semantics must still be testable with fixtures. # 18. Safety / authority invariants - Web/source text is untrusted data. - Never auto-`consult` retrieved Prolog/source text. - Research tools retain ordinary capabilities/authority. - Read-only research does not imply mutation authority. - LLM fallback cannot widen capabilities. - Learned facts cannot become executable trusted configuration without an explicit trust/promotion boundary. - Prompt injection in retrieved content is data, not an instruction to the runtime. - Recursive children inherit/narrow authority and budgets through existing runtime contracts. # Proposed implementation slices ## S1 — Research state + claim/evidence graph Add generic symbolic research-run, goal, claim, evidence, source, gap, and provenance contracts. ## S2 — Deterministic research agenda Add frontier selection, dependency handling, budgets, progress/novelty accounting, and stop reasons. ## S3 — Research strategy registry Add extensible Prolog rule packs for decomposition/evidence requirements and strategy selection. ## S4 — Generic search/fetch evidence adapters Compose existing tool/MCP surfaces into a generic research-source contract. ## S5 — Deterministic query compiler Symbolic goal -> search/query candidates using aliases, ontology/lexicon, dates, source classes, and previous observations. ## S6 — Evidence normalization + contradiction engine Normalize observations, track provenance, derive support/conflict/mixed/unknown states, and generate evidence gaps. ## S7 — Recursive gap-filling loop Drive autonomous continuation entirely from research graph state. ## S8 — Source quality + corroboration Add configurable evidence-quality and independence rules. ## S9 — Deterministic NL front door DCG/template/entity/alias based intent + question normalization with explicit ambiguity. ## S10 — Optional LLM fallback Add strictly bounded fallback policy and trace events for parsing/rendering/optional summarization. ## S11 — Conversation projection Natural conversational inspection and continuation over the symbolic research state. ## S12 — Durable learning / strategy metrics Persist safe, provenance-carrying learned facts and research-strategy statistics without promoting untrusted executable code. Child issues should be created as each slice reaches a concrete design boundary. Do not implement the entire epic as one PR. # Acceptance criteria - [ ] A complete autonomous research run can execute with **zero model/provider calls**. - [ ] With model providers disabled, the engine can recursively decompose a nontrivial fixture question, search fixture sources, acquire evidence, detect an evidence gap, recurse, resolve or preserve uncertainty, and terminate with a structured result. - [ ] Research planning/agenda/query generation/evidence evaluation/convergence are implemented in Prolog or deterministic host algorithms, not prompts. - [ ] Research state is inspectable as ordinary symbolic data. - [ ] Claims retain evidence/provenance and distinguish observed vs derived facts. - [ ] Contradictory evidence is preserved and can trigger recursive investigation. - [ ] Source duplication/independence can affect corroboration decisions. - [ ] Repeated searches/non-progress are detected and bounded. - [ ] Depth, node, source, tool, time, and byte budgets are enforced. - [ ] Cancellation terminates recursive work through the canonical runtime. - [ ] Fixed-source fixtures provide deterministic recursive-research tests. - [ ] Natural-language questions can be handled through deterministic normalization for a useful bounded grammar. - [ ] Ambiguous/unhandled language returns a structured outcome instead of fabricated semantics. - [ ] LLM fallback is optional, policy-controlled, usage-traced, and cannot widen authority. - [ ] Conversation mode can use an LLM renderer without making model output authoritative evidence. - [ ] `why`, support, conflict, uncertainty, provenance, branch history, and stop reason are inspectable. - [ ] Existing canonical tool/capability/authority/effect/async/graph/trace systems are reused rather than duplicated. - [ ] Full deterministic runtime gate remains green. # North-star test The strongest integration test should deliberately provide **no LLM credentials/provider at all**. Fixture question requires multiple hops and conflicting evidence: ```text Q -> discover entity alias -> find source A -> derive intermediate claim -> notice missing corroboration -> search independent source B -> encounter conflicting source C -> resolve conflict using date/version/provenance evidence -> answer with support + remaining uncertainty ``` The run passes only if the runtime performs the complete recursive investigation and produces a structured evidence-backed conclusion with `model_calls = 0`. Then run the same research session with conversation fallback enabled and prove that the optional model changes **presentation only**, not the underlying claim/evidence result. ## Non-goals - No requirement to eliminate LLM support from Prolog-RLM generally. - No attempt to build a human-level unrestricted English parser in the first slice. - No opaque model-generated research plans as the default engine. - No model-generated arbitrary executable Prolog. - No second scheduler/agent runtime/tool system. - No fake certainty from source scores. - No assumption that every open-ended question is decidable. ## End state The intended architecture is: ```text optional LLM conversation ^ | user -> NL -> symbolic research engine -> evidence/proofs -> answer | ^ v | tools/sources | | | +-> claim/evidence graph | +-> gaps -> recursive research ``` **Prolog owns the research intelligence.** The LLM is a useful language adapter when desired, not the thing keeping the system alive.

nsaspy closed this issue

2026-09-08 01:29:22 +00:00

nsaspy commented

2026-09-08 01:46:15 +00:00

Author

Owner

Closing as out of scope for prolog-rlm. The request was intended for Zara, specifically its Prolog-first / natural-language autonomous research behavior. This issue was created in the wrong repository and should not be implemented here.

Closing as **out of scope** for `prolog-rlm`. The request was intended for **Zara**, specifically its Prolog-first / natural-language autonomous research behavior. This issue was created in the wrong repository and should not be implemented here.