nsaspy/prolog-rlm

Fork 0

[EPIC] Add first-class Project / SPEC / source-knowledge APIs #437

New issue

Closed

opened 2026-09-10 21:18:11 +00:00 by nsaspy · 1 comment

nsaspy commented

2026-09-10 21:18:11 +00:00

Owner

Goal

Add the reusable Prolog-native foundation for representing a project, observing its source/code structure, defining a domain-neutral specification, freezing that specification, and verifying it against explicit evidence.

Project
  -> files / languages / artifacts
  -> parser/analyzer backends
  -> versioned source observations / CST / symbols / relations
  -> project knowledge base

SPEC
  -> requirements / invariants / expected artifacts / relations / observations
  -> validate
  -> freeze

VERIFY
  -> compare frozen SPEC against current observations/evidence
  -> structured satisfied / violated / unknown / blocked / error

Planning and execution are optional consumers. A Project may be indexed without a SPEC. A SPEC may be created/frozen without a Project. VERIFY may evaluate a frozen SPEC against existing evidence without planning or execution.

Architectural boundary

This epic owns epistemic project/source/specification concepts, not trusted project policy or coding-agent UX.

It is distinct from:

#74/#75/#77: durable trusted scoped state, project identity for policy/security, and safe project configuration;
#68-#71: prompt/workflow compilation and execution;
#56: generic proof/evidence result acceptance;
#53/#57: authority and external-effect identity.

Where #75 provides a canonical ProjectIdentity, this API should be able to reference/reuse it rather than inventing a competing security-sensitive identity. Source-observation identity, parser generations, file hashes, and syntax-node identities remain separate epistemic/versioning concepts.

Required Project model

Establish small normalized entities/relations equivalent in semantics to:

project(Project, Meta).
project_file(Project, File).

file(File, Path).
file_language(File, Language).
file_hash(File, Hash).

language(Language).
language_parser(Language, Backend).

Do not encode all metadata into a single file(Language, Path, ...) mega-term. Files, languages, parser backends, parse generations, symbols, and evidence need stable independent identities and relations.

The model must support multiple projects, changed/moved files, unknown/ambiguous languages, generated/vendor/excluded metadata, embedded language regions, non-code artifacts, and versioned observations.

Parser/analyzer architecture

Use a pluggable source-analysis boundary.

language_parser(python, tree_sitter).
language_parser(javascript, tree_sitter).
language_parser(nim, tree_sitter).
language_parser(common_lisp, tree_sitter).
language_parser(prolog, swi_native).

Tree-sitter integration is a direct SWI-Prolog <-> Tree-sitter C API binding. Do not add Python, Node, Nim, or another runtime merely as glue.

Tree-sitter owns concrete/incremental polyglot syntax parsing. Prolog owns normalized facts, language adapters, semantics, inference, SPEC, and VERIFY.

For Prolog source, prefer SWI-native source/xref analysis for authoritative semantic facts where it is stronger than a generic Tree-sitter grammar.

Source observation model

The persistent/inspectable project KB must not depend on raw native pointer identity.

Provide normalized versioned observations equivalent to:

parse(Parse, File).
parse_language(Parse, Language).
parse_backend(Parse, Backend).
parse_generation(Parse, Generation).
parse_status(Parse, Status).

syntax_node(Node, Parse, Type).
syntax_parent(Node, Parent).
source_span(Node, File, StartByte, EndByte).

symbol(Symbol, Kind, Name).
symbol_definition(Symbol, Node).
symbol_defined_in(Symbol, File).
reference(Reference, Kind, Symbol, Node).

calls(Caller, Callee).
imports(Source, Target).

Preserve the distinction between low-level CST observations, normalized cross-language facts, richer language-specific facts, and derived Prolog relations/inferences. Do not flatten every language into one impoverished universal AST.

Provenance and freshness

Every material source-derived fact must be traceable to project, file, content hash/generation, parser backend/grammar identity, query/extractor identity where applicable, and source range.

A changed file must not leave old facts silently looking current. Stale observations may remain inspectable, but current queries and VERIFY need deterministic freshness semantics.

First-class SPEC contract

A SPEC is domain-neutral declarative data. Coding/project requirements are one useful specialization, not the definition of SPEC.

Required semantics:

create/build independently of planning/execution;
schema/semantic validation;
explicit requirements/invariants/constraints with provenance;
freeze into an immutable/versioned specification;
frozen requirements cannot silently mutate during planning, execution, repair, or verification;
changing a frozen SPEC creates a new version/superseding SPEC;
requirements may refer to Project/source concepts;
callers can inspect/query SPEC as ordinary Prolog data.

Public API direction:

spec_create(+Input, +Options, -Outcome).
spec_validate(+Spec, -Outcome).
spec_freeze(+Spec, -FrozenSpec).
spec_requirement(+Spec, ?Requirement).
spec_status(+Spec, -Status).

Exact naming/arity should follow repository conventions.

First-class VERIFY contract

VERIFY evaluates a frozen specification against observations/evidence and does not require PLAN or EXECUTE.

Outcomes should distinguish:

satisfied(Evidence)
violated(Evidence)
unknown(Reason)
blocked(Reason)
error(Reason)

Verifier results preserve evidence/provenance and integrate with #56 without collapsing proof exists into premises are trustworthy. Worker/model self-report is not source evidence when a requirement asks for observable source/runtime state.

Tree-sitter child issues

#94 — Add direct SWI-Prolog Tree-sitter C FFI
#95 — Add Project file/language facts and Tree-sitter grammar registry
#96 — Project Tree-sitter CST into versioned Prolog syntax facts
#97 — Expose Tree-sitter query/capture APIs to Prolog
#98 — Normalize Tree-sitter captures into symbols, references, and source relations
#99 — Add incremental Tree-sitter reparsing, changed-range invalidation, and source freshness

Dependency shape:

#93 epic
  |
  +--> #94 direct Tree-sitter FFI
  |      |
  |      +--> #95 Project/file/language/grammar registry
  |              |
  |              +--> #96 versioned CST facts
  |                       |
  |                       +--> #97 query/capture API
  |                                |
  |                                +--> #98 semantic facts
  |                                         |
  |                                         +--> #99 incremental freshness
  |
  +--> first-class SPEC API
  |
  +--> first-class VERIFY API
          +--> consumes current Project/source evidence
          +--> composes with #56

The SPEC and VERIFY contracts remain tracked by this epic and should be decomposed into focused implementation issues when their concrete API design is ready. Tree-sitter is an evidence source for them, not a mandatory dependency for domain-neutral SPEC use.

Security / authority invariants

Parsing a repository grants no execution authority.
Loading a grammar grants no tool capability.
Source text and Tree-sitter query data are not arbitrary Prolog callables.
Do not auto-consult arbitrary project Prolog as trusted executable policy.
Parser output is observation with provenance, not automatically trusted host policy.
Model-generated SPEC content cannot widen #53 authority or #75/#76 trusted persisted policy.
FFI resource lifetime bugs must fail structurally where possible rather than expose stale native pointers as valid KB entities.

Integration acceptance

A host can register/index at least two projects without cross-project fact leakage.
Files have independent identities, paths, languages, hashes/generations, and parser assignments.
At least three non-Prolog languages are parsed through the same generic Tree-sitter FFI/API without language-specific C wrappers.
Prolog can traverse generic syntax nodes and execute Tree-sitter queries/captures.
Parser errors/incomplete source are observations rather than requiring a fully valid file.
Source facts carry deterministic file/range/generation/parser provenance.
Editing a file produces a new parse generation and cannot leave stale observations looking current.
A normalized symbol/definition/reference layer can be queried independently of raw CST traversal.
A SPEC can be created, validated, frozen, inspected, and versioned without PLAN/EXECUTE.
VERIFY can evaluate a frozen SPEC against current Project observations without PLAN/EXECUTE.
Frozen requirements cannot be mutated in place by repair/replanning/execution.
Missing/ambiguous/stale evidence remains distinct from verified success or failure.
Deterministic fixtures cover grammar loading, malformed source, multiple languages, stale generations, incremental edits, SPEC freezing, and evidence-backed verification.

Non-goals

No coding-agent TUI.
No mandatory SPEC -> PLAN -> EXECUTE workflow.
No requirement that every Project has a SPEC.
No requirement that every SPEC describes code.
No giant universal AST ontology before concrete adapters justify it.
No Python/Node/Nim runtime dependency for the Tree-sitter bridge.
No arbitrary source-code execution as part of indexing.
No replacement for #74/#75 trusted scoped project policy/state.
No replacement for #68 prompt/workflow compilation.

References

#56 proof/evidence verifier boundary
#68-#71 compiler/context/workflow pipeline
#74-#77 project-scoped trusted state/policy
#53 host authority
#57 effect identity
SWI library(prolog_source) / library(prolog_xref) for native Prolog analysis
Tree-sitter C API + query/incremental parsing model

## Goal Add the reusable Prolog-native foundation for representing a project, observing its source/code structure, defining a domain-neutral specification, freezing that specification, and verifying it against explicit evidence. ```text Project -> files / languages / artifacts -> parser/analyzer backends -> versioned source observations / CST / symbols / relations -> project knowledge base SPEC -> requirements / invariants / expected artifacts / relations / observations -> validate -> freeze VERIFY -> compare frozen SPEC against current observations/evidence -> structured satisfied / violated / unknown / blocked / error ``` Planning and execution are optional consumers. A Project may be indexed without a SPEC. A SPEC may be created/frozen without a Project. VERIFY may evaluate a frozen SPEC against existing evidence without planning or execution. ## Architectural boundary This epic owns **epistemic project/source/specification concepts**, not trusted project policy or coding-agent UX. It is distinct from: - #74/#75/#77: durable trusted scoped state, project identity for policy/security, and safe project configuration; - #68-#71: prompt/workflow compilation and execution; - #56: generic proof/evidence result acceptance; - #53/#57: authority and external-effect identity. Where #75 provides a canonical `ProjectIdentity`, this API should be able to reference/reuse it rather than inventing a competing security-sensitive identity. Source-observation identity, parser generations, file hashes, and syntax-node identities remain separate epistemic/versioning concepts. ## Required Project model Establish small normalized entities/relations equivalent in semantics to: ```prolog project(Project, Meta). project_file(Project, File). file(File, Path). file_language(File, Language). file_hash(File, Hash). language(Language). language_parser(Language, Backend). ``` Do not encode all metadata into a single `file(Language, Path, ...)` mega-term. Files, languages, parser backends, parse generations, symbols, and evidence need stable independent identities and relations. The model must support multiple projects, changed/moved files, unknown/ambiguous languages, generated/vendor/excluded metadata, embedded language regions, non-code artifacts, and versioned observations. ## Parser/analyzer architecture Use a pluggable source-analysis boundary. ```prolog language_parser(python, tree_sitter). language_parser(javascript, tree_sitter). language_parser(nim, tree_sitter). language_parser(common_lisp, tree_sitter). language_parser(prolog, swi_native). ``` Tree-sitter integration is a **direct SWI-Prolog <-> Tree-sitter C API binding**. Do not add Python, Node, Nim, or another runtime merely as glue. Tree-sitter owns concrete/incremental polyglot syntax parsing. Prolog owns normalized facts, language adapters, semantics, inference, SPEC, and VERIFY. For Prolog source, prefer SWI-native source/xref analysis for authoritative semantic facts where it is stronger than a generic Tree-sitter grammar. ## Source observation model The persistent/inspectable project KB must not depend on raw native pointer identity. Provide normalized versioned observations equivalent to: ```prolog parse(Parse, File). parse_language(Parse, Language). parse_backend(Parse, Backend). parse_generation(Parse, Generation). parse_status(Parse, Status). syntax_node(Node, Parse, Type). syntax_parent(Node, Parent). source_span(Node, File, StartByte, EndByte). symbol(Symbol, Kind, Name). symbol_definition(Symbol, Node). symbol_defined_in(Symbol, File). reference(Reference, Kind, Symbol, Node). calls(Caller, Callee). imports(Source, Target). ``` Preserve the distinction between low-level CST observations, normalized cross-language facts, richer language-specific facts, and derived Prolog relations/inferences. Do not flatten every language into one impoverished universal AST. ## Provenance and freshness Every material source-derived fact must be traceable to project, file, content hash/generation, parser backend/grammar identity, query/extractor identity where applicable, and source range. A changed file must not leave old facts silently looking current. Stale observations may remain inspectable, but current queries and VERIFY need deterministic freshness semantics. ## First-class SPEC contract A SPEC is domain-neutral declarative data. Coding/project requirements are one useful specialization, not the definition of SPEC. Required semantics: - create/build independently of planning/execution; - schema/semantic validation; - explicit requirements/invariants/constraints with provenance; - freeze into an immutable/versioned specification; - frozen requirements cannot silently mutate during planning, execution, repair, or verification; - changing a frozen SPEC creates a new version/superseding SPEC; - requirements may refer to Project/source concepts; - callers can inspect/query SPEC as ordinary Prolog data. Public API direction: ```prolog spec_create(+Input, +Options, -Outcome). spec_validate(+Spec, -Outcome). spec_freeze(+Spec, -FrozenSpec). spec_requirement(+Spec, ?Requirement). spec_status(+Spec, -Status). ``` Exact naming/arity should follow repository conventions. ## First-class VERIFY contract VERIFY evaluates a frozen specification against observations/evidence and does not require PLAN or EXECUTE. Outcomes should distinguish: ```text satisfied(Evidence) violated(Evidence) unknown(Reason) blocked(Reason) error(Reason) ``` Verifier results preserve evidence/provenance and integrate with #56 without collapsing `proof exists` into `premises are trustworthy`. Worker/model self-report is not source evidence when a requirement asks for observable source/runtime state. ## Tree-sitter child issues - [x] #94 — **Add direct SWI-Prolog Tree-sitter C FFI** - [ ] #95 — **Add Project file/language facts and Tree-sitter grammar registry** - [ ] #96 — **Project Tree-sitter CST into versioned Prolog syntax facts** - [ ] #97 — **Expose Tree-sitter query/capture APIs to Prolog** - [ ] #98 — **Normalize Tree-sitter captures into symbols, references, and source relations** - [ ] #99 — **Add incremental Tree-sitter reparsing, changed-range invalidation, and source freshness** Dependency shape: ```text #93 epic | +--> #94 direct Tree-sitter FFI | | | +--> #95 Project/file/language/grammar registry | | | +--> #96 versioned CST facts | | | +--> #97 query/capture API | | | +--> #98 semantic facts | | | +--> #99 incremental freshness | +--> first-class SPEC API | +--> first-class VERIFY API +--> consumes current Project/source evidence +--> composes with #56 ``` The SPEC and VERIFY contracts remain tracked by this epic and should be decomposed into focused implementation issues when their concrete API design is ready. Tree-sitter is an evidence source for them, not a mandatory dependency for domain-neutral SPEC use. ## Security / authority invariants - Parsing a repository grants no execution authority. - Loading a grammar grants no tool capability. - Source text and Tree-sitter query data are not arbitrary Prolog callables. - Do not auto-`consult` arbitrary project Prolog as trusted executable policy. - Parser output is observation with provenance, not automatically trusted host policy. - Model-generated SPEC content cannot widen #53 authority or #75/#76 trusted persisted policy. - FFI resource lifetime bugs must fail structurally where possible rather than expose stale native pointers as valid KB entities. ## Integration acceptance - [ ] A host can register/index at least two projects without cross-project fact leakage. - [ ] Files have independent identities, paths, languages, hashes/generations, and parser assignments. - [ ] At least three non-Prolog languages are parsed through the same generic Tree-sitter FFI/API without language-specific C wrappers. - [ ] Prolog can traverse generic syntax nodes and execute Tree-sitter queries/captures. - [ ] Parser errors/incomplete source are observations rather than requiring a fully valid file. - [ ] Source facts carry deterministic file/range/generation/parser provenance. - [ ] Editing a file produces a new parse generation and cannot leave stale observations looking current. - [ ] A normalized symbol/definition/reference layer can be queried independently of raw CST traversal. - [ ] A SPEC can be created, validated, frozen, inspected, and versioned without PLAN/EXECUTE. - [ ] VERIFY can evaluate a frozen SPEC against current Project observations without PLAN/EXECUTE. - [ ] Frozen requirements cannot be mutated in place by repair/replanning/execution. - [ ] Missing/ambiguous/stale evidence remains distinct from verified success or failure. - [ ] Deterministic fixtures cover grammar loading, malformed source, multiple languages, stale generations, incremental edits, SPEC freezing, and evidence-backed verification. ## Non-goals - No coding-agent TUI. - No mandatory SPEC -> PLAN -> EXECUTE workflow. - No requirement that every Project has a SPEC. - No requirement that every SPEC describes code. - No giant universal AST ontology before concrete adapters justify it. - No Python/Node/Nim runtime dependency for the Tree-sitter bridge. - No arbitrary source-code execution as part of indexing. - No replacement for #74/#75 trusted scoped project policy/state. - No replacement for #68 prompt/workflow compilation. ## References - #56 proof/evidence verifier boundary - #68-#71 compiler/context/workflow pipeline - #74-#77 project-scoped trusted state/policy - #53 host authority - #57 effect identity - SWI `library(prolog_source)` / `library(prolog_xref)` for native Prolog analysis - Tree-sitter C API + query/incremental parsing model

nsaspy commented

2026-09-10 21:23:35 +00:00

Author

Owner

Duplicate of #93 (pre-existing Forgejo mirror with GitHub number parity). Closing this accidental duplicate created by today's open-state sync; #93 stays canonical on Forgejo.

nsaspy closed this issue

2026-09-10 21:23:35 +00:00