[EPIC] Add first-class Project / SPEC / source-knowledge APIs #93

Open
opened 2026-08-18 14:13:09 +00:00 by lost-rob0t · 2 comments
lost-rob0t commented 2026-08-18 14:13:09 +00:00 (Migrated from github.com)

Goal

Add the reusable Prolog-native foundation for representing a project, observing its source/code structure, defining a domain-neutral specification, freezing that specification, and verifying it against explicit evidence.

Project
  -> files / languages / artifacts
  -> parser/analyzer backends
  -> versioned source observations / CST / symbols / relations
  -> project knowledge base

SPEC
  -> requirements / invariants / expected artifacts / relations / observations
  -> validate
  -> freeze

VERIFY
  -> compare frozen SPEC against current observations/evidence
  -> structured satisfied / violated / unknown / blocked / error

Planning and execution are optional consumers. A Project may be indexed without a SPEC. A SPEC may be created/frozen without a Project. VERIFY may evaluate a frozen SPEC against existing evidence without planning or execution.

Architectural boundary

This epic owns epistemic project/source/specification concepts, not trusted project policy or coding-agent UX.

It is distinct from:

  • #74/#75/#77: durable trusted scoped state, project identity for policy/security, and safe project configuration;
  • #68-#71: prompt/workflow compilation and execution;
  • #56: generic proof/evidence result acceptance;
  • #53/#57: authority and external-effect identity.

Where #75 provides a canonical ProjectIdentity, this API should be able to reference/reuse it rather than inventing a competing security-sensitive identity. Source-observation identity, parser generations, file hashes, and syntax-node identities remain separate epistemic/versioning concepts.

Required Project model

Establish small normalized entities/relations equivalent in semantics to:

project(Project, Meta).
project_file(Project, File).

file(File, Path).
file_language(File, Language).
file_hash(File, Hash).

language(Language).
language_parser(Language, Backend).

Do not encode all metadata into a single file(Language, Path, ...) mega-term. Files, languages, parser backends, parse generations, symbols, and evidence need stable independent identities and relations.

The model must support multiple projects, changed/moved files, unknown/ambiguous languages, generated/vendor/excluded metadata, embedded language regions, non-code artifacts, and versioned observations.

Parser/analyzer architecture

Use a pluggable source-analysis boundary.

language_parser(python, tree_sitter).
language_parser(javascript, tree_sitter).
language_parser(nim, tree_sitter).
language_parser(common_lisp, tree_sitter).
language_parser(prolog, swi_native).

Tree-sitter integration is a direct SWI-Prolog <-> Tree-sitter C API binding. Do not add Python, Node, Nim, or another runtime merely as glue.

Tree-sitter owns concrete/incremental polyglot syntax parsing. Prolog owns normalized facts, language adapters, semantics, inference, SPEC, and VERIFY.

For Prolog source, prefer SWI-native source/xref analysis for authoritative semantic facts where it is stronger than a generic Tree-sitter grammar.

Source observation model

The persistent/inspectable project KB must not depend on raw native pointer identity.

Provide normalized versioned observations equivalent to:

parse(Parse, File).
parse_language(Parse, Language).
parse_backend(Parse, Backend).
parse_generation(Parse, Generation).
parse_status(Parse, Status).

syntax_node(Node, Parse, Type).
syntax_parent(Node, Parent).
source_span(Node, File, StartByte, EndByte).

symbol(Symbol, Kind, Name).
symbol_definition(Symbol, Node).
symbol_defined_in(Symbol, File).
reference(Reference, Kind, Symbol, Node).

calls(Caller, Callee).
imports(Source, Target).

Preserve the distinction between low-level CST observations, normalized cross-language facts, richer language-specific facts, and derived Prolog relations/inferences. Do not flatten every language into one impoverished universal AST.

Provenance and freshness

Every material source-derived fact must be traceable to project, file, content hash/generation, parser backend/grammar identity, query/extractor identity where applicable, and source range.

A changed file must not leave old facts silently looking current. Stale observations may remain inspectable, but current queries and VERIFY need deterministic freshness semantics.

First-class SPEC contract

A SPEC is domain-neutral declarative data. Coding/project requirements are one useful specialization, not the definition of SPEC.

Required semantics:

  • create/build independently of planning/execution;
  • schema/semantic validation;
  • explicit requirements/invariants/constraints with provenance;
  • freeze into an immutable/versioned specification;
  • frozen requirements cannot silently mutate during planning, execution, repair, or verification;
  • changing a frozen SPEC creates a new version/superseding SPEC;
  • requirements may refer to Project/source concepts;
  • callers can inspect/query SPEC as ordinary Prolog data.

Public API direction:

spec_create(+Input, +Options, -Outcome).
spec_validate(+Spec, -Outcome).
spec_freeze(+Spec, -FrozenSpec).
spec_requirement(+Spec, ?Requirement).
spec_status(+Spec, -Status).

Exact naming/arity should follow repository conventions.

First-class VERIFY contract

VERIFY evaluates a frozen specification against observations/evidence and does not require PLAN or EXECUTE.

Outcomes should distinguish:

satisfied(Evidence)
violated(Evidence)
unknown(Reason)
blocked(Reason)
error(Reason)

Verifier results preserve evidence/provenance and integrate with #56 without collapsing proof exists into premises are trustworthy. Worker/model self-report is not source evidence when a requirement asks for observable source/runtime state.

Tree-sitter child issues

  • #94 — Add direct SWI-Prolog Tree-sitter C FFI
  • #95 — Add Project file/language facts and Tree-sitter grammar registry
  • #96 — Project Tree-sitter CST into versioned Prolog syntax facts
  • #97 — Expose Tree-sitter query/capture APIs to Prolog
  • #98 — Normalize Tree-sitter captures into symbols, references, and source relations
  • #99 — Add incremental Tree-sitter reparsing, changed-range invalidation, and source freshness

Dependency shape:

#93 epic
  |
  +--> #94 direct Tree-sitter FFI
  |      |
  |      +--> #95 Project/file/language/grammar registry
  |              |
  |              +--> #96 versioned CST facts
  |                       |
  |                       +--> #97 query/capture API
  |                                |
  |                                +--> #98 semantic facts
  |                                         |
  |                                         +--> #99 incremental freshness
  |
  +--> first-class SPEC API
  |
  +--> first-class VERIFY API
          +--> consumes current Project/source evidence
          +--> composes with #56

The SPEC and VERIFY contracts remain tracked by this epic and should be decomposed into focused implementation issues when their concrete API design is ready. Tree-sitter is an evidence source for them, not a mandatory dependency for domain-neutral SPEC use.

Security / authority invariants

  • Parsing a repository grants no execution authority.
  • Loading a grammar grants no tool capability.
  • Source text and Tree-sitter query data are not arbitrary Prolog callables.
  • Do not auto-consult arbitrary project Prolog as trusted executable policy.
  • Parser output is observation with provenance, not automatically trusted host policy.
  • Model-generated SPEC content cannot widen #53 authority or #75/#76 trusted persisted policy.
  • FFI resource lifetime bugs must fail structurally where possible rather than expose stale native pointers as valid KB entities.

Integration acceptance

  • A host can register/index at least two projects without cross-project fact leakage.
  • Files have independent identities, paths, languages, hashes/generations, and parser assignments.
  • At least three non-Prolog languages are parsed through the same generic Tree-sitter FFI/API without language-specific C wrappers.
  • Prolog can traverse generic syntax nodes and execute Tree-sitter queries/captures.
  • Parser errors/incomplete source are observations rather than requiring a fully valid file.
  • Source facts carry deterministic file/range/generation/parser provenance.
  • Editing a file produces a new parse generation and cannot leave stale observations looking current.
  • A normalized symbol/definition/reference layer can be queried independently of raw CST traversal.
  • A SPEC can be created, validated, frozen, inspected, and versioned without PLAN/EXECUTE.
  • VERIFY can evaluate a frozen SPEC against current Project observations without PLAN/EXECUTE.
  • Frozen requirements cannot be mutated in place by repair/replanning/execution.
  • Missing/ambiguous/stale evidence remains distinct from verified success or failure.
  • Deterministic fixtures cover grammar loading, malformed source, multiple languages, stale generations, incremental edits, SPEC freezing, and evidence-backed verification.

Non-goals

  • No coding-agent TUI.
  • No mandatory SPEC -> PLAN -> EXECUTE workflow.
  • No requirement that every Project has a SPEC.
  • No requirement that every SPEC describes code.
  • No giant universal AST ontology before concrete adapters justify it.
  • No Python/Node/Nim runtime dependency for the Tree-sitter bridge.
  • No arbitrary source-code execution as part of indexing.
  • No replacement for #74/#75 trusted scoped project policy/state.
  • No replacement for #68 prompt/workflow compilation.

References

  • #56 proof/evidence verifier boundary
  • #68-#71 compiler/context/workflow pipeline
  • #74-#77 project-scoped trusted state/policy
  • #53 host authority
  • #57 effect identity
  • SWI library(prolog_source) / library(prolog_xref) for native Prolog analysis
  • Tree-sitter C API + query/incremental parsing model
## Goal Add the reusable Prolog-native foundation for representing a project, observing its source/code structure, defining a domain-neutral specification, freezing that specification, and verifying it against explicit evidence. ```text Project -> files / languages / artifacts -> parser/analyzer backends -> versioned source observations / CST / symbols / relations -> project knowledge base SPEC -> requirements / invariants / expected artifacts / relations / observations -> validate -> freeze VERIFY -> compare frozen SPEC against current observations/evidence -> structured satisfied / violated / unknown / blocked / error ``` Planning and execution are optional consumers. A Project may be indexed without a SPEC. A SPEC may be created/frozen without a Project. VERIFY may evaluate a frozen SPEC against existing evidence without planning or execution. ## Architectural boundary This epic owns **epistemic project/source/specification concepts**, not trusted project policy or coding-agent UX. It is distinct from: - #74/#75/#77: durable trusted scoped state, project identity for policy/security, and safe project configuration; - #68-#71: prompt/workflow compilation and execution; - #56: generic proof/evidence result acceptance; - #53/#57: authority and external-effect identity. Where #75 provides a canonical `ProjectIdentity`, this API should be able to reference/reuse it rather than inventing a competing security-sensitive identity. Source-observation identity, parser generations, file hashes, and syntax-node identities remain separate epistemic/versioning concepts. ## Required Project model Establish small normalized entities/relations equivalent in semantics to: ```prolog project(Project, Meta). project_file(Project, File). file(File, Path). file_language(File, Language). file_hash(File, Hash). language(Language). language_parser(Language, Backend). ``` Do not encode all metadata into a single `file(Language, Path, ...)` mega-term. Files, languages, parser backends, parse generations, symbols, and evidence need stable independent identities and relations. The model must support multiple projects, changed/moved files, unknown/ambiguous languages, generated/vendor/excluded metadata, embedded language regions, non-code artifacts, and versioned observations. ## Parser/analyzer architecture Use a pluggable source-analysis boundary. ```prolog language_parser(python, tree_sitter). language_parser(javascript, tree_sitter). language_parser(nim, tree_sitter). language_parser(common_lisp, tree_sitter). language_parser(prolog, swi_native). ``` Tree-sitter integration is a **direct SWI-Prolog <-> Tree-sitter C API binding**. Do not add Python, Node, Nim, or another runtime merely as glue. Tree-sitter owns concrete/incremental polyglot syntax parsing. Prolog owns normalized facts, language adapters, semantics, inference, SPEC, and VERIFY. For Prolog source, prefer SWI-native source/xref analysis for authoritative semantic facts where it is stronger than a generic Tree-sitter grammar. ## Source observation model The persistent/inspectable project KB must not depend on raw native pointer identity. Provide normalized versioned observations equivalent to: ```prolog parse(Parse, File). parse_language(Parse, Language). parse_backend(Parse, Backend). parse_generation(Parse, Generation). parse_status(Parse, Status). syntax_node(Node, Parse, Type). syntax_parent(Node, Parent). source_span(Node, File, StartByte, EndByte). symbol(Symbol, Kind, Name). symbol_definition(Symbol, Node). symbol_defined_in(Symbol, File). reference(Reference, Kind, Symbol, Node). calls(Caller, Callee). imports(Source, Target). ``` Preserve the distinction between low-level CST observations, normalized cross-language facts, richer language-specific facts, and derived Prolog relations/inferences. Do not flatten every language into one impoverished universal AST. ## Provenance and freshness Every material source-derived fact must be traceable to project, file, content hash/generation, parser backend/grammar identity, query/extractor identity where applicable, and source range. A changed file must not leave old facts silently looking current. Stale observations may remain inspectable, but current queries and VERIFY need deterministic freshness semantics. ## First-class SPEC contract A SPEC is domain-neutral declarative data. Coding/project requirements are one useful specialization, not the definition of SPEC. Required semantics: - create/build independently of planning/execution; - schema/semantic validation; - explicit requirements/invariants/constraints with provenance; - freeze into an immutable/versioned specification; - frozen requirements cannot silently mutate during planning, execution, repair, or verification; - changing a frozen SPEC creates a new version/superseding SPEC; - requirements may refer to Project/source concepts; - callers can inspect/query SPEC as ordinary Prolog data. Public API direction: ```prolog spec_create(+Input, +Options, -Outcome). spec_validate(+Spec, -Outcome). spec_freeze(+Spec, -FrozenSpec). spec_requirement(+Spec, ?Requirement). spec_status(+Spec, -Status). ``` Exact naming/arity should follow repository conventions. ## First-class VERIFY contract VERIFY evaluates a frozen specification against observations/evidence and does not require PLAN or EXECUTE. Outcomes should distinguish: ```text satisfied(Evidence) violated(Evidence) unknown(Reason) blocked(Reason) error(Reason) ``` Verifier results preserve evidence/provenance and integrate with #56 without collapsing `proof exists` into `premises are trustworthy`. Worker/model self-report is not source evidence when a requirement asks for observable source/runtime state. ## Tree-sitter child issues - [x] #94 — **Add direct SWI-Prolog Tree-sitter C FFI** - [ ] #95 — **Add Project file/language facts and Tree-sitter grammar registry** - [ ] #96 — **Project Tree-sitter CST into versioned Prolog syntax facts** - [ ] #97 — **Expose Tree-sitter query/capture APIs to Prolog** - [ ] #98 — **Normalize Tree-sitter captures into symbols, references, and source relations** - [ ] #99 — **Add incremental Tree-sitter reparsing, changed-range invalidation, and source freshness** Dependency shape: ```text #93 epic | +--> #94 direct Tree-sitter FFI | | | +--> #95 Project/file/language/grammar registry | | | +--> #96 versioned CST facts | | | +--> #97 query/capture API | | | +--> #98 semantic facts | | | +--> #99 incremental freshness | +--> first-class SPEC API | +--> first-class VERIFY API +--> consumes current Project/source evidence +--> composes with #56 ``` The SPEC and VERIFY contracts remain tracked by this epic and should be decomposed into focused implementation issues when their concrete API design is ready. Tree-sitter is an evidence source for them, not a mandatory dependency for domain-neutral SPEC use. ## Security / authority invariants - Parsing a repository grants no execution authority. - Loading a grammar grants no tool capability. - Source text and Tree-sitter query data are not arbitrary Prolog callables. - Do not auto-`consult` arbitrary project Prolog as trusted executable policy. - Parser output is observation with provenance, not automatically trusted host policy. - Model-generated SPEC content cannot widen #53 authority or #75/#76 trusted persisted policy. - FFI resource lifetime bugs must fail structurally where possible rather than expose stale native pointers as valid KB entities. ## Integration acceptance - [ ] A host can register/index at least two projects without cross-project fact leakage. - [ ] Files have independent identities, paths, languages, hashes/generations, and parser assignments. - [ ] At least three non-Prolog languages are parsed through the same generic Tree-sitter FFI/API without language-specific C wrappers. - [ ] Prolog can traverse generic syntax nodes and execute Tree-sitter queries/captures. - [ ] Parser errors/incomplete source are observations rather than requiring a fully valid file. - [ ] Source facts carry deterministic file/range/generation/parser provenance. - [ ] Editing a file produces a new parse generation and cannot leave stale observations looking current. - [ ] A normalized symbol/definition/reference layer can be queried independently of raw CST traversal. - [ ] A SPEC can be created, validated, frozen, inspected, and versioned without PLAN/EXECUTE. - [ ] VERIFY can evaluate a frozen SPEC against current Project observations without PLAN/EXECUTE. - [ ] Frozen requirements cannot be mutated in place by repair/replanning/execution. - [ ] Missing/ambiguous/stale evidence remains distinct from verified success or failure. - [ ] Deterministic fixtures cover grammar loading, malformed source, multiple languages, stale generations, incremental edits, SPEC freezing, and evidence-backed verification. ## Non-goals - No coding-agent TUI. - No mandatory SPEC -> PLAN -> EXECUTE workflow. - No requirement that every Project has a SPEC. - No requirement that every SPEC describes code. - No giant universal AST ontology before concrete adapters justify it. - No Python/Node/Nim runtime dependency for the Tree-sitter bridge. - No arbitrary source-code execution as part of indexing. - No replacement for #74/#75 trusted scoped project policy/state. - No replacement for #68 prompt/workflow compilation. ## References - #56 proof/evidence verifier boundary - #68-#71 compiler/context/workflow pipeline - #74-#77 project-scoped trusted state/policy - #53 host authority - #57 effect identity - SWI `library(prolog_source)` / `library(prolog_xref)` for native Prolog analysis - Tree-sitter C API + query/incremental parsing model
Owner

Operator clarification: on-disk project-local KB under $PROJECT_ROOT/.kb/ (first recorded 2026-09-02)

The epic's "persistent/inspectable project KB" language was never given a concrete persistence design. A repository/issue-wide check on 2026-09-02 found no doc, issue, or research record specifying where project KB state lives on disk. Recording the operator-clarified intent here so future RAGE slices consume it from the epic instead of rediscovering it:

Requirement (operator, 2026-09-02)

  • Project KB state that Prolog generates (parse observations, query extractions, derived facts) persists to disk under $PROJECT_ROOT/.kb/<kb tree> — project-local, the writable workspace being observed.
  • The nix store (grammar bundle, FFI library, RLM_TREE_SITTER_GRAMMAR_DIR) is inputs, read-only by nature; it is never KB state. A read-only project root (e.g. store-built tree) must fail with a structured persistence error, never silently degrade to memory-only.
  • Mechanics follow the repository's established persistence precedent: backend-neutral persistence interface with library(persistency) as the initial/local adapter only (per TODO.md and docs/artifact-runtime.md), journal/snapshot under the .kb tree, reload on registry open, deterministic replay.
  • On-disk project KB facts are project epistemic state — a distinct semantic class from durable artifacts, graph checkpoints, MCP configuration, and effect journals. Same disk, different semantics; no shared generic fact bucket.
  • Fresh-process restart fixtures are required wherever a slice introduces .kb persistence (crash/restart semantics are observable behavior, not implementation detail).

Where this lands

  • This is not new scope: it is the previously-unwritten storage half of this epic's project knowledge base goal, first written down in research/tree-sitter-query-capture.org (RLM-RESEARCH-012, branch rage/97-query-capture-apis, base 156bfe9b2caffa1b967e77bee51263ee9e85fb50, currently PENDING research approval in the ADADR control plane).
  • Slice #97 (Tree-sitter query/capture APIs) introduces the .kb layer for its query-extraction facts (own rlm_project_query sibling module, not inside the #96 CST module) and defines the reusable layout.
  • #219 (retrieval intelligence, embeddings, source-aware KB) is a later consumer of this layer.
  • Epic body "Source observation model" may cite this comment when the durable KB child issue is decomposed; the SPEC/VERIFY slices should treat .kb reload as the freshness substrate for post-restart evidence.
## Operator clarification: on-disk project-local KB under `$PROJECT_ROOT/.kb/` (first recorded 2026-09-02) The epic's "persistent/inspectable project KB" language was never given a concrete persistence design. A repository/issue-wide check on 2026-09-02 found no doc, issue, or research record specifying where project KB state lives on disk. Recording the operator-clarified intent here so future RAGE slices consume it from the epic instead of rediscovering it: ### Requirement (operator, 2026-09-02) - Project KB state that Prolog generates (parse observations, query extractions, derived facts) **persists to disk under `$PROJECT_ROOT/.kb/<kb tree>`** — project-local, the writable workspace being observed. - The nix store (grammar bundle, FFI library, `RLM_TREE_SITTER_GRAMMAR_DIR`) is **inputs**, read-only by nature; it is never KB state. A read-only project root (e.g. store-built tree) must fail with a structured persistence error, never silently degrade to memory-only. - Mechanics follow the repository's established persistence precedent: backend-neutral persistence interface with `library(persistency)` as the initial/local adapter only (per TODO.md and `docs/artifact-runtime.md`), journal/snapshot under the `.kb` tree, reload on registry open, deterministic replay. - On-disk project KB facts are **project epistemic state** — a distinct semantic class from durable artifacts, graph checkpoints, MCP configuration, and effect journals. Same disk, different semantics; no shared generic fact bucket. - Fresh-process restart fixtures are required wherever a slice introduces `.kb` persistence (crash/restart semantics are observable behavior, not implementation detail). ### Where this lands - This is **not** new scope: it is the previously-unwritten storage half of this epic's project knowledge base goal, first written down in `research/tree-sitter-query-capture.org` (`RLM-RESEARCH-012`, branch `rage/97-query-capture-apis`, base `156bfe9b2caffa1b967e77bee51263ee9e85fb50`, currently `PENDING` research approval in the ADADR control plane). - Slice #97 (Tree-sitter query/capture APIs) introduces the `.kb` layer for its query-extraction facts (own `rlm_project_query` sibling module, not inside the #96 CST module) and defines the reusable layout. - #219 (retrieval intelligence, embeddings, source-aware KB) is a later consumer of this layer. - Epic body "Source observation model" may cite this comment when the durable KB child issue is decomposed; the SPEC/VERIFY slices should treat `.kb` reload as the freshness substrate for post-restart evidence.
Owner

Downstream proving consumer: Zara Android projects

lost-rob0t/zara#653 is now the concrete product proving ground for persistent project contexts and project-scoped chat under the frozen Android design (lost-rob0t/zara#648, design PR lost-rob0t/zara#649).

Direction is intentionally Zara-first for product behavior: Zara implements the end-to-end UX/adapter/tests first and should reuse this epic's existing project/source semantics where they already fit. If Zara #653 exposes a genuinely missing domain-neutral seam, the Prolog-RLM follow-up must point to the concrete Zara implementation and RED-first fixtures before generalizing it here.

Do not move Android navigation/product policy upstream and do not make #93 a speculative blocker for the Zara implementation.

### Downstream proving consumer: Zara Android projects `lost-rob0t/zara#653` is now the concrete product proving ground for persistent project contexts and project-scoped chat under the frozen Android design (`lost-rob0t/zara#648`, design PR `lost-rob0t/zara#649`). Direction is intentionally **Zara-first for product behavior**: Zara implements the end-to-end UX/adapter/tests first and should reuse this epic's existing project/source semantics where they already fit. If Zara #653 exposes a genuinely missing domain-neutral seam, the Prolog-RLM follow-up must point to the concrete Zara implementation and RED-first fixtures before generalizing it here. Do not move Android navigation/product policy upstream and do not make #93 a speculative blocker for the Zara implementation.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
nsaspy/prolog-rlm#93
No description provided.