nsaspy/prolog-rlm

Fork 0

[P1] Project Tree-sitter CST into versioned Prolog syntax facts #96

New issue

Closed

opened 2026-08-18 14:14:50 +00:00 by lost-rob0t · 1 comment

lost-rob0t commented

2026-08-18 14:14:50 +00:00

(Migrated from github.com)

Parent: #93
Depends on: #94, #95

Goal

Turn a Tree-sitter parse tree into ordinary inspectable, versioned Prolog syntax observations tied to a Project file and parse generation.

Tree-sitter produces a concrete syntax tree. This issue owns the generic CST/syntax fact layer. It does not pretend that grammar node names already form a language-neutral semantic AST.

Parse generation model

Provide records equivalent in semantics to:

parse(Parse, File).
parse_language(Parse, Language).
parse_backend(Parse, tree_sitter).
parse_grammar(Parse, GrammarRef).
parse_generation(Parse, Generation).
parse_content_hash(Parse, Hash).
parse_status(Parse, Status).

A new source version must produce a distinguishable parse generation. Old generations may remain inspectable but must not silently satisfy queries that request current observations.

Syntax fact model

Provide generic facts equivalent to:

syntax_node(Node, Parse, Type).
syntax_named(Node).
syntax_parent(Node, Parent).
syntax_child(Parent, Index, Child).
syntax_field(Parent, FieldName, Child).
source_span(Node, File, StartByte, EndByte).
source_points(Node, StartPoint, EndPoint).
syntax_error(Node, ErrorKind).

Exact predicates may be optimized/refined. The semantic contract matters more than preserving these names.

Node identity must be stable within the parse generation and must not be a raw native pointer. If deterministic node identities across equivalent reparses are feasible, document the policy; do not promise cross-generation identity without proof.

Projection modes

Support at least two explicit modes if useful:

direct/lazy traversal backed by the native TSTree for temporary analysis;
materialized Prolog syntax observations for project KB persistence/inspection.

Do not accidentally make materialization of every punctuation token mandatory for every workflow. Provide a way to materialize named nodes or selected node classes when that is the appropriate bounded representation.

Source text policy

Do not duplicate every node's full source text into the KB.

Store source spans and obtain bounded text by range when requested. Text extraction must validate that the requested node/span belongs to the corresponding file content generation.

Provide semantics equivalent to:

syntax_node_text(+Node, -Text).

with a bounded/explicit source-generation lookup rather than trusting stale byte ranges against changed file contents.

Error and recovery observations

Represent Tree-sitter recovery information rather than failing the entire file whenever source is incomplete.

Preserve at least:

error-bearing nodes;
missing nodes/tokens where available;
root/tree error status;
byte/source ranges for errors.

This is important for live agent edits where the file may be temporarily invalid.

Bounded indexing

The Project API must be able to apply explicit limits to projection/indexing, including where applicable:

max source bytes;
max materialized nodes;
max nesting/traversal work;
excluded/generated/vendor file policy from #95;
cancellation/timeout integration using existing runtime conventions where work becomes substantial.

Limit exhaustion must return a structured partial/blocked outcome; it must not silently claim a complete AST.

Provenance

Every materialized syntax fact must resolve back to:

Project
File
file content hash/generation
Parse generation
Tree-sitter grammar identity/version
source range

The syntax layer is observational evidence, not trusted executable policy.

Acceptance criteria

A parsed Python fixture can be materialized into Prolog syntax-node/parent/field/range facts.
The same generic projection works for JavaScript and at least one additional grammar without language-specific C projection code.
Node identity never exposes/requires a raw pointer.
Current-vs-stale parse generations are distinguishable.
Named-only projection avoids mandatory punctuation-token explosion.
Source text lookup is range/generation checked.
Malformed/incomplete source yields inspectable error/missing-node observations instead of whole-file parse failure where Tree-sitter recovers.
Project/file/parse/grammar provenance is queryable for each materialized node.
Index limits produce explicit incomplete/blocked outcomes.
Two files containing structurally identical code do not collide in node identity/provenance.
Deterministic tests cover empty files, malformed files, Unicode/byte offsets, deeply nested syntax, named/anonymous nodes, fields, stale generations, and bounded projection.

Non-goals

No cross-language symbol ontology in this issue.
No function/call/import semantics inferred solely from generic node names.
No mandatory full-tree persistence for every Project.
No source execution.

Refs #93 #94 #95

Parent: #93 Depends on: #94, #95 ## Goal Turn a Tree-sitter parse tree into ordinary inspectable, versioned Prolog syntax observations tied to a Project file and parse generation. Tree-sitter produces a concrete syntax tree. This issue owns the **generic CST/syntax fact layer**. It does not pretend that grammar node names already form a language-neutral semantic AST. ## Parse generation model Provide records equivalent in semantics to: ```prolog parse(Parse, File). parse_language(Parse, Language). parse_backend(Parse, tree_sitter). parse_grammar(Parse, GrammarRef). parse_generation(Parse, Generation). parse_content_hash(Parse, Hash). parse_status(Parse, Status). ``` A new source version must produce a distinguishable parse generation. Old generations may remain inspectable but must not silently satisfy queries that request current observations. ## Syntax fact model Provide generic facts equivalent to: ```prolog syntax_node(Node, Parse, Type). syntax_named(Node). syntax_parent(Node, Parent). syntax_child(Parent, Index, Child). syntax_field(Parent, FieldName, Child). source_span(Node, File, StartByte, EndByte). source_points(Node, StartPoint, EndPoint). syntax_error(Node, ErrorKind). ``` Exact predicates may be optimized/refined. The semantic contract matters more than preserving these names. Node identity must be stable **within the parse generation** and must not be a raw native pointer. If deterministic node identities across equivalent reparses are feasible, document the policy; do not promise cross-generation identity without proof. ## Projection modes Support at least two explicit modes if useful: 1. direct/lazy traversal backed by the native `TSTree` for temporary analysis; 2. materialized Prolog syntax observations for project KB persistence/inspection. Do not accidentally make materialization of every punctuation token mandatory for every workflow. Provide a way to materialize named nodes or selected node classes when that is the appropriate bounded representation. ## Source text policy Do not duplicate every node's full source text into the KB. Store source spans and obtain bounded text by range when requested. Text extraction must validate that the requested node/span belongs to the corresponding file content generation. Provide semantics equivalent to: ```prolog syntax_node_text(+Node, -Text). ``` with a bounded/explicit source-generation lookup rather than trusting stale byte ranges against changed file contents. ## Error and recovery observations Represent Tree-sitter recovery information rather than failing the entire file whenever source is incomplete. Preserve at least: - error-bearing nodes; - missing nodes/tokens where available; - root/tree error status; - byte/source ranges for errors. This is important for live agent edits where the file may be temporarily invalid. ## Bounded indexing The Project API must be able to apply explicit limits to projection/indexing, including where applicable: - max source bytes; - max materialized nodes; - max nesting/traversal work; - excluded/generated/vendor file policy from #95; - cancellation/timeout integration using existing runtime conventions where work becomes substantial. Limit exhaustion must return a structured partial/blocked outcome; it must not silently claim a complete AST. ## Provenance Every materialized syntax fact must resolve back to: ```text Project File file content hash/generation Parse generation Tree-sitter grammar identity/version source range ``` The syntax layer is observational evidence, not trusted executable policy. ## Acceptance criteria - [ ] A parsed Python fixture can be materialized into Prolog syntax-node/parent/field/range facts. - [ ] The same generic projection works for JavaScript and at least one additional grammar without language-specific C projection code. - [ ] Node identity never exposes/requires a raw pointer. - [ ] Current-vs-stale parse generations are distinguishable. - [ ] Named-only projection avoids mandatory punctuation-token explosion. - [ ] Source text lookup is range/generation checked. - [ ] Malformed/incomplete source yields inspectable error/missing-node observations instead of whole-file parse failure where Tree-sitter recovers. - [ ] Project/file/parse/grammar provenance is queryable for each materialized node. - [ ] Index limits produce explicit incomplete/blocked outcomes. - [ ] Two files containing structurally identical code do not collide in node identity/provenance. - [ ] Deterministic tests cover empty files, malformed files, Unicode/byte offsets, deeply nested syntax, named/anonymous nodes, fields, stale generations, and bounded projection. ## Non-goals - No cross-language symbol ontology in this issue. - No function/call/import semantics inferred solely from generic node names. - No mandatory full-tree persistence for every Project. - No source execution. Refs #93 #94 #95

nsaspy closed this issue

2026-09-04 14:36:16 +00:00

nsaspy referenced this issue

2026-09-04 14:36:16 +00:00

[P1] Expose Tree-sitter query/capture APIs to Prolog #97

nsaspy referenced this issue

2026-09-04 14:36:32 +00:00

[EPIC] Add first-class Project / SPEC / source-knowledge APIs #93

nsaspy commented

2026-09-04 14:36:33 +00:00

Owner

Closed by PR #342 (rebase-merge).

Merge SHA: 156bfe9b2caffa1b967e77bee51263ee9e85fb50 on main

Exact-head gate evidence (recorded at PR comment): PlUnit 96 suites 1179/1179; deterministic 16/16; deep-experiment 15/15; paid deep-integration 6/6 at z-ai/glm-5.3-flash; demo --json; git diff --check; research-approval 19/19; nix flake check all-pass (native 21/21 + FFI); all required CI green including the pinned paid OpenRouter lane.

Successor slices: #97 (query/capture APIs — handoff posted), then #98, #99 under epic #93.

Closed by PR #342 (rebase-merge). **Merge SHA**: `156bfe9b2caffa1b967e77bee51263ee9e85fb50` on `main` **Exact-head gate evidence** (recorded at [PR comment](https://github.com/lost-rob0t/prolog-rlm/pull/342#issuecomment-5515109326)): PlUnit 96 suites 1179/1179; deterministic 16/16; deep-experiment 15/15; paid deep-integration 6/6 at `z-ai/glm-5.3-flash`; demo --json; git diff --check; research-approval 19/19; nix flake check all-pass (native 21/21 + FFI); all required CI green including the pinned paid OpenRouter lane. Successor slices: #97 (query/capture APIs — handoff posted), then #98, #99 under epic #93.

nsaspy referenced this issue

2026-09-04 14:36:33 +00:00

[P1] Expose Tree-sitter query/capture APIs to Prolog #97

nsaspy referenced this issue

2026-09-04 14:36:34 +00:00

[P1] Expose Tree-sitter query/capture APIs to Prolog #97

nsaspy referenced this issue from a pull request that will close it,

2026-09-04 22:14:00 +00:00

Add versioned project syntax observations #363

nsaspy referenced this issue

2026-09-08 01:29:22 +00:00

[EPIC][expert] Project symbolic knowledge expert: fresh source facts, relations, provenance, and explainable project queries #418