[P1] Add direct SWI-Prolog Tree-sitter C FFI #94

New issue

Closed

opened 2026-08-18 14:13:51 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-18 14:13:51 +00:00

(Migrated from github.com)

Parent: #93

Goal

Add a small owned foreign interface that binds SWI-Prolog directly to the Tree-sitter C runtime. No Python, Node, Nim, subprocess, or parser service sits between Prolog and libtree-sitter.

This issue owns mechanical parser access and native resource lifetime, not language-specific semantic extraction.

Required architecture

SWI-Prolog
   |
   | SWI FLI
   v
small C binding
   |
   v
libtree-sitter
   |
   v
TSLanguage / TSParser / TSTree / TSNode

Keep the C layer deliberately stupid. Tree-sitter mechanics belong here; project facts, language adapters, inference, SPEC, and VERIFY stay in Prolog.

Generic grammar contract

The FFI must operate on generic TSLanguage * values rather than compile one Prolog predicate per programming language.

Support loading a compatible generated grammar through a generic mechanism equivalent to:

ts_language_load(+LibraryPath, +EntrySymbol, -Language).
ts_language_abi(+Language, -Abi).
ts_language_compatible(+Language, -Outcome).

A grammar such as Python may export tree_sitter_python; another grammar exports its own symbol. The generic FFI must not require a C source edit to add each grammar.

Dynamic loading must fail structurally for missing libraries, missing symbols, or incompatible Tree-sitter language ABI.

Resource model

Use SWI foreign blobs or an equivalent owned foreign-resource representation for long-lived native objects.

At minimum define safe wrappers/lifetimes for:

TSLanguage / loaded grammar handle;
TSParser;
TSTree;
query resources later consumed by the query issue.

Do not expose naked integer addresses as the supported public object representation.

TSNode is a value referring into a tree. Its wrapper must retain/validate the owning tree lifetime so a node cannot remain apparently usable after its tree is freed.

Finalizers/explicit cleanup must be deterministic and idempotent where applicable.

Initial public surface

Provide repository-convention predicates equivalent to:

ts_parser_create(-Parser).
ts_parser_set_language(+Parser, +Language, -Outcome).
ts_parse_string(+Parser, +Source, -Tree).

ts_tree_root(+Tree, -Node).
ts_node_type(+Node, -Type).
ts_node_named(+Node).
ts_node_has_error(+Node).
ts_node_start_byte(+Node, -Byte).
ts_node_end_byte(+Node, -Byte).
ts_node_start_point(+Node, -Point).
ts_node_end_point(+Node, -Point).
ts_node_child_count(+Node, -Count).
ts_node_child(+Node, +Index, -Child).
ts_node_named_child(+Node, +Index, -Child).
ts_node_field(+Node, +FieldName, -Child).

Exact names/arity should follow repository conventions. Avoid a giant binding surface in the first slice.

Error semantics

Return/throw repository-consistent structured errors for at least:

invalid/stale foreign handle;
parser without language;
incompatible grammar ABI;
dynamic library load failure;
grammar symbol lookup failure;
allocation failure where observable;
invalid node access/index;
invalid source encoding/input where Tree-sitter requires a defined policy.

Do not leak opaque native stderr as the API contract.

Threading / reentrancy

Document ownership and thread-safety assumptions for parser/tree objects under SWI's threaded runtime. Do not share a mutable TSParser across concurrent operations unless the wrapper deliberately serializes or proves that use safe.

The first implementation may require one parser per logical parse worker/context.

Build integration

add the C source/build integration required by the current SWI project layout;
pin/document the supported Tree-sitter runtime/API version policy;
keep deterministic CI able to build the binding from a clean checkout/environment;
do not vendor hundreds of language grammars into this issue.

Acceptance criteria

SWI loads the foreign library from a clean project setup.
Prolog can create a parser, assign a grammar, parse source, obtain the root node, and traverse children.
The same FFI parses at least three different grammars without language-specific C wrappers.
Missing grammar library/symbol fails structurally.
Unsupported grammar ABI fails before parsing rather than crashing.
Node type/range/child/field access works through Prolog predicates.
Tree/parser/blob finalization is deterministic and double cleanup is safe.
A retained node cannot become a silent use-after-free when its tree would otherwise be reclaimed.
Concurrent fixture coverage proves the documented parser ownership model.
Malformed/incomplete source still returns a Tree-sitter tree and error-bearing nodes where the grammar/runtime supports recovery.
No Python, Node, Nim, or subprocess is required for parsing.

Non-goals

No project/file ontology in C.
No language detection in C.
No normalized symbol/call/import extraction in C.
No arbitrary repository code execution.
No Tree-sitter query-pack semantics in this issue beyond whatever primitive handle support is needed for later extension.

Refs #93

Parent: #93 ## Goal Add a small owned foreign interface that binds SWI-Prolog directly to the Tree-sitter C runtime. No Python, Node, Nim, subprocess, or parser service sits between Prolog and `libtree-sitter`. This issue owns **mechanical parser access and native resource lifetime**, not language-specific semantic extraction. ## Required architecture ```text SWI-Prolog | | SWI FLI v small C binding | v libtree-sitter | v TSLanguage / TSParser / TSTree / TSNode ``` Keep the C layer deliberately stupid. Tree-sitter mechanics belong here; project facts, language adapters, inference, SPEC, and VERIFY stay in Prolog. ## Generic grammar contract The FFI must operate on generic `TSLanguage *` values rather than compile one Prolog predicate per programming language. Support loading a compatible generated grammar through a generic mechanism equivalent to: ```prolog ts_language_load(+LibraryPath, +EntrySymbol, -Language). ts_language_abi(+Language, -Abi). ts_language_compatible(+Language, -Outcome). ``` A grammar such as Python may export `tree_sitter_python`; another grammar exports its own symbol. The generic FFI must not require a C source edit to add each grammar. Dynamic loading must fail structurally for missing libraries, missing symbols, or incompatible Tree-sitter language ABI. ## Resource model Use SWI foreign blobs or an equivalent owned foreign-resource representation for long-lived native objects. At minimum define safe wrappers/lifetimes for: - `TSLanguage` / loaded grammar handle; - `TSParser`; - `TSTree`; - query resources later consumed by the query issue. Do not expose naked integer addresses as the supported public object representation. `TSNode` is a value referring into a tree. Its wrapper must retain/validate the owning tree lifetime so a node cannot remain apparently usable after its tree is freed. Finalizers/explicit cleanup must be deterministic and idempotent where applicable. ## Initial public surface Provide repository-convention predicates equivalent to: ```prolog ts_parser_create(-Parser). ts_parser_set_language(+Parser, +Language, -Outcome). ts_parse_string(+Parser, +Source, -Tree). ts_tree_root(+Tree, -Node). ts_node_type(+Node, -Type). ts_node_named(+Node). ts_node_has_error(+Node). ts_node_start_byte(+Node, -Byte). ts_node_end_byte(+Node, -Byte). ts_node_start_point(+Node, -Point). ts_node_end_point(+Node, -Point). ts_node_child_count(+Node, -Count). ts_node_child(+Node, +Index, -Child). ts_node_named_child(+Node, +Index, -Child). ts_node_field(+Node, +FieldName, -Child). ``` Exact names/arity should follow repository conventions. Avoid a giant binding surface in the first slice. ## Error semantics Return/throw repository-consistent structured errors for at least: - invalid/stale foreign handle; - parser without language; - incompatible grammar ABI; - dynamic library load failure; - grammar symbol lookup failure; - allocation failure where observable; - invalid node access/index; - invalid source encoding/input where Tree-sitter requires a defined policy. Do not leak opaque native stderr as the API contract. ## Threading / reentrancy Document ownership and thread-safety assumptions for parser/tree objects under SWI's threaded runtime. Do not share a mutable `TSParser` across concurrent operations unless the wrapper deliberately serializes or proves that use safe. The first implementation may require one parser per logical parse worker/context. ## Build integration - add the C source/build integration required by the current SWI project layout; - pin/document the supported Tree-sitter runtime/API version policy; - keep deterministic CI able to build the binding from a clean checkout/environment; - do not vendor hundreds of language grammars into this issue. ## Acceptance criteria - [ ] SWI loads the foreign library from a clean project setup. - [ ] Prolog can create a parser, assign a grammar, parse source, obtain the root node, and traverse children. - [ ] The same FFI parses at least three different grammars without language-specific C wrappers. - [ ] Missing grammar library/symbol fails structurally. - [ ] Unsupported grammar ABI fails before parsing rather than crashing. - [ ] Node type/range/child/field access works through Prolog predicates. - [ ] Tree/parser/blob finalization is deterministic and double cleanup is safe. - [ ] A retained node cannot become a silent use-after-free when its tree would otherwise be reclaimed. - [ ] Concurrent fixture coverage proves the documented parser ownership model. - [ ] Malformed/incomplete source still returns a Tree-sitter tree and error-bearing nodes where the grammar/runtime supports recovery. - [ ] No Python, Node, Nim, or subprocess is required for parsing. ## Non-goals - No project/file ontology in C. - No language detection in C. - No normalized symbol/call/import extraction in C. - No arbitrary repository code execution. - No Tree-sitter query-pack semantics in this issue beyond whatever primitive handle support is needed for later extension. Refs #93