[P1] Add Project file/language facts and Tree-sitter grammar registry #95

New issue

Closed

opened 2026-08-18 14:14:18 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-18 14:14:18 +00:00

(Migrated from github.com)

Parent: #93
Depends on: #94

Goal

Define the Prolog-side mapping from Project -> File -> Language -> parser backend -> Tree-sitter grammar so arbitrary compatible grammars can be registered without modifying the C FFI.

This issue owns the declarative Project/source registry, not CST traversal or semantic extraction.

Required Project/file facts

Provide normalized facts/API equivalent in semantics to:

project(Project, Meta).
project_file(Project, File).

file(File, Path).
file_language(File, Language).
file_hash(File, Hash).
file_generation(File, Generation).

language(Language).
language_parser(Language, Backend).

A file must have an identity distinct from its path so later facts can refer to one versioned source artifact without encoding everything into one compound term.

Support at least:

known language;
unknown language;
ambiguous language evidence;
explicit host override;
excluded/vendor/generated metadata;
embedded/injected language regions later without changing the base file identity model.

Tree-sitter grammar registry

Provide a Prolog registry equivalent to:

tree_sitter_grammar(
    Language,
    grammar{
        library: Library,
        symbol: EntrySymbol,
        abi: Abi,
        version: Version,
        provenance: Provenance
    }
).

Exact representation may differ. The registry must remain declarative and inspectable.

Required operations should include semantics equivalent to:

ts_grammar_register(+Language, +GrammarSpec, -Outcome).
ts_grammar_unregister(+Language, -Outcome).
ts_grammar(+Language, -Grammar).
ts_grammars(-Grammars).
parser_for_file(+File, -Backend).
grammar_for_file(+File, -Grammar).

Loading/registering a grammar does not grant filesystem/process/network/tool authority.

Language detection

Represent detection evidence explicitly rather than hiding it in an irreversible imperative guess.

Support evidence forms equivalent to:

extension_language('.py', python).
language_evidence(File, python, extension('.py'), Confidence).
language_evidence(File, shell, shebang('/usr/bin/env bash'), Confidence).

The first implementation may use extensions/shebangs plus explicit overrides. Preserve enough structure so stronger detectors can be added without rewriting the file model.

Define deterministic resolution for:

no evidence -> unknown;
one clear candidate;
conflicting candidates;
explicit trusted host override;
unsupported language/known language with no installed grammar.

Do not silently treat unknown and unsupported as the same state.

Parser backend selection

Tree-sitter is one backend, not the ontology itself.

Support facts equivalent to:

language_parser(python, tree_sitter).
language_parser(javascript, tree_sitter).
language_parser(nim, tree_sitter).
language_parser(common_lisp, tree_sitter).
language_parser(prolog, swi_native).

Do not route authoritative Prolog semantic parsing through Tree-sitter merely for symmetry. A later SWI-native analyzer may provide stronger Prolog facts while Tree-sitter remains optionally available for editor-style syntax observations.

Version/provenance

Grammar registration and parse selection must expose enough metadata to later prove which parser produced an observation:

backend identity;
grammar/library identity;
grammar ABI/version where available;
host/provider origin;
file content hash/generation.

Do not make raw filesystem library paths the sole durable grammar identity.

Acceptance criteria

Two projects can register/index files without cross-project leakage.
File identity is distinct from path and can carry hash/generation metadata.
Language detection distinguishes known, unknown, ambiguous, unsupported, and explicit override states.
Tree-sitter grammars are registered through data, not hard-coded language-specific C predicates.
At least Python, JavaScript, and one additional grammar can be selected through the same registry mechanism.
Adding a fourth compatible grammar requires no FFI source-code change.
Grammar lookup validates #94 compatibility before use.
parser_for_file/2 can select Tree-sitter or another backend such as swi_native.
Grammar registration/loading changes no capabilities or authority.
Parser/grammar provenance is available to the later parse-generation model.
Deterministic tests cover extension detection, host override, ambiguity, missing grammar, incompatible grammar, multiple projects, and registry unload/reload.

Non-goals

No syntax-node persistence yet; see the AST/CST issue.
No Tree-sitter query packs yet.
No symbol/reference extraction yet.
No auto-download/install of arbitrary grammars from model input.
No coding-agent UI.

Refs #93 #94 #75

Parent: #93 Depends on: #94 ## Goal Define the Prolog-side mapping from Project -> File -> Language -> parser backend -> Tree-sitter grammar so arbitrary compatible grammars can be registered without modifying the C FFI. This issue owns the **declarative Project/source registry**, not CST traversal or semantic extraction. ## Required Project/file facts Provide normalized facts/API equivalent in semantics to: ```prolog project(Project, Meta). project_file(Project, File). file(File, Path). file_language(File, Language). file_hash(File, Hash). file_generation(File, Generation). language(Language). language_parser(Language, Backend). ``` A file must have an identity distinct from its path so later facts can refer to one versioned source artifact without encoding everything into one compound term. Support at least: - known language; - unknown language; - ambiguous language evidence; - explicit host override; - excluded/vendor/generated metadata; - embedded/injected language regions later without changing the base file identity model. ## Tree-sitter grammar registry Provide a Prolog registry equivalent to: ```prolog tree_sitter_grammar( Language, grammar{ library: Library, symbol: EntrySymbol, abi: Abi, version: Version, provenance: Provenance } ). ``` Exact representation may differ. The registry must remain declarative and inspectable. Required operations should include semantics equivalent to: ```prolog ts_grammar_register(+Language, +GrammarSpec, -Outcome). ts_grammar_unregister(+Language, -Outcome). ts_grammar(+Language, -Grammar). ts_grammars(-Grammars). parser_for_file(+File, -Backend). grammar_for_file(+File, -Grammar). ``` Loading/registering a grammar does not grant filesystem/process/network/tool authority. ## Language detection Represent detection evidence explicitly rather than hiding it in an irreversible imperative guess. Support evidence forms equivalent to: ```prolog extension_language('.py', python). language_evidence(File, python, extension('.py'), Confidence). language_evidence(File, shell, shebang('/usr/bin/env bash'), Confidence). ``` The first implementation may use extensions/shebangs plus explicit overrides. Preserve enough structure so stronger detectors can be added without rewriting the file model. Define deterministic resolution for: - no evidence -> unknown; - one clear candidate; - conflicting candidates; - explicit trusted host override; - unsupported language/known language with no installed grammar. Do not silently treat `unknown` and `unsupported` as the same state. ## Parser backend selection Tree-sitter is one backend, not the ontology itself. Support facts equivalent to: ```prolog language_parser(python, tree_sitter). language_parser(javascript, tree_sitter). language_parser(nim, tree_sitter). language_parser(common_lisp, tree_sitter). language_parser(prolog, swi_native). ``` Do not route authoritative Prolog semantic parsing through Tree-sitter merely for symmetry. A later SWI-native analyzer may provide stronger Prolog facts while Tree-sitter remains optionally available for editor-style syntax observations. ## Version/provenance Grammar registration and parse selection must expose enough metadata to later prove which parser produced an observation: - backend identity; - grammar/library identity; - grammar ABI/version where available; - host/provider origin; - file content hash/generation. Do not make raw filesystem library paths the sole durable grammar identity. ## Acceptance criteria - [ ] Two projects can register/index files without cross-project leakage. - [ ] File identity is distinct from path and can carry hash/generation metadata. - [ ] Language detection distinguishes known, unknown, ambiguous, unsupported, and explicit override states. - [ ] Tree-sitter grammars are registered through data, not hard-coded language-specific C predicates. - [ ] At least Python, JavaScript, and one additional grammar can be selected through the same registry mechanism. - [ ] Adding a fourth compatible grammar requires no FFI source-code change. - [ ] Grammar lookup validates #94 compatibility before use. - [ ] `parser_for_file/2` can select Tree-sitter or another backend such as `swi_native`. - [ ] Grammar registration/loading changes no capabilities or authority. - [ ] Parser/grammar provenance is available to the later parse-generation model. - [ ] Deterministic tests cover extension detection, host override, ambiguity, missing grammar, incompatible grammar, multiple projects, and registry unload/reload. ## Non-goals - No syntax-node persistence yet; see the AST/CST issue. - No Tree-sitter query packs yet. - No symbol/reference extraction yet. - No auto-download/install of arbitrary grammars from model input. - No coding-agent UI. Refs #93 #94 #75