[P1] Normalize Tree-sitter captures into symbols, references, and source relations #438

New issue

Closed

opened 2026-09-10 21:18:12 +00:00 by nsaspy · 1 comment

nsaspy commented

2026-09-10 21:18:12 +00:00

Owner

Parent: #93
Depends on: #95, #96, #97

Goal

Build the first semantic source-knowledge layer above raw Tree-sitter syntax/captures: definitions, symbols, references, containment, calls, imports, exports, and selected language-specific relations represented as ordinary Prolog facts with provenance.

This is where Tree-sitter syntax becomes useful to SPEC/VERIFY and project reasoning.

Layering rule

Keep three levels distinct:

Tree-sitter CST
    -> language-specific query captures
    -> normalized common facts + richer language-specific facts

Do not infer semantics directly in the C FFI and do not pretend every language has one identical AST.

Common fact vocabulary

Define a deliberately small common model equivalent in semantics to:

symbol(Symbol, Kind, Name).
symbol_defined_in(Symbol, File).
symbol_definition(Symbol, Node).

reference(Reference, Kind, SymbolOrName, Node).
contains(Parent, Child).

calls(Caller, CalleeOrReference).
imports(Source, Target).
exports(Source, Symbol).

Exact terms may change after fixture-driven design. Add common concepts only when they have defensible cross-language meaning.

Allow language-specific facts alongside the common model, for example predicate arity, macro definitions, implementation/interface relations, decorators/attributes, module forms, or language-native binding details.

Identity and unresolved references

A definition may yield a stable source symbol identity within a project/source generation. A reference may initially resolve only to a textual/structural target.

Do not fabricate successful symbol resolution when static information is insufficient.

Represent at least the semantic distinction between:

resolved reference;
unresolved reference;
ambiguous reference;
external/unknown target;
syntactic call/import observation whose semantic target is not yet resolved.

Later language servers/type checkers/indexes may enrich these facts without invalidating the basic Tree-sitter observation.

Language adapters

Implement initial query+normalization adapters for at least three languages with meaningfully different syntax families, including Python and JavaScript/TypeScript plus one additional language selected from current project priorities.

Adapters live in Prolog/query-pack code, not language-specific C wrappers.

Each adapter should declare supported extraction capabilities, e.g.:

language_analysis_capability(python, definitions).
language_analysis_capability(python, calls).
language_analysis_capability(python, imports).

Unsupported extraction must be distinguishable from an analyzed file containing zero matches.

Provenance

Every semantic fact must be traceable to its source observations:

Project/File;
content hash/generation;
parse generation;
grammar identity/version;
query/extractor identity/version;
one or more syntax-node/source spans supporting the fact.

Derived Prolog relations must remain distinguishable from directly extracted observations.

Cross-file/project resolution

Provide a bounded initial resolver where semantics are defensible, but keep extraction separate from global resolution.

For example:

extracted import target text
  !=
proved module identity

Project-level symbol/reference resolution can use file/language/module facts and return ambiguous/unknown explicitly.

Do not require a full compiler/typechecker for the first slice.

Acceptance criteria

Python fixtures produce normalized function/class definitions, references/calls, and imports with source provenance.
JavaScript/TypeScript fixtures produce normalized definitions/references/imports through the same common predicates where semantics align.
At least one third language proves the adapter API is not Python/JS-specific.
No language-specific C extraction code is required.
Resolved, unresolved, ambiguous, external, and unsupported-analysis states remain distinguishable.
A file with zero definitions is distinguishable from a language adapter that does not support definition extraction.
Common facts and language-specific richer facts can coexist without one flattening the other.
Every extracted semantic fact can report source/grammar/query provenance.
Derived cross-file relations are marked as derived rather than masquerading as direct syntax observations.
Deterministic fixtures cover nested definitions, aliases, anonymous constructs, import variants, shadowed names, unresolved calls, duplicate names across files, and ambiguous targets.

Non-goals

No claim of compiler-grade name/type resolution for every language.
No giant universal AST schema.
No LSP dependency in this issue.
No execution of indexed source.

Refs #93 #95 #96 #97

Parent: #93 Depends on: #95, #96, #97 ## Goal Build the first semantic source-knowledge layer above raw Tree-sitter syntax/captures: definitions, symbols, references, containment, calls, imports, exports, and selected language-specific relations represented as ordinary Prolog facts with provenance. This is where Tree-sitter syntax becomes useful to SPEC/VERIFY and project reasoning. ## Layering rule Keep three levels distinct: ```text Tree-sitter CST -> language-specific query captures -> normalized common facts + richer language-specific facts ``` Do not infer semantics directly in the C FFI and do not pretend every language has one identical AST. ## Common fact vocabulary Define a deliberately small common model equivalent in semantics to: ```prolog symbol(Symbol, Kind, Name). symbol_defined_in(Symbol, File). symbol_definition(Symbol, Node). reference(Reference, Kind, SymbolOrName, Node). contains(Parent, Child). calls(Caller, CalleeOrReference). imports(Source, Target). exports(Source, Symbol). ``` Exact terms may change after fixture-driven design. Add common concepts only when they have defensible cross-language meaning. Allow language-specific facts alongside the common model, for example predicate arity, macro definitions, implementation/interface relations, decorators/attributes, module forms, or language-native binding details. ## Identity and unresolved references A definition may yield a stable source symbol identity within a project/source generation. A reference may initially resolve only to a textual/structural target. Do not fabricate successful symbol resolution when static information is insufficient. Represent at least the semantic distinction between: - resolved reference; - unresolved reference; - ambiguous reference; - external/unknown target; - syntactic call/import observation whose semantic target is not yet resolved. Later language servers/type checkers/indexes may enrich these facts without invalidating the basic Tree-sitter observation. ## Language adapters Implement initial query+normalization adapters for at least three languages with meaningfully different syntax families, including Python and JavaScript/TypeScript plus one additional language selected from current project priorities. Adapters live in Prolog/query-pack code, not language-specific C wrappers. Each adapter should declare supported extraction capabilities, e.g.: ```prolog language_analysis_capability(python, definitions). language_analysis_capability(python, calls). language_analysis_capability(python, imports). ``` Unsupported extraction must be distinguishable from an analyzed file containing zero matches. ## Provenance Every semantic fact must be traceable to its source observations: - Project/File; - content hash/generation; - parse generation; - grammar identity/version; - query/extractor identity/version; - one or more syntax-node/source spans supporting the fact. Derived Prolog relations must remain distinguishable from directly extracted observations. ## Cross-file/project resolution Provide a bounded initial resolver where semantics are defensible, but keep extraction separate from global resolution. For example: ```text extracted import target text != proved module identity ``` Project-level symbol/reference resolution can use file/language/module facts and return ambiguous/unknown explicitly. Do not require a full compiler/typechecker for the first slice. ## Acceptance criteria - [ ] Python fixtures produce normalized function/class definitions, references/calls, and imports with source provenance. - [ ] JavaScript/TypeScript fixtures produce normalized definitions/references/imports through the same common predicates where semantics align. - [ ] At least one third language proves the adapter API is not Python/JS-specific. - [ ] No language-specific C extraction code is required. - [ ] Resolved, unresolved, ambiguous, external, and unsupported-analysis states remain distinguishable. - [ ] A file with zero definitions is distinguishable from a language adapter that does not support definition extraction. - [ ] Common facts and language-specific richer facts can coexist without one flattening the other. - [ ] Every extracted semantic fact can report source/grammar/query provenance. - [ ] Derived cross-file relations are marked as derived rather than masquerading as direct syntax observations. - [ ] Deterministic fixtures cover nested definitions, aliases, anonymous constructs, import variants, shadowed names, unresolved calls, duplicate names across files, and ambiguous targets. ## Non-goals - No claim of compiler-grade name/type resolution for every language. - No giant universal AST schema. - No LSP dependency in this issue. - No execution of indexed source. Refs #93 #95 #96 #97

nsaspy commented

2026-09-10 21:23:35 +00:00

Author

Owner

Duplicate of #98 (pre-existing Forgejo mirror with GitHub number parity). Closing this accidental duplicate created by today's open-state sync; #98 stays canonical on Forgejo.

nsaspy closed this issue

2026-09-10 21:23:35 +00:00