feat: implement #98 semantic project knowledge normalization #458

Merged

nsaspy merged 5 commits from rage/98-semantic-project-knowledge into main 2026-09-11 05:33:42 +00:00

Conversation 2 Commits 5 Files changed 26 +3363 -24

nsaspy commented 2026-09-11 00:51:07 +00:00

Owner

Copy link

#98 — Normalize captures into semantic project facts

Runtime invariant implemented

The #97 grouped capture layer now has a semantic normalization layer:
rlm_project_semantic turns the current extraction's grouped matches into
closed definitions/references/calls/imports/exports observations with exact
provenance (project, file hash/generation, content hash, parse identity and
generation, grammar ref, pack identity/sha/purpose, adapter identity, syntax
node, byte/point spans). Extraction and resolution stay separate operations
with explicit resolved/unresolved/ambiguous/external states; duplicate names
stay distinct project_symbol/4 identities and surface as ambiguous.
Derived containment, bounded cycle-safe call reachability, and file
dependency relations are computed on demand and expose their supporting
observation ids. The public surface emits the #288-compatible
symbol_index{project, kinds, definitions, coherence} over still-current
knowledge only, consumable by plan_graph_resolve_symbol/3 directly.

Freshness reuses #97 currentness: stale extractions cannot answer current
queries, pack replacement invalidates dependent semantic state, and
currentness is never journaled — the #97 extraction journal stays
authoritative, so fresh-process restarts cannot resurrect superseded
semantic facts as current (proven by kill/restart fixtures, including the
stale path).

Zero-matches and unsupported-capability states remain distinct via
language_analysis_capability/2. Common Lisp proves the adapter surface is
not Python/JS-shaped (keyword-disambiguated kinds, S-expression calls, and
imports/exports declared unsupported). No model calls, no scheduler, no
consult/1, no new authority; #377/#380 are deliberately untouched.

Non-goals

• No incremental reparsing/coherent snapshot work (#99 owns it).
• No expert registry or project_knowledge_expert (#377/#380).
• No Prolog/xref extraction backend (documented future backend).
• No Chroma/vector retrieval (#381).

Tests / evidence

• New deterministic suite test/rlm_project_semantic_test.pl covering all
  #98 acceptance points plus two fresh-process restart fixtures, registered
  through the native tree-sitter gate: 84/84 green (nix).
• Latent #97 FFI defect fixed (multi-capture matches silently dropped) with
  a regression test in rlm_tree_sitter_query_test.
• Full local gate green via prolog-verify: check_runtime, load_all,
  deterministic corpus, native suite, benchmark deterministic 16/16, CLI
  demo smoke, git diff --check.

Follow-up scope

#99 minimum coherent snapshot/freshness work -> #377 expert registry ->
#380 project knowledge expert -> #381 retrieval expert.

#98 — Normalize captures into semantic project facts ## Runtime invariant implemented The #97 grouped capture layer now has a semantic normalization layer: `rlm_project_semantic` turns the current extraction's grouped matches into closed definitions/references/calls/imports/exports observations with exact provenance (project, file hash/generation, content hash, parse identity and generation, grammar ref, pack identity/sha/purpose, adapter identity, syntax node, byte/point spans). Extraction and resolution stay separate operations with explicit resolved/unresolved/ambiguous/external states; duplicate names stay distinct `project_symbol/4` identities and surface as `ambiguous`. Derived containment, bounded cycle-safe call reachability, and file dependency relations are computed on demand and expose their supporting observation ids. The public surface emits the #288-compatible `symbol_index{project, kinds, definitions, coherence}` over still-current knowledge only, consumable by `plan_graph_resolve_symbol/3` directly. Freshness reuses #97 currentness: stale extractions cannot answer current queries, pack replacement invalidates dependent semantic state, and currentness is never journaled — the #97 extraction journal stays authoritative, so fresh-process restarts cannot resurrect superseded semantic facts as current (proven by kill/restart fixtures, including the stale path). Zero-matches and unsupported-capability states remain distinct via `language_analysis_capability/2`. Common Lisp proves the adapter surface is not Python/JS-shaped (keyword-disambiguated kinds, S-expression calls, and imports/exports declared `unsupported`). No model calls, no scheduler, no `consult/1`, no new authority; #377/#380 are deliberately untouched. ## Non-goals - No incremental reparsing/coherent snapshot work (#99 owns it). - No expert registry or `project_knowledge_expert` (#377/#380). - No Prolog/xref extraction backend (documented future backend). - No Chroma/vector retrieval (#381). ## Tests / evidence - New deterministic suite `test/rlm_project_semantic_test.pl` covering all #98 acceptance points plus two fresh-process restart fixtures, registered through the native tree-sitter gate: 84/84 green (nix). - Latent #97 FFI defect fixed (multi-capture matches silently dropped) with a regression test in `rlm_tree_sitter_query_test`. - Full local gate green via prolog-verify: check_runtime, load_all, deterministic corpus, native suite, benchmark deterministic 16/16, CLI demo smoke, git diff --check. ## Follow-up scope #99 minimum coherent snapshot/freshness work -> #377 expert registry -> #380 project knowledge expert -> #381 retrieval expert.

nsaspy added 1 commit 2026-09-11 00:51:07 +00:00

feat: implement #98 semantic project knowledge normalization ... 42a7cf64ed

Realization slice for issue #98 (parent #93): normalize grouped #97 query
captures into closed semantic project facts.

- prolog/rlm_project_semantic.pl: project_semantic_normalize (canonical
  async execute -> Future -> sync await) turns the current extraction's
  grouped matches into definitions/references/calls/imports/exports with
  full provenance (project, file hash/generation, content hash, parse
  identity/generation, grammar ref, pack identity/sha/purpose, adapter
  identity, syntax node, byte/point spans); knowledge_state current/stale/
  none; bounded separate resolution (resolved/unresolved/ambiguous/
  external); derived containment, cycle-safe bounded call reachability,
  and file-dependency relations expose their supporting observation ids;
  symbol_index emits the #288-compatible symbol_index{kinds, definitions,
  coherence} over still-current knowledge only.
- prolog/rlm_project_semantic_pack.pl: inert standard adapter packs for
  Python, JavaScript, and Common Lisp with the @relation[.kind] +
  @relation.role capture protocol and language_analysis_capability/2;
  zero matches vs unsupported capabilities stay distinct.
- prolog/rlm_project_semantic_persist.pl: .kb/project-semantic journal
  (sync(close)); currentness is never journaled - the #97 extraction
  journal stays authoritative, so restarts cannot resurrect superseded
  semantic facts as current; stale fencing across restarts is proven by
  fresh-process fixtures.
- c/rlm_tree_sitter_query.c: fix latent #97 FFI defect - unify_match reused
  the capture argument slot across iterations, silently dropping every
  multi-capture match; regression test added (semantic adapters require
  grouped multi-capture matches).
- rlm_plan_graph: resolve_supported reads the index via get_dict so the #98
  index may carry coherence/provenance keys; project_query gains read
  accessors (extraction_record, active_packs) for normalization; registry
  destroy clears semantic state.
- Tests: deterministic TDD suite for all #98 acceptance points plus two
  fresh-process restart fixtures; registered through the native tree-sitter
  gate (84/84).
- Docs: docs/project-semantic.md; layering notes in project-query.md;
  README section; roadmap/TODO reconciliation. Durable KB entries under
  .prolog/kb/.

Full local gate green via prolog-verify (check_runtime, load_all,
deterministic corpus, native suite, benchmark deterministic 16/16, CLI
demo smoke, git diff --check).

Closes #98

nsaspy added 1 commit 2026-09-11 01:20:33 +00:00

ci: re-trigger PR gates after paid-lane variable provisioning d868c11087

nsaspy added 1 commit 2026-09-11 04:57:43 +00:00

fix: #98 review — atomic semantic journal, complete index coherence, incomplete resolution, hydration, host adapters, true project isolation ... fa41aa355e

Operator-review fixes for #98:

- persist: one atomic semantic_publication/3 row per extraction publication
  (crash cannot half-journal an extraction); publish/snapshot self-attach the
  target journal so multi-project processes never cross-write.  The bare
  persistency query form was silently failing as an assert on an empty
  journal (root cause of the silent async goal_failed cascade).
- hydrate_rows consumes the publication-pair row shape and observation ids
  restart at 1 per publication, so re-normalization re-journals an identical
  row and hydration replay is idempotent.
- host-registered adapters (e.g. typescript) shadow the standard pack table
  via project_semantic_adapter_register/3.
- every public read (knowledge_state/resolve/symbol_index/file_dependencies/
  definitions) best-effort hydrates the project journal; registry_clear keeps
  the shared attachment while other registries use it.
- resolve reports ok(incomplete(unresolved|ambiguous, missing(Files))) when
  registered project files lack current semantic knowledge; symbol_index
  coherence is partial(stale(Files), unindexed(Files)) and complete only when
  both are empty.

Tests: true two-project isolation, shadowing, partial-index-never-complete,
ambiguous-resolve-reports-missing, TS host-adapter e2e; restart phase2 proves
knowledge_state self-hydrates before any other read.

Evidence: native tree-sitter suite 88/88; full prolog-verify gate pass
(check_runtime, load_all, run_tests 1235, benchmark deterministic, CLI demo,
git diff --check).

nsaspy commented 2026-09-11 04:58:19 +00:00

Author

Owner

Copy link

Review-fix push (fa41aa3) — 6 operator-review findings addressed, root-caused from the 21-failure regression:

Runtime fixes

1. persist: one atomic semantic_publication/3 row per extraction publication; publish/snapshot self-attach the target journal (multi-project processes never cross-write). The silent goal_failed cascade was a bare persistency query used as an assert — plain failure, no exception, invisible through the async facade.
2. hydrate_rows/2 consumes the publication-pair row shape; observation ids restart at 1 per publication, so re-normalization re-journals an identical row and hydration replay is idempotent (fixes duplicate project_symbol facts across re-publication/restart).
3. Host-registered adapters (typescript) shadow the standard pack table via project_semantic_adapter_register/3 — capabilities alone no longer overstate the surface.
4. Every public read (knowledge_state/resolve/symbol_index/file_dependencies/definitions) best-effort hydrates; registry_clear keeps the shared journal attachment while other registries use it.
5. resolve returns ok(incomplete(unresolved|ambiguous, missing(Files))) when registered project files lack current semantic knowledge; symbol_index coherence is partial(stale(Files), unindexed(Files)), complete only when both are empty.

Test corrections (runtime was right, tests were wrong): registry_clear now expects current after clear (public reads self-hydrate, matching the phase2 self-hydration proof); subset-dict pattern replaced with Index2.coherence == complete; ambiguous-resolve fixture now carries two same-name definitions so ambiguity is genuine (2 symbols) with missing files reported.

Evidence: native tree-sitter suite 88/88; full prolog-verify gate pass (check_runtime, load_all, run_tests 1235, benchmark deterministic, CLI demo, git diff --check). rlm_async untouched.

Review-fix push (fa41aa3) — 6 operator-review findings addressed, root-caused from the 21-failure regression: **Runtime fixes** 1. persist: one atomic `semantic_publication/3` row per extraction publication; publish/snapshot self-attach the target journal (multi-project processes never cross-write). The silent `goal_failed` cascade was a bare persistency *query* used as an assert — plain failure, no exception, invisible through the async facade. 2. `hydrate_rows/2` consumes the publication-pair row shape; observation ids restart at 1 per publication, so re-normalization re-journals an identical row and hydration replay is idempotent (fixes duplicate `project_symbol` facts across re-publication/restart). 3. Host-registered adapters (typescript) shadow the standard pack table via `project_semantic_adapter_register/3` — capabilities alone no longer overstate the surface. 4. Every public read (`knowledge_state`/`resolve`/`symbol_index`/`file_dependencies`/`definitions`) best-effort hydrates; `registry_clear` keeps the shared journal attachment while other registries use it. 5. `resolve` returns `ok(incomplete(unresolved|ambiguous, missing(Files)))` when registered project files lack current semantic knowledge; symbol_index coherence is `partial(stale(Files), unindexed(Files))`, complete only when both are empty. **Test corrections** (runtime was right, tests were wrong): registry_clear now expects `current` after clear (public reads self-hydrate, matching the phase2 self-hydration proof); subset-dict pattern replaced with `Index2.coherence == complete`; ambiguous-resolve fixture now carries two same-name definitions so ambiguity is genuine (2 symbols) with missing files reported. **Evidence**: native tree-sitter suite 88/88; full prolog-verify gate pass (check_runtime, load_all, run_tests 1235, benchmark deterministic, CLI demo, git diff --check). rlm_async untouched.

nsaspy added 1 commit 2026-09-11 05:13:06 +00:00

ci: re-run PR gates after environmental nix-flake runner failure (flake check/build/pack-load/develop-load all pass locally) fbdb276a4d

nsaspy added 1 commit 2026-09-11 05:21:19 +00:00

ci: install nix via determinate script in the flake workflow ... dfc96cb120

Forgejo runner jobs execute in Docker containers without systemd;
DeterminateSystems/nix-installer-action@v20 hangs on its systemd init there
(consistent ~5m27s failure, nix steps never reached) on both gate pushes of
this PR. Use the same --init none + direct nix-daemon install that
tree-sitter.yml already uses successfully; the four flake checks are
unchanged (flake check, build, packaged library load, packaged wrapper load)
and all pass locally.

nsaspy merged commit 51d2530f0e into main 2026-09-11 05:33:42 +00:00

nsaspy referenced this pull request from a commit 2026-09-11 05:33:44 +00:00

Merge pull request 'fix: #98 review — semantic knowledge review fixes' (#458) from rage/98-semantic-project-knowledge into main

nsaspy commented 2026-09-11 05:34:06 +00:00

Author

Owner

Copy link

Merged on green: all six forgejo gates success at dfc96cb (deterministic CI x2, pack-install, tree-sitter FFI, paid OpenRouter lane, Nix flake after the runner install fix in dfc96cb — the installer action hung on systemd-less docker runners; the four flake checks themselves are unchanged and were passing locally throughout). Merge commit 51d2530f0e.

Merged on green: all six forgejo gates success at dfc96cb (deterministic CI x2, pack-install, tree-sitter FFI, paid OpenRouter lane, Nix flake after the runner install fix in dfc96cb — the installer action hung on systemd-less docker runners; the four flake checks themselves are unchanged and were passing locally throughout). Merge commit 51d2530f0e18.

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something isn't working

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature or request

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

invalid

This doesn't seem right

  

question

Further information is requested

  

wontfix

This will not be worked on

No labels

bug

documentation

duplicate

enhancement

good first issue

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

nsaspy/prolog-rlm!458

No description provided.