[semantic-compiler] Library/tool integration for large documents, chunked compilation, projections, and host adapters

nsaspy commented

2026-09-08 01:29:19 +00:00

Owner

Parent: #388
Depends on: #392, #393, #394
Related: #395

Goal

Finish the reusable library/tool surface around the general semantic compiler so hosts can compile single strings, documents, streams/logs, or chunked corpora without depending on symbolic-memory.

Public surface

Provide semantics equivalent to:

semantic_compile(+Text, +Options, -Outcome).
semantic_compile_segment(+SourceRef, +Span, +Text, +Options, -Outcome).
semantic_compile_document(+Document, +Options, -Outcome).
semantic_compile_stream(+Source, +Options, -Outcome).
semantic_validate(+CandidateIR, +Options, -Outcome).
semantic_normalize(+ValidatedIR, +Options, -Outcome).
semantic_lower(+ValidatedIR, +Options, -Outcome).
semantic_projection(+CompiledPackage, +Goal, +Options, -Outcome).
semantic_explain(+RecordOrResult, -Outcome).

Exact naming follows repo conventions.

Large-document semantics

Support:

deterministic segmentation;
stable segment IDs/spans;
bounded provider windows;
overlap/continuation metadata;
document-level entity/coreference table;
cross-segment rule/procedure/event linking;
resumable compilation checkpoints where the host supplies storage;
idempotent segment fingerprints;
bounded concurrency/cancellation;
partial failure reporting without silently publishing an incoherent document package.

The compiler itself may return a package/stream of accepted semantic records; persistence remains caller-owned.

Tool adapter

Expose semantic compilation as a normal capability-gated tool using the same library semantics.

The tool must not:

grant execution authority to compiled actions;
auto-consult generated Prolog;
hide provider/model usage;
require symbolic-memory.

Projection

Allow a host to request a bounded projection from an already compiled package for a current goal, selecting relevant entities/rules/events/procedures/constraints plus provenance refs rather than sending the entire compiled document to a model.

This projection may compose with #381 retrieval and the prompt compiler, but must remain usable as a pure library operation for in-memory packages.

Structured input adapters

Support optional deterministic adapters for sources such as:

JSON/JSONL message/event streams;
LLM logs;
Markdown/Org/document headings;
web-result records;
issue/PR comment streams;
timestamped traces.

Adapters preserve source structure/provenance and then feed semantic content through the same compiler.

Acceptance

Single-string library call compiles without persistent store.
Multi-page document compiles through bounded segments with cross-segment identity preserved.
JSONL/log adapter preserves event metadata and semantic content.
Cancellation leaves structured partial outcome and no hidden continuation.
Same source/segment/config yields stable work fingerprint where compiler frontend output is fixed.
Tool adapter and direct library call produce equivalent normalized packages.
Projection returns relevant mixed knowledge classes rather than full package.
Provider usage reflects only actual model calls.
Compiler runs with symbolic-memory absent.

Refs #388 #392 #393 #394 #395 #381.

Parent: #388 Depends on: #392, #393, #394 Related: #395 ## Goal Finish the reusable library/tool surface around the general semantic compiler so hosts can compile single strings, documents, streams/logs, or chunked corpora without depending on symbolic-memory. ## Public surface Provide semantics equivalent to: ```prolog semantic_compile(+Text, +Options, -Outcome). semantic_compile_segment(+SourceRef, +Span, +Text, +Options, -Outcome). semantic_compile_document(+Document, +Options, -Outcome). semantic_compile_stream(+Source, +Options, -Outcome). semantic_validate(+CandidateIR, +Options, -Outcome). semantic_normalize(+ValidatedIR, +Options, -Outcome). semantic_lower(+ValidatedIR, +Options, -Outcome). semantic_projection(+CompiledPackage, +Goal, +Options, -Outcome). semantic_explain(+RecordOrResult, -Outcome). ``` Exact naming follows repo conventions. ## Large-document semantics Support: - deterministic segmentation; - stable segment IDs/spans; - bounded provider windows; - overlap/continuation metadata; - document-level entity/coreference table; - cross-segment rule/procedure/event linking; - resumable compilation checkpoints where the host supplies storage; - idempotent segment fingerprints; - bounded concurrency/cancellation; - partial failure reporting without silently publishing an incoherent document package. The compiler itself may return a package/stream of accepted semantic records; persistence remains caller-owned. ## Tool adapter Expose semantic compilation as a normal capability-gated tool using the same library semantics. The tool must not: - grant execution authority to compiled actions; - auto-consult generated Prolog; - hide provider/model usage; - require symbolic-memory. ## Projection Allow a host to request a bounded projection from an already compiled package for a current goal, selecting relevant entities/rules/events/procedures/constraints plus provenance refs rather than sending the entire compiled document to a model. This projection may compose with #381 retrieval and the prompt compiler, but must remain usable as a pure library operation for in-memory packages. ## Structured input adapters Support optional deterministic adapters for sources such as: - JSON/JSONL message/event streams; - LLM logs; - Markdown/Org/document headings; - web-result records; - issue/PR comment streams; - timestamped traces. Adapters preserve source structure/provenance and then feed semantic content through the same compiler. ## Acceptance - [ ] Single-string library call compiles without persistent store. - [ ] Multi-page document compiles through bounded segments with cross-segment identity preserved. - [ ] JSONL/log adapter preserves event metadata and semantic content. - [ ] Cancellation leaves structured partial outcome and no hidden continuation. - [ ] Same source/segment/config yields stable work fingerprint where compiler frontend output is fixed. - [ ] Tool adapter and direct library call produce equivalent normalized packages. - [ ] Projection returns relevant mixed knowledge classes rather than full package. - [ ] Provider usage reflects only actual model calls. - [ ] Compiler runs with symbolic-memory absent. Refs #388 #392 #393 #394 #395 #381.

nsaspy referenced this issue

2026-09-08 01:46:38 +00:00

[EPIC][expert] Retrieval / evidence expert: symbolic source selection, evidence gaps, and explainable retrieval strategy #417

nsaspy referenced this issue

2026-09-08 01:46:38 +00:00

[EPIC][expert] Retrieval / evidence expert: symbolic source selection, evidence gaps, and explainable retrieval strategy #417

nsaspy commented

2026-09-08 01:47:26 +00:00

Author

Owner

MACHINE-SPIRIT #398C/#398D handoff — projections become replayable query modules

Depth-1 research changes the intended meaning of semantic_projection/4.

A projection should not be an untyped bag of top-K semantic records. It should be a bounded query-specific semantic module containing or referencing:

source theories / versions
exported query signature
query class / bindings
admitted theory mappings / bridges traversed
reasoning profiles used
selected relevant base assertions/rules/events/procedures/constraints
included materialized consequences and freshness generation
known unsupported/omitted semantics
provenance refs
deductive-plan/query-module fingerprint

This makes a projection replayable and explainable: downstream Expert/Prompt/Model consumers can see which semantic environment produced it rather than receiving context stripped of theory boundaries.

Large-document / corpus implication

Chunked source compilation may produce many source-local theories/contexts and versioned vocabulary references. Cross-segment linking should create explicit semantic mappings/bridges rather than silently rewriting all identifiers into one document-global ontology.

Query integration implication

Where a query is already symbolic, the compiler/runtime should let deductive demand narrow the projection before embeddings/model relevance ranking are used. A known Horn/procedure/constraint query should be able to derive its relevant theory/rule slice with zero provider calls.

Acceptance additions suggested by #398C/D

Projection receipt identifies exact theory edges/mappings traversed.
Projection can exclude a source theory because its mapping/import contract is unverified.
Projection fingerprint is stable for fixed source semantic generation/query/options.
Derived records report freshness/materialization generation.
Replaying the receipt against the same semantic generation yields an equivalent bounded module.
A large corpus narrow symbolic query does not require union/materialization of the entire compiled corpus.

Full research is in #398C/#398D and parent #397.

## MACHINE-SPIRIT #398C/#398D handoff — projections become replayable query modules Depth-1 research changes the intended meaning of `semantic_projection/4`. A projection should not be an untyped bag of top-K semantic records. It should be a bounded **query-specific semantic module** containing or referencing: ```text source theories / versions exported query signature query class / bindings admitted theory mappings / bridges traversed reasoning profiles used selected relevant base assertions/rules/events/procedures/constraints included materialized consequences and freshness generation known unsupported/omitted semantics provenance refs deductive-plan/query-module fingerprint ``` This makes a projection replayable and explainable: downstream Expert/Prompt/Model consumers can see which semantic environment produced it rather than receiving context stripped of theory boundaries. ### Large-document / corpus implication Chunked source compilation may produce many source-local theories/contexts and versioned vocabulary references. Cross-segment linking should create explicit semantic mappings/bridges rather than silently rewriting all identifiers into one document-global ontology. ### Query integration implication Where a query is already symbolic, the compiler/runtime should let deductive demand narrow the projection before embeddings/model relevance ranking are used. A known Horn/procedure/constraint query should be able to derive its relevant theory/rule slice with zero provider calls. ### Acceptance additions suggested by #398C/D - [ ] Projection receipt identifies exact theory edges/mappings traversed. - [ ] Projection can exclude a source theory because its mapping/import contract is unverified. - [ ] Projection fingerprint is stable for fixed source semantic generation/query/options. - [ ] Derived records report freshness/materialization generation. - [ ] Replaying the receipt against the same semantic generation yields an equivalent bounded module. - [ ] A large corpus narrow symbolic query does not require union/materialization of the entire compiled corpus. Full research is in #398C/#398D and parent #397.

nsaspy commented

2026-09-08 01:47:26 +00:00

Author

Owner

MACHINE-SPIRIT #399C library/tool delta — analyzer/adaptor interoperability surface

The public library surface should gain optional semantics equivalent to:

semantic_analyzer_capabilities(+AnalyzerId, -Manifest).
semantic_select_analyzers(+Language,+Genre,+Risk,+RequiredPhenomena,+Budget,-Plan).
semantic_import_external(+Formalism,+Payload,+Options,-Outcome,-ConversionReceipt).
semantic_export_external(+Package,+TargetFormalism,+Options,-Payload,-LossReport).
semantic_mapping_status(+SourceSchema,+TargetSchema,-Outcome).

Requirements:

analyzers remain optional capabilities; no single AMR/UCCA/DRS/etc. dependency is mandatory;
adapter/manifests and mapping bundles are versioned/fingerprinted;
native parser payloads may be referenced/content-addressed rather than copied into durable semantic records;
common graph serialization must not erase formalism identity;
import/export must report weakened/dropped semantic dimensions and introduced assumptions;
unknown mapping/version compatibility fails explicitly rather than optimistic conversion;
direct semantic_compile remains usable with the admitted analyzer subset available;
source genre affects routing, never authority or truth promotion;
external labels/predicates remain inert data and cannot resolve to host calls/capabilities.

Full rationale and sources are in #399C.

## MACHINE-SPIRIT #399C library/tool delta — analyzer/adaptor interoperability surface The public library surface should gain optional semantics equivalent to: ```prolog semantic_analyzer_capabilities(+AnalyzerId, -Manifest). semantic_select_analyzers(+Language,+Genre,+Risk,+RequiredPhenomena,+Budget,-Plan). semantic_import_external(+Formalism,+Payload,+Options,-Outcome,-ConversionReceipt). semantic_export_external(+Package,+TargetFormalism,+Options,-Payload,-LossReport). semantic_mapping_status(+SourceSchema,+TargetSchema,-Outcome). ``` Requirements: - analyzers remain optional capabilities; no single AMR/UCCA/DRS/etc. dependency is mandatory; - adapter/manifests and mapping bundles are versioned/fingerprinted; - native parser payloads may be referenced/content-addressed rather than copied into durable semantic records; - common graph serialization must not erase formalism identity; - import/export must report weakened/dropped semantic dimensions and introduced assumptions; - unknown mapping/version compatibility fails explicitly rather than optimistic conversion; - direct `semantic_compile` remains usable with the admitted analyzer subset available; - source genre affects routing, never authority or truth promotion; - external labels/predicates remain inert data and cannot resolve to host calls/capabilities. Full rationale and sources are in #399C.

nsaspy commented

2026-09-08 01:47:27 +00:00

Author

Owner

MACHINE-SPIRIT #399D library/tool delta — resumable content-addressed compile execution

Depth 2D adds an execution layer beneath the existing public semantic APIs without making persistence mandatory.

Additional API direction

Semantics equivalent to:

semantic_plan_compile(+SourceDescriptor,+RequiredProfile,+Budget,+AvailableAnalyzers,-Plan,-PlanReceipt).
semantic_execute_plan(+Plan,+Options,-Outcome,-CompileReceipt).
semantic_compile_resume(+ReceiptOrCheckpoint,+Options,-Outcome).
semantic_compile_diff(+PriorGeneration,+SourceDelta,+Options,-Outcome).
semantic_artifact_status(+ArtifactRef,-Status).

Exact names follow repository style.

Cache/artifact SPI

The compiler should accept an optional host-supplied immutable artifact/cache interface. Implementations may be memory-only, filesystem CAS, Symbolic Memory-backed, remote/shared, or disabled. Correctness must not depend on cache availability.

Cacheable artifacts include deterministic source-adapter output, analyzer/native formalism output, SCL/SCC shards, mapping artifacts, validation/export/loss receipts and retained model candidate artifacts where policy permits.

Hard replay distinction:

exact_artifact_replay   retained immutable outputs, no re-inference
deterministic_rebuild   pinned deterministic stages reproduce digests
semantic_recompile      provider/nondeterministic stage invoked again; no identity promise

A model request fingerprint alone cannot satisfy exact replay.

Partial outcome contract

Compilation should expose typed material gaps rather than one generic failed/succeeded bit. Callers may request completeness profiles such as no_material_gaps([attribution,negation,quantifier_scope]). Successful completed artifacts remain reusable across retry/cancellation.

Parallelism

Segment-local analysis can fan out; ordered/dynamic discourse reconciliation remains dependency-aware. Worker interleaving may not choose semantic winners. Existing runtime concurrency/cancellation machinery executes the plan; do not create another global scheduler.

Full design + D1-D13 fixtures: #399D.

## MACHINE-SPIRIT #399D library/tool delta — resumable content-addressed compile execution Depth 2D adds an execution layer beneath the existing public semantic APIs without making persistence mandatory. ### Additional API direction Semantics equivalent to: ```prolog semantic_plan_compile(+SourceDescriptor,+RequiredProfile,+Budget,+AvailableAnalyzers,-Plan,-PlanReceipt). semantic_execute_plan(+Plan,+Options,-Outcome,-CompileReceipt). semantic_compile_resume(+ReceiptOrCheckpoint,+Options,-Outcome). semantic_compile_diff(+PriorGeneration,+SourceDelta,+Options,-Outcome). semantic_artifact_status(+ArtifactRef,-Status). ``` Exact names follow repository style. ### Cache/artifact SPI The compiler should accept an optional host-supplied immutable artifact/cache interface. Implementations may be memory-only, filesystem CAS, Symbolic Memory-backed, remote/shared, or disabled. Correctness must not depend on cache availability. Cacheable artifacts include deterministic source-adapter output, analyzer/native formalism output, SCL/SCC shards, mapping artifacts, validation/export/loss receipts and retained model candidate artifacts where policy permits. Hard replay distinction: ```text exact_artifact_replay retained immutable outputs, no re-inference deterministic_rebuild pinned deterministic stages reproduce digests semantic_recompile provider/nondeterministic stage invoked again; no identity promise ``` A model request fingerprint alone cannot satisfy exact replay. ### Partial outcome contract Compilation should expose typed material gaps rather than one generic failed/succeeded bit. Callers may request completeness profiles such as `no_material_gaps([attribution,negation,quantifier_scope])`. Successful completed artifacts remain reusable across retry/cancellation. ### Parallelism Segment-local analysis can fan out; ordered/dynamic discourse reconciliation remains dependency-aware. Worker interleaving may not choose semantic winners. Existing runtime concurrency/cancellation machinery executes the plan; do not create another global scheduler. Full design + D1-D13 fixtures: #399D.

nsaspy commented

2026-09-08 01:47:27 +00:00

Author

Owner

Machine Spirit #403A handoff — `semantic_projection` contract

The package-local semantic_projection(+CompiledPackage,+Goal,+Options,-Outcome) should use the same TAPS semantics as world-model-backed projection, while remaining a pure in-memory library operation with no symbolic-memory/vector/model requirement.

Required result distinctions:

support_closure = closed | partial | resource_limited | stale
search_coverage = exhaustive_for_declared_fragment | bounded_structural | heuristic | partial_backend | unknown

Projection flow: hard scope/profile/temporal admission → exact/structural seed discovery → mandatory rule/premise/guard/justification/counterevidence closure → optional soft periphery. Top-k/budget packing may drop a candidate, but must not retain a derived candidate while silently omitting its mandatory dependencies.

Return a replayable receipt with package fingerprint, projection options/profile, selected seeds, dependencies, frontier/gaps and deterministic ordering metadata. Embedding/graph/learned retrieval may be pluggable candidate sensors but cannot be required for exact symbolic projection or change truth/authority semantics.

This makes direct package projection and symbolic-memory #7 projection differential-testable under identical semantic fixtures. Refs #403 #381 #394.

## Machine Spirit #403A handoff — `semantic_projection` contract The package-local `semantic_projection(+CompiledPackage,+Goal,+Options,-Outcome)` should use the same **TAPS** semantics as world-model-backed projection, while remaining a pure in-memory library operation with no symbolic-memory/vector/model requirement. Required result distinctions: ```text support_closure = closed | partial | resource_limited | stale search_coverage = exhaustive_for_declared_fragment | bounded_structural | heuristic | partial_backend | unknown ``` Projection flow: hard scope/profile/temporal admission → exact/structural seed discovery → mandatory rule/premise/guard/justification/counterevidence closure → optional soft periphery. Top-k/budget packing may drop a candidate, but must not retain a derived candidate while silently omitting its mandatory dependencies. Return a replayable receipt with package fingerprint, projection options/profile, selected seeds, dependencies, frontier/gaps and deterministic ordering metadata. Embedding/graph/learned retrieval may be pluggable candidate sensors but cannot be required for exact symbolic projection or change truth/authority semantics. This makes direct package projection and symbolic-memory #7 projection differential-testable under identical semantic fixtures. Refs #403 #381 #394.

nsaspy commented

2026-09-08 01:47:27 +00:00

Author

Owner

Machine Spirit #403B handoff — adversarially robust `semantic_projection`

For pure in-memory compiled packages, semantic_projection should preserve the same OATH-TAPS control boundary established in #403B:

trusted QueryIR/profile determines required evidence lanes/obligations;
raw source/query/model-generated search strings cannot set mandatory, pinned, trust, identity or authority;
selected records must be validated as members of the exact compiled package/version before reasoning;
support closure remains distinct from candidate-search coverage;
when triggered, projection reports opposition/defeater, freshness, identity and source-independence coverage/gaps;
ranked similarity never implies identity/warrant;
closure budget exhaustion returns structured partial/resource-limited state instead of a conclusion with missing dependencies.

The in-memory surface need not implement durable-ledger mechanics, but it should preserve the same semantic receipt shape so hosts do not get weaker safety merely because persistence is absent. Refs #403 #395 #381.

## Machine Spirit #403B handoff — adversarially robust `semantic_projection` For pure in-memory compiled packages, `semantic_projection` should preserve the same OATH-TAPS control boundary established in #403B: - trusted QueryIR/profile determines required evidence lanes/obligations; - raw source/query/model-generated search strings cannot set `mandatory`, `pinned`, trust, identity or authority; - selected records must be validated as members of the exact compiled package/version before reasoning; - support closure remains distinct from candidate-search coverage; - when triggered, projection reports opposition/defeater, freshness, identity and source-independence coverage/gaps; - ranked similarity never implies identity/warrant; - closure budget exhaustion returns structured partial/resource-limited state instead of a conclusion with missing dependencies. The in-memory surface need not implement durable-ledger mechanics, but it should preserve the same semantic receipt shape so hosts do not get weaker safety merely because persistence is absent. Refs #403 #395 #381.

nsaspy commented

2026-09-08 01:47:27 +00:00

Author

Owner

MACHINE-SPIRIT #403C handoff — in-memory projection interoperability

semantic_projection should expose the same MOSAIC-TAPS contract as durable/remote retrievers where applicable: typed candidate envelopes, explicit coverage/freshness, QueryIR translation/preservation receipts, and backend-local ordering semantics. A pure compiled package can naturally advertise a strong exact/local fragment; it must not fake vector/remote coverage it does not have.

For identical admitted package/query/profile fixtures, #403C requires semantic equivalence between the pure #396 projection path and the symbolic-memory #7 local adapter path, with differences limited to storage/provenance envelope details. Preserve model_calls = 0 for known compiled-semantic projection.

Refs #397 #403 #381 #395 and symbolic-memory#7/#9.

### MACHINE-SPIRIT #403C handoff — in-memory projection interoperability `semantic_projection` should expose the same MOSAIC-TAPS contract as durable/remote retrievers where applicable: typed candidate envelopes, explicit coverage/freshness, QueryIR translation/preservation receipts, and backend-local ordering semantics. A pure compiled package can naturally advertise a strong exact/local fragment; it must not fake vector/remote coverage it does not have. For identical admitted package/query/profile fixtures, #403C requires semantic equivalence between the pure #396 projection path and the symbolic-memory #7 local adapter path, with differences limited to storage/provenance envelope details. Preserve `model_calls = 0` for known compiled-semantic projection. Refs #397 #403 #381 #395 and symbolic-memory#7/#9.

nsaspy commented

2026-09-08 01:47:27 +00:00

Author

Owner

Machine Spirit #403D handoff — in-memory projection equivalence

semantic_projection/3 for an already-compiled in-memory package should implement the same logical projection/receipt semantics as RACE-TAPS without requiring symbolic-memory or remote federation.

For pure in-memory packages, require deterministic QueryIR/obligation planning, hard admission, mandatory dependency closure, explicit support/search/opposition coverage, stable tie-breaking, byte/work bounds and a replayable strategy receipt. Physical optimization may be trivial/local, but the semantic contract must remain equivalent to #403 so callers do not get a weaker meaning model merely because the package is in memory.

Differential fixture: same finite semantic package + goal/profile/budget through semantic_projection and an equivalent one-backend #381/RACE projection must normalize to the same selected semantic records, closure state, gaps and coverage classes with model_calls = 0.

## Machine Spirit #403D handoff — in-memory projection equivalence `semantic_projection/3` for an already-compiled in-memory package should implement the same **logical projection/receipt semantics** as RACE-TAPS without requiring symbolic-memory or remote federation. For pure in-memory packages, require deterministic QueryIR/obligation planning, hard admission, mandatory dependency closure, explicit support/search/opposition coverage, stable tie-breaking, byte/work bounds and a replayable strategy receipt. Physical optimization may be trivial/local, but the semantic contract must remain equivalent to #403 so callers do not get a weaker meaning model merely because the package is in memory. Differential fixture: same finite semantic package + goal/profile/budget through `semantic_projection` and an equivalent one-backend #381/RACE projection must normalize to the same selected semantic records, closure state, gaps and coverage classes with `model_calls = 0`.

nsaspy referenced this issue

2026-09-08 01:47:30 +00:00

[AUTO-RESEARCH][MACHINE-SPIRIT] Eight-depth × four-subpass design program for general symbolic intelligence #401

nsaspy referenced this issue

2026-09-08 01:47:35 +00:00

[MACHINE-SPIRIT 1/8] Knowledge-representation foundations: logic, frames, DL/Datalog, events, typed open vocabularies #400

nsaspy referenced this issue

2026-09-08 01:47:35 +00:00

[MACHINE-SPIRIT 1/8] Knowledge-representation foundations: logic, frames, DL/Datalog, events, typed open vocabularies #400

nsaspy referenced this issue

2026-09-08 01:47:46 +00:00

[MACHINE-SPIRIT 4/8] Procedural intelligence and expert systems: production rules, blackboards, HTN planning, CBR, diagnosis and repair #397