[MACHINE-SPIRIT 2/8] Semantic compilation from language: discourse, AMR/DRS/SRL/OpenIE, constrained generation, semantic-loss control #399
Labels
No labels
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
nsaspy/prolog-rlm#399
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Parent: #397
Related: #388 #393 #395 #396
Downstream: lost-rob0t/symbolic-memory#5 #8
Distinct research mandate
Study natural-language-to-symbolic semantic compilation as its own design problem. Compare semantic parsing traditions and modern structured-generation approaches while preserving discourse, scope, attribution, ambiguity, pragmatic/commitment status, multilingual/formalism differences, and exact source evidence.
This depth is research/design only. It does not solve durable truth maintenance or claim an executable Machine Spirit implementation.
Four-subpass status — DESIGN-COMPLETE
A — foundations / representative formalisms: COMPLETE
Evidence-anchored multi-view Semantic Compilation Lattice (SCL) over exact source anchors. Analyzer outputs are observations rather than truth; ambiguity remains packed; symbolic discourse state crosses segments; deterministic source structure is extracted before model use; bounded LLM induction/reconciliation is candidate generation, not truth.
Primary result is preserved in the #399A issue comment and canonical #393/#395 handoffs.
B — semantic/adversarial deepening: COMPLETE
Compiler-local Semantic Commitment Calculus (SCC) preserves holder/context, linguistic force, projectivity/opacity, defeasibility and derivation license. Presupposition, implicature, ellipsis, intensionality, indexicals, quotation, genericity/habituality, metonymy and temporal/discourse interpretation cannot flatten into ordinary assertions.
semantic overpromotionis a first-class conformance failure.Primary result is preserved in the #399B issue comment and canonical #393/#395 + symbolic-memory handoffs.
C — composition / heterogeneity / interoperability: COMPLETE
Federated Semantic Interoperability Fabric keeps native analyzer/formalism observations auditable; common graphs are transport rather than universal semantics; analyzer capability manifests, typed semantic mapping/alignment edges, mapping/loss profiles, multilingual translation divergence, language/formalism category lattices, versioned mapping bundles and explicit import/export loss receipts mediate composition.
Mapping confidence is representational evidence, not world truth. Representation interoperability never grants authority.
Primary result is preserved in the #399C issue comment and canonical #393/#395/#396 + symbolic-memory handoffs.
D — computational realization / scaling / conformance: COMPLETE
Adopt a compiler-local Semantic Build Graph (SBG) + demanded incremental semantic state.
Surviving decisions:
exact_artifact_replay,deterministic_rebuild, andsemantic_recompile; a model request hash alone is never proof of exact replay;Full sources, candidate architectures, algorithms, adversarial review, metrics and D1-D13 conformance fixtures are preserved in the #399D issue comment.
Reconciled Depth-2 architecture
Canonical handoffs
Depth completion status
Depth 2 is design-complete and implementation-unproven.
Do not infer Machine Spirit acceptance from these research results. The next research depth is #400A — epistemics / truth maintenance / non-monotonic reasoning.
Refs #388 #392 #393 #394 #395 #396 #397 #400.
Depth 2 / subpass A — semantic-compilation foundations: multi-view evidence-anchored compiler
This is #399A only. It establishes the foundations/baseline architecture for language -> symbolic compilation. It does not consume #399B/#399C/#399D and does not close #399.
Research question
Given the intended inputs — LLM logs, Wikipedia/encyclopedic prose, news, manuals, scientific papers, chats, issue/PR histories, policies, event traces, and arbitrary text — what frontend architecture can compile language into the #392 semantic substrate while preserving discourse, scope, attribution, quantification, modality, temporal/event structure, procedures/rules, ambiguity, and exact source evidence?
The crucial design constraint is stronger than ordinary information extraction: a compiler output is intended to become reusable symbolic knowledge after the source text is no longer in the reasoning context. A representation that is merely useful for search or QA is therefore insufficient if it silently drops meaning required for later inference.
Primary/authoritative sources inspected
Classical / explicit semantic representations:
Modern constrained/LLM generation:
What #392/#393 already get right
The current architecture already has several critical invariants that survive this pass:
What #393 is currently missing
Its pipeline is written as a mostly linear sequence:
That makes an architectural assumption that each early phase can commit to one interpretation before later phases run. Natural language does not behave that cleanly.
Examples:
if,only if,unless, generic tense, habituals and conditionals interact with rule induction;The compiler therefore needs parallel evidence views + explicit candidate hypotheses, not destructive phase-by-phase commitment.
Candidate architecture A — direct constrained generation into #392 IR
Strengths
Fatal weakness
Syntax-constrained != semantically faithful. The model can emit a perfectly legal IR package that invents a cause, chooses the wrong antecedent, loses
only if, promotes an attributed claim, collapses an existential witness, or silently chooses one quantifier scope.The richer the #392 schema becomes, the more a direct generator can produce well-typed hallucinations.
Decision
Useful as one frontend, absolutely insufficient as the canonical compiler architecture.
Candidate architecture B — one linguistically deep pivot representation, then deterministic lowering
Use something DRS/MRS-like as the compiler's universal pivot:
Strengths
Weaknesses
Decision
A formal DRS/MRS-like view is extremely valuable, but should be a semantic evidence layer, not the sole canonical pivot.
Candidate architecture C — evidence-anchored multi-view semantic compiler (preferred)
The compiler should construct an intermediate Semantic Compilation Lattice (SCL) containing observations and competing interpretation hypotheses before committing normalized records to #392.
This is not another long-term knowledge representation. It is an ephemeral/versioned compiler IR whose purpose is to preserve evidence, alternatives and transformation lineage.
The system does not require every frontend analyzer to exist for every language. Each adapter advertises its capabilities and confidence/evidence class. The lattice works with whatever evidence is available.
Core SCL types
Conceptual only:
Possible observations include:
Observations never equal truth. They are evidence used to build semantic hypotheses.
Why this fits #392B's contextual theory design
Compilation naturally creates at least two contexts:
An asserted proposition in the final candidate package requires an auditable path back through SCL evidence to exact source spans. A generalized rule inferred from several sentences gets a different induction class than an explicitly stated conditional.
This directly supports:
without pretending these have identical epistemic status.
Semantic Compilation Lattice dimensions
A hypothesis may branch independently along dimensions such as:
Do NOT eagerly enumerate the Cartesian product of every ambiguity. Use a packed forest/lattice with shared substructure and compatibility constraints.
Example:
The compiler can preserve competing
he/herbindings and modal/temporal scopes as linked hypothesis choices rather than fabricating one globally committed interpretation.Final #392 lowering can either:
Failure to know is a valid compiler result.
Discourse state must be explicit
For each document/stream maintain an ephemeral discourse workspace:
This is inspired more by DRT's evolving discourse representation than by ordinary chunk embedding/retrieval.
Cross-segment compilation should pass symbolic discourse state, not merely overlap text windows.
For huge LLM logs, for example:
are deterministic inputs to the next segment's semantic analysis.
This substantially reduces semantic amnesia without stuffing the whole corpus into the model.
Source structure is a first-class compiler sensor
Before any LLM pass, consume deterministic information already present in source formats.
Examples:
LLM logs
News/web
Wikipedia
manuals/docs
A model should never be asked to rediscover metadata that the input adapter already knows exactly.
LLM role: semantic inducer/reconciler, not sole parser
The LLM is best used for phenomena where deterministic parsers/signatures are insufficient:
It should consume bounded SCL evidence and emit typed candidate hypotheses under constrained decoding.
Conceptually:
Not:
Grammar-constrained decoding contract
Grammar/schema constraints provide:
They do not provide semantic truth.
After generation, deterministic validators must check at least:
Explicit vs implied vs generalized knowledge
#393 needs a stronger required classification. Recommended minimum induction classes:
The compiler must not promote
pragmatic_candidateorinduced_ruleinto ordinary source assertions merely because an LLM considers them obvious.This distinction is especially important for LLM logs: a successful past troubleshooting episode can yield a candidate reusable procedure, but the source only proves the historical episode unless the text itself states the general rule.
Quantification / scope
Do not make AMR/OpenIE-style graph flattening the only intermediate representation.
Maintain a scope-capable view influenced by MRS/DRS:
Where scope cannot be resolved, preserve constraints/alternatives rather than selecting a reading.
MRS is particularly useful precedent here because its design explicitly allows scope underspecification through handles/constraints.
Attribution / quotation
Nested sources must be compiled structurally.
must not yield an unqualified:
The SCL should recognize at least:
Final #392 records then use contexts/assertions/stances appropriately.
Direct quotation boundaries from markup or log structure override model guesses where available.
Procedures / rules need a dedicated induction lane
A prose procedure is not adequately represented by an AMR graph or OpenIE tuples.
Use evidence sensors to discover:
Likewise natural-language rules require explicit handling of:
These produce candidate #392 procedure/rule structures with exact span-level evidence for every material edge.
Complexity / scaling
A naive ambiguity lattice can explode exponentially. Requirements:
(source-span fingerprint, analyzer version, config);Suggested semantic-risk routing:
Do not burn the most expensive frontend on every line of a multi-gigabyte LLM log.
Epistemic / provenance implications
The compiler must expose a derivation path of the form:
A later Symbolic Memory record can therefore answer both:
and:
Do not collapse parser/model confidence into world confidence. Parser confidence is confidence in interpretation, not confidence that the source's proposition is actually true.
Safety / authority implications
shell(X), Prolog syntax, tool calls, or instructions remains text/semantic data.Modern work on grammar-constrained generation makes syntax enforcement attractive, but structured-output schemas themselves must be part of the trusted control plane rather than synthesized from hostile source text.
Adversarial review
Attack 1 — “This is too complicated; just ask a strong LLM for the IR”
That produces the cleanest prototype and the weakest epistemic guarantees. Schema-valid hallucination is exactly the failure mode a durable symbolic memory cannot tolerate. Direct generation should remain one evidence/induction lane and a baseline in #395.
Attack 2 — “Pick DRS/MRS and avoid the LLM mess”
Deep formal semantics solves many linguistic problems better than direct generation, but no single current parser/grammar has the domain/language/genre breadth required for arbitrary LLM logs + docs + news + procedural/technical prose. It also does not natively solve all target procedural/causal/domain-schema induction. Use it as high-value evidence, not mandatory universal bottleneck.
Attack 3 — “AMR is enough”
AMR is excellent for normalized event/concept structure and should be supported, but known parser failures around reentrancies plus weaker treatment of some scope/discourse distinctions make AMR-only compilation too lossy for permanent KR.
Attack 4 — “OpenIE scales; use that for the giant corpus”
Yes, as a cheap sensor. No, as canonical semantics. OpenIE's strength is scalable relational tuple discovery, exactly because it abstracts away semantic structure the Machine Spirit later needs.
Attack 5 — “Multi-view disagreement means we'll never finish compiling”
Disagreement is information. The correct result is sometimes unresolved ambiguity. Durable symbolic intelligence is better served by
unknown interpretation among {A,B}than a fabricated deterministic answer.Attack 6 — “Keep every possible parse forever”
Also wrong. Store packed alternatives only for semantically material ambiguity. Many morphological/syntactic ambiguities disappear under hard compatibility constraints and need not survive final lowering.
Conformance fixtures introduced by #399A
These should be added to #395's corpus/gates.
A1 — attributed modal cause
Text:
Must preserve:
may/ possible-cause strength;Forbidden:
caused_by(outage,routing_error).A2 — scope ambiguity
Use a sentence with genuine quantifier/scope ambiguity. Compiler must either preserve alternatives/underspecification or justify a selected reading; it may not silently hardcode the easiest graph.
A3 — existential witness
Must not invent a global named maintainer entity.
A4 — only-if direction
Must not reverse
only ifinto an ordinary sufficient-condition rule.A5 — nested quotation
Publication A reports that Person B says Organization C denied P. The resulting assertions/contexts must retain all source levels.
A6 — historical case vs induced rule
Must preserve the historical episode. A reusable
hash_failure -> update_lockfilerule, if proposed, must beinduced_rule, notsource_explicit.A7 — cross-segment coreference
Entity introduced in segment N and pronoun/alias used in N+K. Compile with bounded text windows but persistent discourse state. Identity must remain linked or explicitly ambiguous.
A8 — adversarial source instruction
Must be represented as source content/possibly an instruction proposition only; cannot alter compiler control flow or emit executable host syntax.
A9 — parser disagreement
Fixture where AMR/SRL/deep-parser views disagree on a material argument/reentrancy. Compiler must record disagreement and cannot claim high-certainty normalization without reconciliation evidence.
A10 — semantic round trip
Compile a mixed passage, discard original text from the answering context, and prove #392 queries recover the intended actor/event/rule/modal/attribution distinctions. Exact serialization is not required; semantic invariants are.
Concrete canonical changes recommended
#393
Replace the conceptual linear multipass pipeline with an evidence-anchored Semantic Compilation Lattice architecture:
Add the explicit induction-class taxonomy (
source_explicit,compositionally_entailed,source_presupposed,pragmatic_candidate,cross_sentence_synthesized,induced_rule,compiler_repair).Add hard invariant: analyzer/model confidence is interpretation confidence, never source/world truth confidence.
#395
Add parser-disagreement, scope-underspecification, induction-class correctness, nested-attribution, historical-case-vs-generalized-rule and source-injection fixtures. Measure semantic coverage by phenomenon/invariant, not one canonical graph serialization.
Add ablation lanes:
so complexity is justified empirically rather than doctrinally.
#388
Clarify public API outcome should be able to return:
A parseable candidate package alone is not sufficient success.
Preferred #399A architecture decision
Adopt the evidence-anchored multi-view Semantic Compilation Lattice as the baseline architecture for subsequent depth-2 attacks.
It combines the strongest ideas from different traditions without pretending they are equivalent:
No analyzer is authoritative by itself.
#399B should attack this design specifically on hard semantic ambiguity and discourse phenomena: presupposition, implicature, ellipsis, intensionality, nested modalities, genericity/habituality, indexicals, metonymy, temporal discourse, quoted/indirect speech, rule-vs-example classification, and when not to compile an inference at all.
Status: #399A design-complete; #399 remains OPEN. Next required subpass: #399B.
Depth 2 / subpass B — semantic/adversarial deepening: commitment, projectivity, defeasibility, intensional scope
This is #399B only. It attacks the #399A Semantic Compilation Lattice (SCL) on phenomena where preserving alternative parses is still not enough: presupposition, implicature, ellipsis, intensionality, genericity/habituality, indexicals, nested modality, quotation/reported speech, temporal discourse and metonymy. It does not consume #399C/#399D and does not close #399.
Research question
#399A established that semantic compilation should preserve multiple evidence views and packed hypotheses instead of committing early. The harder question is:
This matters because a compiler for durable symbolic knowledge must distinguish materially different licenses for content:
Treating those as equivalent propositions with different confidence scores is semantically wrong.
Primary / authoritative sources inspected
Presupposition / projectivity
Implicature / cancellable pragmatic inference
Scope / rich symbolic semantics
Discourse / incremental interpretation
Quotation / report opacity
Temporal discourse
Genericity / habituals
forallrewrite.Metonymy / coercion
What #399A gets right
Keep these A decisions:
source_explicit,source_presupposed,pragmatic_candidate,induced_rule;But the A classification is currently too flat.
source_presupposedandpragmatic_candidateare labels on output origin; they do not formally state projection, holder, context, cancellation, opacity, or whether content is even eligible for promotion into an ordinary world assertion.Candidate architecture A — eager enriched semantics
Compile every recovered meaning into ordinary #392 propositions/assertions, attaching metadata such as:
Strengths
Failure
This collapses distinctions that are structural, not scalar:
A metadata enum is insufficient if the runtime can still accidentally query all forms as ordinary truth.
Reject as canonical design.
Candidate architecture B — literal semantics + disconnected pragmatic sidecar
Maintain two products:
Only literal semantics can enter ordinary symbolic reasoning.
Strengths
Failure
Natural-language interpretation is not cleanly separable this way. Presupposition resolution, ellipsis, metonymy, indexical anchoring and discourse relations often participate in determining what proposition was expressed at all. A disconnected sidecar cannot adequately represent their interactions with scope, reference and attribution.
Example:
Resolving
The White Housemetonymically,itanaphorically, and the presupposition triggered bychangedare coupled. There is no single fully specified "literal proposition" before those phenomena are handled.Keep the safety instinct, reject the disconnected architecture.
Candidate architecture C — Semantic Commitment Calculus inside SCL (preferred)
Extend #399A's SCL with a typed Semantic Commitment Calculus (SCC). The SCL continues to represent competing interpretations; SCC states the discourse/epistemic license of each content hypothesis.
Conceptual compiler-local records:
Candidate
Forcevocabulary:This is not a universal truth-value lattice. It describes how linguistic content entered the interpretation and what commitments it licenses.
Projectivitymay be structured as:Defeasibilitymay distinguish:The exact final enum can be smaller; the invariant is that these dimensions are not collapsed into one confidence number.
Hard semantic rules by phenomenon
1. Presupposition — trigger != assertion
A trigger creates a presuppositional content hypothesis plus a resolution obligation.
Resolution choices may include:
If a source says:
Alice previously smokedis notsource_explicit(asserted). It is a presuppositional commitment derived from a trigger and must retain that provenance.If embedded:
projection may be filtered/satisfied. The compiler cannot blindly export the presupposition globally.
Canonical requirement: presuppositions lower into #392 only with an explicit
presupposed_by/source-contextstance or equivalent context/derivation structure, never as unqualified assertions.2. Conversational implicature — candidate, cancellable, never automatic world truth
For:
A possible scalar implicature
not all services passedis a pragmatic candidate, not source-explicit truth.The continuation:
cancels the implicature without contradicting the asserted
somecontent.Therefore:
Machine Spirit may preserve useful pragmatic candidates for retrieval/explanation, but ordinary world queries must not treat them as assertions unless a separate trusted epistemic policy promotes/corroborates them.
3. Ellipsis — reconstruction is interpretation evidence
For:
The compiler may reconstruct
Bob deployed the API, but must preserve:The reconstructed proposition can be compositionally licensed while still being marked
reconstructed; the source did not literally contain that clause.Do not rewrite source spans or pretend reconstructed tokens existed.
4. Intensionality / attitudes — embedded content stays inside attitude contexts
For:
and separately:
Machine Spirit must not derive that Lois believes Clark Kent can fly merely from actual-world identity. Substitution under belief/report/modal contexts is not generally extensional.
Represent an explicit attitude context:
Preserve unresolved
de_re/de_dictoalternatives where the language permits both.No embedded belief/wish/fear/plan proposition leaks upward into the actual-world source theory without an explicit bridge.
5. Indexicals — resolve from utterance context or stay unresolved
I,you,here,now,today,this system, etc. depend on an utterance/context anchor.LLM logs provide unusually strong deterministic anchors:
Use those before model inference.
Conceptually:
If speaker/time/location is missing, preserve an unresolved indexical. Never invent an anchor to make the IR prettier.
6. Direct / indirect / mixed quotation — wording identity != reported content
Maintain separate layers:
Direct quotation can support an exact-form assertion (
speaker uttered string S) while the semantic interpretation of S remains separately compiled in the quoted speaker context.Indirect speech supports attributed content but not verbatim wording.
Mixed quotation requires a metalinguistic relation over form/use; Maier's analysis is direct evidence that simple proposition embedding is insufficient.
Hard rule:
7. Genericity / habituality — never silently
forallExamples:
Do not compile these as:
Instead preserve a generic/habitual operator or dedicated semantic form with its restrictor, scope and contextual normality/frequency information.
Conceptual:
A later defeasible-reasoning profile may map a generic to a default rule, but that is a reasoning-profile decision, not semantic compilation truth.
This avoids converting natural-language regularities into exceptionless executable rules.
8. Metonymy — preserve surface referent and coerced reading separately
For:
The surface NP identifies one entity/concept while the predicate selection strongly suggests an institution/spokesperson or driver/vehicle operation relation.
Do not globally rewrite:
Instead represent a contextual coercion/role bridge:
Identity remains separate. This dovetails with symbolic-memory#10's requirement for reversible identity/concept reconciliation rather than destructive merging.
9. Temporal discourse — constraints, not invented timestamps
TimeML's useful lesson is explicit event/time objects plus underspecified temporal relations.
Distinguish evidence strength:
Mere sentence order must not become
before(E1,E2)unless the genre/discourse model explicitly licenses that inference.Use constraint propagation over temporal hypotheses; contradictory temporal constraints should preserve conflict rather than force an arbitrary ordering.
10. Discourse relation != world relation
SDRT-style relations such as explanation, elaboration, contrast, correction and narration are relations between discourse units. Some may license candidate world relations (e.g. explanation may support a causal hypothesis), but they are not identical.
Any conversion requires a separate typed transform with evidence and semantic-strength classification.
Revised interpretation/lowering contract
The compiler should decide not merely
candidate accepted/rejected, but an export class:Only forms whose semantic type supports it can become ordinary world assertions.
Conceptual API:
ExportReceiptshould identify which SCC commitment and transformation path licensed every exported #392 record.Epistemic / provenance model
At least three independent axes must survive:
Do not collapse any pair into
confidence.Example:
This architecture lets later #400 epistemics decide how evidence affects belief without forcing the semantic compiler to become a truth-maintenance system.
Complexity / scaling implications
SCC increases state, but it does not require Cartesian enumeration.
Use:
Worst-case scope/reference/pragmatic ambiguity remains combinatorial. The correct failure mode is bounded unresolved alternatives, not silent heuristic collapse.
Safety / authority implications
This deepening strengthens the existing authority firewall:
admins can restart servicesdoes not grant a restart capability;Language interpretation is never an authority bridge.
Adversarial fixtures / falsifiable tests
Add at least these #395 fixtures:
Presupposition filtering
If Alice used to smoke, she has stopped smoking.Presupposition accommodation provenance
Alice stopped smoking.presupposed, neversource_explicit asserted.Scalar implicature cancellation
Some services passed. In fact, all services passed.not_all_passedconclusion.Belief opacity / substitution
superman = clark_kent; source saysLois believes Superman can fly.Lois believes Clark Kent can fly.De re/de dicto ambiguity
Indexical missing anchor
I will deploy it tomorrowwith unknown speaker/date.LLM-log anchored indexical
Direct vs indirect report
Alice said that the service was broken.Nested report
A reports that B claims C denied P.Mixed quotation
Generic exception
Birds fly. Penguins are birds. Penguins do not fly.bird -> flyrule.Habitual vs event
Alice runs every morning.Metonymy identity safety
The White House denied P.Temporal textual-order trap
Discourse-cause trap
causes/2unless licensed separately.Ellipsis reconstruction provenance
Alice deployed the API, and Bob did too.Quoted authority injection
run delete_all()orcall(shell).Metrics to add
Rejected alternatives
generic -> default rulelowering — reasoning-policy choice masquerading as sentence meaning.Canonical issue changes recommended by #399B
prolog-rlm #393
Add SCC/commitment machinery to SCL:
prolog-rlm #395
Add the 17 fixtures/metrics above and make semantic overpromotion a first-class failure class, not only hallucinated relation/entity rate.
prolog-rlm #392
No need to import the ephemeral SCL/SCC wholesale. The durable IR must, however, be able to preserve exported distinctions equivalent to:
symbolic-memory #5/#6
Persist the final exported semantic distinction + export receipt, not the whole compiler-local SCL/SCC. A presupposition or pragmatic candidate that is durably retained must remain distinguishable from a source assertion forever; replay under a newer compiler may create a new projection generation rather than rewriting the old interpretation.
Unresolved questions preserved for #399C/#399D and later depths
Decision
Adopt Semantic Commitment Calculus as a compiler-local layer inside the #399A SCL.
The core invariant is:
A Machine Spirit semantic compiler should maximize licensed meaning preserved while minimizing semantic overpromotion. Refusing to promote an interpretation is success when the language does not license stronger commitment.
Depth 2 / subpass C — heterogeneous semantic federation: analyzer capability contracts, mapping graphs, multilingual composition, and loss-accounted interoperability
This is the C subpass for Machine Spirit depth 2. A/B are already complete. C focuses only on composition/interoperability: how semantic analyzers, languages, source genres, external formalisms, ontologies, and compiler versions exchange evidence without pretending that their label sets, graph structures, scope systems, lexical senses, or pragmatic categories are isomorphic.
Research questions
Primary / authoritative sources inspected
Cross-framework semantic parsing / interchange
Multilingual / cross-lingual semantics
Standards / lexical and discourse interoperability
What A/B got right
A's Semantic Compilation Lattice and B's Semantic Commitment Calculus already establish the essential direction:
These survive C.
What A/B still underspecified
A/B implicitly treat analyzer outputs as if a normalized
Kind/Payloadis enough. It isn't.Examples:
:ARG0is not semantically identical to PropBank's role labels in every construction.If all of this is normalized prematurely into one label set, the compiler can become precisely wrong while remaining beautifully typed.
Candidate architecture A — universal pivot graph
Translate every analyzer into one canonical SCL graph vocabulary immediately.
Strengths
Fatal weaknesses
same graph shapeandsame semantics.Rejected as canonical architecture. A pivot transport graph is useful; a pivot meaning ontology is not.
Candidate architecture B — pairwise translators between analyzers/formalisms
Maintain converters such as AMR↔UCCA, AMR↔DRS, SRL↔AMR, etc.
Strengths
Weaknesses
Retain as optional specialized adapters, not the core architecture.
Candidate architecture C — federated native evidence + typed semantic mapping graph (preferred)
Keep every analyzer/source representation in a native evidence envelope. Add an explicit mapping/alignment graph that states how pieces of different representations correspond, with typed mapping strength, provenance, version and loss declarations.
New compiler-local objects
Conceptual shapes:
Native payload may be stored by reference/hash if large; the SCL does not need to copy every external AST into Prolog terms.
Mapping relations must be semantic, not just
same_asAt minimum mappings need relations equivalent to:
equivalentmust always be qualified by which semantic dimensions/profile are being claimed equivalent.Example:
This prevents a graph simplification from being mistaken for a truth-preserving full semantic equivalence.
Analyzer capability negotiation
A compiler should select analyzers because they claim/test support for the needed semantic phenomena, not because their binary happens to be installed.
Conceptual API:
Capability dimensions should cover things such as:
Support levels need more than boolean:
A language-independent parser architecture does not mean the semantics or training evidence are language-independent.
Multilingual design: interlingual hypotheses, not language erasure
The Parallel Meaning Bank and cross-lingual AMR show that shared representations across languages are useful. But projection typically depends on alignment and often on an assumption that translation is meaning-preserving.
Machine Spirit should therefore represent parallel/cross-lingual relationships explicitly:
Useful relations include:
Translation is evidence for semantic correspondence, never identity by fiat.
A multilingual lexical resource (LMF/OntoLex/BabelNet-style) can propose sense/concept bridges, but these remain versioned mappings with provenance.
Cross-linguistic category lattice
Van Gysel et al.'s central observation should become an explicit Machine Spirit principle: semantic categories across languages can be:
Therefore the open vocabulary/ontology mapping layer should support a partial-order/lattice-like relationship between semantic categories rather than forcing each language-specific label to one global enum.
Example:
A source-specific
MOTIONobservation can map to a broader category while candidate hypotheses retain the unresolved refinement. The compiler does not guess which subtype holds merely because another ontology has more detailed labels.Source genre is a routing/context feature, not a truth semantics
Adapters may declare genre capabilities/policies:
Genre can change which analyzers are prioritized and which failure modes are high risk:
But genre must never directly promote semantic commitment.
manualdoes not mean every imperative is authorized.newsdoes not make a publisher's assertion true.Wikipediadoes not convert prose to globally admitted facts.External format import/export rule
Use MRP/GrAF/Salt-like generic graph transport when useful, but keep format semantics in adapter manifests.
Import
Never:
Export
Every external export should return:
If #392 semantics cannot be represented faithfully in AMR/RDF/UCCA/etc., the exporter must say exactly what was dropped/weakened rather than returning a clean-looking lossy graph with no warning.
Version compatibility / semantic migration
Analyzer/formalism/compiler versions are semantic inputs.
Hard invariants:
unmapped/unsupported, not optimistic equivalence.Conceptual form:
Mapping bundles can be admitted/revoked independently of analyzer installation.
Repository ownership boundaries
prolog-rlmOwns:
It does not need to vendor every parser/ontology/lexicon.
symbolic-memoryOwns durable persistence of:
It should not persist the entire ephemeral SCL native analyzer graph by default. Native artifacts may be optionally archived/referenced when needed for replay/audit.
external analyzers/resources
Remain independent components behind adapters. Their output is never executable authority.
Performance / scaling implications
Naively comparing every analyzer node to every other analyzer node is quadratic or worse. Avoid that.
Anchored mapping first
Use exact/overlapping source spans, token/character anchors, event/entity candidate IDs and discourse-unit IDs to limit possible cross-view mappings.
Mapping bundles
Compile declarative mapping bundles into indexed transforms keyed by:
Lazy composition
Do not normalize every native observation eagerly. Materialize mappings only for:
Shared mapping DAG
Multiple hypotheses may share the same mapping/alignment evidence. Store one mapping edge/DAG and reference it.
Cross-lingual work
Word/sentence alignment can be expensive; cache by source+translation+model/alignment-version fingerprint. Do not recompute translation alignment merely because another semantic analyzer runs.
Mapping composition
Multi-hop mappings accumulate loss. A composed map must calculate a conservative cumulative loss profile; do not mark a chain exact because every individual edge had a plausible label.
Epistemic / provenance implications
C adds a fourth provenance axis to A/B's model:
Example:
The world status of P is not changed merely because mapping confidence is high. Conversely a low-confidence mapping does not mean the source itself is uncertain.
Mapping confidence is representation correspondence confidence.
Safety / authority implications
execute,shell,delete, etc. does not map to host capabilities;exact_correspondenceis semantic correspondence only, never authority equivalence.Representation interoperability is never an authority bridge.
Adversarial review
Attack: “MRP proved we can normalize all frameworks to one graph.”
No. MRP proved a common abstract graph serialization/evaluation substrate is useful. It preserved framework identity because the graph abstractions encode different semantics. Treating serialization uniformity as semantic uniformity would be category error.
Attack: “Use AMR as the interlingua.”
AMR is useful, but cross-lingual AMR work itself relies on projection/transfer and incomplete non-English supervision. It also does not preserve every discourse/scope/pragmatic distinction Machine Spirit needs. It should be one view, not the truth substrate.
Attack: “Use DRS as the universal logical form.”
DRS is stronger for scope/discourse, but it still does not provide a universal mapping for every lexical/frame/procedural/pragmatic distinction, and not every language/parser has comparable DRS support.
Attack: “Just map labels by embedding similarity.”
Embeddings are useful candidate sensors. They cannot establish logical equivalence, narrower/broader relations, invertibility or safe migration. Embedding-proposed mappings must remain hypotheses until validated/admitted.
Attack: “Translation gives us an interlingua.”
Translation can alter specificity, implicature, information structure, lexicalization and even content. Treat parallel text as alignment evidence plus possible divergence, not proposition identity.
Attack: “Persist every native parser graph forever for perfect audit.”
Could be enormous and unnecessary. Canonical durability needs source, accepted #392 semantics and replay-critical compiler/mapping receipts. Native analyzer blobs may be content-addressed optional artifacts with retention policy.
Attack: “Mapping bundles are executable code.”
They must not be arbitrary callable Prolog. Use closed mapping constructors/validated transforms, analogous to #392's semantic-profile boundary.
Rejected alternatives
Falsifiable conformance fixtures introduced by C
Cross-framework
Multilingual
overlaps/mapping set, not forced hierarchy.Versioning
Genre/source
restartto operation concept. Forbidden: host restart capability grant.Export
Recommended canonical issue changes
#393
Add a first-class Semantic Interoperability Fabric within SCL:
#395
Add interoperability metrics:
#396
The library/tool surface should expose analyzer registration/introspection and external import/export, but must keep analyzers optional and adapters capability-gated. Direct semantic compilation must still work with whatever admitted analyzer subset is available.
symbolic-memory #5 / #8
Durable projections should bind not only compiler/model/schema fingerprints but also the analyzer/mapping bundle versions and replay-critical interoperability receipts. Recompilation under a newer mapping bundle is a new projection generation, not a rewrite of historical semantics.
Preferred C design, compactly
Native representations survive long enough to be auditable. Mapping is knowledge with provenance, not preprocessing glue. A common graph is transport; semantic equivalence must be earned.
Subpass conclusion
Depth 2C is design-complete, not implementation-proven.
The strongest surviving decision is to make semantic interoperability a federated evidence/mapping problem, not a normalization problem. Machine Spirit can ingest English logs, multilingual Wikipedia, news translations, AMR/UCCA/DRS/SRL outputs and changing ontology/parser versions while preserving where correspondences are exact, approximate, broader/narrower, lossy, version-bound or unresolved.
Next required work is #399D — computational realization/scaling/conformance. D should attack whether SCL + SCC + the interoperability fabric can actually run over large corpora with bounded analyzer scheduling, cached mapping transforms, deterministic replay, measurable semantic loss, and honest partial failure without turning the compiler into an unbounded research project per paragraph.
Depth 2 / subpass D — computational realization: content-addressed Semantic Build Graph, demand-driven discourse incrementality, and budgeted conformance
This is #399D only. It completes Depth 2's computational/scaling attack over the A/B/C design. It does not claim an implementation exists or that Machine Spirit acceptance is met.
Research question
A/B/C intentionally made semantic compilation richer: multi-view analyzer evidence (SCL), packed ambiguity, explicit discourse state, Semantic Commitment Calculus (SCC), multilingual/formalism mappings, and loss-accounted interoperability. The resulting danger is obvious: a naive implementation can become computationally absurd.
The D question is therefore:
The answer should make semantic compilation behave more like an incremental compiler/database engine than a sequence of giant NLP prompts.
Primary / authoritative sources inspected
Incremental computation and dependency tracking
Heterogeneous NLP pipelines / shared typed analysis state
Packed ambiguity
LLM serving / constrained structured generation
Content-addressed computation precedent
What A/B/C get right—and where they are computationally underspecified
A/B/C survive:
But these designs still leave implementation traps:
segment changedcould be interpreted asrerun every analyzer + model pass + every later segment.packed ambiguityhas no hard resource contract yet.discourse state is reusabledoes not define invalidation when an earlier span changes or a later span resolves an earlier reference.same config -> same outputis false unless exact candidate artifacts are retained/replayed.Candidate architecture A — eager fixed pipeline
Strengths
Rejection
At corpus scale it repeats expensive work, makes version migration all-or-nothing, gives no principled selective invalidation, and cannot exploit unchanged analyzer outputs across retries/revisions. It also encourages silent quality reduction through ad hoc
skip expensive parserflags.Candidate architecture B — pure streaming/dataflow compiler
Treat all source/analyzer/hypothesis/mapping/export changes as streams and maintain every semantic view incrementally via a Differential/Dataflow-style engine.
Strengths
Weaknesses
semantic_compile/3works with no persistent service.Decision
Use dataflow/IVM concepts for state and dependency propagation, not a mandatory distributed stream processor.
Candidate architecture C — content-addressed static build DAG
Every semantic stage is a build action. Inputs + tool/version/config digests determine a work key; unchanged outputs are reused.
Strengths
Weaknesses
Candidate architecture D — Semantic Build Graph (SBG) + demanded incremental semantic state (preferred)
Combine content-addressed build actions with dynamically recorded semantic dependencies and demand-driven materialization.
The SBG is compiler execution metadata, not a third durable knowledge base and not a second agent scheduler.
prolog-rlm#376already requires one canonical runtime/scheduler/capability boundary. Therefore SBG nodes are typed work-plan data executed through existing async/Futures/cancellation/concurrency machinery (or synchronously for direct library callers). Do not build an independent daemon/scheduler in the semantic compiler.1. Stage identity and content-addressed artifacts
Every cacheable stage has an explicit semantic work identity equivalent to:
Not every field applies to every stage. Only actual dependencies enter the key.
Examples:
Artifacts should be immutable and content-addressable where practical:
The library needs a small cache/artifact SPI, not a mandatory database. A caller may supply:
Correctness cannot depend on cache presence.
2. Determinism classes — stop lying about replay
The compiler must classify stages independently:
This produces three different replay claims:
exact artifact replay
All required output artifacts were retained. Replay consumes those immutable artifacts and performs zero analyzer/model calls for them.
deterministic rebuild
All stages in the requested closure are deterministic under pinned inputs/versions/configs. Re-execution must reproduce artifact digests; mismatch is a conformance failure.
semantic recompile
A nondeterministic/provider stage must be re-invoked because its artifact was not retained. The result may be compared against semantic invariants, but bit-identical or semantically identical output is not promised.
A model request hash is not enough to claim deterministic replay. Provider/model ID, prompt, decoding config and schema identify the request, but the exact returned candidate artifact/response digest is what enables exact historical replay.
This distinction must flow downstream into Symbolic Memory projection lineage.
3. Two-phase execution: parallel local analysis, ordered/dynamic discourse reconciliation
Most expensive local observations are segment-local and parallelizable:
Run these under bounded worker pools/batching where capabilities allow.
Do not claim arbitrary discourse analysis is embarrassingly parallel. Coreference, quotation stacks, temporal anchoring, ellipsis and procedure continuity create cross-segment dependencies.
Recommended execution:
A semantic frontier is not “all previous text.” It is a bounded symbolic continuation containing referenced entities/events/context frames/open obligations plus digests of its dependencies.
4. Incremental discourse invalidation
A simple
segment i changed -> invalidate all later segmentsrule is safe but often wasteful. A simpleonly recompile segment irule is wrong.Use dynamic dependency capture.
Every discourse/SCL decision records which prior frontier items/source hypotheses it consumed. When source or an upstream analysis changes:
This is the self-adjusting-computation/build-system lesson applied to semantic compilation.
Later text can resolve earlier material (cataphora,
the former, retrospective clarification, delayed quote attribution). Handle this through explicit open obligations/backlinks:A later resolution creates a dependency edge back to the affected earlier hypothesis/export. That may invalidate a bounded dependency closure rather than requiring a full-document reset.
Where dependency cycles form, maintain/recompute the affected strongly connected semantic region to a stable digest/fixed point under explicit iteration bounds. Exhaustion yields unresolved/cycle-limit state, never silent convergence.
5. Packed ambiguity has a resource contract
A/B correctly reject Cartesian enumeration, but D requires hard bounds.
Represent alternatives as shared hypergraph/lattice structures. Budget structure, not just “top K parses”:
When an ambiguity budget is exhausted, allowed outcomes are:
interpretation_only/unresolved;Forbidden outcome: select the highest-scoring interpretation only because memory/time ran out and publish it as if ambiguity was resolved.
Track ambiguity debt explicitly, e.g. unresolved material alternative sets and which semantic dimensions they affect. A high-risk unresolved negation/quantifier/authority attribution should count differently from a low-impact lexical synonym ambiguity.
6. Analyzer scheduling is a typed compile plan, not a hidden heuristic
Build on #399C's analyzer manifests.
Each analyzer advertises at least:
Planner API direction:
Scheduling policy:
V1 should prefer inspectable rule/cost tables over a learned router. A learned scheduler may later recommend plans, but cannot silently lower a required semantic-quality contract.
Example risk classes include:
A “fast” budget may legally return more unresolved content. It may not return less truthful metadata about what remains unresolved.
7. LLM stage throughput
Model-backed induction/reconciliation should support:
PagedAttention/vLLM and modern structured-generation engines show why the runtime should expose batching/constrained-generation capability, but the semantic compiler must not depend on one serving engine.
Model inference remains the expensive exception. Cache exact candidate artifacts aggressively subject to privacy/retention policy.
8. Publication barriers and partial failure
Partial failure must be typed by semantic region/dimension.
Conceptual outcome:
Possible status semantics:
A caller can require a completeness profile, e.g.
no_material_gaps([attribution,negation,quantifier_scope]). If an analyzer/provider fails on one of those dimensions, the compiler must not publish a clean-looking package.Successful deterministic artifacts from the run remain reusable on retry. Failure of one deep analyzer must not force rerunning tokenization/source adapters.
9. Streaming / corpus revisions are semantic generations, not destructive updates
For logs/news/Wikipedia revisions, source arrival is an append/update stream. The semantic compiler should accept deltas but emit immutable compile generations.
A source revision may:
The SBG computes the affected work closure. The resulting semantic package generation links the prior generation plus source delta/compile receipt. Symbolic Memory decides durable lifecycle/supersession; Prolog-RLM does not mutate historical memory.
This follows the Dataflow lesson that late data/revisions are normal rather than exceptional.
10. Execution metrics: quality, cost and invalidation together
Do not benchmark only
documents/sec.Every conformance/benchmark run should report at least:
throughput / latency
resource / model cost
incremental behavior
semantic quality under budget
Reuse #395 metrics plus:
The important curve is semantic quality vs cost/latency, not maximum raw throughput.
11. Deterministic replay / conformance fixtures
Depth 2D requires at least these falsifiable fixtures.
D1 — exact no-op replay
Compile a corpus with retained artifacts. Re-run identical source/analyzer/schema/mapping/config.
Required:
model_calls = 0for retained model artifacts;D2 — one-span local edit
Change one independent paragraph near the end of a long document.
Required:
D3 — upstream discourse edit
Change the referent/speaker in an early segment that later segments actually reference.
Required:
D4 — late resolution of earlier ambiguity
Later segment resolves an earlier pronoun/quotation/source ambiguity.
Required:
D5 — analyzer version bump
Bump one analyzer version only.
Required:
D6 — mapping bundle bump
Change only multilingual/ontology mapping rules.
Required:
D7 — provider nondeterminism honesty
Delete a retained LLM candidate artifact but keep its request fingerprint.
Required:
semantic_recompile/equivalent, not exact replay;D8 — ambiguity explosion
Adversarial syntactic/coreference/scope fixture produces combinatorial interpretations.
Required:
D9 — partial analyzer failure
Deep attribution analyzer fails for one region while cheap analyzers succeed.
Required:
D10 — cancellation/resume
Cancel a large document after arbitrary completed nodes.
Required:
D11 — parallelism determinism
Compile identical corpus with different worker counts/interleavings.
Required:
D12 — quality-budget monotonic honesty
Run
cheap,standard,deepbudgets.Required:
D13 — full-vs-incremental differential oracle
For every edit fixture, compare incremental result with a clean from-scratch compile under the same retained nondeterministic artifacts.
Required: semantic equivalence under #395 invariants. This is the core correctness oracle for change propagation.
12. Repository ownership boundaries
prolog-rlmOwns:
It does not own a durable corpus database or a second global scheduler.
symbolic-memoryOwns:
Symbolic Memory may provide the cache/CAS implementation to upstream calls, but the upstream compiler remains usable with an in-memory/no-persistence implementation.
existing expert/runtime plane
#376-#387remains the one expert/runtime authority/scheduling plane. Semantic compile planning may be implemented by a local Prolog expert/typed plan but must not create a competing expert registry, authority model, effect runner or process scheduler.13. Safety / authority implications
first answer winsis forbidden).Adversarial review
Attack: “This is just Bazel for NLP; overengineered.”
A simple cache would be enough for independent sentence extraction. It is not enough once cross-segment discourse and mapping dependencies can change old outputs. The dynamic dependency graph is the minimum honest mechanism if we want incremental edits without whole-document rebuilds.
Attack: “Just stream everything through Differential Dataflow.”
Excellent for algebraic/incremental views, but model calls and heterogeneous external parsers have side effects, cost, nondeterminism and non-relational payloads. The library must also run without a distributed service. Use delta/IVM concepts in the execution model rather than imposing the substrate.
Attack: “Hash request + model ID and call that reproducible.”
False. A provider may change weights, kernels, serving policy, hidden system prompts, nondeterministic sampling or backend behavior while keeping a public model name. Exact replay requires retained output artifacts; otherwise only request reproduction + semantic re-evaluation can be claimed.
Attack: “Top-K ambiguity is good enough.”
Not as a semantic contract. Top-K is acceptable as an expensive-analyzer scheduling heuristic only if pruned material alternatives remain represented as unresolved/loss debt. Resource pressure cannot manufacture certainty.
Attack: “Compile every segment independently then resolve globally.”
Works for local predicates but fails on discourse-dependent meaning and can make global reconciliation itself the giant expensive pass. Explicit frontier/dependency state permits bounded incremental reconciliation and gives a real invalidation story.
Attack: “Make every stage incremental.”
Bad trade. Some analyzers are cheap enough to rerun; some external models expose no delta API. Incrementality belongs at artifact boundaries. A stage may remain from-scratch internally while its immutable output is reused globally.
Attack: “Learn the analyzer router so it optimizes cost automatically.”
Potential future work, but v1 needs inspectable deterministic scheduling constraints. A learned router that chooses not to inspect negation/attribution is itself a semantic failure risk. Learned scheduling must be subordinate to required semantic profiles and conformance.
Preferred D decision
Adopt the Semantic Build Graph + demanded incremental semantic state as the execution architecture for #393/#396.
Core invariants:
Unresolved questions preserved for later implementation/depths
Those are implementation/research follow-ups, not reasons to weaken the D contract.
Depth-2 conclusion
A/B/C/D now form one coherent architecture:
Depth 2 is design-complete, implementation-unproven. The next Machine Spirit research depth should be #400A (epistemics / truth maintenance / non-monotonic reasoning), after canonical issue handoffs are reconciled.