[semantic-memory] Append-only general semantic ledger: arbitrary IR records, temporal/current views, conflict, supersession, and provenance #43
Labels
No labels
accessibility
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
nsaspy/symbolic-memory#43
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Parent: #4
Depends on: #1, #5
Upstream IR: lost-rob0t/prolog-rlm#392
Goal
Persist the full general semantic IR as immutable append-only knowledge history while exposing efficient namespace-scoped current/history/world-model views.
This is not a fact/rule-only table. Storage must preserve the semantic shape of every supported upstream record: entities/types, relations, rules, defaults/exceptions, procedures, events/states, temporal/causal/modal/constraint knowledge, attributed claims, hypotheses, preferences, examples, and extension vocabulary.
Core invariant
Never rewrite an old semantic record in place to make it look current.
Represent evolution explicitly:
Exact vocabulary follows repository conventions.
Required record identity
Every semantic record must bind to:
Preserve semantic structure
Do not denormalize away distinctions needed for reasoning, especially:
Indexes/materialized tables may optimize these shapes but are rebuildable caches, not the source of truth.
Current/history/world-model views
Provide general semantics equivalent to:
Current views resolve retraction/supersession and temporal validity while preserving explicit conflict/uncertainty. A conflict does not automatically pick a winner.
Entity and relation indexing
Index for efficient lookup by at least:
Do not require every extension relation to get a new storage migration.
Temporal/current semantics
A semantic record may be historically valid but no longer current. Current views must use upstream temporal/validity semantics rather than assuming latest append wins.
Examples:
Both remain in history; current query returns B for current time while a May query returns A.
Conflict and plurality
Preserve:
Conflict links are semantic events/relations, not destructive reconciliation.
Trust semantics
Source trust and semantic interpretation trust are independent. Model-derived semantics remain model-derived even when attached to a trusted user's source memory.
Remembered normative/policy statements remain epistemic records and never become execution authority.
MACHINE-SPIRIT #400A epistemic ledger requirements
Depth 3A establishes that Symbolic Memory must not persist one mutable authoritative
truthvalue for each proposition.Persist/reconstruct a canonical Epistemic Support Graph (ESG) over the semantic ledger:
Required durable distinctions:
At minimum, preserve enough immutable data to rebuild proposition support states equivalent to:
Those are evidence/support summaries, not universal truth values.
Durable epistemic records should include, where applicable:
Ordinary current support indexes, warranted-result caches, WFS tables, answer-set results, argumentation acceptance sets and revision projections are rebuildable derived state unless a promoted result requires a replay receipt.
A source withdrawal deactivates that source's current positive support but does not synthesize negative support. An explicit correction to ¬P may create a new negative support assertion in addition to the withdrawal/supersession relation.
Two independent derivations of P must remain independently addressable so invalidating one premise removes only the dependent derivation.
ATMS-style assumption environments are useful but can grow exponentially; do not require global environment-label materialization. Storage must support query-scoped/bounded environment reconstruction, nogood indexing and subsumption.
Extension durability
Persist unknown/new domain vocabulary using stable versioned references without requiring core code changes. Unknown extensions remain queryable data but cannot become executable host predicates.
Acceptance
Refs #4 #5 and prolog-rlm#392-#394 #397 #400.
#398B storage consequence: the append-only semantic ledger should not flatten
context/theory, existential witness, or justification structures into generic assertion metadata. Index current/history views by theory/context/profile; preserve source assertions independently from domain-theory promotions; persist witness scope/origin without promoting labelled unknowns to global entities; keep canonical dependency/justification records sufficient for explanation/replay while ordinary materialized consequences remain rebuildable caches. See upstreamprolog-rlm#392updated Machine Spirit #398B section.Machine Spirit depth-1 storage consequences (#398C/#398D)
The general semantic ledger needs to persist the semantic theory graph while keeping compiled execution state rebuildable.
Persist canonically
Keep rebuildable/materialized
Materialized state should be keyed to the semantic/theory generations it depends on and report freshness. Appending/retracting/superseding a theory mapping, bridge, assertion or rule invalidates dependent derived state without mutating historical semantic records.
A mapping/alignment change must also remain historical knowledge: do not rewrite old source records into the latest ontology vocabulary.
This directly supports replay of old LLM logs/web/news/wiki semantic projections under the theory/schema version with which they were originally interpreted.
Refs upstream prolog-rlm #397/#398/#392/#394/#396.
Machine Spirit #399B ledger consequence — preserve semantic force, not just proposition payload
The append-only ledger must be able to retain exported semantic records whose proposition payloads may look similar while their linguistic force and derivation are materially different.
Examples that must remain distinguishable forever:
Corrections/new compiler interpretations append new projection lineage/supersession rather than mutating older records. Current/query views may choose which classes participate, but the ledger cannot flatten them into one
fact(P, confidence)representation. In particular, pragmatic candidates and generics must never silently become ordinary world facts merely because they were durably stored.MACHINE-SPIRIT #400B storage handoff — typed defeat, dependence, bitemporal epistemics
Depth 3B refines the #400A ESG persistence contract. Persist enough immutable history to rebuild, when applicable:
rebut,undermine,undercut, deterministicinvalidate, with target refs and provenance;copied_from,syndicates,derived_from,shared_origin, plus evidence and certainty/status for the dependence claim;Hard invariants:
Ten copied/syndicated articles may remain ten immutable assertion episodes while exposing one or a few independent evidence-origin components under an explicit dependence policy.
Full B research and adversarial fixtures are on prolog-rlm#400. #400C/#400D still own cross-profile composition and execution/scaling details.
MACHINE-SPIRIT #400C durability handoff — federated epistemic provenance
Depth 3C adds a durable interoperability requirement: Symbolic Memory must preserve enough information to replay cross-theory epistemic federation without ever materializing one merged global truth graph.
Persist/reconstruct, where promoted/durable:
evidence_only, skeptical-warrant-as-support, credulous candidate, attributed acceptance, revision input, argument import, inert reference, etc.);exact_preserving,sound_but_incomplete,lossy_declared,status_only,noncomposable,unknown_compatibility);known_at(T)replays the mappings and bridge policies actually available at T;Hard invariants:
Ordinary local/federated warrant caches, bridge closures, MCS equilibria, model products, and argumentation acceptance sets remain rebuildable derived state unless a promoted result requires a replay receipt.
Acceptance should include: two memories derived from one original source remain one independent evidence component after federation; identity-induced conflict can be removed by changing the identity assumption without deleting either original support; historic
known_atuses historic mapping/bridge state; local no-equilibrium/failure does not corrupt unrelated ledger state.MACHINE-SPIRIT #400D handoff — derived epistemic cache/frontier requirements
Depth 3D makes Symbolic Memory the durable source for canonical epistemic history while keeping federation execution state rebuildable.
Persist enough immutable lineage/index data to rebuild query-local epistemic frontiers and invalidate derived state by dependency:
Do not persist one global model product or mandatory ATMS environment closure.
Derived/cache objects may include:
They are disposable unless a promoted decision/action requires a replay receipt.
Cache/frontier identity must include theory/profile/evidence frontier,
valid_at+known_atcontext, interface signature, identity/dependence generation, bridge/mapping/closure generation and translator/backend contract version where material.Propagation-stop invariant: if a changed local theory recomputes to the same projected bridge-interface digest, downstream derived state remains valid; internal churn must not force global invalidation.
Historical replay must remain possible after later corrections/mappings/profile changes.
known_at(Tpast)uses the historically available evidence/mapping/profile state, not today's cache.Add acceptance for cold/warm/rebuilt and incremental/full differential equivalence, identity/bridge/profile scoped invalidation, same-interface propagation stop, and no destructive cache writes to canonical ledger history.
Full design and sources are on prolog-rlm#400D.
Machine Spirit #401A procedural-memory handoff
Depth 4A distinguishes reusable procedural knowledge from episodic procedural experience and from run-time control state.
Durable ledger should be able to preserve/version, where upstream #392 exposes them:
Hard storage distinctions:
Do not make transient Rete match memories, coordination-board agendas, planner frontier nodes, ordinary decomposition caches or active diagnostic search state canonical semantic truth; those are rebuildable/run-local unless a replay/promotion receipt specifically requires retention.
Verified cases can later support CBR retrieval and #404 candidate procedure/rule promotion, but Symbolic Memory must not itself turn successful cases into trusted procedural policy. Full design and A1-A20 fixtures are on prolog-rlm#401.
Machine Spirit #401B downstream durability handoff — procedural policy history
Depth 4B refines durable procedural memory. Preserve append-only history sufficient to reconstruct why a monitored procedure was selected, how it behaved, and whether later evidence suggests a method defect, without storing one mutable
current best proceduretruth value.Durable records should support, where material:
Ordinary live execution frontiers, schedules, guard-match caches and current resource leases are run-local/rebuildable unless retained in a final execution/audit receipt.
Hard durability rules:
Promotion/supersession of trusted procedural knowledge remains a later governed operation (#404 upstream), never an automatic consequence of case retention.
Full rationale/sources and B1-B28 fixtures: lost-rob0t/prolog-rlm#401 Depth-4B.
Machine Spirit #401C handoff — durable procedural interoperability lineage
Depth 4C (prolog-rlm#401) selects PIFF/PSC for heterogeneous procedure federation. The append-only semantic ledger should preserve enough history to reconstruct procedure interoperability without rewriting source-native meaning:
exact_on, refine/abstract/project/lossy/incompatible/etc.);Hard invariant: mapping/adapter upgrades create new projection lineage; old executions still resolve the exact historical mapping. Concrete current capability/authority remains runtime/host state, not durable memory truth. See prolog-rlm#401 C1-C28; #401D will settle cache/incremental execution details.
Upstream Machine Spirit #401D handoff — procedural execution/replay durability
Prolog-RLM Depth 4 is now design-complete with DPEC — Demand-Driven Procedural Execution & Conformance Graph.
Durability boundary for this ledger:
Durable / append-only where material
Rebuildable derived state by default
Hard invariant:
Changing PSC/bridge/binding/expert versions creates new projection/run lineage and never rewrites prior history. If DPEC recomputation yields the same selected observable-interface digest, downstream derived invalidation may stop, but original version lineage remains inspectable.
Full #401D research and D1-D34 conformance fixtures are on lost-rob0t/prolog-rlm#401.
Machine Spirit #402A durable-ledger handoff
Depth 5A is now complete on
prolog-rlm#402and strengthens this issue's storage contract without changing repository ownership.Adopt the BSLPF foundation: immutable/versioned semantic objects + an authoritative append-only bitemporal semantic ledger + rebuildable/materialized projections.
Concrete #6 deltas for later implementation/design reconciliation:
known_at/ transaction time comes from immutable commit history; worldvalid_atremains an independent semantic time axis;symbolic_memory_storage.plimplementation is a bootstrap backend only, not the semantic source-of-truth architecture.This intentionally preserves the older
SYMBOLIC-MEMORY-RESEARCH-006-persistence-and-concurrency.orgas historical backend/bootstrap research while superseding its earlier position that complete replayable semantic event history could be optional at the world-model contract layer.Refs: lost-rob0t/prolog-rlm#402 (#402A), #392, #400, #401, and this issue's parent #4.
MACHINE-SPIRIT #402B canonical durability refinement — GRIE
Depth 5B refines this ledger contract with GRIE — Governed Redaction & Integrity Envelope. Treat this as a hard clarification of
append-only:Required durable distinctions now include:
#6 owns durable lifecycle/admission/integrity state sufficient to represent: retention/encryption-scoped payload envelopes; authenticated admission receipts; redaction/sanitization completion (
complete|partial|pending|blocked|failed); replay capability (exact|semantic_only|commitment_only|redacted|corrupt|unavailable); versioned schema/upcaster interpretation receipts; integrity checkpoints/recovery lineage; and duplicate classes distinguishing transport retry from independent/correlated evidence.A redaction receipt must not copy sensitive plaintext into the audit trail, and global content hashes/dedup may not leak equality across authority-isolated namespaces. A corrupted/stale projection is disposable and may never repair/replace canonical ledger authority. Canonical segment corruption is an explicit integrity failure requiring authenticated recovery lineage.
Physical erasure/sanitization remains a backend capability: a tombstone is never proof bytes are gone from replicas/backups/media. See upstream prolog-rlm#402B for the full primary research and B1-B30 conformance fixtures.
MACHINE-SPIRIT #402C handoff — Causal Ledger Federation Mesh
Depth 5C requires #6's append-only semantic ledger contract to distinguish a logical ledger identity from physical replicas/backups/mirrors and from genuinely independent ledgers.
Add durable/reconstructible support for typed records equivalent to:
Hard requirements:
CRDT/consensus replication may implement an explicit replica group but is not the semantic merge rule for independent ledgers. Full research + C1-C34 fixtures: prolog-rlm#402. Refs #4 #7 #9 #10.
Machine Spirit #402D handoff: adopt SAFIRE as the computational durability target beneath this issue. Canonical ledger state should become segmented/append-only with dense per-ledger sequence, sparse/hierarchical federation cuts, exact projection dependency generations, atomic projection checkpoints, explicit freshness, typed replica profiles, bounded causal traversal, anti-entropy inside replica/mirror relationships only, and cold/warm/rebuilt/incremental differential conformance. The current whole-snapshot rewrite backend remains bootstrap-only. See prolog-rlm#402 D1-D40. Hard boundary: projections never repair canonical ledger authority; Bloom/index hits never become semantic evidence; replicas never manufacture corroboration.
Upstream Machine Spirit #403B handoff — retrieval inputs from durable provenance/dependence
prolog-rlm#403B/OATH-TAPS depends on the durable world model preserving enough canonical metadata for adversarially honest retrieval:trusted,pinned,priority, etc.).Indexes remain rebuildable sensors. An index hit absent from the exact admitted ledger cut is an index inconsistency, not evidence. Refs lost-rob0t/prolog-rlm#403 #400 #402 and symbolic-memory#7/#10.
MACHINE-SPIRIT #404A handoff — EPOCH learning ledger
Depth 7A requires the durable world model to persist learning history without collapsing experience into trusted rules.
Canonical durable classes should be able to represent:
Experience must bind to exact semantic/federation cut, epistemic profile, procedure/strategy/expert versions, observations, VERIFY/Review/diagnosis/resource/provider receipts and replay lineage. Derived strategy-performance aggregates/indexes may remain rebuildable; the source episodes and material promotion lifecycle are durable.
Hard distinction:
Correlated/mirrored episodes must remain distinguishable for promotion-independence accounting. Ref upstream prolog-rlm#404A EPOCH.
Machine Spirit #404B / CITADEL-EPOCH durable-learning handoff
The append-only semantic ledger must persist learning/promotion history as first-class semantic/control provenance rather than one mutable
confidencefield.Persist/reconstruct, where applicable:
generation,selection,validation,audit,post_admission) plus origin/source-dependence groups;History remains append-only/bitemporal: confirmed drift or a new counterexample changes the current admission projection but does not rewrite what was promoted previously or why. Derived aggregate scores/current promotion views remain rebuildable state. Refs prolog-rlm#404B CITADEL-EPOCH.
Machine Spirit #404C handoff — durable learned-artifact federation
PACT-EPOCH requires #6 to persist imported Learning Artifact Contracts (LACs) and their lifecycle append-only, including source/local promotion states side by side, learner kind, source environment/applicability assumptions, evidence-use summaries, source promotion/evaluation attestations, transfer assessments, drift/calibration epochs and opaque origin/dependence claims.
Hard storage invariant:
Imported source receipts remain immutable evidence; target promotion creates a distinct local receipt/version. Retraction/drift/mapping changes invalidate affected transfer projections without rewriting source history. Full C design: prolog-rlm#404C.
Implementation decomposition
#6 remains the canonical semantic-ledger/current-history requirement. It is now executed through:
#6 should not absorb the full scalable backend or query stack in one PR. Canonical history remains authoritative; all current/support/query indexes are rebuildable derived state.
MACHINE-SPIRIT #404D durable-ledger refinement
Depth 7D extends the append-only ledger contract to learned-state history. Do not add one mutable
learned=true/admitted=trueproperty to semantic records.Persist direct lifecycle/evidence events sufficient to reconstruct:
Current learned status/exposure closure/validation eligibility remain rebuildable views over canonical history. Historical queries must resolve the exact candidate/policy/verifier/attestation versions active at the requested frontier.
A promotion transition is valid only against its frozen snapshot/dependency generations; a later source withdrawal or mapping correction appends invalidation/suspension rather than rewriting the old decision. Refs prolog-rlm#404D and symbolic-memory#16/#35/#36.
prolog-rlm#405B AEGIS-MS durable handoff: persist direct influence/control-generation/replay-lineage edges needed to reconstruct end-to-end acceptance history, alongside existing source/compiler/logical/lifecycle/profile provenance. Canonical state should store direct edges/events; transitive influence/acceptance closure remains query-scoped/rebuildable and generation-keyed. Material changes to identity, redaction, integrity, admission/control refs or source lifecycle must invalidate dependent derived acceptance/query artifacts. This is provenance/history storage only: Symbolic Memory does not become an authority policy engine or final verifier.
Machine Spirit #405C / CONCORDAT durable-ledger handoff
Cross-domain semantic history must preserve the origin domain + source ledger/frontier + bridge/appraisal generations, not flatten imported records into indistinguishable local assertions.
Persist enough to reconstruct:
Hard invariants from #405C:
Canonical research: lost-rob0t/prolog-rlm#405C.