[MACHINE-SPIRIT 5/8] Durable world-model architecture: event sourcing, append-only semantics, incremental inference, identity and ontology evolution #396

Closed
opened 2026-09-08 01:29:18 +00:00 by nsaspy · 5 comments
Owner

Parent: #397
Related: #388 #392 #394
Downstream: symbolic-memory#4 #6 #10

Distinct research mandate

Treat persistence and evolving world models as the primary problem: event sourcing, bitemporal data, append-only ledgers, materialized views, incremental/differential computation, dependency-directed invalidation, provenance indexing, entity/ontology evolution, scalable log/index storage, integrity, redaction, governed retention and federation.

Four-subpass status — DESIGN-COMPLETE

A — BSLPF: COMPLETE

Bitemporal Semantic Ledger + Projection Fabric establishes immutable/versioned semantic objects, an authoritative append-only bitemporal semantic ledger, atomic/idempotent commits, exact local frontiers, crash/replay semantics, reversible identity/schema/ontology mappings and rebuildable projections/checkpoints/indexes.

Hard distinction:

semantic object identity
    != assertion/support episode identity
    != ledger event identity
    != commit/batch identity
    != materialized-view identity

Valid time and transaction/knowledge time remain independent. Current/support/warrant/entity/procedure/retrieval indexes are rebuildable views, never canonical truth. Full A research and A1-A24 fixtures are preserved in the #402A comment.

B — GRIE: COMPLETE

Governed Redaction & Integrity Envelope corrects the naive assumption that append-only history implies permanent payload retrievability.

append-only logical commitment history
    != permanent payload retrievability

GRIE separates correction, withdrawal, invalidation, quarantine, archive, access revocation, redaction, cryptographic erase and physical purge; adds authenticated admission receipts, explicit sanitization states, replay capability classes, schema/upcaster interpretation lineage, duplicate/corroboration/dependence taxonomy, fail-closed projections, canonical-ledger integrity failure/recovery lineage, privacy-scoped content commitments and shared provenance DAGs. Full B mechanisms and B1-B30 fixtures are preserved in #402B.

C — CLFM: COMPLETE

Causal Ledger Federation Mesh keeps independent ledgers as separate authority/order/retention/integrity domains.

Hard distinctions include:

local commit order != cross-ledger causal order
replica != backup != mirror != independent ledger
replica convergence != semantic reconciliation
receipt/inclusion proof != proposition truth
redaction notice != remote sanitization completion
federation cut != globally serializable snapshot

Cross-ledger causality comes only from evidence-backed references/import observations/checkpoint ancestry. Federation cuts name exact per-ledger frontiers/checkpoints and mapping/admission/redaction/integrity generations. Replica groups cannot manufacture independent corroboration; divergent single-ledger ancestry is an integrity/fork conflict. External PROV/SCITT-style material interoperates through typed loss-aware adapters. Full C design and C1-C34 fixtures are preserved in #402C.

D — SAFIRE: COMPLETE

Adopt SAFIRE — Sparse Append-only Frontier & Incremental Replay Engine as the computational realization of BSLPF + GRIE + CLFM.

immutable semantic objects
          +
segmented per-ledger append logs
          +
commit/checkpoint manifests
          ↓
sparse / hierarchical frontier descriptors
          ↓
delta feeds
          ↓
versioned projection DAG
          ↓
current / epistemic / procedural / retrieval indexes

D establishes:

  • dense monotonically ordered coordinates inside each logical ledger; no universal global vector clock across independent ledgers;
  • sparse federation cuts, optionally represented by immutable shared hierarchical frontier manifests whose exact leaves remain reconstructible;
  • DVV/ITC-style causality metadata only inside declared optimistic replica profiles, never as federation truth/order;
  • explicit causal-edge indexes and demanded bounded traversal; no mandatory global transitive-closure materialization;
  • probabilistic Bloom-style indexes as negative storage accelerators only — never semantic/causal/authority evidence;
  • immutable committed segments + persistent cursor/range indexes so billion-record operation never requires whole-store SWI-Prolog heap materialization;
  • exact projection dependency generations and atomic projection checkpoint publication;
  • direct delta maintenance for simple indexes and optional DBSP/differential-dataflow-style incrementalization for recursive/rich projections;
  • generation-bump + stale-partition + bounded demanded/background rebuild for huge invalidations instead of eager rewriting of all derived rows;
  • explicit projection freshness (exact, lagging, stale_dependency, rebuilding, integrity_blocked, etc.) carrying requested/materialized cuts;
  • distinct single_writer_local, consensus_replica_group, optimistic_replica_group, and mirror/archive profiles;
  • Merkle/range anti-entropy only inside declared replica/mirror relationships; incompatible single-history ancestry fails as fork/integrity conflict rather than auto-union;
  • deterministic semantic crash boundaries: durable ledger commit always outranks projection state, and unpublished/torn projections remain disposable;
  • partition behavior determined by replica/federation profile rather than timestamps or guesswork;
  • redaction obligations executed as durable idempotent attempt/receipt histories without upgrading remote ACK to sanitization proof;
  • shared provenance DAGs and demanded explanation expansion;
  • deterministic replay receipts fingerprinting exact cuts/manifests, semantic objects, schema/upcaster/mapping/admission/redaction/integrity/profile generations and projection implementation versions;
  • hard differential oracle: cold rebuild == warm query == cache-drop/rebuild == incrementally maintained result for the same cut/dependency generations;
  • structured resource_limited/lag/integrity failure instead of implicit LLM fallback.

Primary D evidence includes O'Neil et al. 1996 LSM-trees, Bigtable 2006, Dynamo 2007, DVV 2010/2011, Interval Tree Clocks 2008, Raft 2014, Differential Dataflow 2013, DBSP 2023 and Scalable Bloom Filters 2007. Full algorithms, candidate architectures, complexity analysis, typed APIs, rejected shortcuts and D1-D40 falsifiable fixtures are preserved in the #402D comment.

Reconciled Depth-5 architecture

BSLPF
  canonical bitemporal history + rebuildable projections
        ↓
GRIE
  governed payload/redaction/integrity/admission/replay semantics
        ↓
CLFM
  independent-ledger causality/federation/cuts/remote obligations
        ↓
SAFIRE
  segmented storage + sparse cuts + projection DAG + typed replica profiles
  + anti-entropy + bounded indexing + deterministic replay/conformance

Repository boundary

symbolic-memory owns physical/logical storage adapters, segments/checkpoints/frontiers, replica/anti-entropy profiles, CLFM persistence/cuts, GRIE redaction maintenance, projection freshness/invalidation, scalable indexes/cursors and storage failure-injection conformance.

prolog-rlm owns #392 semantic meaning/schema/profile identity, #394 reasoning/epistemic semantics, #395 semantic conformance and DPEC procedural execution semantics. Dependency remains symbolic-memory -> prolog-rlm; SAFIRE is neither a new reasoner nor a second plan scheduler.

Design-completion boundary

Depth 5 is design-complete only. The current whole-snapshot symbolic_memory_storage.pl backend does not implement SAFIRE. No claim of billion-record performance, anti-entropy correctness, distributed serializability, remote sanitization, deterministic crash recovery, or Machine Spirit acceptance follows from this design prose.

Unresolved implementation inputs

  • first physical backend profile and exact segment encoding;
  • manifest/cut canonicalization format;
  • compaction under governed redaction;
  • incremental-index vs DBSP/differential threshold;
  • first optimistic replica/consensus implementation choices;
  • anti-entropy authentication/range-tree details;
  • measured append/query/rebuild/provenance benchmark thresholds;
  • backup/restore/integrity-checkpoint retention policy;
  • deployment-specific deletion interaction with immutable backups.

Advance rule

A/B/C/D are complete and reconciled. This issue may close as completed for design completion only.

Exact next research work: Depth 6 / #403A — retrieval / symbolic-attention foundations.

Do not start #403B/C/D until #403A is durably complete.

Parent: #397 Related: #388 #392 #394 Downstream: symbolic-memory#4 #6 #10 ## Distinct research mandate Treat persistence and evolving world models as the primary problem: event sourcing, bitemporal data, append-only ledgers, materialized views, incremental/differential computation, dependency-directed invalidation, provenance indexing, entity/ontology evolution, scalable log/index storage, integrity, redaction, governed retention and federation. ## Four-subpass status — DESIGN-COMPLETE ### A — BSLPF: COMPLETE **Bitemporal Semantic Ledger + Projection Fabric** establishes immutable/versioned semantic objects, an authoritative append-only bitemporal semantic ledger, atomic/idempotent commits, exact local frontiers, crash/replay semantics, reversible identity/schema/ontology mappings and rebuildable projections/checkpoints/indexes. Hard distinction: ```text semantic object identity != assertion/support episode identity != ledger event identity != commit/batch identity != materialized-view identity ``` Valid time and transaction/knowledge time remain independent. Current/support/warrant/entity/procedure/retrieval indexes are rebuildable views, never canonical truth. Full A research and A1-A24 fixtures are preserved in the #402A comment. ### B — GRIE: COMPLETE **Governed Redaction & Integrity Envelope** corrects the naive assumption that append-only history implies permanent payload retrievability. ```text append-only logical commitment history != permanent payload retrievability ``` GRIE separates correction, withdrawal, invalidation, quarantine, archive, access revocation, redaction, cryptographic erase and physical purge; adds authenticated admission receipts, explicit sanitization states, replay capability classes, schema/upcaster interpretation lineage, duplicate/corroboration/dependence taxonomy, fail-closed projections, canonical-ledger integrity failure/recovery lineage, privacy-scoped content commitments and shared provenance DAGs. Full B mechanisms and B1-B30 fixtures are preserved in #402B. ### C — CLFM: COMPLETE **Causal Ledger Federation Mesh** keeps independent ledgers as separate authority/order/retention/integrity domains. Hard distinctions include: ```text local commit order != cross-ledger causal order replica != backup != mirror != independent ledger replica convergence != semantic reconciliation receipt/inclusion proof != proposition truth redaction notice != remote sanitization completion federation cut != globally serializable snapshot ``` Cross-ledger causality comes only from evidence-backed references/import observations/checkpoint ancestry. Federation cuts name exact per-ledger frontiers/checkpoints and mapping/admission/redaction/integrity generations. Replica groups cannot manufacture independent corroboration; divergent single-ledger ancestry is an integrity/fork conflict. External PROV/SCITT-style material interoperates through typed loss-aware adapters. Full C design and C1-C34 fixtures are preserved in #402C. ### D — SAFIRE: COMPLETE Adopt **SAFIRE — Sparse Append-only Frontier & Incremental Replay Engine** as the computational realization of BSLPF + GRIE + CLFM. ```text immutable semantic objects + segmented per-ledger append logs + commit/checkpoint manifests ↓ sparse / hierarchical frontier descriptors ↓ delta feeds ↓ versioned projection DAG ↓ current / epistemic / procedural / retrieval indexes ``` D establishes: - dense monotonically ordered coordinates inside each logical ledger; no universal global vector clock across independent ledgers; - sparse federation cuts, optionally represented by immutable shared hierarchical frontier manifests whose exact leaves remain reconstructible; - DVV/ITC-style causality metadata only inside declared optimistic replica profiles, never as federation truth/order; - explicit causal-edge indexes and demanded bounded traversal; no mandatory global transitive-closure materialization; - probabilistic Bloom-style indexes as negative storage accelerators only — never semantic/causal/authority evidence; - immutable committed segments + persistent cursor/range indexes so billion-record operation never requires whole-store SWI-Prolog heap materialization; - exact projection dependency generations and atomic projection checkpoint publication; - direct delta maintenance for simple indexes and optional DBSP/differential-dataflow-style incrementalization for recursive/rich projections; - generation-bump + stale-partition + bounded demanded/background rebuild for huge invalidations instead of eager rewriting of all derived rows; - explicit projection freshness (`exact`, `lagging`, `stale_dependency`, `rebuilding`, `integrity_blocked`, etc.) carrying requested/materialized cuts; - distinct `single_writer_local`, `consensus_replica_group`, `optimistic_replica_group`, and `mirror/archive` profiles; - Merkle/range anti-entropy only inside declared replica/mirror relationships; incompatible single-history ancestry fails as fork/integrity conflict rather than auto-union; - deterministic semantic crash boundaries: durable ledger commit always outranks projection state, and unpublished/torn projections remain disposable; - partition behavior determined by replica/federation profile rather than timestamps or guesswork; - redaction obligations executed as durable idempotent attempt/receipt histories without upgrading remote ACK to sanitization proof; - shared provenance DAGs and demanded explanation expansion; - deterministic replay receipts fingerprinting exact cuts/manifests, semantic objects, schema/upcaster/mapping/admission/redaction/integrity/profile generations and projection implementation versions; - hard differential oracle: cold rebuild == warm query == cache-drop/rebuild == incrementally maintained result for the same cut/dependency generations; - structured `resource_limited`/lag/integrity failure instead of implicit LLM fallback. Primary D evidence includes O'Neil et al. 1996 LSM-trees, Bigtable 2006, Dynamo 2007, DVV 2010/2011, Interval Tree Clocks 2008, Raft 2014, Differential Dataflow 2013, DBSP 2023 and Scalable Bloom Filters 2007. Full algorithms, candidate architectures, complexity analysis, typed APIs, rejected shortcuts and **D1-D40 falsifiable fixtures** are preserved in the #402D comment. ## Reconciled Depth-5 architecture ```text BSLPF canonical bitemporal history + rebuildable projections ↓ GRIE governed payload/redaction/integrity/admission/replay semantics ↓ CLFM independent-ledger causality/federation/cuts/remote obligations ↓ SAFIRE segmented storage + sparse cuts + projection DAG + typed replica profiles + anti-entropy + bounded indexing + deterministic replay/conformance ``` ## Repository boundary `symbolic-memory` owns physical/logical storage adapters, segments/checkpoints/frontiers, replica/anti-entropy profiles, CLFM persistence/cuts, GRIE redaction maintenance, projection freshness/invalidation, scalable indexes/cursors and storage failure-injection conformance. `prolog-rlm` owns #392 semantic meaning/schema/profile identity, #394 reasoning/epistemic semantics, #395 semantic conformance and DPEC procedural execution semantics. Dependency remains `symbolic-memory -> prolog-rlm`; SAFIRE is neither a new reasoner nor a second plan scheduler. ## Design-completion boundary **Depth 5 is design-complete only.** The current whole-snapshot `symbolic_memory_storage.pl` backend does not implement SAFIRE. No claim of billion-record performance, anti-entropy correctness, distributed serializability, remote sanitization, deterministic crash recovery, or Machine Spirit acceptance follows from this design prose. ## Unresolved implementation inputs - first physical backend profile and exact segment encoding; - manifest/cut canonicalization format; - compaction under governed redaction; - incremental-index vs DBSP/differential threshold; - first optimistic replica/consensus implementation choices; - anti-entropy authentication/range-tree details; - measured append/query/rebuild/provenance benchmark thresholds; - backup/restore/integrity-checkpoint retention policy; - deployment-specific deletion interaction with immutable backups. ## Advance rule A/B/C/D are complete and reconciled. This issue may close as `completed` for **design completion only**. **Exact next research work: Depth 6 / #403A — retrieval / symbolic-attention foundations.** Do not start #403B/C/D until #403A is durably complete.
Author
Owner

Machine Spirit program progression

Depth 4 / #401 A-B-C-D is now durably design-complete and reconciled. This issue is therefore the first incomplete numbered depth.

Exact next research subpass: #402A — foundations / representative formalisms only.

Do not execute #402B/C/D in the same research run. #402A should independently compare durable world-model/storage foundations (event sourcing, bitemporal/append-only ledgers, authoritative history vs rebuildable materialized views, incremental/differential derivation state, provenance indexes, crash/replay/idempotency, reversible identity/schema/ontology evolution) before later adversarial/composition/computational subpasses attack it.

This comment marks queue progression only; #402A is not yet complete.

## Machine Spirit program progression Depth 4 / #401 A-B-C-D is now durably design-complete and reconciled. This issue is therefore the first incomplete numbered depth. **Exact next research subpass: #402A — foundations / representative formalisms only.** Do not execute #402B/C/D in the same research run. #402A should independently compare durable world-model/storage foundations (event sourcing, bitemporal/append-only ledgers, authoritative history vs rebuildable materialized views, incremental/differential derivation state, provenance indexes, crash/replay/idempotency, reversible identity/schema/ontology evolution) before later adversarial/composition/computational subpasses attack it. This comment marks queue progression only; #402A is not yet complete.
Author
Owner

#402A — foundations / representative formalisms — COMPLETE

Scope / research questions

This A subpass treats the durable world model itself as the primary design problem. It asks:

  1. What is canonical durable knowledge: current rows, immutable facts, or semantic events/history?
  2. How do we preserve world-valid time separately from when Machine Spirit learned/committed knowledge?
  3. How should proposition/object identity, assertion/episode identity, ledger-event identity, and materialized-view identity differ?
  4. What must survive a crash so current/query/index state can be rebuilt deterministically?
  5. Which state is authoritative and which state is safely disposable/rebuildable?
  6. How do identity, ontology, schema, compiler and reasoning-profile evolution occur without rewriting history?
  7. What storage architecture remains plausible from millions toward billions of semantic records without baking one backend into Prolog-RLM's semantic contract?
  8. How do provenance/explanation, authority boundaries, idempotency and exact historical replay survive this persistence model?

Current-repo comparison

The current architecture already points toward append-only semantics:

  • #392 separates proposition/assertion/context/justification identity and says canonical semantic state differs from derived/materialized state.
  • #399D makes compiler artifacts/version fingerprints and replay classes explicit.
  • #400 requires immutable epistemic inputs, bitemporal valid_at vs known_at, and rebuildable warrant/query caches.
  • #401 requires immutable procedure/PSC/mapping versions plus replayable execution/VERIFY lineage while DPEC search/monitor state remains derived.
  • symbolic-memory#4/#6 require append-only semantic history and rebuildable current/world-model views; #10 already requires reversible identity links.

But the current symbolic_memory_storage.pl implementation is materially weaker than that contract: a mutation runs under one process-wide mutex, snapshots all dynamic state, serializes the whole snapshot(...) term to a temporary file, and renames that file over the previous snapshot. That is a fine bootstrap persistence implementation, but it is not the durable semantic world-model architecture and cannot be treated as such at scale.

There is also a useful historical disagreement to preserve: research/SYMBOLIC-MEMORY-RESEARCH-006-persistence-and-concurrency.org (2026-08-29) explicitly did not require full event sourcing for v1 and favored a replaceable storage adapter with library(persistency), SQLite, or RocksDB candidates. The new semantic/epistemic/procedural contracts created by #398-#401 materially strengthen the requirement: an audit side-log plus mutable current snapshot is no longer sufficient to reconstruct all accepted lifecycle, bitemporal, identity, provenance and replay semantics. The old research remains valuable backend/bootstrap evidence; this A pass supersedes only its assumption that full replayable semantic history can be optional at the world-model contract layer.


Primary evidence

Representative primary/authoritative sources used in this subpass:

  1. Torp, Jensen, Snodgrass — “Effective Timestamping in Databases,” VLDB Journal 8(3-4), 2000. Valid time and transaction time are distinct temporal aspects and need transaction-level timestamp semantics. DOI: https://doi.org/10.1007/s007780050008 ; VLDB record: https://www.vldb.org/vldb_journal/index.php/component/article_manager/article/685
  2. McSherry, Murray, Isaacs, Isard — “Differential Dataflow,” CIDR 2013. Incremental computation over changing inputs, including nested iteration/SCC-style workloads: https://www.cidrdb.org/cidr2013/Papers/CIDR13_Paper111.pdf
  3. Budiu, Chajed, McSherry, Ryzhyk, Tannen — “DBSP: Automatic Incremental View Maintenance for Rich Query Languages,” PVLDB 16(7), 2023. General incrementalization for rich relational/recursive query languages: https://www.vldb.org/pvldb/vol16/p1601-budiu.pdf
  4. Green, Karvounarakis, Tannen — “Provenance Semirings,” PODS 2007. Algebraic provenance over relational/Datalog derivations. DOI: https://doi.org/10.1145/1265530.1265535
  5. Hartung, Groß, Rahm — “Rule-based Generation of Diff Evolution Mappings between Ontology Versions,” 2010. Expressive, invertible ontology-version mappings including merge/split/subgraph changes: https://arxiv.org/abs/1010.0122
  6. PostgreSQL WAL documentation (current PostgreSQL 18 docs, 2026). Durable log-before-data discipline, recovery/checkpoint semantics, and replay: https://www.postgresql.org/docs/18/runtime-config-wal.html and https://www.postgresql.org/docs/16/wal-intro.html
  7. SQLite WAL documentation (current). WAL as committed-but-not-yet-checkpointed roll-forward state, with checkpoint/concurrency behavior: https://sqlite.org/wal.html and https://www.sqlite.org/walformat.html
  8. Kurrent/EventStoreDB client documentation. Explicit event UUIDs and duplicate-event suppression demonstrate the useful distinction between event identity and payload: https://docs.kurrent.io/clients/node/legacy/v6.2/appending-events
  9. Apache Kafka design documentation. Durable ordered log offsets and idempotent producer/transaction mechanisms are useful implementation precedents, but log compaction is explicitly not sufficient as the sole Machine Spirit canonical-history contract: https://kafka.apache.org/41/design/design/
  10. Datomic architecture/documentation. Immutable facts/history and transaction-oriented time are a strong representative immutable-database architecture: https://www.datomic.com/

These sources do not imply one required physical database. They establish useful semantics and mechanisms that the logical storage contract can adopt independently of backend choice.


Candidate architectures

Candidate 1 — mutable snapshot + append-only audit side-log

mutable current semantic snapshot
           +
append-only audit/events

This is closest to the current bootstrap implementation and old Research-006 direction.

Strengths

  • very simple reads;
  • small implementation surface;
  • snapshots/checkpoints are easy to understand;
  • SQLite/Postgres or even a local Prolog snapshot can make the mutable state transactional.

Fatal weakness as canonical architecture

The audit log can drift into “debug metadata” rather than the complete reconstruction authority. If a lifecycle edge, source dependence, identity mapping, correction, projection-generation change or promoted derivation receipt is represented only by mutation of the snapshot, historical semantics become unrecoverable. A snapshot + incomplete audit is not event sourcing and cannot satisfy exact historical epistemic replay.

Rejected as the canonical world-model contract. It remains valid as a bootstrap/materialized-view implementation beneath a stronger contract.

Candidate 2 — pure event-sourced semantic store

every semantic change
       ↓
append event containing full semantic payload
       ↓
all current/query state projected from events

Strengths

  • excellent audit/replay story;
  • lifecycle corrections naturally append;
  • deterministic view rebuild is conceptually clean;
  • write path is sequential/transactional.

Weaknesses

  • repeats large semantic payloads and source/compiler metadata across episodes;
  • encourages conflating an abstract proposition/formula with an assertion episode or lifecycle event;
  • event schema evolution becomes expensive if event payloads are the only semantic-object representation;
  • identical semantic content from multiple sources needs distinct episodes while also benefiting from structural dedup;
  • graph/provenance queries become awkward if every shared semantic object is repeatedly embedded in events.

Viable but not preferred.

Candidate 3 — immutable semantic object store + authoritative bitemporal ledger + rebuildable projections (preferred)

immutable/versioned semantic objects
   proposition / formula / procedure / context /
   vocabulary / provenance payload / receipt
                │
                │ referenced by
                ▼
AUTHORITATIVE BITEMPORAL SEMANTIC LEDGER
  append-only committed event batches
  lifecycle + assertion episodes + mapping decisions
  valid-time + transaction/knowledge-time
                │
                ▼
rebuildable/materialized world-model projections
  current indexes / support state / identity view /
  temporal indexes / derivations / retrieval indexes /
  DPEC/EQP query caches

This architecture combines immutable-database/event-sourcing strengths without making semantic payload identity equal to event identity.

Preferred name for the A-level architecture: Bitemporal Semantic Ledger + Projection Fabric (BSLPF).


Core distinctions — hard

semantic object identity
    != assertion/support episode identity
    != ledger event identity
    != commit/batch identity
    != materialized view-row identity
    != query/result receipt identity

Example: two sources independently assert the exact same structural proposition P.

  • P may have one structural proposition fingerprint under one #392 vocabulary/schema generation;
  • the two source assertion episodes are distinct;
  • their ledger events are distinct;
  • they may have separate provenance/support edges;
  • the current support view may summarize them into one proposition row with two independent support components.

A content hash therefore must not be the universal event ID. Identical payload does not imply the same occurrence.


Preferred typed model

Conceptual only; exact representation follows repository conventions.

sem_object(
    ObjectId,
    Kind,
    SemanticSchema,
    VocabularyVersion,
    PayloadDigest,
    PayloadRef,
    Meta).

sem_ledger_event(
    EventId,
    LedgerId,
    Sequence,
    CommitId,
    CommandId,
    EventType,
    ObjectRefs,
    ValidTime,
    CommitTime,
    CausalRefs,
    ProvenanceRefs,
    EventSchema,
    Checksum).

sem_commit(
    CommitId,
    LedgerId,
    FirstSequence,
    LastSequence,
    IdempotencyKey,
    PrincipalRef,
    Namespace,
    ExpectedFrontier,
    CommittedAt,
    Digest).

sem_view_checkpoint(
    ViewId,
    ViewVersion,
    LedgerFrontier,
    DependencyVersions,
    StateDigest,
    CreatedAt).

Representative event classes include:

source_committed
projection_committed
assertion_added
support_added
support_withdrawn
correction_added
supersession_added
conflict_added
justification_promoted
procedure_version_added
execution_receipt_added
verify_receipt_added
identity_link_added
identity_link_withdrawn
ontology_mapping_added
schema_mapping_added
projection_invalidated

The list is not an executable open event vocabulary: event envelope kinds must be closed/versioned enough to protect replay semantics. Domain semantic vocabulary remains open inside #392 objects.


Bitemporal contract

Machine Spirit requires at least two independent axes:

valid time
    when the proposition/state/event applies in the modeled world

transaction / knowledge time
    when this ledger durably learned/committed that episode

Therefore:

What is warranted now about 2025-01-20?

and:

What did Machine Spirit know on 2025-01-20?

are different queries.

A correction received in February may assert that P was false throughout January. It can alter the current view of January-valid truth without rewriting the January transaction-time history. The February event remains visibly later in the ledger.

CommitTime must not rely on wall clock as the sole ordering primitive. The ledger needs a monotonically ordered logical position such as (LedgerId, Sequence). Wall-clock timestamp is metadata. Backdated valid-time facts are ordinary and must never reorder transaction history.

For A, do not require one distributed global total order. A local ledger may use a global sequence for simplicity, but the semantic contract should require stable per-ledger order + explicit commit/causal references. Federation/distributed ordering can be attacked in C/D.


Append / idempotency / crash semantics

Preferred API semantics:

memory_ledger_append(
    +Ledger,
    +EventBatch,
    +ExpectedFrontier,
    +IdempotencyKey,
    -CommitReceipt).

memory_ledger_scan(+Ledger, +FromSequence, +Options, -Events).
memory_projection_rebuild(+Projection, +TargetFrontier, -Receipt).
memory_projection_checkpoint(+Projection, -Checkpoint).
memory_semantic_as_of(+Query, +ValidAt, +KnownAt, +Profile, -Outcome).

Atomic batch

A commit batch is all-or-nothing. Partial committed semantic packages are forbidden.

Optimistic frontier/CAS

ExpectedFrontier permits stale concurrent writers to fail/rebase rather than silently overwrite logical history.

Idempotency

The caller provides a stable command/idempotency key where retry is possible.

same key + same canonical batch digest
    -> return original commit receipt / no duplicate append

same key + different batch digest
    -> idempotency conflict

Do not claim magical exactly-once transport. The safer contract is at-least-once delivery safe because committed commands/events are idempotently recognized.

Crash classes

  1. crash before durable commit -> batch absent;
  2. crash after durable ledger commit but before projection update -> ledger is authoritative; replay updates projection;
  3. crash after projection update but before checkpoint metadata -> projection can be discarded/rebuilt or reconciled against ledger frontier;
  4. corrupted/incompatible projection -> reject as cache; rebuild from ledger + immutable objects;
  5. corrupted canonical ledger/object segment -> integrity failure, never silently trust a materialized view as replacement authority.

WAL systems provide the physical crash-recovery precedent; Machine Spirit's semantic ledger is a logical domain log, not necessarily the database engine's physical WAL.


Authoritative vs rebuildable state

Canonical durable

At minimum:

  • exact source/source-version identity and immutable source reference;
  • immutable semantic object payloads needed to interpret accepted #392 knowledge;
  • assertion/support/lifecycle episodes;
  • context/theory/bridge identities when durable;
  • bitemporal validity + commit history;
  • identity/alignment/ontology mapping decisions and their lifecycle;
  • semantic compiler/projection lineage needed for replay;
  • materially promoted derivation/acceptance/procedure/execution/VERIFY receipts;
  • exact schema/vocabulary/profile/mapping version refs required to interpret history;
  • append commits/idempotency identities and checksums.

Rebuildable derived

  • current-active rows;
  • WFS tables / stable-model outputs / argumentation acceptance sets;
  • transitive closures;
  • current entity canonicalization view;
  • Rete/applicability state;
  • DPEC packed search forests/monitor caches;
  • EQP/federation interface caches;
  • retrieval/vector/full-text indexes;
  • provenance reachability indexes;
  • hot materialized temporal/current views;
  • ordinary intermediate proof DAG expansions.

A derived structure may be persisted for speed, but its persistence never makes it canonical truth.


Incremental/differential projections

Depth 2 and 3 already established dependency-directed incremental compilation/reasoning. Depth 5A generalizes the persistence substrate:

append semantic delta
       ↓
projection dependency graph
       ↓
apply delta to affected views
       ↓
record new frontier + view digest

Differential Dataflow and DBSP are strong foundations for maintaining recursive/rich views from deltas. The architecture should therefore expose delta/frontier semantics without requiring those particular runtimes.

A projection is a function of:

ledger frontier
+ projection code/version
+ semantic schema/vocabulary versions
+ reasoning/mapping/profile versions
+ namespace/admission policy

If any of those change, the old projection may become stale even when the base ledger did not.

A view checkpoint must identify all of them. “Database row exists” is not enough freshness evidence.


Provenance model

Green/Karvounarakis/Tannen motivates factorized provenance for alternative/conjunctive derivations. Machine Spirit should retain canonical source/compiler/logical/lifecycle/profile/runtime provenance classes from #392/#400, but ordinary expanded proof trees remain derived.

The ledger records provenance-bearing edges/receipts sufficient to reconstruct why. Provenance indexes can then support:

record -> source span
result -> justification -> premises -> assertion episodes
current support -> independent source/dependence components
procedure run -> method/PSC/binding -> effects -> observations -> VERIFY

Never combine provenance, source trust, probability, compiler confidence and authority into one scalar.


Identity / ontology / schema evolution

Identity

Cross-memory identity remains a reversible semantic relation, not destructive row coalescing.

entity A
entity B
identity_link L: likely_same_as(A,B)

A later event can withdraw/supersede L; A and B never disappear. Current canonicalization is a view.

Ontology evolution

Hartung/Groß/Rahm's invertible diff-evolution mappings are a useful precedent. Persist versioned mappings capable of expressing at least:

rename
exact map
narrower/broader
merge
split
subgraph mapping
incompatible/unresolved

A new ontology version does not rewrite historical semantic objects into the new vocabulary. Query-time or materialized version projections may translate with mapping/loss receipts.

Schema / event evolution

Historical event/object bytes remain interpreted under their original schema version.

Allowed strategies:

  • native decoder for old version;
  • deterministic read-time upcaster/adapter;
  • rebuild a new derived projection under the new schema;
  • append an explicit migration/mapping event if semantic interpretation changed.

Forbidden:

rewrite all historical events in place and pretend they were originally vNext

A schema conversion that changes meaning creates new lineage and a receipt.


Storage/backend implications

A is intentionally logical-architecture first. Do not lock the project to one physical backend yet.

Local/small

SQLite WAL or a purpose-built append file + indexes can plausibly implement the contract for a single-node deployment. The current whole-snapshot rewrite backend remains suitable only as a bootstrap/test backend once it is clearly labeled non-scalable and the semantic contract is not shaped around it.

Serious single-node/server

PostgreSQL-style transactional storage is a strong candidate for authoritative commits, bitemporal/indexed metadata and concurrent readers/writers. An embedded LSM/KV store can be attractive for high-write secondary indexes/object payloads.

Very large

At hundreds of millions/billions of records, expect immutable segmented object/event storage plus partitioned/LSM or relational indexes and separately maintained projection stores. The contract should allow tiering old immutable segments while retaining exact historical addressability.

A hard warning from Kafka-style compaction: physical compaction that destroys superseded semantic history cannot be the only copy of canonical Machine Spirit history. View/index compaction is fine; canonical-history retention requires an explicit retention/governance contract and cannot happen merely because a storage engine wants fewer bytes.

Complexity direction (A-level, not D benchmarks)

With suitable indexes:

  • append is expected amortized sequential/O(1)-ish per event plus index cost;
  • indexed point/current lookup should target O(log N + k) or KV-equivalent, not whole-log scan;
  • recovery work is O(events since checkpoint + affected derived work);
  • full rebuild is intentionally O(history + derivation work) and therefore checkpointing/tiering matters;
  • storage is O(unique immutable objects + ledger episodes + retained receipts + indexes);
  • identity/ontology mappings can create graph closure costs, therefore full canonical closure should not be eagerly materialized as source truth.

Depth 5D must benchmark and tighten these claims.


Repo ownership boundaries

prolog-rlm

Owns:

  • #392 semantic object algebra, canonical semantic fingerprints and schema/profile definitions;
  • #393/#396 compiler artifact/work fingerprints and replay classes;
  • #394 reasoning semantics and derivation receipt formats;
  • #376-#387 expert/procedure/VERIFY semantic/runtime receipt schemas where reusable;
  • no long-term canonical memory database.

symbolic-memory

Owns:

  • authoritative semantic ledger and atomic commit/idempotency/frontier contract;
  • durable binding of semantic objects/events to source memories/namespaces;
  • bitemporal known_at/valid_at history;
  • lifecycle/supersession/retraction/correction persistence;
  • projection checkpoints and materialized current/history views;
  • durable identity/ontology/schema mapping lifecycle;
  • persistent provenance indexes/object-addressability;
  • corpus/backfill append/resume semantics;
  • storage backend implementation/adapters.

Hard dependency rule

symbolic-memory -> prolog-rlm semantic contracts
prolog-rlm -/-> symbolic-memory storage backend

The existing one-scheduler rule remains unchanged. A ledger is not a scheduler and a projection engine is not an expert runtime.


Authority / safety implications

Persistence must preserve the existing safety separation:

remembered semantic action
    != runtime capability

remembered policy statement
    != authority policy

identity link to a principal-like entity
    != authentication principal binding

old successful execution receipt
    != permission to execute again

materialized view row
    != canonical evidence if its frontier/version is stale

A malicious imported source may append semantic content only through the normal compiler/validation/trust boundary. It cannot choose ledger event types, spoof commit principals, forge compiler/profile versions, install schema adapters, or synthesize authority mappings.

Event/commit envelopes therefore belong to trusted storage/runtime code, not model/source-controlled open vocabulary.


Rejected alternatives

  1. Current snapshot as truth + optional audit log — insufficient replay/history semantics.
  2. One giant mutable knowledge graph — destructive merges/retractions destroy source-local and historical meaning.
  3. Content hash as universal ID — collapses repeated independent episodes and cannot represent two identical assertions from different sources/times.
  4. Wall-clock timestamp as transaction order — clock skew/backdating and same-time commits break deterministic replay.
  5. Latest-write-wins current truth — incompatible with #400 conflict/support/default semantics and bitemporal history.
  6. Store every derived consequence forever — provenance/storage explosion; derived state should normally be reproducible projections.
  7. Never store any derivation receipt — breaks replay/explanation for promoted decisions, expert actions and VERIFY evidence.
  8. Rewrite history during schema/ontology migration — destroys auditability and changes what the system historically knew.
  9. Backend-specific semantics exposed to public API — prevents migration from bootstrap snapshot/SQLite to serious stores.
  10. Kafka-style destructive compaction as canonical retention policy — valid for projections/indexes, not a substitute for required semantic history.
  11. Exactly-once transport as correctness assumption — retries/network failures make this brittle; use idempotent commit recognition instead.
  12. A second durable scheduler hidden in the ledger — violates #376/#401; runtime scheduling remains #288/rlm_plan.

Falsifiable A-level conformance fixtures

A1. Independent identical assertion episodes — two sources assert structurally identical P; one proposition object may dedupe, but two source/support episodes and provenance paths remain.

A2. Event identity != content hash — two legitimate same-payload events retain distinct event IDs.

A3. Idempotent retry — same idempotency key + same batch digest commits once and returns the original receipt.

A4. Idempotency collision — same key + different digest is rejected, never silently deduped.

A5. Atomic semantic package — crash/failure halfway through a multi-event batch yields either the complete commit or no commit.

A6. Crash after commit/before view update — restart replays from frontier and produces the same normalized current view as uninterrupted execution.

A7. Delete/rebuild projection — deleting every rebuildable current/index table and replaying canonical state yields the same normalized semantic view.

A8. Backdated correction — a February correction about January changes current January-valid view but known_at(January) still reports the earlier epistemic state.

A9. Withdrawal != negation — withdrawal event deactivates support without creating negative assertion payload.

A10. Wall-clock reversal — intentionally nonmonotonic timestamps do not alter ledger sequence/replay order.

A11. Concurrent expected-frontier conflict — two writers based on the same frontier cannot both silently commit mutually stale state where the transaction contract requires CAS.

A12. Two independent justifications — invalidating one support path leaves the other intact after replay.

A13. Identity-link rollback — add same/likely-same mapping, query through it, withdraw mapping, rebuild; original entity IDs and pre-link history remain.

A14. Ontology rename — new ontology mapping supports current query projection while historical records remain encoded under old vocabulary/version.

A15. Ontology merge/split — mapping receipt records nontrivial merge/split and reverse/history inspection remains possible; no destructive source-ID rewrite.

A16. Old event schema replay — historical event under v1 is decoded/upcast deterministically under a versioned adapter; original bytes/schema ID remain addressable.

A17. Semantic-changing migration — migration that changes meaning creates new lineage/receipt rather than claiming byte-identical history.

A18. Stale checkpoint rejection — projection whose dependency versions/frontier do not match query requirements cannot satisfy current VERIFY/action admission.

A19. View compaction safety — compact/drop/rebuild materialized indexes without deleting canonical ledger history.

A20. Provenance round trip — current query → justification/support → assertion episode → compiler/source span remains resolvable after restart/rebuild.

A21. Execution lineage — a stored method/expert/effect/VERIFY receipt reconstructs which versions/evidence were used without turning the receipt into current authority.

A22. Namespace isolation — identical semantic objects may be shared/deduped physically, but ledger admissions/current views do not leak between namespaces.

A23. Unsupported schema — unknown/incompatible semantic or event schema can be retained/quarantined but cannot enter active current projections.

A24. Known symbolic replay with provider disabled — ledger replay, current-view rebuild and query over already compiled knowledge perform model_calls = 0.

These are architecture fixtures. Depth 5D must convert surviving ones into concrete storage/concurrency/performance gates.


Concrete canonical deltas from A

#402

Adopt BSLPF as the Depth-5A baseline to attack in B/C/D:

immutable semantic objects
        +
authoritative append-only bitemporal semantic ledger
        +
versioned rebuildable projections/checkpoints

symbolic-memory#6

Strengthen the append-only contract:

  • ledger event/commit identity is distinct from semantic-record identity;
  • add atomic commit batch, ordered frontier, idempotency key/digest, and crash/replay semantics;
  • known_at comes from immutable commit history; valid_at remains semantic world time;
  • current/support/entity/procedure/query indexes are versioned materialized projections;
  • projection checkpoints include ledger frontier + all relevant semantic/reasoning/mapping versions;
  • existing whole-snapshot backend is bootstrap implementation only, not semantic architecture.

symbolic-memory#5/#8

Projection/backfill lifecycle should commit through the ledger transaction boundary, with stable command/work IDs. Compiler retries can reuse retained artifacts while projection commits remain independently idempotent.

symbolic-memory#10

Identity/ontology reconciliation becomes explicit append-only mapping events + current mapping projection. Never rewrite old semantic object/entity IDs.

#392/#396

No storage ownership moves upstream. They must expose stable semantic object/schema/compiler fingerprints sufficient for downstream ledger commits and replay.

#400/#401 handoff preservation

Epistemic and procedural query/execution caches stay derived. Persist only canonical evidence/lifecycle and materially promoted/replay-critical receipts.


Unresolved questions preserved for #402B/C/D

  1. Should the first implementation have one global local commit sequence or one sequence per namespace/ledger plus causal refs?
  2. Which events require compare-and-swap/frontier preconditions versus commutative blind append?
  3. How should redaction/legal deletion requirements interact with an append-only semantic history? Crypto-shredding/tombstone/reference indirection may be needed; A does not resolve it.
  4. What is the minimum canonical receipt set versus safely reconstructible derivation state?
  5. How aggressively may immutable semantic payloads be content-deduplicated across namespaces with different read authority?
  6. How should large source blobs and native analyzer artifacts be tiered while preserving exact replay/addressability?
  7. Which schema upcasters are provably semantics-preserving versus merely loss-accounted?
  8. How do distributed/multi-node commits and cross-ledger atomicity work without imposing an unnecessary global order?
  9. How are provenance indexes protected from exponential derivation growth while retaining complete necessary lineage?
  10. What physical backend should be the first serious target: SQLite, PostgreSQL, embedded LSM/KV, or a hybrid?
  11. At what corpus/update rates do relational materialized views stop being enough and differential/dataflow-style maintenance become justified?
  12. What benchmark thresholds define acceptable append latency, restart/rebuild time, history query latency, index size and write amplification?
  13. What retention rules may archive canonical history without violating Machine Spirit replay/history requirements?
  14. How should corrupted canonical segments be detected, backed up, replicated and repaired?
  15. How should event-envelope signing/authentication work if histories are federated between trust domains?

Those are intentionally left for the hostile B pass, composition C pass, and computational D pass.

Completion state

#402A is design-complete only. Implementation is unproven. #402 stays OPEN.

Exact next subpass: #402B — semantic/adversarial deepening of durable world-model semantics.

B must attack append-only assumptions, deletion/redaction, provenance explosions, retroactive corrections, event/model duplication, transaction boundaries, temporal anomalies, stale/poisoned projections, schema migration, corruption, authority spoofing, and the distinction between historical record integrity and current epistemic acceptance.

Do not start #402C or #403 until #402B and then #402C/#402D are durably complete.

## #402A — foundations / representative formalisms — COMPLETE ### Scope / research questions This A subpass treats the durable world model itself as the primary design problem. It asks: 1. What is canonical durable knowledge: current rows, immutable facts, or semantic events/history? 2. How do we preserve **world-valid time** separately from **when Machine Spirit learned/committed knowledge**? 3. How should proposition/object identity, assertion/episode identity, ledger-event identity, and materialized-view identity differ? 4. What must survive a crash so current/query/index state can be rebuilt deterministically? 5. Which state is authoritative and which state is safely disposable/rebuildable? 6. How do identity, ontology, schema, compiler and reasoning-profile evolution occur without rewriting history? 7. What storage architecture remains plausible from millions toward billions of semantic records without baking one backend into Prolog-RLM's semantic contract? 8. How do provenance/explanation, authority boundaries, idempotency and exact historical replay survive this persistence model? ### Current-repo comparison The current architecture already points toward append-only semantics: - #392 separates proposition/assertion/context/justification identity and says canonical semantic state differs from derived/materialized state. - #399D makes compiler artifacts/version fingerprints and replay classes explicit. - #400 requires immutable epistemic inputs, bitemporal `valid_at` vs `known_at`, and rebuildable warrant/query caches. - #401 requires immutable procedure/PSC/mapping versions plus replayable execution/VERIFY lineage while DPEC search/monitor state remains derived. - symbolic-memory#4/#6 require append-only semantic history and rebuildable current/world-model views; #10 already requires reversible identity links. But the current `symbolic_memory_storage.pl` implementation is materially weaker than that contract: a mutation runs under one process-wide mutex, snapshots all dynamic state, serializes the whole `snapshot(...)` term to a temporary file, and renames that file over the previous snapshot. That is a fine bootstrap persistence implementation, but it is **not** the durable semantic world-model architecture and cannot be treated as such at scale. There is also a useful historical disagreement to preserve: `research/SYMBOLIC-MEMORY-RESEARCH-006-persistence-and-concurrency.org` (2026-08-29) explicitly did **not** require full event sourcing for v1 and favored a replaceable storage adapter with `library(persistency)`, SQLite, or RocksDB candidates. The new semantic/epistemic/procedural contracts created by #398-#401 materially strengthen the requirement: an audit side-log plus mutable current snapshot is no longer sufficient to reconstruct all accepted lifecycle, bitemporal, identity, provenance and replay semantics. The old research remains valuable backend/bootstrap evidence; this A pass supersedes only its assumption that full replayable semantic history can be optional at the world-model contract layer. --- ## Primary evidence Representative primary/authoritative sources used in this subpass: 1. **Torp, Jensen, Snodgrass — “Effective Timestamping in Databases,” VLDB Journal 8(3-4), 2000.** Valid time and transaction time are distinct temporal aspects and need transaction-level timestamp semantics. DOI: https://doi.org/10.1007/s007780050008 ; VLDB record: https://www.vldb.org/vldb_journal/index.php/component/article_manager/article/685 2. **McSherry, Murray, Isaacs, Isard — “Differential Dataflow,” CIDR 2013.** Incremental computation over changing inputs, including nested iteration/SCC-style workloads: https://www.cidrdb.org/cidr2013/Papers/CIDR13_Paper111.pdf 3. **Budiu, Chajed, McSherry, Ryzhyk, Tannen — “DBSP: Automatic Incremental View Maintenance for Rich Query Languages,” PVLDB 16(7), 2023.** General incrementalization for rich relational/recursive query languages: https://www.vldb.org/pvldb/vol16/p1601-budiu.pdf 4. **Green, Karvounarakis, Tannen — “Provenance Semirings,” PODS 2007.** Algebraic provenance over relational/Datalog derivations. DOI: https://doi.org/10.1145/1265530.1265535 5. **Hartung, Groß, Rahm — “Rule-based Generation of Diff Evolution Mappings between Ontology Versions,” 2010.** Expressive, invertible ontology-version mappings including merge/split/subgraph changes: https://arxiv.org/abs/1010.0122 6. **PostgreSQL WAL documentation (current PostgreSQL 18 docs, 2026).** Durable log-before-data discipline, recovery/checkpoint semantics, and replay: https://www.postgresql.org/docs/18/runtime-config-wal.html and https://www.postgresql.org/docs/16/wal-intro.html 7. **SQLite WAL documentation (current).** WAL as committed-but-not-yet-checkpointed roll-forward state, with checkpoint/concurrency behavior: https://sqlite.org/wal.html and https://www.sqlite.org/walformat.html 8. **Kurrent/EventStoreDB client documentation.** Explicit event UUIDs and duplicate-event suppression demonstrate the useful distinction between event identity and payload: https://docs.kurrent.io/clients/node/legacy/v6.2/appending-events 9. **Apache Kafka design documentation.** Durable ordered log offsets and idempotent producer/transaction mechanisms are useful implementation precedents, but log compaction is explicitly *not* sufficient as the sole Machine Spirit canonical-history contract: https://kafka.apache.org/41/design/design/ 10. **Datomic architecture/documentation.** Immutable facts/history and transaction-oriented time are a strong representative immutable-database architecture: https://www.datomic.com/ These sources do not imply one required physical database. They establish useful semantics and mechanisms that the logical storage contract can adopt independently of backend choice. --- # Candidate architectures ## Candidate 1 — mutable snapshot + append-only audit side-log ```text mutable current semantic snapshot + append-only audit/events ``` This is closest to the current bootstrap implementation and old Research-006 direction. ### Strengths - very simple reads; - small implementation surface; - snapshots/checkpoints are easy to understand; - SQLite/Postgres or even a local Prolog snapshot can make the mutable state transactional. ### Fatal weakness as canonical architecture The audit log can drift into “debug metadata” rather than the **complete reconstruction authority**. If a lifecycle edge, source dependence, identity mapping, correction, projection-generation change or promoted derivation receipt is represented only by mutation of the snapshot, historical semantics become unrecoverable. A snapshot + incomplete audit is not event sourcing and cannot satisfy exact historical epistemic replay. **Rejected as the canonical world-model contract.** It remains valid as a bootstrap/materialized-view implementation beneath a stronger contract. ## Candidate 2 — pure event-sourced semantic store ```text every semantic change ↓ append event containing full semantic payload ↓ all current/query state projected from events ``` ### Strengths - excellent audit/replay story; - lifecycle corrections naturally append; - deterministic view rebuild is conceptually clean; - write path is sequential/transactional. ### Weaknesses - repeats large semantic payloads and source/compiler metadata across episodes; - encourages conflating an abstract proposition/formula with an assertion episode or lifecycle event; - event schema evolution becomes expensive if event payloads are the only semantic-object representation; - identical semantic content from multiple sources needs distinct episodes while also benefiting from structural dedup; - graph/provenance queries become awkward if every shared semantic object is repeatedly embedded in events. **Viable but not preferred.** ## Candidate 3 — immutable semantic object store + authoritative bitemporal ledger + rebuildable projections (**preferred**) ```text immutable/versioned semantic objects proposition / formula / procedure / context / vocabulary / provenance payload / receipt │ │ referenced by ▼ AUTHORITATIVE BITEMPORAL SEMANTIC LEDGER append-only committed event batches lifecycle + assertion episodes + mapping decisions valid-time + transaction/knowledge-time │ ▼ rebuildable/materialized world-model projections current indexes / support state / identity view / temporal indexes / derivations / retrieval indexes / DPEC/EQP query caches ``` This architecture combines immutable-database/event-sourcing strengths without making semantic payload identity equal to event identity. **Preferred name for the A-level architecture: `Bitemporal Semantic Ledger + Projection Fabric (BSLPF)`.** --- # Core distinctions — hard ```text semantic object identity != assertion/support episode identity != ledger event identity != commit/batch identity != materialized view-row identity != query/result receipt identity ``` Example: two sources independently assert the exact same structural proposition `P`. - `P` may have one structural proposition fingerprint under one #392 vocabulary/schema generation; - the two source assertion episodes are distinct; - their ledger events are distinct; - they may have separate provenance/support edges; - the current support view may summarize them into one proposition row with two independent support components. A content hash therefore **must not be the universal event ID**. Identical payload does not imply the same occurrence. --- # Preferred typed model Conceptual only; exact representation follows repository conventions. ```prolog sem_object( ObjectId, Kind, SemanticSchema, VocabularyVersion, PayloadDigest, PayloadRef, Meta). sem_ledger_event( EventId, LedgerId, Sequence, CommitId, CommandId, EventType, ObjectRefs, ValidTime, CommitTime, CausalRefs, ProvenanceRefs, EventSchema, Checksum). sem_commit( CommitId, LedgerId, FirstSequence, LastSequence, IdempotencyKey, PrincipalRef, Namespace, ExpectedFrontier, CommittedAt, Digest). sem_view_checkpoint( ViewId, ViewVersion, LedgerFrontier, DependencyVersions, StateDigest, CreatedAt). ``` Representative event classes include: ```text source_committed projection_committed assertion_added support_added support_withdrawn correction_added supersession_added conflict_added justification_promoted procedure_version_added execution_receipt_added verify_receipt_added identity_link_added identity_link_withdrawn ontology_mapping_added schema_mapping_added projection_invalidated ``` The list is not an executable open event vocabulary: event envelope kinds must be closed/versioned enough to protect replay semantics. Domain semantic vocabulary remains open inside #392 objects. --- # Bitemporal contract Machine Spirit requires at least two independent axes: ```text valid time when the proposition/state/event applies in the modeled world transaction / knowledge time when this ledger durably learned/committed that episode ``` Therefore: ```text What is warranted now about 2025-01-20? ``` and: ```text What did Machine Spirit know on 2025-01-20? ``` are different queries. A correction received in February may assert that `P` was false throughout January. It can alter the **current view of January-valid truth** without rewriting the January transaction-time history. The February event remains visibly later in the ledger. `CommitTime` must not rely on wall clock as the sole ordering primitive. The ledger needs a monotonically ordered logical position such as `(LedgerId, Sequence)`. Wall-clock timestamp is metadata. Backdated valid-time facts are ordinary and must never reorder transaction history. For A, do **not** require one distributed global total order. A local ledger may use a global sequence for simplicity, but the semantic contract should require stable per-ledger order + explicit commit/causal references. Federation/distributed ordering can be attacked in C/D. --- # Append / idempotency / crash semantics Preferred API semantics: ```prolog memory_ledger_append( +Ledger, +EventBatch, +ExpectedFrontier, +IdempotencyKey, -CommitReceipt). memory_ledger_scan(+Ledger, +FromSequence, +Options, -Events). memory_projection_rebuild(+Projection, +TargetFrontier, -Receipt). memory_projection_checkpoint(+Projection, -Checkpoint). memory_semantic_as_of(+Query, +ValidAt, +KnownAt, +Profile, -Outcome). ``` ### Atomic batch A commit batch is all-or-nothing. Partial committed semantic packages are forbidden. ### Optimistic frontier/CAS `ExpectedFrontier` permits stale concurrent writers to fail/rebase rather than silently overwrite logical history. ### Idempotency The caller provides a stable command/idempotency key where retry is possible. ```text same key + same canonical batch digest -> return original commit receipt / no duplicate append same key + different batch digest -> idempotency conflict ``` Do not claim magical exactly-once transport. The safer contract is **at-least-once delivery safe because committed commands/events are idempotently recognized**. ### Crash classes 1. crash before durable commit -> batch absent; 2. crash after durable ledger commit but before projection update -> ledger is authoritative; replay updates projection; 3. crash after projection update but before checkpoint metadata -> projection can be discarded/rebuilt or reconciled against ledger frontier; 4. corrupted/incompatible projection -> reject as cache; rebuild from ledger + immutable objects; 5. corrupted canonical ledger/object segment -> integrity failure, never silently trust a materialized view as replacement authority. WAL systems provide the physical crash-recovery precedent; Machine Spirit's semantic ledger is a **logical domain log**, not necessarily the database engine's physical WAL. --- # Authoritative vs rebuildable state ## Canonical durable At minimum: - exact source/source-version identity and immutable source reference; - immutable semantic object payloads needed to interpret accepted #392 knowledge; - assertion/support/lifecycle episodes; - context/theory/bridge identities when durable; - bitemporal validity + commit history; - identity/alignment/ontology mapping decisions and their lifecycle; - semantic compiler/projection lineage needed for replay; - materially promoted derivation/acceptance/procedure/execution/VERIFY receipts; - exact schema/vocabulary/profile/mapping version refs required to interpret history; - append commits/idempotency identities and checksums. ## Rebuildable derived - current-active rows; - WFS tables / stable-model outputs / argumentation acceptance sets; - transitive closures; - current entity canonicalization view; - Rete/applicability state; - DPEC packed search forests/monitor caches; - EQP/federation interface caches; - retrieval/vector/full-text indexes; - provenance reachability indexes; - hot materialized temporal/current views; - ordinary intermediate proof DAG expansions. A derived structure may be persisted for speed, but its persistence never makes it canonical truth. --- # Incremental/differential projections Depth 2 and 3 already established dependency-directed incremental compilation/reasoning. Depth 5A generalizes the persistence substrate: ```text append semantic delta ↓ projection dependency graph ↓ apply delta to affected views ↓ record new frontier + view digest ``` Differential Dataflow and DBSP are strong foundations for maintaining recursive/rich views from deltas. The architecture should therefore expose **delta/frontier semantics** without requiring those particular runtimes. A projection is a function of: ```text ledger frontier + projection code/version + semantic schema/vocabulary versions + reasoning/mapping/profile versions + namespace/admission policy ``` If any of those change, the old projection may become stale even when the base ledger did not. A view checkpoint must identify all of them. “Database row exists” is not enough freshness evidence. --- # Provenance model Green/Karvounarakis/Tannen motivates factorized provenance for alternative/conjunctive derivations. Machine Spirit should retain canonical source/compiler/logical/lifecycle/profile/runtime provenance classes from #392/#400, but ordinary expanded proof trees remain derived. The ledger records provenance-bearing **edges/receipts sufficient to reconstruct why**. Provenance indexes can then support: ```text record -> source span result -> justification -> premises -> assertion episodes current support -> independent source/dependence components procedure run -> method/PSC/binding -> effects -> observations -> VERIFY ``` Never combine provenance, source trust, probability, compiler confidence and authority into one scalar. --- # Identity / ontology / schema evolution ## Identity Cross-memory identity remains a reversible semantic relation, not destructive row coalescing. ```text entity A entity B identity_link L: likely_same_as(A,B) ``` A later event can withdraw/supersede `L`; A and B never disappear. Current canonicalization is a view. ## Ontology evolution Hartung/Groß/Rahm's invertible diff-evolution mappings are a useful precedent. Persist versioned mappings capable of expressing at least: ```text rename exact map narrower/broader merge split subgraph mapping incompatible/unresolved ``` A new ontology version does not rewrite historical semantic objects into the new vocabulary. Query-time or materialized **version projections** may translate with mapping/loss receipts. ## Schema / event evolution Historical event/object bytes remain interpreted under their original schema version. Allowed strategies: - native decoder for old version; - deterministic read-time upcaster/adapter; - rebuild a new derived projection under the new schema; - append an explicit migration/mapping event if semantic interpretation changed. Forbidden: ```text rewrite all historical events in place and pretend they were originally vNext ``` A schema conversion that changes meaning creates new lineage and a receipt. --- # Storage/backend implications A is intentionally **logical-architecture first**. Do not lock the project to one physical backend yet. ### Local/small SQLite WAL or a purpose-built append file + indexes can plausibly implement the contract for a single-node deployment. The current whole-snapshot rewrite backend remains suitable only as a bootstrap/test backend once it is clearly labeled non-scalable and the semantic contract is not shaped around it. ### Serious single-node/server PostgreSQL-style transactional storage is a strong candidate for authoritative commits, bitemporal/indexed metadata and concurrent readers/writers. An embedded LSM/KV store can be attractive for high-write secondary indexes/object payloads. ### Very large At hundreds of millions/billions of records, expect immutable segmented object/event storage plus partitioned/LSM or relational indexes and separately maintained projection stores. The contract should allow tiering old immutable segments while retaining exact historical addressability. A hard warning from Kafka-style compaction: **physical compaction that destroys superseded semantic history cannot be the only copy of canonical Machine Spirit history**. View/index compaction is fine; canonical-history retention requires an explicit retention/governance contract and cannot happen merely because a storage engine wants fewer bytes. ### Complexity direction (A-level, not D benchmarks) With suitable indexes: - append is expected amortized sequential/O(1)-ish per event plus index cost; - indexed point/current lookup should target `O(log N + k)` or KV-equivalent, not whole-log scan; - recovery work is `O(events since checkpoint + affected derived work)`; - full rebuild is intentionally `O(history + derivation work)` and therefore checkpointing/tiering matters; - storage is `O(unique immutable objects + ledger episodes + retained receipts + indexes)`; - identity/ontology mappings can create graph closure costs, therefore full canonical closure should not be eagerly materialized as source truth. Depth 5D must benchmark and tighten these claims. --- # Repo ownership boundaries ### `prolog-rlm` Owns: - #392 semantic object algebra, canonical semantic fingerprints and schema/profile definitions; - #393/#396 compiler artifact/work fingerprints and replay classes; - #394 reasoning semantics and derivation receipt formats; - #376-#387 expert/procedure/VERIFY semantic/runtime receipt schemas where reusable; - no long-term canonical memory database. ### `symbolic-memory` Owns: - authoritative semantic ledger and atomic commit/idempotency/frontier contract; - durable binding of semantic objects/events to source memories/namespaces; - bitemporal `known_at`/`valid_at` history; - lifecycle/supersession/retraction/correction persistence; - projection checkpoints and materialized current/history views; - durable identity/ontology/schema mapping lifecycle; - persistent provenance indexes/object-addressability; - corpus/backfill append/resume semantics; - storage backend implementation/adapters. ### Hard dependency rule ```text symbolic-memory -> prolog-rlm semantic contracts prolog-rlm -/-> symbolic-memory storage backend ``` The existing one-scheduler rule remains unchanged. A ledger is not a scheduler and a projection engine is not an expert runtime. --- # Authority / safety implications Persistence must preserve the existing safety separation: ```text remembered semantic action != runtime capability remembered policy statement != authority policy identity link to a principal-like entity != authentication principal binding old successful execution receipt != permission to execute again materialized view row != canonical evidence if its frontier/version is stale ``` A malicious imported source may append semantic **content** only through the normal compiler/validation/trust boundary. It cannot choose ledger event types, spoof commit principals, forge compiler/profile versions, install schema adapters, or synthesize authority mappings. Event/commit envelopes therefore belong to trusted storage/runtime code, not model/source-controlled open vocabulary. --- # Rejected alternatives 1. **Current snapshot as truth + optional audit log** — insufficient replay/history semantics. 2. **One giant mutable knowledge graph** — destructive merges/retractions destroy source-local and historical meaning. 3. **Content hash as universal ID** — collapses repeated independent episodes and cannot represent two identical assertions from different sources/times. 4. **Wall-clock timestamp as transaction order** — clock skew/backdating and same-time commits break deterministic replay. 5. **Latest-write-wins current truth** — incompatible with #400 conflict/support/default semantics and bitemporal history. 6. **Store every derived consequence forever** — provenance/storage explosion; derived state should normally be reproducible projections. 7. **Never store any derivation receipt** — breaks replay/explanation for promoted decisions, expert actions and VERIFY evidence. 8. **Rewrite history during schema/ontology migration** — destroys auditability and changes what the system historically knew. 9. **Backend-specific semantics exposed to public API** — prevents migration from bootstrap snapshot/SQLite to serious stores. 10. **Kafka-style destructive compaction as canonical retention policy** — valid for projections/indexes, not a substitute for required semantic history. 11. **Exactly-once transport as correctness assumption** — retries/network failures make this brittle; use idempotent commit recognition instead. 12. **A second durable scheduler hidden in the ledger** — violates #376/#401; runtime scheduling remains #288/`rlm_plan`. --- # Falsifiable A-level conformance fixtures A1. **Independent identical assertion episodes** — two sources assert structurally identical P; one proposition object may dedupe, but two source/support episodes and provenance paths remain. A2. **Event identity != content hash** — two legitimate same-payload events retain distinct event IDs. A3. **Idempotent retry** — same idempotency key + same batch digest commits once and returns the original receipt. A4. **Idempotency collision** — same key + different digest is rejected, never silently deduped. A5. **Atomic semantic package** — crash/failure halfway through a multi-event batch yields either the complete commit or no commit. A6. **Crash after commit/before view update** — restart replays from frontier and produces the same normalized current view as uninterrupted execution. A7. **Delete/rebuild projection** — deleting every rebuildable current/index table and replaying canonical state yields the same normalized semantic view. A8. **Backdated correction** — a February correction about January changes current January-valid view but `known_at(January)` still reports the earlier epistemic state. A9. **Withdrawal != negation** — withdrawal event deactivates support without creating negative assertion payload. A10. **Wall-clock reversal** — intentionally nonmonotonic timestamps do not alter ledger sequence/replay order. A11. **Concurrent expected-frontier conflict** — two writers based on the same frontier cannot both silently commit mutually stale state where the transaction contract requires CAS. A12. **Two independent justifications** — invalidating one support path leaves the other intact after replay. A13. **Identity-link rollback** — add `same/likely-same` mapping, query through it, withdraw mapping, rebuild; original entity IDs and pre-link history remain. A14. **Ontology rename** — new ontology mapping supports current query projection while historical records remain encoded under old vocabulary/version. A15. **Ontology merge/split** — mapping receipt records nontrivial merge/split and reverse/history inspection remains possible; no destructive source-ID rewrite. A16. **Old event schema replay** — historical event under v1 is decoded/upcast deterministically under a versioned adapter; original bytes/schema ID remain addressable. A17. **Semantic-changing migration** — migration that changes meaning creates new lineage/receipt rather than claiming byte-identical history. A18. **Stale checkpoint rejection** — projection whose dependency versions/frontier do not match query requirements cannot satisfy current VERIFY/action admission. A19. **View compaction safety** — compact/drop/rebuild materialized indexes without deleting canonical ledger history. A20. **Provenance round trip** — current query → justification/support → assertion episode → compiler/source span remains resolvable after restart/rebuild. A21. **Execution lineage** — a stored method/expert/effect/VERIFY receipt reconstructs which versions/evidence were used without turning the receipt into current authority. A22. **Namespace isolation** — identical semantic objects may be shared/deduped physically, but ledger admissions/current views do not leak between namespaces. A23. **Unsupported schema** — unknown/incompatible semantic or event schema can be retained/quarantined but cannot enter active current projections. A24. **Known symbolic replay with provider disabled** — ledger replay, current-view rebuild and query over already compiled knowledge perform `model_calls = 0`. These are architecture fixtures. Depth 5D must convert surviving ones into concrete storage/concurrency/performance gates. --- # Concrete canonical deltas from A ### #402 Adopt **BSLPF** as the Depth-5A baseline to attack in B/C/D: ```text immutable semantic objects + authoritative append-only bitemporal semantic ledger + versioned rebuildable projections/checkpoints ``` ### symbolic-memory#6 Strengthen the append-only contract: - ledger event/commit identity is distinct from semantic-record identity; - add atomic commit batch, ordered frontier, idempotency key/digest, and crash/replay semantics; - `known_at` comes from immutable commit history; `valid_at` remains semantic world time; - current/support/entity/procedure/query indexes are versioned materialized projections; - projection checkpoints include ledger frontier + all relevant semantic/reasoning/mapping versions; - existing whole-snapshot backend is bootstrap implementation only, not semantic architecture. ### symbolic-memory#5/#8 Projection/backfill lifecycle should commit through the ledger transaction boundary, with stable command/work IDs. Compiler retries can reuse retained artifacts while projection commits remain independently idempotent. ### symbolic-memory#10 Identity/ontology reconciliation becomes explicit append-only mapping events + current mapping projection. Never rewrite old semantic object/entity IDs. ### #392/#396 No storage ownership moves upstream. They must expose stable semantic object/schema/compiler fingerprints sufficient for downstream ledger commits and replay. ### #400/#401 handoff preservation Epistemic and procedural query/execution caches stay derived. Persist only canonical evidence/lifecycle and materially promoted/replay-critical receipts. --- # Unresolved questions preserved for #402B/C/D 1. Should the first implementation have one global local commit sequence or one sequence per namespace/ledger plus causal refs? 2. Which events require compare-and-swap/frontier preconditions versus commutative blind append? 3. How should redaction/legal deletion requirements interact with an append-only semantic history? Crypto-shredding/tombstone/reference indirection may be needed; A does not resolve it. 4. What is the minimum canonical receipt set versus safely reconstructible derivation state? 5. How aggressively may immutable semantic payloads be content-deduplicated across namespaces with different read authority? 6. How should large source blobs and native analyzer artifacts be tiered while preserving exact replay/addressability? 7. Which schema upcasters are provably semantics-preserving versus merely loss-accounted? 8. How do distributed/multi-node commits and cross-ledger atomicity work without imposing an unnecessary global order? 9. How are provenance indexes protected from exponential derivation growth while retaining complete necessary lineage? 10. What physical backend should be the first serious target: SQLite, PostgreSQL, embedded LSM/KV, or a hybrid? 11. At what corpus/update rates do relational materialized views stop being enough and differential/dataflow-style maintenance become justified? 12. What benchmark thresholds define acceptable append latency, restart/rebuild time, history query latency, index size and write amplification? 13. What retention rules may archive canonical history without violating Machine Spirit replay/history requirements? 14. How should corrupted canonical segments be detected, backed up, replicated and repaired? 15. How should event-envelope signing/authentication work if histories are federated between trust domains? Those are intentionally left for the hostile B pass, composition C pass, and computational D pass. ## Completion state **#402A is design-complete only. Implementation is unproven. #402 stays OPEN.** Exact next subpass: **#402B — semantic/adversarial deepening of durable world-model semantics.** B must attack append-only assumptions, deletion/redaction, provenance explosions, retroactive corrections, event/model duplication, transaction boundaries, temporal anomalies, stale/poisoned projections, schema migration, corruption, authority spoofing, and the distinction between historical record integrity and current epistemic acceptance. Do **not** start #402C or #403 until #402B and then #402C/#402D are durably complete.
Author
Owner

#402B — semantic/adversarial deepening — COMPLETE

Scope / attack surface

This B subpass attacks the assumptions behind #402A's BSLPF — Bitemporal Semantic Ledger + Projection Fabric rather than repeating its storage survey. The questions are:

  1. Can an append-only semantic world model support actual erasure/redaction without lying about what remains recoverable?
  2. How must correction, withdrawal, invalidation, quarantine, redaction, archival and physical deletion differ?
  3. What does historical replay mean after payload erasure, schema evolution, corruption, or later discovery that an event was unauthorized/poisoned?
  4. Can event/source identity and content deduplication create privacy or corroboration bugs?
  5. How are event schema/upcaster changes prevented from silently rewriting historical meaning?
  6. How do projection caches fail safely when stale, poisoned or corrupted?
  7. What integrity properties can an append-only ledger actually guarantee, and what still requires monitoring/authentication/recovery?
  8. How do actor identity, provenance and registration/commit receipts avoid authority spoofing?
  9. How can provenance remain complete enough for explanation/replay without exploding into duplicated proof trees?
  10. Which guarantees belong to the semantic contract versus a physical backend's sanitization/replication machinery?

Current-state comparison

#402A already established the right baseline separation: immutable/versioned semantic objects, ordered bitemporal ledger events and rebuildable projections. #400 requires valid_at vs known_at, explicit support/defeat lineage and derived warrant caches; #401 requires immutable procedure/PSC versions plus material execution/VERIFY/replay lineage. The current Symbolic Memory issues already distinguish withdrawal from negation and current views from history.

The hidden failure is that append-only was still overloaded. Historical research SYMBOLIC-MEMORY-RESEARCH-008 explicitly says forgetting usually differs from deletion and reserves physical erase for policy/user requirements. Research-009 treats persistent memory as security-sensitive execution input and requires selective poisoning repair. Research-006's bootstrap storage analysis also never established a strong physical-erasure/tamper-evidence contract.

Therefore BSLPF needs an explicit governance/integrity layer rather than interpreting logical append-only history as permanent recoverability of every historical payload.


Primary / authoritative evidence

  1. EU General Data Protection Regulation, Regulation (EU) 2016/679, Article 17 (2016). Article 17 defines circumstances where a controller must erase personal data without undue delay and also defines exceptions. This is a concrete counterexample to a design that promises every plaintext event payload is immortal: https://eur-lex.europa.eu/eli/reg/2016/679
  2. NIST SP 800-88 Rev. 2, Guidelines for Media Sanitization (final 2025-09-26). Sanitization is defined in terms of making access to target data infeasible for a given level of effort; the revision explicitly includes cryptographic erase and sanitization validation. Logical deletion in a semantic ledger is therefore not automatically physical sanitization: https://csrc.nist.gov/pubs/sp/800/88/r2/final
  3. Peterson et al., “Secure Deletion for a Versioning File System,” FAST 2005. Demonstrates that versioned storage requires deliberate deletion architecture; authenticated encryption plus destruction/overwriting of small key/stub material can make selected historical versions unrecoverable without rewriting all shared blocks: https://www.usenix.org/conference/fast-05/secure-deletion-versioning-file-system
  4. Reardon, Capkun, Basin, “Data Node Encrypted File System: Efficient Secure Deletion for Flash Memory,” USENIX Security 2012. Shows why simple overwrite assumptions fail on flash and uses encryption-based techniques for secure deletion: https://www.usenix.org/conference/usenixsecurity12/technical-sessions/presentation/reardon
  5. RFC 9162, Certificate Transparency Version 2.0 (2021). Merkle consistency proofs efficiently establish that a later log state extends an earlier one, but the RFC also warns that inconsistent/split views require auditing/monitoring. Append-only structure is tamper-evidence machinery, not semantic truth or complete protection from a malicious log: https://www.rfc-editor.org/rfc/rfc9162.html
  6. RFC 9943, SCITT architecture (2026). Separates an issuer's signed statement from a Transparency Service registration/receipt. A client can register on behalf of an issuer, and a receipt proves registration, not authorship. This is directly useful for preventing principal=admin fields inside imported history from becoming authenticated local authority: https://www.rfc-editor.org/rfc/rfc9943.html
  7. Overeem, Spoor, Jansen, Brinkkemper, “An empirical characterization of event sourced systems and their schema evolution — Lessons from industry,” JSS 178, 2021. Study of 19 systems/25 engineers identifies rebuilding projections and privacy as event-sourcing challenges and reports five schema-evolution techniques: versioned events, weak schema, upcasting, in-place transform, copy-and-transform. Upcasting retains immutable stored events; in-place transformation explicitly gives up immutability/audit guarantees: https://doi.org/10.1016/j.jss.2021.110970
  8. W3C PROV-O Recommendation (2013). Provenance bundles are first-class entities and can themselves carry provenance, supporting provenance-of-provenance and shared provenance structures rather than duplicated opaque metadata blobs: https://www.w3.org/TR/prov-o/

These sources establish mechanisms and failure boundaries; they do not imply that Symbolic Memory is automatically legally compliant or that one cryptographic/logging mechanism is universally sufficient.


Candidate architectures

Candidate B1 — physically immutable plaintext event log

all event envelopes + all historical payloads
             ↓
permanent append-only store

Advantage

Maximum naive replay simplicity.

Fatal problems

  • conflicts with legitimate erasure/sanitization requirements;
  • poisoned/secrets-containing payloads remain recoverable forever;
  • backups/replicas make “delete” dishonest;
  • content-addressed global hashes can preserve equality/fingerprinting side channels after nominal deletion;
  • an event claiming a principal remains spoofable unless identity is independently authenticated.

Rejected. Logical historical integrity cannot require perpetual recoverability of every payload byte.

Candidate B2 — mutable history with in-place redaction

ledger row
  ↓ delete/replace sensitive fields in place
  ↓ append audit marker

Advantage

Straightforward privacy/delete semantics and ordinary database tooling.

Fatal problems

  • breaks exact historical replay and append-only integrity;
  • cannot distinguish authorized redaction from illicit tampering without an independent commitment history;
  • cache/checkpoint/proof roots become ambiguous after mutation;
  • schema “fixes” can silently rewrite the past.

Rejected as canonical. Controlled physical compaction/migration may use copy-transform, but it creates a new ledger generation with an explicit migration receipt rather than pretending the old history never existed.

Candidate B3 — BSLPF + Governed Redaction & Integrity Envelope (preferred)

Call the B-level refinement GRIE — Governed Redaction & Integrity Envelope.

immutable/versioned semantic identity
          +
retention/encryption-scoped payload objects
          ↓
AUTHORITATIVE BITEMPORAL CONTROL LEDGER
  append-only commitments + lifecycle events
  issuer/authentication/authorization receipts
  correction/withdrawal/quarantine/redaction events
  schema/upcaster/migration receipts
  segment/checkpoint integrity commitments
          ↓
     admissibility / retention gate
          ↓
rebuildable current/history/epistemic/procedural views

The hard refinement is:

append-only logical commitment history
    != permanent payload retrievability

The ledger can preserve that event/object E existed, was accepted at frontier F, and was later redacted under policy/authority R, while the protected payload becomes cryptographically or physically unrecoverable where the backend/policy actually supports that guarantee.


Hard semantic distinctions added by B

correction      != withdrawal
withdrawal      != explicit negation
invalidation    != contradiction
quarantine      != deletion
redaction       != semantic retraction
access revoke   != physical sanitization
archive         != erase
crypto erase    != proof every replica is sanitized
current reject  != historical nonexistence
ledger receipt  != issuer identity
commit integrity != semantic truth

A correction changes current interpretation/support through a new event. A redaction changes payload availability. Quarantine removes an item from active admissibility while preserving it for controlled investigation. Integrity failure says the system cannot trust/replay material; it does not manufacture not(P).


Typed conceptual model

sem_payload_envelope(
    PayloadRef,
    ObjectId,
    Namespace,
    StorageClass,
    EncryptionDomain,
    RetentionProfile,
    PayloadDigestRef,
    AvailabilityState).

sem_admission_receipt(
    AdmissionId,
    EventId,
    ClaimedIssuer,
    AuthenticatedSubmitter,
    AuthenticationEvidence,
    AuthorizationDecision,
    PolicyVersion,
    AdmittedAtFrontier).

sem_lifecycle_event(
    LifecycleEventId,
    TargetRef,
    Kind,                 % withdraw|correct|invalidate|quarantine|redact|purge|archive...
    ReasonClass,
    RequestedBy,
    AuthorizedBy,
    Validity,
    CommitRef,
    ReceiptRefs).

sem_redaction_receipt(
    ReceiptId,
    TargetRefs,
    Mode,                 % access_revoke|payload_redact|crypto_erase|physical_purge
    PolicyRef,
    AuthorityRef,
    BackendEvidence,
    CompletionState,      % complete|partial|pending|blocked|failed
    RetainedCommitmentRef,
    At).

sem_schema_interpretation(
    InterpretationId,
    RawEventRef,
    InputSchema,
    UpcasterVersion,
    OutputSchema,
    Relation,             % representation_preserving|semantic_migration|lossy
    InputDigest,
    OutputDigest,
    ReceiptRef).

sem_integrity_checkpoint(
    CheckpointId,
    Ledger,
    Frontier,
    PreviousCheckpoint,
    CommitmentRoot,
    Algorithm,
    SignerOrWitness,
    CreatedAt).

sem_replay_capability(
    SubjectRef,
    Frontier,
    Class,                % exact|semantic_only|commitment_only|redacted|corrupt|unavailable
    MissingRefs,
    Reason).

Exact representation remains implementation work; the semantic distinctions are canonical B output.


Mechanisms / guarantees

1. Governed erasure and redaction

A delete request is a lifecycle command with its own authorization/policy evaluation. It must not execute because remembered text says delete this.

Candidate modes:

logical_withdrawal
archive
access_revoke
payload_redact
crypto_erase
physical_purge

The backend advertises which guarantees it can actually make. A successful semantic redaction receipt cannot claim NIST-style sanitization unless the physical/storage layer performed and validated an appropriate method. Backups/replicas/holds may yield partial, pending, or blocked rather than fake success.

Redaction receipts themselves must avoid copying the sensitive value back into the audit trail. Low-entropy plaintext hashes can also leak equality/dictionary information; externally visible/global content digests are therefore not mandatory. Namespace/trust-domain keyed commitments or opaque object IDs are safer where equality disclosure is sensitive.

2. Historical truth after erasure

Bitemporal history now gains a third independent concern: payload availability/replay capability.

What did Machine Spirit accept on Jan 20?

may legitimately answer:

accepted(E) at frontier F
payload = redacted
replay = commitment_only
redaction receipt = R

It must not reconstruct erased plaintext from a derived cache or pretend exact replay remains possible.

Thus:

known_at != currently_accepted != currently_retrievable

3. Schema evolution / upcasting

Original raw event/object bytes remain bound to their original schema. An upcaster is a versioned derived interpretation.

raw event v1
    ↓ upcaster U7
interpreted representation v3

If U7 is representation-preserving, current projectors may consume it. If it changes meaning, that is a semantic migration/mapping and gets new lineage; it cannot masquerade as the original historical event.

Exact replay requires the original raw material plus the exact interpretation/upcaster artifact. If either is unavailable, replay capability is downgraded explicitly. Unknown/incompatible schema goes to preservation-only/quarantine, never active current knowledge.

4. Duplicate taxonomy

Do not use one generic dedup operation. Distinguish:

transport_retry_duplicate       -> same command key + same digest; no new occurrence
structural_semantic_duplicate   -> same proposition/object shape
same_source_repeat              -> repeated source episode; policy-dependent
independent_corroboration       -> distinct evidence origin; retain both supports
correlated_copy                 -> distinct episode but shared evidence origin
conflicting_same_id             -> hard integrity/idempotency failure

Content equality therefore never proves evidence independence.

5. Projection poisoning/staleness

A materialized view checkpoint is accepted only if its key matches:

ledger frontier
view implementation/version
semantic schema/profile versions
identity/mapping generation
admission/redaction/integrity generation
state digest

A projection can be thrown away. It can never heal or overwrite a corrupted canonical ledger. Action/VERIFY-sensitive queries must reject stale or integrity-failed projections rather than serving them as truth.

Derived projection output cannot promote itself into canonical semantic history without crossing the normal validation/admission command boundary.

6. Ledger integrity

Checksums detect accidental corruption; hash chains/Merkle commitments can make unauthorized alteration detectable; signed/witnessed checkpoints can make equivocation harder to hide. RFC 9162 is useful precisely because it also shows the limitation: a log can present inconsistent views unless roots are compared/monitored.

Therefore BSLPF/GRIE promises detectable integrity under its configured checkpoint/witness profile, not magic prevention of all tampering.

If canonical ledger/object integrity fails:

integrity_failure

is a first-class system state. Do not reconstruct canonical authority from a projection. Recovery must use an authenticated backup/replica/checkpoint lineage and create a recovery receipt.

7. Transaction/publication boundaries

The event batch must be atomic at the smallest semantic invariant boundary. Immutable payload objects may be staged before commit, but a frontier may advance only when every required object/reference is present and digest-valid.

Examples that belong in one publication commit:

  • assertion episode + source/provenance bindings required to interpret it;
  • correction + target lifecycle edge when the correction would otherwise become misleadingly active;
  • projection generation publication + all mandatory record refs;
  • redaction state transition + its authorization/completion receipt where policy requires atomic visibility.

Unrelated knowledge should not be forced into one mega-transaction. B defines atomic publication groups, not global serial transactions across the whole world model.

8. Authority authenticity

Imported event metadata such as principal(system) is merely a claim. Preserve separately:

claimed issuer
cryptographically/authentically established issuer where available
authenticated local submitter
local authorization/admission decision
registration/commit receipt

SCITT is a strong precedent: issuer signature and transparency-service receipt answer different questions. Imported signed knowledge may raise provenance quality under policy, but can never grant local host authority merely because an issuer name matches a local role.

9. Provenance scaling

Canonical history preserves atomic provenance edges and material lifecycle/admission receipts. It does not duplicate expanded proof trees on every derived record.

Use shared/hash-consed provenance DAGs/bundles, query-local expansion and compact summaries. Classify retained derivation material roughly as:

canonical_minimum
reproducibility_required
rebuildable_derived

A rebuildable proof/cache may be evicted if all inputs + reasoner/profile versions needed to regenerate it remain available. A promoted action/VERIFY/learning receipt may require stronger retention. If privacy policy redacts a source, explanations degrade to authorized opaque commitments rather than leaking the source through provenance.


Complexity / scaling implications

  • append cost stays approximately O(batch size) plus integrity-checkpoint overhead;
  • Merkle/checkpoint inclusion/consistency proofs can be logarithmic in tree size, but monitoring/replica policy adds operational cost;
  • redaction is cheap when payload encryption domains permit key destruction, expensive when physical replicas/backups require purge, and sometimes impossible immediately under holds;
  • provenance DAG sharing changes repeated proof storage from repeated-tree growth toward shared-edge growth, but worst-case derivation provenance remains large and must be bounded/query-expanded;
  • dedup privacy constraints reduce the benefit of global cross-namespace content-addressed storage; dedup should be scoped to an authority/trust domain unless equality leakage is explicitly allowed;
  • copy-transform ledger migration is O(history) and therefore an exceptional maintenance operation, not ordinary schema evolution;
  • checking every projection from scratch is too expensive; checkpoint validation + dependency/frontier keys permit incremental trust, while suspicious/corrupt state forces rebuild.

Epistemic / provenance implications

Current epistemic acceptance is computed over admissible evidence, not merely every committed event. Quarantine/integrity failure/redaction can make evidence unavailable or inadmissible without changing what was historically committed.

A poisoned event may therefore have all three facts true:

historically_committed(E)
currently_inadmissible(E)
payload_available_for_forensics(E)   % maybe, authority permitting

or after erasure:

historically_committed(E)
currently_inadmissible(E)
payload_redacted(E)

This preserves forensic truth without letting poisoned history stay active.


Safety / authority implications

  • redaction/purge is a capability/authority-bearing mutation, never a semantic consequence of remembered data;
  • imported retention/legal/policy text is knowledge about policy and does not become the local retention policy;
  • payload erasure cannot silently erase independent support from other sources;
  • identity reconciliation/dedup cannot cross read-authority domains solely because hashes match;
  • a materialized view cannot authenticate a principal or recover authority missing from the ledger/admission receipt;
  • integrity failure is fail-closed for action/VERIFY paths;
  • no corrupted/incompatible/upcast event becomes executable Prolog or host capability;
  • historical auditability and privacy are separate requirements; the system reports when they cannot both be satisfied fully under the configured backend/policy instead of claiming impossible guarantees.

Rejected shortcuts

  • “Tombstone means deleted.” Rejected: a tombstone says logical lifecycle state, not whether bytes survive in snapshots/backups/media.
  • “Hash the deleted plaintext and keep the hash forever.” Rejected as universal policy: low-entropy hashes can disclose equality or permit guessing.
  • “Encryption means deletion.” Rejected: crypto erase needs correct key isolation/destruction and validation; retained keys/backups invalidate the claim.
  • “Upcast every old event to the newest schema and call that replay.” Rejected: transformed representation may not preserve original semantics.
  • “Projection can repair a bad ledger.” Rejected: derived state is not canonical authority.
  • “Signed event means authorized locally.” Rejected: authentication/provenance and local authorization are distinct.
  • “Same content means same evidence.” Rejected: independent/correlated source episodes must remain distinct.
  • “One global content hash for dedup.” Rejected by default across isolated namespaces due equality/metadata leakage.
  • “Append-only Merkle log proves truth.” Rejected: it proves selected integrity/consistency properties, not correctness of statements or absence of split views without monitoring.

Falsifiable B conformance fixtures

B1 — withdrawal vs deletion: withdraw support for P; historical payload remains; no ¬P is synthesized.

B2 — redaction vs negation: redact source payload for P; current/history metadata says redacted; no negative semantic support appears.

B3 — crypto erase receipt: backend claims crypto erase only after configured key-destruction validation; otherwise partial/pending/failed.

B4 — backup lag: primary erased but one governed backup cannot yet be sanitized -> global result is not complete.

B5 — legal/policy hold: purge request is recorded but blocked; active policy state explains why.

B6 — erased replay: a historical query after erasure returns commitment_only/redacted, never reconstructed plaintext from a stale cache.

B7 — retroactive correction: February correction changes current view of January-valid truth while known_at(January) still reports the earlier accepted state.

B8 — correction vs world change: correction of bad January data does not create a February world-state transition.

B9 — transport retry: same idempotency key + same batch digest returns original receipt/no duplicate event.

B10 — idempotency collision: same key + different batch digest is a hard conflict.

B11 — independent duplicate: two independent sources assert structurally identical P; proposition may dedup, support episodes do not.

B12 — copied duplicate: ten articles copied from one origin produce ten episodes but one dependence component.

B13 — cross-namespace equality leak: unauthorized caller cannot discover that a private namespace stores equal payload via global object ID/hash behavior.

B14 — poisoned projection: mutate/corrupt a materialized current view; checkpoint mismatch causes rejection/rebuild, not canonical append.

B15 — stale view: action/VERIFY query against stale frontier fails/refreshes rather than using cached truth.

B16 — corrupted ledger segment: canonical integrity failure cannot be healed from a projection; authenticated recovery lineage is required.

B17 — Merkle consistency: later checkpoint is accepted only with valid configured consistency relation to prior checkpoint.

B18 — split-view detection fixture: two incompatible signed checkpoint roots at same/logically comparable frontier produce equivocation/integrity alarm, not arbitrary winner.

B19 — spoofed principal field: imported event claiming system/admin remains untrusted claim unless issuer/authentication/local admission establish it.

B20 — signed-but-unauthorized: cryptographically valid issuer statement remains locally inadmissible if policy denies it.

B21 — representation-preserving upcast: original event stays immutable; exact versioned upcaster produces deterministic equivalent representation.

B22 — semantic upcast: an upcaster that changes meaning must be labeled migration/lossy and cannot satisfy exact historical replay.

B23 — missing old upcaster: historical exact replay reports degraded/unavailable instead of silently using today's semantics.

B24 — provenance redaction: explanation for an authorized caller can use surviving commitment/lifecycle refs without leaking erased source bytes.

B25 — atomic projection publish: crash midway through staged immutable-object writes exposes no active half-package before commit/frontier advance.

B26 — atomic correction: current view never observes replacement active without its required correction/supersession relation when those are one publication invariant.

B27 — quarantine selective repair: poisoned source and dependent derived views are invalidated/rebuilt; unrelated evidence remains active.

B28 — cache cannot promote itself: corrupted/recomputed projection data cannot create canonical events except through validated append/admission API.

B29 — read authority on history: knowing an event/object/commit ID does not reveal redacted/private payload or existence beyond allowed metadata policy.

B30 — integrity is not truth: a perfectly valid inclusion/commitment proof for false/untrusted P leaves P epistemically untrusted unless its support/admission semantics warrant it.


Canonical issue changes / ownership

lost-rob0t/prolog-rlm

  • #402: BSLPF is refined by GRIE; B marked complete, C next.
  • #392: semantic IR continues to own immutable semantic meaning/version identity; it must not absorb physical retention/authentication policy.
  • #394: epistemic reasoning consumes only evidence admitted under the memory frontier/integrity/admissibility contract; redaction/quarantine/integrity failure are not negation.
  • #395: later persistence conformance should include the B fixtures above where they intersect semantic replay/current/history behavior.
  • #376/#384: authority and VERIFY remain independent of memory provenance; stale/integrity-failed projections cannot satisfy high-authority evidence requirements.

lost-rob0t/symbolic-memory

  • #6 owns ledger events, payload-availability/redaction lifecycle, admission receipts, exact frontiers, integrity checkpoints, schema interpretation lineage and rebuildable projections.
  • #5 owns projection publication atomicity and compiler/upcaster lineage; a projection cannot be ready when its source/schema/admission state is quarantined/integrity-failed.
  • #7 owns query result distinctions among current acceptance, historical commitment, retrievability and replay capability.
  • #9 owns capability-gated redaction/history/source-expansion surfaces; transport must not flatten redacted/quarantined/integrity_failure to not-found/false.
  • #10 owns reversible identity/alignment and must not use cross-authority global content equality as implicit merge evidence.

Physical sanitization implementation stays behind the Symbolic Memory storage adapter/backend capability boundary. Prolog-RLM does not become a database, KMS or transparency-log service.


Unresolved questions preserved for C/D

  1. C: how are multiple ledgers/namespaces/backends federated when some support cryptographic erase, some immutable retention, some external transparency receipts, and some weaker integrity semantics?
  2. C: how should replicated/federated ledgers exchange redaction notices without leaking deleted content or falsely claiming remote erasure?
  3. C: what is the cross-ledger causal/order model; which checkpoint/witness roots are portable?
  4. C: how are external W3C PROV / SCITT / database provenance identities mapped without conflating issuer, registrar, submitter and local authority?
  5. C: what are safe cross-namespace dedup/content-addressing boundaries?
  6. D: exact integrity structure — hash chain, Merkle tree, segmented Merkle forest, signed checkpoints, external witness, or combinations — needs benchmarked realization.
  7. D: key hierarchy/granularity for crypto erase without absurd key-management overhead.
  8. D: backup/replica purge protocol and measurable completion proof.
  9. D: scalable shared provenance representation and retention thresholds.
  10. D: projection/checkpoint verification cost and rebuild thresholds at millions/billions of events.
  11. D: corruption repair/replica selection and deterministic replay after partial loss.
  12. D: empirically justified publication batch sizes/frontier granularity.

Completion statement

#402B is design-complete only. It does not prove implementation, physical sanitization, tamper resistance, privacy compliance, or Machine Spirit acceptance.

The surviving architecture is:

A: BSLPF
  immutable semantic identity + bitemporal authoritative history + rebuildable views
                       ↓
B: GRIE
  governed payload availability/redaction + authenticated admission +
  schema-interpretation lineage + integrity checkpoints + fail-closed projections

Exact next research subpass: #402C — composition / heterogeneity / interoperability.

Do not start #402D or Depth 6/#403 until C is durably complete.

## #402B — semantic/adversarial deepening — COMPLETE ### Scope / attack surface This B subpass attacks the assumptions behind #402A's **BSLPF — Bitemporal Semantic Ledger + Projection Fabric** rather than repeating its storage survey. The questions are: 1. Can an append-only semantic world model support actual erasure/redaction without lying about what remains recoverable? 2. How must correction, withdrawal, invalidation, quarantine, redaction, archival and physical deletion differ? 3. What does historical replay mean after payload erasure, schema evolution, corruption, or later discovery that an event was unauthorized/poisoned? 4. Can event/source identity and content deduplication create privacy or corroboration bugs? 5. How are event schema/upcaster changes prevented from silently rewriting historical meaning? 6. How do projection caches fail safely when stale, poisoned or corrupted? 7. What integrity properties can an append-only ledger actually guarantee, and what still requires monitoring/authentication/recovery? 8. How do actor identity, provenance and registration/commit receipts avoid authority spoofing? 9. How can provenance remain complete enough for explanation/replay without exploding into duplicated proof trees? 10. Which guarantees belong to the semantic contract versus a physical backend's sanitization/replication machinery? ### Current-state comparison #402A already established the right baseline separation: immutable/versioned semantic objects, ordered bitemporal ledger events and rebuildable projections. #400 requires `valid_at` vs `known_at`, explicit support/defeat lineage and derived warrant caches; #401 requires immutable procedure/PSC versions plus material execution/VERIFY/replay lineage. The current Symbolic Memory issues already distinguish withdrawal from negation and current views from history. The hidden failure is that **`append-only` was still overloaded**. Historical research `SYMBOLIC-MEMORY-RESEARCH-008` explicitly says forgetting usually differs from deletion and reserves physical erase for policy/user requirements. Research-009 treats persistent memory as security-sensitive execution input and requires selective poisoning repair. Research-006's bootstrap storage analysis also never established a strong physical-erasure/tamper-evidence contract. Therefore BSLPF needs an explicit governance/integrity layer rather than interpreting logical append-only history as permanent recoverability of every historical payload. --- ## Primary / authoritative evidence 1. **EU General Data Protection Regulation, Regulation (EU) 2016/679, Article 17 (2016).** Article 17 defines circumstances where a controller must erase personal data without undue delay and also defines exceptions. This is a concrete counterexample to a design that promises every plaintext event payload is immortal: https://eur-lex.europa.eu/eli/reg/2016/679 2. **NIST SP 800-88 Rev. 2, Guidelines for Media Sanitization (final 2025-09-26).** Sanitization is defined in terms of making access to target data infeasible for a given level of effort; the revision explicitly includes cryptographic erase and sanitization validation. Logical deletion in a semantic ledger is therefore not automatically physical sanitization: https://csrc.nist.gov/pubs/sp/800/88/r2/final 3. **Peterson et al., “Secure Deletion for a Versioning File System,” FAST 2005.** Demonstrates that versioned storage requires deliberate deletion architecture; authenticated encryption plus destruction/overwriting of small key/stub material can make selected historical versions unrecoverable without rewriting all shared blocks: https://www.usenix.org/conference/fast-05/secure-deletion-versioning-file-system 4. **Reardon, Capkun, Basin, “Data Node Encrypted File System: Efficient Secure Deletion for Flash Memory,” USENIX Security 2012.** Shows why simple overwrite assumptions fail on flash and uses encryption-based techniques for secure deletion: https://www.usenix.org/conference/usenixsecurity12/technical-sessions/presentation/reardon 5. **RFC 9162, Certificate Transparency Version 2.0 (2021).** Merkle consistency proofs efficiently establish that a later log state extends an earlier one, but the RFC also warns that inconsistent/split views require auditing/monitoring. Append-only structure is tamper-evidence machinery, not semantic truth or complete protection from a malicious log: https://www.rfc-editor.org/rfc/rfc9162.html 6. **RFC 9943, SCITT architecture (2026).** Separates an issuer's signed statement from a Transparency Service registration/receipt. A client can register on behalf of an issuer, and a receipt proves registration, not authorship. This is directly useful for preventing `principal=admin` fields inside imported history from becoming authenticated local authority: https://www.rfc-editor.org/rfc/rfc9943.html 7. **Overeem, Spoor, Jansen, Brinkkemper, “An empirical characterization of event sourced systems and their schema evolution — Lessons from industry,” JSS 178, 2021.** Study of 19 systems/25 engineers identifies rebuilding projections and privacy as event-sourcing challenges and reports five schema-evolution techniques: versioned events, weak schema, upcasting, in-place transform, copy-and-transform. Upcasting retains immutable stored events; in-place transformation explicitly gives up immutability/audit guarantees: https://doi.org/10.1016/j.jss.2021.110970 8. **W3C PROV-O Recommendation (2013).** Provenance bundles are first-class entities and can themselves carry provenance, supporting provenance-of-provenance and shared provenance structures rather than duplicated opaque metadata blobs: https://www.w3.org/TR/prov-o/ These sources establish mechanisms and failure boundaries; they do **not** imply that Symbolic Memory is automatically legally compliant or that one cryptographic/logging mechanism is universally sufficient. --- # Candidate architectures ## Candidate B1 — physically immutable plaintext event log ```text all event envelopes + all historical payloads ↓ permanent append-only store ``` ### Advantage Maximum naive replay simplicity. ### Fatal problems - conflicts with legitimate erasure/sanitization requirements; - poisoned/secrets-containing payloads remain recoverable forever; - backups/replicas make “delete” dishonest; - content-addressed global hashes can preserve equality/fingerprinting side channels after nominal deletion; - an event claiming a principal remains spoofable unless identity is independently authenticated. **Rejected.** Logical historical integrity cannot require perpetual recoverability of every payload byte. ## Candidate B2 — mutable history with in-place redaction ```text ledger row ↓ delete/replace sensitive fields in place ↓ append audit marker ``` ### Advantage Straightforward privacy/delete semantics and ordinary database tooling. ### Fatal problems - breaks exact historical replay and append-only integrity; - cannot distinguish authorized redaction from illicit tampering without an independent commitment history; - cache/checkpoint/proof roots become ambiguous after mutation; - schema “fixes” can silently rewrite the past. **Rejected as canonical.** Controlled physical compaction/migration may use copy-transform, but it creates a new ledger generation with an explicit migration receipt rather than pretending the old history never existed. ## Candidate B3 — BSLPF + Governed Redaction & Integrity Envelope (**preferred**) Call the B-level refinement **GRIE — Governed Redaction & Integrity Envelope**. ```text immutable/versioned semantic identity + retention/encryption-scoped payload objects ↓ AUTHORITATIVE BITEMPORAL CONTROL LEDGER append-only commitments + lifecycle events issuer/authentication/authorization receipts correction/withdrawal/quarantine/redaction events schema/upcaster/migration receipts segment/checkpoint integrity commitments ↓ admissibility / retention gate ↓ rebuildable current/history/epistemic/procedural views ``` The hard refinement is: ```text append-only logical commitment history != permanent payload retrievability ``` The ledger can preserve that event/object `E` existed, was accepted at frontier `F`, and was later redacted under policy/authority `R`, while the protected payload becomes cryptographically or physically unrecoverable where the backend/policy actually supports that guarantee. --- # Hard semantic distinctions added by B ```text correction != withdrawal withdrawal != explicit negation invalidation != contradiction quarantine != deletion redaction != semantic retraction access revoke != physical sanitization archive != erase crypto erase != proof every replica is sanitized current reject != historical nonexistence ledger receipt != issuer identity commit integrity != semantic truth ``` A correction changes current interpretation/support through a new event. A redaction changes payload availability. Quarantine removes an item from active admissibility while preserving it for controlled investigation. Integrity failure says the system cannot trust/replay material; it does not manufacture `not(P)`. --- # Typed conceptual model ```prolog sem_payload_envelope( PayloadRef, ObjectId, Namespace, StorageClass, EncryptionDomain, RetentionProfile, PayloadDigestRef, AvailabilityState). sem_admission_receipt( AdmissionId, EventId, ClaimedIssuer, AuthenticatedSubmitter, AuthenticationEvidence, AuthorizationDecision, PolicyVersion, AdmittedAtFrontier). sem_lifecycle_event( LifecycleEventId, TargetRef, Kind, % withdraw|correct|invalidate|quarantine|redact|purge|archive... ReasonClass, RequestedBy, AuthorizedBy, Validity, CommitRef, ReceiptRefs). sem_redaction_receipt( ReceiptId, TargetRefs, Mode, % access_revoke|payload_redact|crypto_erase|physical_purge PolicyRef, AuthorityRef, BackendEvidence, CompletionState, % complete|partial|pending|blocked|failed RetainedCommitmentRef, At). sem_schema_interpretation( InterpretationId, RawEventRef, InputSchema, UpcasterVersion, OutputSchema, Relation, % representation_preserving|semantic_migration|lossy InputDigest, OutputDigest, ReceiptRef). sem_integrity_checkpoint( CheckpointId, Ledger, Frontier, PreviousCheckpoint, CommitmentRoot, Algorithm, SignerOrWitness, CreatedAt). sem_replay_capability( SubjectRef, Frontier, Class, % exact|semantic_only|commitment_only|redacted|corrupt|unavailable MissingRefs, Reason). ``` Exact representation remains implementation work; the semantic distinctions are canonical B output. --- # Mechanisms / guarantees ## 1. Governed erasure and redaction A delete request is a lifecycle command with its own authorization/policy evaluation. It must not execute because remembered text says `delete this`. Candidate modes: ```text logical_withdrawal archive access_revoke payload_redact crypto_erase physical_purge ``` The backend advertises which guarantees it can actually make. A successful semantic `redaction` receipt cannot claim NIST-style sanitization unless the physical/storage layer performed and validated an appropriate method. Backups/replicas/holds may yield `partial`, `pending`, or `blocked` rather than fake success. Redaction receipts themselves must avoid copying the sensitive value back into the audit trail. Low-entropy plaintext hashes can also leak equality/dictionary information; externally visible/global content digests are therefore not mandatory. Namespace/trust-domain keyed commitments or opaque object IDs are safer where equality disclosure is sensitive. ## 2. Historical truth after erasure Bitemporal history now gains a third independent concern: **payload availability/replay capability**. ```text What did Machine Spirit accept on Jan 20? ``` may legitimately answer: ```text accepted(E) at frontier F payload = redacted replay = commitment_only redaction receipt = R ``` It must not reconstruct erased plaintext from a derived cache or pretend exact replay remains possible. Thus: ```text known_at != currently_accepted != currently_retrievable ``` ## 3. Schema evolution / upcasting Original raw event/object bytes remain bound to their original schema. An upcaster is a **versioned derived interpretation**. ```text raw event v1 ↓ upcaster U7 interpreted representation v3 ``` If U7 is representation-preserving, current projectors may consume it. If it changes meaning, that is a semantic migration/mapping and gets new lineage; it cannot masquerade as the original historical event. Exact replay requires the original raw material plus the exact interpretation/upcaster artifact. If either is unavailable, replay capability is downgraded explicitly. Unknown/incompatible schema goes to preservation-only/quarantine, never active current knowledge. ## 4. Duplicate taxonomy Do not use one generic `dedup` operation. Distinguish: ```text transport_retry_duplicate -> same command key + same digest; no new occurrence structural_semantic_duplicate -> same proposition/object shape same_source_repeat -> repeated source episode; policy-dependent independent_corroboration -> distinct evidence origin; retain both supports correlated_copy -> distinct episode but shared evidence origin conflicting_same_id -> hard integrity/idempotency failure ``` Content equality therefore never proves evidence independence. ## 5. Projection poisoning/staleness A materialized view checkpoint is accepted only if its key matches: ```text ledger frontier view implementation/version semantic schema/profile versions identity/mapping generation admission/redaction/integrity generation state digest ``` A projection can be thrown away. It can never heal or overwrite a corrupted canonical ledger. Action/VERIFY-sensitive queries must reject stale or integrity-failed projections rather than serving them as truth. Derived projection output cannot promote itself into canonical semantic history without crossing the normal validation/admission command boundary. ## 6. Ledger integrity Checksums detect accidental corruption; hash chains/Merkle commitments can make unauthorized alteration detectable; signed/witnessed checkpoints can make equivocation harder to hide. RFC 9162 is useful precisely because it also shows the limitation: a log can present inconsistent views unless roots are compared/monitored. Therefore BSLPF/GRIE promises **detectable integrity under its configured checkpoint/witness profile**, not magic prevention of all tampering. If canonical ledger/object integrity fails: ```text integrity_failure ``` is a first-class system state. Do not reconstruct canonical authority from a projection. Recovery must use an authenticated backup/replica/checkpoint lineage and create a recovery receipt. ## 7. Transaction/publication boundaries The event batch must be atomic at the smallest semantic invariant boundary. Immutable payload objects may be staged before commit, but a frontier may advance only when every required object/reference is present and digest-valid. Examples that belong in one publication commit: - assertion episode + source/provenance bindings required to interpret it; - correction + target lifecycle edge when the correction would otherwise become misleadingly active; - projection generation publication + all mandatory record refs; - redaction state transition + its authorization/completion receipt where policy requires atomic visibility. Unrelated knowledge should not be forced into one mega-transaction. B defines atomic *publication groups*, not global serial transactions across the whole world model. ## 8. Authority authenticity Imported event metadata such as `principal(system)` is merely a claim. Preserve separately: ```text claimed issuer cryptographically/authentically established issuer where available authenticated local submitter local authorization/admission decision registration/commit receipt ``` SCITT is a strong precedent: issuer signature and transparency-service receipt answer different questions. Imported signed knowledge may raise provenance quality under policy, but can never grant local host authority merely because an issuer name matches a local role. ## 9. Provenance scaling Canonical history preserves atomic provenance edges and material lifecycle/admission receipts. It does **not** duplicate expanded proof trees on every derived record. Use shared/hash-consed provenance DAGs/bundles, query-local expansion and compact summaries. Classify retained derivation material roughly as: ```text canonical_minimum reproducibility_required rebuildable_derived ``` A rebuildable proof/cache may be evicted if all inputs + reasoner/profile versions needed to regenerate it remain available. A promoted action/VERIFY/learning receipt may require stronger retention. If privacy policy redacts a source, explanations degrade to authorized opaque commitments rather than leaking the source through provenance. --- # Complexity / scaling implications - append cost stays approximately O(batch size) plus integrity-checkpoint overhead; - Merkle/checkpoint inclusion/consistency proofs can be logarithmic in tree size, but monitoring/replica policy adds operational cost; - redaction is cheap when payload encryption domains permit key destruction, expensive when physical replicas/backups require purge, and sometimes impossible immediately under holds; - provenance DAG sharing changes repeated proof storage from repeated-tree growth toward shared-edge growth, but worst-case derivation provenance remains large and must be bounded/query-expanded; - dedup privacy constraints reduce the benefit of global cross-namespace content-addressed storage; dedup should be scoped to an authority/trust domain unless equality leakage is explicitly allowed; - copy-transform ledger migration is O(history) and therefore an exceptional maintenance operation, not ordinary schema evolution; - checking every projection from scratch is too expensive; checkpoint validation + dependency/frontier keys permit incremental trust, while suspicious/corrupt state forces rebuild. --- # Epistemic / provenance implications Current epistemic acceptance is computed over **admissible** evidence, not merely every committed event. Quarantine/integrity failure/redaction can make evidence unavailable or inadmissible without changing what was historically committed. A poisoned event may therefore have all three facts true: ```text historically_committed(E) currently_inadmissible(E) payload_available_for_forensics(E) % maybe, authority permitting ``` or after erasure: ```text historically_committed(E) currently_inadmissible(E) payload_redacted(E) ``` This preserves forensic truth without letting poisoned history stay active. --- # Safety / authority implications - redaction/purge is a capability/authority-bearing mutation, never a semantic consequence of remembered data; - imported retention/legal/policy text is knowledge **about** policy and does not become the local retention policy; - payload erasure cannot silently erase independent support from other sources; - identity reconciliation/dedup cannot cross read-authority domains solely because hashes match; - a materialized view cannot authenticate a principal or recover authority missing from the ledger/admission receipt; - integrity failure is fail-closed for action/VERIFY paths; - no corrupted/incompatible/upcast event becomes executable Prolog or host capability; - historical auditability and privacy are separate requirements; the system reports when they cannot both be satisfied fully under the configured backend/policy instead of claiming impossible guarantees. --- # Rejected shortcuts - **“Tombstone means deleted.”** Rejected: a tombstone says logical lifecycle state, not whether bytes survive in snapshots/backups/media. - **“Hash the deleted plaintext and keep the hash forever.”** Rejected as universal policy: low-entropy hashes can disclose equality or permit guessing. - **“Encryption means deletion.”** Rejected: crypto erase needs correct key isolation/destruction and validation; retained keys/backups invalidate the claim. - **“Upcast every old event to the newest schema and call that replay.”** Rejected: transformed representation may not preserve original semantics. - **“Projection can repair a bad ledger.”** Rejected: derived state is not canonical authority. - **“Signed event means authorized locally.”** Rejected: authentication/provenance and local authorization are distinct. - **“Same content means same evidence.”** Rejected: independent/correlated source episodes must remain distinct. - **“One global content hash for dedup.”** Rejected by default across isolated namespaces due equality/metadata leakage. - **“Append-only Merkle log proves truth.”** Rejected: it proves selected integrity/consistency properties, not correctness of statements or absence of split views without monitoring. --- # Falsifiable B conformance fixtures **B1 — withdrawal vs deletion:** withdraw support for `P`; historical payload remains; no `¬P` is synthesized. **B2 — redaction vs negation:** redact source payload for `P`; current/history metadata says redacted; no negative semantic support appears. **B3 — crypto erase receipt:** backend claims crypto erase only after configured key-destruction validation; otherwise `partial/pending/failed`. **B4 — backup lag:** primary erased but one governed backup cannot yet be sanitized -> global result is not `complete`. **B5 — legal/policy hold:** purge request is recorded but blocked; active policy state explains why. **B6 — erased replay:** a historical query after erasure returns `commitment_only/redacted`, never reconstructed plaintext from a stale cache. **B7 — retroactive correction:** February correction changes current view of January-valid truth while `known_at(January)` still reports the earlier accepted state. **B8 — correction vs world change:** correction of bad January data does not create a February world-state transition. **B9 — transport retry:** same idempotency key + same batch digest returns original receipt/no duplicate event. **B10 — idempotency collision:** same key + different batch digest is a hard conflict. **B11 — independent duplicate:** two independent sources assert structurally identical `P`; proposition may dedup, support episodes do not. **B12 — copied duplicate:** ten articles copied from one origin produce ten episodes but one dependence component. **B13 — cross-namespace equality leak:** unauthorized caller cannot discover that a private namespace stores equal payload via global object ID/hash behavior. **B14 — poisoned projection:** mutate/corrupt a materialized current view; checkpoint mismatch causes rejection/rebuild, not canonical append. **B15 — stale view:** action/VERIFY query against stale frontier fails/refreshes rather than using cached truth. **B16 — corrupted ledger segment:** canonical integrity failure cannot be healed from a projection; authenticated recovery lineage is required. **B17 — Merkle consistency:** later checkpoint is accepted only with valid configured consistency relation to prior checkpoint. **B18 — split-view detection fixture:** two incompatible signed checkpoint roots at same/logically comparable frontier produce equivocation/integrity alarm, not arbitrary winner. **B19 — spoofed principal field:** imported event claiming `system/admin` remains untrusted claim unless issuer/authentication/local admission establish it. **B20 — signed-but-unauthorized:** cryptographically valid issuer statement remains locally inadmissible if policy denies it. **B21 — representation-preserving upcast:** original event stays immutable; exact versioned upcaster produces deterministic equivalent representation. **B22 — semantic upcast:** an upcaster that changes meaning must be labeled migration/lossy and cannot satisfy exact historical replay. **B23 — missing old upcaster:** historical exact replay reports degraded/unavailable instead of silently using today's semantics. **B24 — provenance redaction:** explanation for an authorized caller can use surviving commitment/lifecycle refs without leaking erased source bytes. **B25 — atomic projection publish:** crash midway through staged immutable-object writes exposes no active half-package before commit/frontier advance. **B26 — atomic correction:** current view never observes replacement active without its required correction/supersession relation when those are one publication invariant. **B27 — quarantine selective repair:** poisoned source and dependent derived views are invalidated/rebuilt; unrelated evidence remains active. **B28 — cache cannot promote itself:** corrupted/recomputed projection data cannot create canonical events except through validated append/admission API. **B29 — read authority on history:** knowing an event/object/commit ID does not reveal redacted/private payload or existence beyond allowed metadata policy. **B30 — integrity is not truth:** a perfectly valid inclusion/commitment proof for false/untrusted `P` leaves `P` epistemically untrusted unless its support/admission semantics warrant it. --- # Canonical issue changes / ownership ### `lost-rob0t/prolog-rlm` - **#402:** BSLPF is refined by GRIE; B marked complete, C next. - **#392:** semantic IR continues to own immutable semantic meaning/version identity; it must not absorb physical retention/authentication policy. - **#394:** epistemic reasoning consumes only evidence admitted under the memory frontier/integrity/admissibility contract; redaction/quarantine/integrity failure are not negation. - **#395:** later persistence conformance should include the B fixtures above where they intersect semantic replay/current/history behavior. - **#376/#384:** authority and VERIFY remain independent of memory provenance; stale/integrity-failed projections cannot satisfy high-authority evidence requirements. ### `lost-rob0t/symbolic-memory` - **#6 owns** ledger events, payload-availability/redaction lifecycle, admission receipts, exact frontiers, integrity checkpoints, schema interpretation lineage and rebuildable projections. - **#5 owns** projection publication atomicity and compiler/upcaster lineage; a projection cannot be ready when its source/schema/admission state is quarantined/integrity-failed. - **#7 owns** query result distinctions among current acceptance, historical commitment, retrievability and replay capability. - **#9 owns** capability-gated redaction/history/source-expansion surfaces; transport must not flatten `redacted/quarantined/integrity_failure` to not-found/false. - **#10 owns** reversible identity/alignment and must not use cross-authority global content equality as implicit merge evidence. Physical sanitization implementation stays behind the Symbolic Memory storage adapter/backend capability boundary. Prolog-RLM does not become a database, KMS or transparency-log service. --- # Unresolved questions preserved for C/D 1. C: how are multiple ledgers/namespaces/backends federated when some support cryptographic erase, some immutable retention, some external transparency receipts, and some weaker integrity semantics? 2. C: how should replicated/federated ledgers exchange redaction notices without leaking deleted content or falsely claiming remote erasure? 3. C: what is the cross-ledger causal/order model; which checkpoint/witness roots are portable? 4. C: how are external W3C PROV / SCITT / database provenance identities mapped without conflating issuer, registrar, submitter and local authority? 5. C: what are safe cross-namespace dedup/content-addressing boundaries? 6. D: exact integrity structure — hash chain, Merkle tree, segmented Merkle forest, signed checkpoints, external witness, or combinations — needs benchmarked realization. 7. D: key hierarchy/granularity for crypto erase without absurd key-management overhead. 8. D: backup/replica purge protocol and measurable completion proof. 9. D: scalable shared provenance representation and retention thresholds. 10. D: projection/checkpoint verification cost and rebuild thresholds at millions/billions of events. 11. D: corruption repair/replica selection and deterministic replay after partial loss. 12. D: empirically justified publication batch sizes/frontier granularity. ## Completion statement **#402B is design-complete only.** It does not prove implementation, physical sanitization, tamper resistance, privacy compliance, or Machine Spirit acceptance. The surviving architecture is: ```text A: BSLPF immutable semantic identity + bitemporal authoritative history + rebuildable views ↓ B: GRIE governed payload availability/redaction + authenticated admission + schema-interpretation lineage + integrity checkpoints + fail-closed projections ``` **Exact next research subpass: #402C — composition / heterogeneity / interoperability.** Do not start #402D or Depth 6/#403 until C is durably complete.
Author
Owner

MACHINE-SPIRIT #402C — composition / heterogeneity / interoperability

Status: COMPLETE (design only; implementation unproven).

Research scope / questions

This subpass attacks the assumption that multiple durable world-model stores can be treated as one bigger append-only log. It asks:

  1. What ordering can be claimed across independently governed ledgers without inventing a global clock?
  2. How do replicas, backups, mirrors, transparency logs and genuinely independent ledgers differ semantically?
  3. How can redaction/retention obligations propagate without one store falsely claiming another store physically erased data?
  4. How do heterogeneous integrity/transparency mechanisms compose without collapsing to a weakest-common-denominator Boolean?
  5. How should external statements/provenance be imported while preserving issuer, registration, admission and local authority boundaries?
  6. How do identity, deduplication and corroboration work across namespaces without leaking equality or double-counting copies?
  7. What exact world-model cut/frontier is required for deterministic federated replay?

Current-system comparison

#402A established BSLPF: immutable/versioned semantic objects + authoritative bitemporal ledger + rebuildable projections. #402B added GRIE: payload retrievability, redaction/sanitization, admission, replay and integrity are separate dimensions.

Upstream #392 already treats contexts/theories/bridges and identity mappings as explicit versioned semantic objects; #400C likewise established that heterogeneous epistemic theories exchange status-bearing objects through typed bridges rather than sharing one global truth relation. Downstream symbolic-memory#6 still describes one append-only semantic ledger/current-history view, and the current implementation is even narrower: prolog/symbolic_memory_storage.pl serializes one in-process dynamic-predicate snapshot under a mutex and atomically renames the complete file. That bootstrap remains useful but cannot define federation semantics.

Historical SYMBOLIC-MEMORY-RESEARCH-006 explicitly treated replication/remote multi-process access as optional future work and correctly insisted on a replaceable storage adapter. C preserves that backend independence while making the logical federation contract explicit.

Primary / authoritative evidence

  • Leslie Lamport, “Time, Clocks, and the Ordering of Events in a Distributed System,” CACM 21(7), July 1978 — partial causal order is fundamental; a total order may be constructed but must not be confused with causality. https://lamport.azurewebsites.net/pubs/time-clocks.pdf
  • Shapiro, Preguiça, Baquero, Zawirski, “A Comprehensive Study of Convergent and Commutative Replicated Data Types,” INRIA RR-7506, January 2011 — strong/eventual convergence is a property of replicas of a shared abstract object under explicit state/operation conditions; it is not a semantic reconciliation rule for unrelated authorities. https://inria.hal.science/inria-00555588
  • W3C, PROV-DM / PROV-O / Linking Across Provenance Bundles, April 30 2013 — provenance is designed for heterogeneous interchange; bundles can remain independently valid “islands” and can be linked without flattening provenance identity. https://www.w3.org/TR/prov-dm/ and https://www.w3.org/TR/prov-links/
  • IETF, RFC 9162 Certificate Transparency v2, December 2021 — Merkle inclusion/consistency can prove append-only log behavior, while consistency of the view shown to all parties is a separate auditing problem. https://www.rfc-editor.org/rfc/rfc9162
  • EU, GDPR Article 17(2), Regulation (EU) 2016/679 — when public personal data must be erased, controllers must take reasonable steps to inform other controllers processing copies/replications; this is an obligation/notification relationship, not proof remote copies have already been physically sanitized. https://eur-lex.europa.eu/eli/reg/2016/679/oj
  • IETF, RFC 9943 SCITT Architecture, June 2026 — a Signed Statement may be registered with multiple independent Transparency Services and carry multiple receipts; receipts prove registration under a TS policy, not semantic truth. SCITT deliberately supports multiple VDS/receipt implementations and leaves relying-party trust policy local. https://www.rfc-editor.org/rfc/rfc9943

Candidate architectures

Candidate 1 — one canonical federation super-ledger

Import every remote event into one globally sequenced append-only log and assign a federation-wide commit number.

Rejected as canonical. It gives convenient snapshots but invents order among unrelated concurrent events, centralizes authority, forces heterogeneous retention/erasure rules into one policy and creates a huge privacy/availability boundary. A sequenced super-ledger can still be an explicit deployment profile where participants intentionally accept one sequencer/consensus domain.

Candidate 2 — CRDT-union world model

Treat all stores as replicas of one convergent replicated semantic object; merge events/state with CRDT rules and tombstones.

Rejected as universal. CRDTs are excellent inside a declared replica group when the same logical object's operations satisfy the convergence conditions. Independent ledgers with different issuers, admission policies, meanings, retention guarantees and source identities are not replicas merely because records look similar. Convergence of bytes/state also does not resolve epistemic conflict or authority.

Candidate 3 — preferred: CLFM — Causal Ledger Federation Mesh

Keep each ledger authoritative for its own history. Federation stores authenticated observations/receipts about remote histories and composes them through typed, versioned federation contracts.

logical ledger A ---- local order/checkpoints ----┐
logical ledger B ---- local order/checkpoints ----┤
SCITT/CT-style log -- signed receipts ------------┤
PROV/source archive -- provenance bundles --------┤
replica group C ---- one logical-ledger identity --┘
                         ↓
                CLFM federation links
       identity / mapping / admission / retention /
       integrity / redaction / replay contracts
                         ↓
             causally closed federation cut
                         ↓
          #400 epistemic federation / queries

Preferred design — hard distinctions

local commit order       != cross-ledger causal order
causal order              != deterministic display order
same wall-clock timestamp != happened-before
replica                   != backup != mirror != independent ledger
replica convergence       != semantic truth reconciliation
remote issuer identity    != local principal/authority
receipt/inclusion proof   != proposition truth
same content hash         != same assertion episode
same proposition          != independent corroboration
redaction notice          != remote erasure completion
remote tombstone          != local semantic negation
federation cut            != globally serializable snapshot

Ordering / causality

Every participating logical ledger declares its order profile. It may provide a strict local commit sequence, causal/version order, checkpoint order or only observed external receipts.

Cross-ledger happens_before is established only by evidence-backed edges, for example:

  • a remote event explicitly references a verified event/receipt/checkpoint from another ledger;
  • an admitted import event records that remote commitment A was observed before local event B was committed;
  • a trusted replication protocol proves ancestry inside one logical-ledger replica group.

Wall-clock comparison alone does not create causality. Unrelated events remain concurrent/unknown_order even if a deterministic renderer sorts them by (time, ledger_id, event_id).

Conceptual records:

sem_ledger_profile(Ledger,
                   AuthorityDomain,
                   Namespace,
                   OrderProfile,
                   RetentionProfile,
                   IntegrityProfile,
                   RedactionProfile,
                   ReplayProfile,
                   Meta).

sem_federation_link(Link,
                    LocalLedger,
                    RemoteLedger,
                    MappingBundle,
                    AdmissionPolicy,
                    IntegrityContract,
                    RetentionContract,
                    RedactionContract,
                    Status,
                    Provenance).

sem_cross_ledger_edge(EdgeId,
                      FromEvent,
                      ToEvent,
                      causal_basis(Basis),
                      EvidenceRefs,
                      Status).

Federation cuts / replay

A federated query never means “latest from everywhere.” It executes against an explicit federation cut:

sem_federation_cut(CutId,
                   [frontier(LedgerA, FrontierA, CheckpointA),
                    frontier(LedgerB, FrontierB, CheckpointB), ...],
                   LinkVersions,
                   MappingVersions,
                   AdmissionGeneration,
                   CreatedAt,
                   Completeness).

Candidate completeness classes:

causally_closed
locally_exact
best_effort
partial_remote_unavailable
incomplete_dependency
integrity_blocked

A cut claiming causally_closed must include every admitted cross-ledger causal predecessor required by its selected events. It does not claim serializability across independent authorities.

Replay receipts therefore bind exact per-ledger frontiers/checkpoints plus link/mapping/admission/redaction/integrity generations. A remote unavailable at query time yields partial/incomplete status rather than treating its missing knowledge as false.

Replica / backup / mirror semantics

CLFM distinguishes storage topology before evidence counting:

  • replica: another physical copy of the same logical ledger. Consensus/log replication or CRDT techniques may maintain it. Replica copies never count as independent semantic corroboration.
  • backup: recovery material for the same ledger identity/frontier. Restoring it does not create a new source episode.
  • mirror/cache/index: derived copy; never canonical authority.
  • fork: histories claiming the same logical ledger identity but incompatible ancestry/checkpoint proofs. This is an integrity/fork state, not two independent sources.
  • independent ledger: separate authority/order/admission history, even if it contains identical semantic objects.

Redaction / retention federation

A redaction request or local GRIE lifecycle event may create federation redaction obligations/notices under negotiated link policy. Each recipient decides/admit/processes the obligation under its own authority and returns its own receipt/status.

sem_redaction_obligation(ObligationId,
                         OriginLedger,
                         OriginEvent,
                         TargetLedger,
                         Scope,
                         RequiredAction,
                         PolicyRef,
                         Status,
                         ReceiptRefs).

Aggregate status must remain per-copy/per-ledger. Example:

A = complete(crypto_erased)
B = pending(backup_retention_window)
C = blocked(immutable_external_archive)

The federation result is partial, not complete. The origin ledger is forbidden from claiming remote physical purge from the fact that it sent a notice.

For immutable transparency services, sensitive/erasable payloads should be represented by commitments/hashes or external references where appropriate before registration. A transparency receipt can remain while the protected payload becomes unavailable under GRIE.

Integrity / transparency composition

Integrity remains a vector of guarantees, not one Boolean:

issuer_signature_verified
registration_receipt_verified
inclusion_verified
append_only_consistency_verified
non_equivocation_observed
local_admission_passed
payload_available
schema_interpretable

A valid SCITT receipt proves the signed statement was registered in the identified TS under its policy; local epistemic trust remains a relying-party decision. One Signed Statement registered in three TSs is one statement origin + three registration receipts, not three independent corroborating sources.

A conflicting checkpoint/fork inside one claimed logical ledger produces integrity_conflict; the system must not silently pick whichever replica answered first.

External provenance interoperability

W3C PROV bundles may be imported as provenance graphs while retaining bundle/source namespace and adapter/version lineage. SCITT Signed/Transparent Statements may be retained as signed external statement/receipt envelopes. Neither format becomes mandatory canonical storage.

Adapters report something equivalent to:

exact
lossless_for(profile)
lossy(fields_or_semantics)
opaque_extension_preserved
unsupported

The original external envelope remains addressable when retention policy permits.

Identity / dedup / privacy

Cross-ledger identity is still explicit reversible knowledge per symbolic-memory#10 and #392; it is never inferred solely from storage equality.

Rules:

  1. same semantic-object fingerprint can deduplicate immutable structure inside an allowed equality domain while keeping assertion/event/source identities separate;
  2. same remote Signed Statement observed through several TSs/mirrors is one assertion origin with several observation/registration paths;
  3. copies of one wire article in several stores belong to one evidence-dependence component unless independent provenance exists;
  4. equality-sensitive hashes/commitments are scoped to federation/trust domains where global equality probing would leak information;
  5. lexical principal/entity equality across ledgers never maps authority.

Repository ownership boundary

Prolog-RLM continues to own semantic IR, theory/identity mapping semantics and #400 epistemic bridge/query semantics. It may represent ledger/federation metadata as inert typed data, but does not own physical replication or remote storage.

symbolic-memory owns logical ledger identities/profiles, federation-link configuration, authenticated remote-import events, federation cuts/frontiers, replica/backup/fork classification, redaction obligation/status lineage, external envelope retention, and rebuildable federated indexes.

Concrete network/SCITT/database adapters may remain host/backend supplied behind typed interfaces. Remote metadata never installs executable Prolog, capabilities or local authority.

Complexity / scaling implications

  • A naïve exact frontier vector is O(number_of_ledgers) per cut. Sparse/hierarchical/dotted versions and checkpoint compaction are D-level optimization questions.
  • Cross-ledger causality is a DAG reachability problem; transitive closure at global scale is too expensive to require canonically. Store direct evidence edges/checkpoints and build demand/incremental indexes.
  • Identity/alignment across n ledgers can become quadratic/combinatorial; preserve candidate mappings and resolve query-locally rather than eager global merging.
  • Redaction fan-out is proportional to known obligated copies/links; unknown external copies can only be reported as unknown, never certified erased.
  • No protocol can obtain a globally consistent current cut from unavailable independent ledgers without coordination/assumptions. CLFM therefore exposes completeness and coordination profile instead of manufacturing consistency.
  • D owns actual anti-entropy, sparse-frontier, checkpoint/index, change-feed and billion-record algorithms.

Epistemic / provenance implications

Federation contributes new evidence structure, not a new truth semantics. #400 EFF remains above CLFM:

CLFM: what durable commitments were observed from which ledgers, with what causal/integrity/admission status?
EFF: under which epistemic profiles/bridges are those commitments warranted for this query?

Thus storage convergence never resolves contradictory claims by itself. Source dependence follows origin lineage through mirrors/replicas/registrations.

Safety / authority

  • authenticated remote issuer != authenticated local principal;
  • remote registration/admission != local admission;
  • imported policy/normative statement != local authority policy;
  • redaction request is typed lifecycle input, not arbitrary deletion authority unless local/pre-negotiated policy admits it;
  • remote mapping/schema/plugin names stay inert; no dynamic call/1 or code installation;
  • federation read/query authority and federation write/import/redaction authority remain separate;
  • source expansion continues to obey namespace/read policy even when an external receipt is public.

Falsifiable / conformance fixtures

C1. Same timestamp on independent ledgers creates no causal edge.
C2. Verified explicit reference A→B yields cross-ledger happens_before(A,B).
C3. Unrelated A/B remain concurrent/unknown-order.
C4. Deterministic display sort does not change causal result.
C5. Two replicas of one logical ledger count as one evidence origin.
C6. Backup restore does not create corroboration.
C7. Mirror/cache copy does not create corroboration.
C8. Divergent histories under one logical-ledger identity produce fork/integrity conflict.
C9. One SCITT Signed Statement with three TS receipts is one statement origin + three registration receipts.
C10. Independent issuers making the same proposition remain distinct assertion episodes.
C11. Valid inclusion/registration receipt does not by itself make proposition epistemically warranted.
C12. Invalid remote receipt blocks admission and records verification failure.
C13. Local redaction complete + remote pending => federation redaction partial.
C14. Redaction notice cannot manufacture semantic negation.
C15. Origin cannot claim target physical purge without target sanitization receipt/status.
C16. Immutable external log retains commitment while erasable payload is unavailable/redacted.
C17. Backup-retention delay remains explicit pending sanitization.
C18. Remote principal named admin gains no local authority.
C19. Same content hash across two ledgers does not merge assertion/event IDs.
C20. Equality-protected namespaces do not expose cross-domain content-hash matches.
C21. Reversible identity mapping can be withdrawn without rewriting either ledger.
C22. PROV bundle import retains bundle namespace/provenance-of-provenance links.
C23. Lossy external mapping exposes loss receipt and cannot claim exact replay.
C24. Unsupported remote schema remains opaque/preserved rather than silently upcast.
C25. Offline remote ledger yields partial/incomplete query, not false.
C26. Causally-closed cut rejects a selected event whose required predecessor is missing.
C27. Same cut/frontiers + versions reproduces normalized federated observation set when retained payloads permit exact replay.
C28. Redacted payload degrades replay capability exactly as declared by GRIE.
C29. Re-import same authenticated remote event is idempotent.
C30. Same remote event identity with a different digest is an integrity conflict.
C31. A valid remote registration policy does not bypass local admission policy.
C32. A federation link update creates new lineage; old historical cuts retain old link/mapping versions.
C33. Replica convergence can succeed while semantic conflict remains visible above it.
C34. Providers disabled: all supported federation/cut/integrity/provenance reasoning fixtures execute with model_calls = 0.

Rejected shortcuts

  • Global wall-clock last-write-wins: destroys causality, provenance and independent authority.
  • Universal CRDT union: confuses replica convergence with semantic reconciliation and deletion semantics.
  • Global content-hash identity: leaks equality and merges independent evidence.
  • Receipt == truth: cryptographic registration/inclusion says nothing sufficient about world truth.
  • Remote tombstone == local deletion: violates local authority/retention and cannot prove physical erasure.
  • One Boolean integrity_ok: hides issuer, inclusion, consistency, admission, payload and schema failures.
  • Treat backups/replicas as extra sources: manufactures fake corroboration.

Canonical issue deltas required by C

  • #402 / #397: mark C complete; queue D only.
  • symbolic-memory#6: add logical ledger profile/federation-link/import/frontier/redaction-obligation lineage; distinguish replica/backup/mirror/fork/independent ledger.
  • symbolic-memory#7: federated query must name/select a federation cut and expose completeness, remote integrity/admission and per-ledger replay state.
  • symbolic-memory#9: native/MCP transport must preserve federation/integrity/redaction/completeness status and never flatten remote identity into local authority.
  • symbolic-memory#10: identity/dedup/corroboration must remain authority-domain scoped and provenance-dependent; replica/receipt duplication cannot count as independent identity/evidence.
  • symbolic-memory#5/#8: external/import work should be idempotent against authenticated remote identity+digest+frontier and retain adapter/link version lineage.

Unresolved questions reserved for #402D

  • exact sparse/dotted/hierarchical frontier representation and cut-validation algorithm;
  • checkpoint/causal-edge indexing thresholds and incremental reachability;
  • replication/anti-entropy protocol profiles and partition behavior;
  • whether one logical ledger may change replication mechanism without changing ledger identity;
  • efficient remote change feeds and backpressure;
  • billion-record storage partitioning and provenance compression;
  • redaction-obligation fan-out/retry scheduling and bounded retention of receipts;
  • exact SCITT/COSE adapter scope and key-discovery/cache policy;
  • reference implementation backend topology (SQLite/PostgreSQL/LSM/segmented log);
  • deterministic cold/warm/rebuild and failure-injection conformance under partitions/crashes.

Completion statement: #402C establishes composition semantics only. It does not claim distributed implementation, global serializability, successful physical erasure, non-equivocation, or Machine Spirit acceptance. Exact next subpass is #402D.

## MACHINE-SPIRIT #402C — composition / heterogeneity / interoperability **Status: COMPLETE (design only; implementation unproven).** ### Research scope / questions This subpass attacks the assumption that multiple durable world-model stores can be treated as one bigger append-only log. It asks: 1. What ordering can be claimed across independently governed ledgers without inventing a global clock? 2. How do replicas, backups, mirrors, transparency logs and genuinely independent ledgers differ semantically? 3. How can redaction/retention obligations propagate without one store falsely claiming another store physically erased data? 4. How do heterogeneous integrity/transparency mechanisms compose without collapsing to a weakest-common-denominator Boolean? 5. How should external statements/provenance be imported while preserving issuer, registration, admission and local authority boundaries? 6. How do identity, deduplication and corroboration work across namespaces without leaking equality or double-counting copies? 7. What exact world-model cut/frontier is required for deterministic federated replay? ### Current-system comparison #402A established BSLPF: immutable/versioned semantic objects + authoritative bitemporal ledger + rebuildable projections. #402B added GRIE: payload retrievability, redaction/sanitization, admission, replay and integrity are separate dimensions. Upstream #392 already treats contexts/theories/bridges and identity mappings as explicit versioned semantic objects; #400C likewise established that heterogeneous epistemic theories exchange status-bearing objects through typed bridges rather than sharing one global truth relation. Downstream `symbolic-memory#6` still describes one append-only semantic ledger/current-history view, and the current implementation is even narrower: `prolog/symbolic_memory_storage.pl` serializes one in-process dynamic-predicate snapshot under a mutex and atomically renames the complete file. That bootstrap remains useful but cannot define federation semantics. Historical `SYMBOLIC-MEMORY-RESEARCH-006` explicitly treated replication/remote multi-process access as optional future work and correctly insisted on a replaceable storage adapter. C preserves that backend independence while making the *logical federation contract* explicit. ### Primary / authoritative evidence - Leslie Lamport, **“Time, Clocks, and the Ordering of Events in a Distributed System,” CACM 21(7), July 1978** — partial causal order is fundamental; a total order may be constructed but must not be confused with causality. https://lamport.azurewebsites.net/pubs/time-clocks.pdf - Shapiro, Preguiça, Baquero, Zawirski, **“A Comprehensive Study of Convergent and Commutative Replicated Data Types,” INRIA RR-7506, January 2011** — strong/eventual convergence is a property of replicas of a shared abstract object under explicit state/operation conditions; it is not a semantic reconciliation rule for unrelated authorities. https://inria.hal.science/inria-00555588 - W3C, **PROV-DM / PROV-O / Linking Across Provenance Bundles, April 30 2013** — provenance is designed for heterogeneous interchange; bundles can remain independently valid “islands” and can be linked without flattening provenance identity. https://www.w3.org/TR/prov-dm/ and https://www.w3.org/TR/prov-links/ - IETF, **RFC 9162 Certificate Transparency v2, December 2021** — Merkle inclusion/consistency can prove append-only log behavior, while consistency of the view shown to all parties is a separate auditing problem. https://www.rfc-editor.org/rfc/rfc9162 - EU, **GDPR Article 17(2), Regulation (EU) 2016/679** — when public personal data must be erased, controllers must take reasonable steps to inform other controllers processing copies/replications; this is an obligation/notification relationship, not proof remote copies have already been physically sanitized. https://eur-lex.europa.eu/eli/reg/2016/679/oj - IETF, **RFC 9943 SCITT Architecture, June 2026** — a Signed Statement may be registered with multiple independent Transparency Services and carry multiple receipts; receipts prove registration under a TS policy, not semantic truth. SCITT deliberately supports multiple VDS/receipt implementations and leaves relying-party trust policy local. https://www.rfc-editor.org/rfc/rfc9943 ### Candidate architectures #### Candidate 1 — one canonical federation super-ledger Import every remote event into one globally sequenced append-only log and assign a federation-wide commit number. **Rejected as canonical.** It gives convenient snapshots but invents order among unrelated concurrent events, centralizes authority, forces heterogeneous retention/erasure rules into one policy and creates a huge privacy/availability boundary. A sequenced super-ledger can still be an *explicit deployment profile* where participants intentionally accept one sequencer/consensus domain. #### Candidate 2 — CRDT-union world model Treat all stores as replicas of one convergent replicated semantic object; merge events/state with CRDT rules and tombstones. **Rejected as universal.** CRDTs are excellent inside a declared replica group when the same logical object's operations satisfy the convergence conditions. Independent ledgers with different issuers, admission policies, meanings, retention guarantees and source identities are not replicas merely because records look similar. Convergence of bytes/state also does not resolve epistemic conflict or authority. #### Candidate 3 — preferred: **CLFM — Causal Ledger Federation Mesh** Keep each ledger authoritative for its own history. Federation stores authenticated observations/receipts about remote histories and composes them through typed, versioned federation contracts. ```text logical ledger A ---- local order/checkpoints ----┐ logical ledger B ---- local order/checkpoints ----┤ SCITT/CT-style log -- signed receipts ------------┤ PROV/source archive -- provenance bundles --------┤ replica group C ---- one logical-ledger identity --┘ ↓ CLFM federation links identity / mapping / admission / retention / integrity / redaction / replay contracts ↓ causally closed federation cut ↓ #400 epistemic federation / queries ``` ### Preferred design — hard distinctions ```text local commit order != cross-ledger causal order causal order != deterministic display order same wall-clock timestamp != happened-before replica != backup != mirror != independent ledger replica convergence != semantic truth reconciliation remote issuer identity != local principal/authority receipt/inclusion proof != proposition truth same content hash != same assertion episode same proposition != independent corroboration redaction notice != remote erasure completion remote tombstone != local semantic negation federation cut != globally serializable snapshot ``` ### Ordering / causality Every participating logical ledger declares its **order profile**. It may provide a strict local commit sequence, causal/version order, checkpoint order or only observed external receipts. Cross-ledger `happens_before` is established only by evidence-backed edges, for example: - a remote event explicitly references a verified event/receipt/checkpoint from another ledger; - an admitted import event records that remote commitment A was observed before local event B was committed; - a trusted replication protocol proves ancestry inside one logical-ledger replica group. Wall-clock comparison alone does not create causality. Unrelated events remain `concurrent/unknown_order` even if a deterministic renderer sorts them by `(time, ledger_id, event_id)`. Conceptual records: ```prolog sem_ledger_profile(Ledger, AuthorityDomain, Namespace, OrderProfile, RetentionProfile, IntegrityProfile, RedactionProfile, ReplayProfile, Meta). sem_federation_link(Link, LocalLedger, RemoteLedger, MappingBundle, AdmissionPolicy, IntegrityContract, RetentionContract, RedactionContract, Status, Provenance). sem_cross_ledger_edge(EdgeId, FromEvent, ToEvent, causal_basis(Basis), EvidenceRefs, Status). ``` ### Federation cuts / replay A federated query never means “latest from everywhere.” It executes against an explicit **federation cut**: ```prolog sem_federation_cut(CutId, [frontier(LedgerA, FrontierA, CheckpointA), frontier(LedgerB, FrontierB, CheckpointB), ...], LinkVersions, MappingVersions, AdmissionGeneration, CreatedAt, Completeness). ``` Candidate completeness classes: ```text causally_closed locally_exact best_effort partial_remote_unavailable incomplete_dependency integrity_blocked ``` A cut claiming `causally_closed` must include every admitted cross-ledger causal predecessor required by its selected events. It does **not** claim serializability across independent authorities. Replay receipts therefore bind exact per-ledger frontiers/checkpoints plus link/mapping/admission/redaction/integrity generations. A remote unavailable at query time yields partial/incomplete status rather than treating its missing knowledge as false. ### Replica / backup / mirror semantics CLFM distinguishes storage topology before evidence counting: - **replica:** another physical copy of the *same logical ledger*. Consensus/log replication or CRDT techniques may maintain it. Replica copies never count as independent semantic corroboration. - **backup:** recovery material for the same ledger identity/frontier. Restoring it does not create a new source episode. - **mirror/cache/index:** derived copy; never canonical authority. - **fork:** histories claiming the same logical ledger identity but incompatible ancestry/checkpoint proofs. This is an integrity/fork state, not two independent sources. - **independent ledger:** separate authority/order/admission history, even if it contains identical semantic objects. ### Redaction / retention federation A redaction request or local GRIE lifecycle event may create **federation redaction obligations/notices** under negotiated link policy. Each recipient decides/admit/processes the obligation under its own authority and returns its own receipt/status. ```prolog sem_redaction_obligation(ObligationId, OriginLedger, OriginEvent, TargetLedger, Scope, RequiredAction, PolicyRef, Status, ReceiptRefs). ``` Aggregate status must remain per-copy/per-ledger. Example: ```text A = complete(crypto_erased) B = pending(backup_retention_window) C = blocked(immutable_external_archive) ``` The federation result is `partial`, not `complete`. The origin ledger is forbidden from claiming remote physical purge from the fact that it sent a notice. For immutable transparency services, sensitive/erasable payloads should be represented by commitments/hashes or external references where appropriate before registration. A transparency receipt can remain while the protected payload becomes unavailable under GRIE. ### Integrity / transparency composition Integrity remains a vector of guarantees, not one Boolean: ```text issuer_signature_verified registration_receipt_verified inclusion_verified append_only_consistency_verified non_equivocation_observed local_admission_passed payload_available schema_interpretable ``` A valid SCITT receipt proves the signed statement was registered in the identified TS under its policy; local epistemic trust remains a relying-party decision. One Signed Statement registered in three TSs is **one statement origin + three registration receipts**, not three independent corroborating sources. A conflicting checkpoint/fork inside one claimed logical ledger produces `integrity_conflict`; the system must not silently pick whichever replica answered first. ### External provenance interoperability W3C PROV bundles may be imported as provenance graphs while retaining bundle/source namespace and adapter/version lineage. SCITT Signed/Transparent Statements may be retained as signed external statement/receipt envelopes. Neither format becomes mandatory canonical storage. Adapters report something equivalent to: ```text exact lossless_for(profile) lossy(fields_or_semantics) opaque_extension_preserved unsupported ``` The original external envelope remains addressable when retention policy permits. ### Identity / dedup / privacy Cross-ledger identity is still explicit reversible knowledge per symbolic-memory#10 and #392; it is never inferred solely from storage equality. Rules: 1. same semantic-object fingerprint can deduplicate immutable structure *inside an allowed equality domain* while keeping assertion/event/source identities separate; 2. same remote Signed Statement observed through several TSs/mirrors is one assertion origin with several observation/registration paths; 3. copies of one wire article in several stores belong to one evidence-dependence component unless independent provenance exists; 4. equality-sensitive hashes/commitments are scoped to federation/trust domains where global equality probing would leak information; 5. lexical principal/entity equality across ledgers never maps authority. ### Repository ownership boundary **Prolog-RLM** continues to own semantic IR, theory/identity mapping semantics and #400 epistemic bridge/query semantics. It may represent ledger/federation metadata as inert typed data, but does not own physical replication or remote storage. **symbolic-memory** owns logical ledger identities/profiles, federation-link configuration, authenticated remote-import events, federation cuts/frontiers, replica/backup/fork classification, redaction obligation/status lineage, external envelope retention, and rebuildable federated indexes. Concrete network/SCITT/database adapters may remain host/backend supplied behind typed interfaces. Remote metadata never installs executable Prolog, capabilities or local authority. ### Complexity / scaling implications - A naïve exact frontier vector is `O(number_of_ledgers)` per cut. Sparse/hierarchical/dotted versions and checkpoint compaction are D-level optimization questions. - Cross-ledger causality is a DAG reachability problem; transitive closure at global scale is too expensive to require canonically. Store direct evidence edges/checkpoints and build demand/incremental indexes. - Identity/alignment across `n` ledgers can become quadratic/combinatorial; preserve candidate mappings and resolve query-locally rather than eager global merging. - Redaction fan-out is proportional to known obligated copies/links; unknown external copies can only be reported as unknown, never certified erased. - No protocol can obtain a globally consistent current cut from unavailable independent ledgers without coordination/assumptions. CLFM therefore exposes completeness and coordination profile instead of manufacturing consistency. - D owns actual anti-entropy, sparse-frontier, checkpoint/index, change-feed and billion-record algorithms. ### Epistemic / provenance implications Federation contributes *new evidence structure*, not a new truth semantics. #400 EFF remains above CLFM: ```text CLFM: what durable commitments were observed from which ledgers, with what causal/integrity/admission status? EFF: under which epistemic profiles/bridges are those commitments warranted for this query? ``` Thus storage convergence never resolves contradictory claims by itself. Source dependence follows origin lineage through mirrors/replicas/registrations. ### Safety / authority - authenticated remote issuer != authenticated local principal; - remote registration/admission != local admission; - imported policy/normative statement != local authority policy; - redaction request is typed lifecycle input, not arbitrary deletion authority unless local/pre-negotiated policy admits it; - remote mapping/schema/plugin names stay inert; no dynamic `call/1` or code installation; - federation read/query authority and federation write/import/redaction authority remain separate; - source expansion continues to obey namespace/read policy even when an external receipt is public. ### Falsifiable / conformance fixtures C1. Same timestamp on independent ledgers creates no causal edge. C2. Verified explicit reference A→B yields cross-ledger `happens_before(A,B)`. C3. Unrelated A/B remain concurrent/unknown-order. C4. Deterministic display sort does not change causal result. C5. Two replicas of one logical ledger count as one evidence origin. C6. Backup restore does not create corroboration. C7. Mirror/cache copy does not create corroboration. C8. Divergent histories under one logical-ledger identity produce fork/integrity conflict. C9. One SCITT Signed Statement with three TS receipts is one statement origin + three registration receipts. C10. Independent issuers making the same proposition remain distinct assertion episodes. C11. Valid inclusion/registration receipt does not by itself make proposition epistemically warranted. C12. Invalid remote receipt blocks admission and records verification failure. C13. Local redaction complete + remote pending => federation redaction `partial`. C14. Redaction notice cannot manufacture semantic negation. C15. Origin cannot claim target physical purge without target sanitization receipt/status. C16. Immutable external log retains commitment while erasable payload is unavailable/redacted. C17. Backup-retention delay remains explicit pending sanitization. C18. Remote principal named `admin` gains no local authority. C19. Same content hash across two ledgers does not merge assertion/event IDs. C20. Equality-protected namespaces do not expose cross-domain content-hash matches. C21. Reversible identity mapping can be withdrawn without rewriting either ledger. C22. PROV bundle import retains bundle namespace/provenance-of-provenance links. C23. Lossy external mapping exposes loss receipt and cannot claim exact replay. C24. Unsupported remote schema remains opaque/preserved rather than silently upcast. C25. Offline remote ledger yields partial/incomplete query, not `false`. C26. Causally-closed cut rejects a selected event whose required predecessor is missing. C27. Same cut/frontiers + versions reproduces normalized federated observation set when retained payloads permit exact replay. C28. Redacted payload degrades replay capability exactly as declared by GRIE. C29. Re-import same authenticated remote event is idempotent. C30. Same remote event identity with a different digest is an integrity conflict. C31. A valid remote registration policy does not bypass local admission policy. C32. A federation link update creates new lineage; old historical cuts retain old link/mapping versions. C33. Replica convergence can succeed while semantic conflict remains visible above it. C34. Providers disabled: all supported federation/cut/integrity/provenance reasoning fixtures execute with `model_calls = 0`. ### Rejected shortcuts - **Global wall-clock last-write-wins:** destroys causality, provenance and independent authority. - **Universal CRDT union:** confuses replica convergence with semantic reconciliation and deletion semantics. - **Global content-hash identity:** leaks equality and merges independent evidence. - **Receipt == truth:** cryptographic registration/inclusion says nothing sufficient about world truth. - **Remote tombstone == local deletion:** violates local authority/retention and cannot prove physical erasure. - **One Boolean `integrity_ok`:** hides issuer, inclusion, consistency, admission, payload and schema failures. - **Treat backups/replicas as extra sources:** manufactures fake corroboration. ### Canonical issue deltas required by C - #402 / #397: mark C complete; queue D only. - symbolic-memory#6: add logical ledger profile/federation-link/import/frontier/redaction-obligation lineage; distinguish replica/backup/mirror/fork/independent ledger. - symbolic-memory#7: federated query must name/select a federation cut and expose completeness, remote integrity/admission and per-ledger replay state. - symbolic-memory#9: native/MCP transport must preserve federation/integrity/redaction/completeness status and never flatten remote identity into local authority. - symbolic-memory#10: identity/dedup/corroboration must remain authority-domain scoped and provenance-dependent; replica/receipt duplication cannot count as independent identity/evidence. - symbolic-memory#5/#8: external/import work should be idempotent against authenticated remote identity+digest+frontier and retain adapter/link version lineage. ### Unresolved questions reserved for #402D - exact sparse/dotted/hierarchical frontier representation and cut-validation algorithm; - checkpoint/causal-edge indexing thresholds and incremental reachability; - replication/anti-entropy protocol profiles and partition behavior; - whether one logical ledger may change replication mechanism without changing ledger identity; - efficient remote change feeds and backpressure; - billion-record storage partitioning and provenance compression; - redaction-obligation fan-out/retry scheduling and bounded retention of receipts; - exact SCITT/COSE adapter scope and key-discovery/cache policy; - reference implementation backend topology (SQLite/PostgreSQL/LSM/segmented log); - deterministic cold/warm/rebuild and failure-injection conformance under partitions/crashes. **Completion statement:** #402C establishes composition semantics only. It does not claim distributed implementation, global serializability, successful physical erasure, non-equivocation, or Machine Spirit acceptance. **Exact next subpass is #402D.**
Author
Owner

#402D — computational realization / scaling / conformance — COMPLETE

Scope / research questions

This D subpass operationalizes A/BSLPF + B/GRIE + C/CLFM. It asks:

  1. How can a ledger/frontier architecture scale from one local store to many independent ledgers without a vector entry for every participant everywhere?
  2. How should causal dependencies be indexed without materializing global transitive closure?
  3. How are current/history/epistemic/procedural projections incrementally maintained and invalidated while the authoritative ledger keeps accepting writes?
  4. Which replication semantics are valid for one logical ledger, and how must they remain distinct from federation of independent ledgers?
  5. What are the exact crash/partition/fork behaviors?
  6. How are redaction obligations executed without confusing notification, attempted deletion and verified sanitization?
  7. What storage/index layout remains plausible at 10^6–10^9 semantic events/objects?
  8. What deterministic replay/differential tests prove cold, warm, rebuilt and incrementally maintained results agree?

Current-repo comparison

The current queue is unambiguous: #397 and this issue say A/B/C are complete and D is next. #392 keeps semantic objects immutable/versioned and separates canonical from materialized state; #394 keeps epistemic inputs canonical while support/warrant tables remain derived. Downstream symbolic-memory#4/#6 require append-only semantic history and rebuildable current views.

The actual current symbolic-memory/prolog/symbolic_memory_storage.pl remains a bootstrap implementation: one process-wide mutex protects dynamic predicates; each transaction snapshots all projects/sources/memories/audits, serializes the entire snapshot(...) term to a temporary file, and renames it over the prior file. That makes transaction cost and restart footprint proportional to total resident state and provides no segmented event log, sparse frontier, causal index, projection checkpoint, anti-entropy or billion-record path. D therefore specifies a replacement semantic storage contract, not a claim that the current backend implements it.


Primary / authoritative evidence

  1. O'Neil, Cheng, Gawlick, O'Neil — “The Log-Structured Merge-Tree (LSM-Tree),” Acta Informatica 33(4), 1996. Write-heavy history/index workloads benefit from deferred batched merge structures rather than random in-place updates. DOI: https://doi.org/10.1007/s002360050048
  2. Chang et al. — “Bigtable: A Distributed Storage System for Structured Data,” OSDI 2006. Demonstrates very large structured stores using ordered key ranges, memtables/immutable SSTables and distributed tablet partitioning: https://research.google/pubs/bigtable-a-distributed-storage-system-for-structured-data/
  3. DeCandia et al. — “Dynamo: Amazon’s Highly Available Key-value Store,” SOSP 2007. Primary precedent for replica versioning, eventual repair and Merkle-tree anti-entropy under highly available replication: https://doi.org/10.1145/1294261.1294281
  4. Preguiça, Baquero, Almeida, Fonte, Gonçalves — “Dotted Version Vectors: Logical Clocks for Optimistic Replication,” 2010/2011. Causality metadata can be bounded by servers registering updates for an object rather than clients/updates: https://arxiv.org/abs/1011.5808
  5. Almeida, Baquero, Fonte — “Interval Tree Clocks,” OPODIS 2008. Dynamic decentralized causality tracking without a preconfigured fixed process-ID vector: https://doi.org/10.1007/978-3-540-92221-6_18
  6. Ongaro & Ousterhout — “In Search of an Understandable Consensus Algorithm,” USENIX ATC 2014. A replicated-log consensus profile can provide one committed history for one logical ledger: https://www.usenix.org/conference/atc14/technical-sessions/presentation/ongaro
  7. McSherry, Murray, Isaacs, Isard — “Differential Dataflow,” CIDR 2013. Incremental nested/iterative computation, including changing graph computations/SCCs: https://www.microsoft.com/en-us/research/publication/differential-dataflow/
  8. Budiu, Chajed, McSherry, Ryzhyk, Tannen — “DBSP: Automatic Incremental View Maintenance for Rich Query Languages,” PVLDB 16(7), 2023. General delta-based incrementalization for rich relational/recursive views: https://doi.org/10.14778/3587136.3587149
  9. Almeida, Baquero, Preguiça, Hutchison — “Scalable Bloom Filters,” Information Processing Letters 101(6), 2007. Space-efficient negative membership filtering with explicit false-positive probability: https://doi.org/10.1016/j.ipl.2006.10.007

These support mechanisms; none is adopted wholesale as Machine Spirit semantics.


Candidate execution architectures

Candidate 1 — one ACID temporal relational database + eager materialized views

semantic/object/event tables
        ↓
SQL temporal/current views
        ↓
triggers/materialized views

Strengths: simple transactional model; good local implementation target; mature indexing, WAL and backup tooling.

Failure as the universal D architecture: binds canonical semantics to one database's temporal/materialized-view behavior; encourages eager global indexes; does not itself solve independent-ledger federation, sparse cuts, replica-vs-source semantics or heterogeneous redaction/integrity profiles. A relational backend remains a valid implementation of the logical contract.

Candidate 2 — distributed event store + global version vector + eager global causal/materialized closure

all ledgers/replicas
    ↓
global version vector
    ↓
causal transitive closure
    ↓
fully current materialized world model

Rejected. Metadata grows with global participants; dynamic/federated authorities are forced into one replication domain; causal-closure maintenance can dwarf useful queries; partitions invite either false global freshness or system-wide blocking.

Candidate 3 — segmented authoritative ledgers + sparse hierarchical cuts + typed replica profiles + demanded incremental projections (preferred)

Name: SAFIRE — Sparse Append-only Frontier & Incremental Replay Engine.

immutable semantic objects
          +
segmented per-ledger append log
          +
commit/checkpoint manifests
          ↓
sparse local/federation frontier descriptors
          ↓
delta feed
          ↓
versioned projection graph
          ↓
indexes/current/epistemic/procedural/retrieval views

Federation remains CLFM above independent ledgers. Replication remains a property inside one logical-ledger identity.


Preferred computational architecture — SAFIRE

1. Local ledger order is dense; federation order is sparse

Every logical ledger owns a monotonically increasing commit/sequence coordinate:

ledger_position(LedgerId, Sequence).

A local checkpoint can summarize a contiguous durable prefix plus segment/manifests. It does not require a vector entry for unrelated ledgers.

A federation cut names only participating ledgers/dependencies:

sem_federation_cut(
    CutId,
    FrontierManifestRef,
    LinkGeneration,
    MappingGeneration,
    AdmissionGeneration,
    RedactionGeneration,
    IntegrityGeneration,
    Completeness,
    Digest).

FrontierManifestRef may be an immutable hierarchical manifest whose leaves are exact (Ledger, Frontier, Checkpoint) tuples. Large cuts can share unchanged manifest subtrees instead of copying a million-entry map into every receipt.

Hard invariant:

sparse/hierarchical cut compression
    != loss of exact participating frontier

The full logical cut must be reconstructible from referenced immutable manifests.

Replica clocks are not federation clocks

Inside an optimistically replicated single logical ledger/object partition, dotted version vectors or an equivalent bounded causality representation are valid candidates. Interval Tree Clocks are a candidate where participants are highly dynamic.

They are not the canonical cross-ledger federation ordering mechanism. Independent ledgers remain separate authorities connected by explicit causal/import links.


2. Do not materialize global causal transitive closure

Canonical causal evidence is the explicit graph:

sem_causal_edge(EdgeId, FromEvent, ToEvent, Kind, Evidence, Receipt).

Indexes provide:

  • event -> direct predecessors/successors;
  • ledger/sequence -> segment;
  • checkpoint -> covered sequence interval;
  • remote reference -> importing event;
  • segment-level summary metadata.

A causality query performs bounded backward/forward traversal from the demanded cut. Frequently reused reachability results may be cached, but global closure is never canonical.

Probabilistic indexes are negative accelerators only

Segment/SSTable Bloom filters or scalable Bloom filters may answer:

"definitely absent"

or

"possibly present"

They can skip storage probes. A false positive may cause extra I/O; it may never create a semantic edge, causal relation, source identity or authorization result.


3. Segmented immutable storage layout

Logical contract:

ledger
 ├─ immutable committed segments
 │   ├─ commit/event envelopes ordered by sequence
 │   ├─ object refs
 │   ├─ checksums / integrity refs
 │   └─ min/max summaries
 ├─ immutable semantic object blobs / rows
 ├─ checkpoint + manifest chain
 └─ rebuildable secondary indexes

Recommended physical shape at scale is backend-neutral but assumes:

  • append batches are sequential/transactional;
  • old committed segments are immutable except governed payload sanitization through GRIE;
  • indexes use sorted/partitioned structures (LSM/B-tree/etc.) rather than loading all records into Prolog dynamic predicates;
  • Prolog-facing APIs use iterators/cursors/page bounds;
  • namespaces/ledger IDs/sequence/time/semantic class/provenance IDs are index dimensions;
  • large source payloads and structural semantic objects may be separated so semantic scans do not drag full text through memory.

A billion-record design cannot have findall/3 over the whole store as its ordinary scan primitive.


4. Projection DAG + exact dependency generations

Every materialized projection declares inputs and semantic dependencies:

sem_projection_descriptor(
    ProjectionId,
    ImplementationVersion,
    InputKinds,
    DependencyGenerations,
    IncrementalMode,
    CanonicalizationProfile).

sem_projection_checkpoint(
    ProjectionId,
    InputCut,
    DependencyGenerations,
    StateDigest,
    PublishedAt,
    Status).

Dependencies include where applicable:

semantic schema / upcaster
vocabulary
identity/mapping
admission
redaction
integrity
context/bridge
reasoning/epistemic profile
procedure/PSC mapping

Delta processing

For a projection at frontier F receiving committed delta Δ:

read Δ
  ↓
validate exact dependency generation
  ↓
compute output delta
  ↓
atomically publish changed index rows + new checkpoint

Simple indexes use direct delta maintenance. Recursive/iterative projections may use DBSP/differential-dataflow-style incrementalization when it wins empirically.

Do not require one universal dataflow runtime for every index.

Invalidation

A dependency generation change invalidates only projections/partitions that declared that dependency. For huge fan-out changes (e.g. ontology/profile generation affecting enormous history), prefer:

generation bump
  + mark affected partitions stale
  + demanded/background bounded rebuild

over eagerly rewriting a billion derived rows in one transaction.

Canonical history remains available while views catch up.


5. Projection freshness is explicit

A query result carries at least:

projection_status(
    Projection,
    requested_cut(Requested),
    materialized_cut(Materialized),
    dependency_generations(Gens),
    freshness(Freshness),
    completeness(Completeness)).

Representative states:

exact
lagging(Delta)
stale_dependency
rebuilding
integrity_blocked
redaction_blocked
unavailable

Never answer from a lagging projection and label it simply current.

Callers may choose policy:

allow_bounded_stale
require_exact_cut
fallback_to_demanded_ledger_evaluation

but the result must expose which path occurred.


6. Replication profiles are explicit and non-interchangeable

single_writer_local

One authoritative local ledger. Physical database WAL/backup is implementation detail.

consensus_replica_group

One logical ledger replicated through a consensus protocol (Raft/Paxos family or equivalent). Only committed log entries become ledger history. Minority partitions cannot fabricate committed history.

optimistic_replica_group

One logical object/partition family may accept concurrent versions during partition. Causality metadata (e.g. DVV) detects concurrency; reconciliation produces explicit retained branches/resolution events. This profile does not promise one write order during the partition.

mirror/archive

Copies committed history but has no authority to originate ledger commits under that identity.

Hard invariant:

replication repair
    != semantic reconciliation
    != cross-ledger federation

Dynamo-style Merkle anti-entropy is appropriate for finding differing key/segment ranges inside a declared replica/mirror relation. A Merkle match proves content equality for the committed range under that digest scheme; it does not prove proposition truth or source independence.


7. Anti-entropy operates on immutable commit/segment identity

Preferred repair flow:

exchange authenticated checkpoint/manifests
        ↓
compare range/segment digests
        ↓
identify differing ranges
        ↓
transfer missing immutable commits/objects
        ↓
validate event IDs/digests/ancestry
        ↓
advance replica checkpoint

Duplicate transfer is harmless through event/commit idempotency.

If two peers claiming the same consensus/single-history ledger identity present incompatible committed ancestry, classify:

fork / integrity conflict

Do not union them automatically.

Optimistic replica profiles may instead retain concurrent branches according to that profile's declared reconciliation semantics.


8. Crash behavior is deterministic at semantic boundaries

Required crash points:

  1. object staged, no commit -> unreachable staging garbage; not history;
  2. event batch encoded, not durably committed -> absent;
  3. ledger commit durable, projection not updated -> commit is authoritative; projection catches up;
  4. projection rows written, checkpoint not atomically published -> those rows are invisible/discardable until a valid checkpoint;
  5. checkpoint published -> readers may use it only if digest + dependency generations validate;
  6. canonical segment corruption -> integrity_failure; projection cannot become replacement authority.

Backends may use SQLite/Postgres/RocksDB/etc. WAL/transactions, but the semantic conformance suite tests the boundaries above independent of backend.


9. Partition behavior follows replica profile, never guesswork

consensus group without quorum
    -> no new committed writes; reads report exact known commit/read policy

optimistic group partition
    -> concurrent versions allowed only if profile says so; later reconciliation explicit

independent federated ledger unavailable
    -> local ledgers continue; federated cut/result becomes partial_remote_unavailable or incomplete_dependency

A remote outage cannot make unrelated local knowledge disappear, and a stale replica cannot advertise itself as globally current.


10. Governed redaction obligations become durable idempotent work

GRIE/CLFM redaction obligations are canonical ledger objects. Execution state is append-only attempt/receipt history:

sem_redaction_attempt(
    AttemptId,
    ObligationId,
    TargetLedger,
    ExpectedTargetState,
    ActionClass,
    StartedAt,
    Result,
    RemoteReceipt,
    RetryClass).

Workers may retry at-least-once. The obligation ID + target + requested action make retries idempotent where the target supports it.

Possible terminal/nonterminal states remain explicit:

pending
in_progress
complete(logical_redaction)
complete(crypto_erased)
complete(physical_purge)
partial
blocked
failed
unknown_remote_state

No local worker can upgrade remote notice accepted to remote physical purge complete without evidence defined by that target profile.

Maintenance execution is storage/runtime work; it does not create a second Prolog-RLM plan scheduler and does not grant semantic actions host authority.


11. Billion-record assumptions / bounds

The semantic contract must make these complexity expectations falsifiable rather than promise magic constants.

Append

For batch size b, logical append work should be O(b) plus backend index/log factors. It must not serialize or rewrite O(total_history).

Catch-up

Projection catch-up from frontier F to F+n should consume the n committed deltas plus affected dependency closure. A small append must not normally force a full replay.

Current/query indexes

Point/range lookup must use persistent indexes. No requirement may imply loading all events/objects into SWI-Prolog heap.

Federation cut

Logical size is O(k) for the k participating ledger frontiers. Hierarchical immutable manifests may make incremental storage proportional to changed leaves/subtrees rather than copying all k entries per receipt.

Causal reachability

Demanded traversal cost is proportional to visited explicit edges/checkpoint summaries plus index lookup costs. Global O(V^2) reachability materialization is forbidden as a baseline requirement.

Anti-entropy

Merkle/range summaries should localize repair to differing ranges rather than resend complete history. Exact complexity depends on tree/segment layout; benchmark it instead of encoding a false universal bound.

Provenance

Durable provenance uses shared DAG/object references. Explanation expands only demanded paths under budgets; it does not duplicate full proof trees into every event.


12. Deterministic replay receipt

A replayable result/checkpoint fingerprints:

ledger/federation cut
exact commit/checkpoint manifests
semantic object digests needed by the result
schema/upcaster versions
identity/mapping generations
admission/redaction/integrity generations
reasoning/procedural profile versions
projection implementation/version
canonicalization version
query/options/budget class where semantically relevant

Canonical result hashing sorts/normalizes unordered sets before digesting. Scheduler/thread completion order must not change semantic digest.

Hard differential oracle:

cold full rebuild
    == warm cached query
    == cache dropped + rebuild
    == incrementally maintained result

for the same exact cut + dependency generations.

If a budget prevents completion, compare structured failure/completeness semantics, not a partial result masquerading as complete.


Falsifiable D conformance fixtures

SAFIRE-D1 — append complexity class

Append 1 event to histories of 10^3 and 10^7 records; instrumentation must prove no full-history serialization/rewrite path.

D2 — crash before ledger commit

Injected crash leaves no committed event/frontier advance.

D3 — crash after ledger commit before projection

Restart replays delta and reaches same projection digest as clean execution.

D4 — torn/unpublished projection

Projection rows without valid checkpoint are never visible as authoritative current state.

D5 — projection corruption

Corrupt projection -> reject/rebuild; canonical ledger unchanged.

D6 — canonical segment corruption

Return integrity_failure; projection cannot repair canonical authority.

D7 — duplicate append retry

Same idempotency key + same digest -> same commit receipt/no duplicate event.

D8 — conflicting retry

Same key + different digest -> hard idempotency conflict.

D9 — sparse frontier

Query over 3 ledgers in a federation containing 100k registered ledgers produces a cut logically naming only required participants/dependencies, not a 100k fixed vector.

D10 — cut manifest exactness

Compressed/hierarchical cut reconstructs exact same leaf frontier set and digest after restart.

D11 — wall clock non-causality

Later timestamp on independent ledger creates no causal edge.

D12 — explicit causal import

Remote event reference/import creates exactly the declared cross-ledger dependency.

D13 — Bloom false positive safety

Forced probabilistic-index false positive may add a storage probe but cannot add semantic/causal result.

D14 — no global closure

Instrumentation on sparse causal graph shows query-local traversal/cache, no mandatory global transitive-closure build.

D15 — incremental current view

One assertion append updates only affected current-view partition/dependencies.

D16 — incremental/full oracle

Incremental projection digest == clean full rebuild digest.

D17 — cold/warm/rebuilt oracle

Identical result/status/receipt semantics for cold, warm and rebuilt paths.

D18 — dependency generation invalidation

Identity/upcaster/profile generation change invalidates only declared consumers; unaffected projection digests stay valid.

D19 — massive invalidation lazy generation

Large ontology/profile change marks affected partition/generation stale without atomically rewriting all derived rows.

D20 — stale result honesty

Lagging projection reports exact materialized cut and never labels itself current.

D21 — exact-cut query

require_exact_cut either catches up/demand-evaluates to that cut or returns explicit inability; never silently serves older state.

D22 — consensus minority partition

Minority cannot create a committed canonical ledger event.

D23 — optimistic concurrent updates

Allowed partition writes remain concurrent branches until explicit reconciliation; no fake total order from timestamps.

D24 — replica != corroboration

Three replicas of one logical ledger contribute one source/assertion episode, not three independent evidence components.

D25 — mirror cannot originate

Mirror/archive refuses canonical commit under source-ledger identity.

D26 — Merkle anti-entropy localization

One damaged/missing segment causes transfer of differing range only; identical ranges are not resent.

D27 — incompatible ancestry

Peers claiming one single-history ledger with incompatible committed ancestry -> fork/integrity conflict, never automatic union.

D28 — remote federation outage

Local exact queries continue; cross-ledger query reports partial/incomplete cut.

D29 — redaction retry idempotency

Repeated obligation attempt cannot duplicate semantic lifecycle effect.

D30 — redaction honesty

Remote ACK without sanitization evidence remains notice/partial state, never physical-purge complete.

D31 — erased payload resurrection defense

Cache/rebuild/anti-entropy cannot restore GRIE-erased plaintext from a stale replica lacking authority to reintroduce it.

D32 — schema/upcaster replay

Historical cut under exact original interpretation and current re-interpretation remain separately reproducible where payload survives.

D33 — billion-record cursor contract

Synthetic/real large backend test proves bounded page/cursor APIs and no findall(all_history)/whole-store heap materialization on ordinary query path.

D34 — provenance DAG sharing

N derived results sharing one provenance subgraph store references rather than N full copied proof trees.

D35 — deterministic concurrency

Different projection-worker completion schedules produce identical canonicalized projection/result digest at same cut.

D36 — resource limit semantics

Projection/rebuild/causal traversal budget exhaustion yields resource_limited/lag status, not false/unknown/current.

D37 — replica clock boundary

DVV/ITC metadata inside replica profile cannot be interpreted as independent-ledger federation order.

D38 — authority isolation

Remote commit/receipt/replica metadata cannot create local host capability or principal authority.

D39 — zero-model durable known task

After semantic projection exists, restart + ledger replay + bounded retrieval + known reasoning/procedure path completes with model_calls = 0.

D40 — end-to-end failure injection matrix

Randomized crashes at the six semantic durability boundaries followed by restart must converge to either the last fully committed cut or a later fully committed cut, never a hybrid projection state.


Rejected shortcuts

  1. One global vector clock for every ledger — metadata/federation coupling scales with participants and falsely treats independent authorities as one replication set.
  2. Wall-clock ordering as causality — clock skew/concurrency destroys the guarantee.
  3. Global transitive causal closure — unacceptable baseline storage/recompute cost; demanded traversal + caches suffice.
  4. One universal CRDT world model — replica convergence is not semantic/epistemic reconciliation.
  5. One universal consensus protocol — independent ledgers and high-availability optimistic replicas have materially different requirements.
  6. Projection tables as backup truth — violates BSLPF/GRIE integrity boundary.
  7. Bloom/filter/index hit as semantic evidence — probabilistic/derived index state is never authority.
  8. Eager rewrite on schema/ontology/profile change — catastrophic fan-out; use version generations + bounded rebuild.
  9. Remote redaction ACK == sanitization proof — false claim about another authority/storage medium.
  10. Full Prolog heap as the billion-record index — current bootstrap technique, not scalable architecture.
  11. Automatic LLM fallback on storage/reasoning exhaustion — resource failure remains structured failure; no provider call is implied.

Repository ownership / typed API boundary

lost-rob0t/symbolic-memory owns

  • ledger/object physical adapter and logical commit API;
  • segment/checkpoint/frontier manifests;
  • replica profiles + anti-entropy implementation;
  • CLFM federation-link persistence/cuts;
  • GRIE redaction-obligation persistence/maintenance;
  • projection checkpoint/invalidation/freshness state;
  • scalable persistent indexes/cursor APIs;
  • storage crash/rebuild/failure-injection conformance.

lost-rob0t/prolog-rlm owns

  • #392 semantic object meaning/schema/profile identities;
  • #394 epistemic/reasoning projection semantics;
  • #395 conformance expectations for semantic reasoning;
  • #401/DPEC procedural execution semantics;
  • consumers of bounded memory projections.

Dependency stays symbolic-memory -> prolog-rlm. SAFIRE does not become a second reasoning semantics or plan scheduler.

Conceptual storage-facing API:

memory_ledger_append(+Ledger,+Batch,+ExpectedFrontier,+Idempotency,-Receipt).
memory_ledger_scan(+Ledger,+Range,+Filter,+Cursor,-Page,-NextCursor).
memory_cut_open(+Scope,+Options,-Cut,-Receipt).
memory_projection_status(+Projection,-Status).
memory_projection_catchup(+Projection,+TargetCut,+Budget,-Receipt).
memory_projection_rebuild(+Projection,+TargetCut,+Budget,-Receipt).
memory_replica_sync(+ReplicaGroup,+Peer,+Budget,-Receipt).
memory_redaction_process(+Obligation,+Budget,-Receipt).
memory_storage_verify(+Scope,+Profile,-Receipt).

Exact implementation follows repo conventions; these are semantic contracts, not frozen exported predicate names.


Depth-5 reconciliation

A/B/C/D now survive as one design:

BSLPF
  canonical bitemporal semantic history + rebuildable projections
        ↓ refined by
GRIE
  payload retrievability/redaction/integrity/admission/replay capability
        ↓ composed by
CLFM
  independent-ledger causality/federation/cuts/remote obligations
        ↓ realized by
SAFIRE
  segmented logs + sparse cuts + projection DAG + typed replica profiles
  + anti-entropy + bounded indexes + deterministic replay/conformance

Depth 5 is therefore design-complete only. Implementation remains unproven. The current snapshot backend does not satisfy SAFIRE yet, and no billion-record, anti-entropy, crash-matrix, sanitization or deterministic-rebuild evidence exists merely because this design is recorded.

Canonical handoffs required from this D pass

  • #397: mark Depth 5 A/B/C/D design-complete; queue #403A only.
  • #402: mark D complete / depth design-complete, then close as completed after reconciliation.
  • symbolic-memory#4: adopt BSLPF+GRIE+CLFM+SAFIRE as durable-world-model direction.
  • symbolic-memory#5: projection commit/checkpoint must be atomic and frontier/generation keyed.
  • symbolic-memory#6: implement segmented ledger/frontier/causal/projection/replica contracts rather than whole-snapshot rewrite as canonical architecture.
  • symbolic-memory#7/#9: query/MCP surfaces expose exact requested/materialized cut, freshness/completeness and replay/integrity/redaction status.
  • symbolic-memory#8: backfill/corpus ingestion uses idempotent batches, cursors/frontiers and resumable delta processing.
  • symbolic-memory#10: identity/mapping generation is a first-class projection invalidation dimension.

Unresolved questions deliberately moved forward

  1. first physical backend profile: SQLite/Postgres vs embedded LSM/other;
  2. exact immutable segment encoding and compaction rules under GRIE redaction;
  3. exact manifest tree and cut canonicalization format;
  4. when to use ordinary incremental indexes versus DBSP/differential dataflow;
  5. whether the first optimistic replica implementation needs DVV, ITC or can defer optimistic multi-writer replication;
  6. consensus implementation/dependency choice if/when replicated logical ledgers are implemented;
  7. exact anti-entropy range tree and authentication scheme;
  8. benchmark thresholds for append latency, catch-up lag, rebuild throughput, point/range query latency and provenance expansion;
  9. operational backup/restore and integrity-checkpoint retention policy;
  10. how legal/policy deletion interacts with immutable off-site backups under deployment-specific requirements.

Those are implementation/research inputs, not reasons to keep #402D open.

#402D complete. Depth 5 A/B/C/D reconciled. Exact next legal research subpass after canonical updates: #403A — retrieval / symbolic attention foundations.

## #402D — computational realization / scaling / conformance — COMPLETE ### Scope / research questions This D subpass operationalizes A/BSLPF + B/GRIE + C/CLFM. It asks: 1. How can a ledger/frontier architecture scale from one local store to many independent ledgers without a vector entry for every participant everywhere? 2. How should causal dependencies be indexed without materializing global transitive closure? 3. How are current/history/epistemic/procedural projections incrementally maintained and invalidated while the authoritative ledger keeps accepting writes? 4. Which replication semantics are valid for one logical ledger, and how must they remain distinct from federation of independent ledgers? 5. What are the exact crash/partition/fork behaviors? 6. How are redaction obligations executed without confusing notification, attempted deletion and verified sanitization? 7. What storage/index layout remains plausible at 10^6–10^9 semantic events/objects? 8. What deterministic replay/differential tests prove cold, warm, rebuilt and incrementally maintained results agree? ### Current-repo comparison The current queue is unambiguous: #397 and this issue say A/B/C are complete and D is next. #392 keeps semantic objects immutable/versioned and separates canonical from materialized state; #394 keeps epistemic inputs canonical while support/warrant tables remain derived. Downstream symbolic-memory#4/#6 require append-only semantic history and rebuildable current views. The actual current `symbolic-memory/prolog/symbolic_memory_storage.pl` remains a bootstrap implementation: one process-wide mutex protects dynamic predicates; each transaction snapshots all projects/sources/memories/audits, serializes the entire `snapshot(...)` term to a temporary file, and renames it over the prior file. That makes transaction cost and restart footprint proportional to total resident state and provides no segmented event log, sparse frontier, causal index, projection checkpoint, anti-entropy or billion-record path. D therefore specifies a replacement *semantic storage contract*, not a claim that the current backend implements it. --- ## Primary / authoritative evidence 1. **O'Neil, Cheng, Gawlick, O'Neil — “The Log-Structured Merge-Tree (LSM-Tree),” Acta Informatica 33(4), 1996.** Write-heavy history/index workloads benefit from deferred batched merge structures rather than random in-place updates. DOI: https://doi.org/10.1007/s002360050048 2. **Chang et al. — “Bigtable: A Distributed Storage System for Structured Data,” OSDI 2006.** Demonstrates very large structured stores using ordered key ranges, memtables/immutable SSTables and distributed tablet partitioning: https://research.google/pubs/bigtable-a-distributed-storage-system-for-structured-data/ 3. **DeCandia et al. — “Dynamo: Amazon’s Highly Available Key-value Store,” SOSP 2007.** Primary precedent for replica versioning, eventual repair and Merkle-tree anti-entropy under highly available replication: https://doi.org/10.1145/1294261.1294281 4. **Preguiça, Baquero, Almeida, Fonte, Gonçalves — “Dotted Version Vectors: Logical Clocks for Optimistic Replication,” 2010/2011.** Causality metadata can be bounded by servers registering updates for an object rather than clients/updates: https://arxiv.org/abs/1011.5808 5. **Almeida, Baquero, Fonte — “Interval Tree Clocks,” OPODIS 2008.** Dynamic decentralized causality tracking without a preconfigured fixed process-ID vector: https://doi.org/10.1007/978-3-540-92221-6_18 6. **Ongaro & Ousterhout — “In Search of an Understandable Consensus Algorithm,” USENIX ATC 2014.** A replicated-log consensus profile can provide one committed history for one logical ledger: https://www.usenix.org/conference/atc14/technical-sessions/presentation/ongaro 7. **McSherry, Murray, Isaacs, Isard — “Differential Dataflow,” CIDR 2013.** Incremental nested/iterative computation, including changing graph computations/SCCs: https://www.microsoft.com/en-us/research/publication/differential-dataflow/ 8. **Budiu, Chajed, McSherry, Ryzhyk, Tannen — “DBSP: Automatic Incremental View Maintenance for Rich Query Languages,” PVLDB 16(7), 2023.** General delta-based incrementalization for rich relational/recursive views: https://doi.org/10.14778/3587136.3587149 9. **Almeida, Baquero, Preguiça, Hutchison — “Scalable Bloom Filters,” Information Processing Letters 101(6), 2007.** Space-efficient negative membership filtering with explicit false-positive probability: https://doi.org/10.1016/j.ipl.2006.10.007 These support mechanisms; none is adopted wholesale as Machine Spirit semantics. --- # Candidate execution architectures ## Candidate 1 — one ACID temporal relational database + eager materialized views ```text semantic/object/event tables ↓ SQL temporal/current views ↓ triggers/materialized views ``` **Strengths:** simple transactional model; good local implementation target; mature indexing, WAL and backup tooling. **Failure as the universal D architecture:** binds canonical semantics to one database's temporal/materialized-view behavior; encourages eager global indexes; does not itself solve independent-ledger federation, sparse cuts, replica-vs-source semantics or heterogeneous redaction/integrity profiles. A relational backend remains a valid implementation of the logical contract. ## Candidate 2 — distributed event store + global version vector + eager global causal/materialized closure ```text all ledgers/replicas ↓ global version vector ↓ causal transitive closure ↓ fully current materialized world model ``` **Rejected.** Metadata grows with global participants; dynamic/federated authorities are forced into one replication domain; causal-closure maintenance can dwarf useful queries; partitions invite either false global freshness or system-wide blocking. ## Candidate 3 — segmented authoritative ledgers + sparse hierarchical cuts + typed replica profiles + demanded incremental projections (**preferred**) Name: **SAFIRE — Sparse Append-only Frontier & Incremental Replay Engine**. ```text immutable semantic objects + segmented per-ledger append log + commit/checkpoint manifests ↓ sparse local/federation frontier descriptors ↓ delta feed ↓ versioned projection graph ↓ indexes/current/epistemic/procedural/retrieval views ``` Federation remains CLFM above independent ledgers. Replication remains a property *inside one logical-ledger identity*. --- # Preferred computational architecture — SAFIRE ## 1. Local ledger order is dense; federation order is sparse Every logical ledger owns a monotonically increasing commit/sequence coordinate: ```prolog ledger_position(LedgerId, Sequence). ``` A local checkpoint can summarize a contiguous durable prefix plus segment/manifests. It does not require a vector entry for unrelated ledgers. A federation cut names only participating ledgers/dependencies: ```prolog sem_federation_cut( CutId, FrontierManifestRef, LinkGeneration, MappingGeneration, AdmissionGeneration, RedactionGeneration, IntegrityGeneration, Completeness, Digest). ``` `FrontierManifestRef` may be an immutable hierarchical manifest whose leaves are exact `(Ledger, Frontier, Checkpoint)` tuples. Large cuts can share unchanged manifest subtrees instead of copying a million-entry map into every receipt. Hard invariant: ```text sparse/hierarchical cut compression != loss of exact participating frontier ``` The full logical cut must be reconstructible from referenced immutable manifests. ### Replica clocks are not federation clocks Inside an **optimistically replicated single logical ledger/object partition**, dotted version vectors or an equivalent bounded causality representation are valid candidates. Interval Tree Clocks are a candidate where participants are highly dynamic. They are *not* the canonical cross-ledger federation ordering mechanism. Independent ledgers remain separate authorities connected by explicit causal/import links. --- ## 2. Do not materialize global causal transitive closure Canonical causal evidence is the explicit graph: ```prolog sem_causal_edge(EdgeId, FromEvent, ToEvent, Kind, Evidence, Receipt). ``` Indexes provide: - event -> direct predecessors/successors; - ledger/sequence -> segment; - checkpoint -> covered sequence interval; - remote reference -> importing event; - segment-level summary metadata. A causality query performs bounded backward/forward traversal from the demanded cut. Frequently reused reachability results may be cached, but global closure is never canonical. ### Probabilistic indexes are negative accelerators only Segment/SSTable Bloom filters or scalable Bloom filters may answer: ```text "definitely absent" ``` or ```text "possibly present" ``` They can skip storage probes. A false positive may cause extra I/O; it may **never create a semantic edge, causal relation, source identity or authorization result**. --- ## 3. Segmented immutable storage layout Logical contract: ```text ledger ├─ immutable committed segments │ ├─ commit/event envelopes ordered by sequence │ ├─ object refs │ ├─ checksums / integrity refs │ └─ min/max summaries ├─ immutable semantic object blobs / rows ├─ checkpoint + manifest chain └─ rebuildable secondary indexes ``` Recommended physical shape at scale is backend-neutral but assumes: - append batches are sequential/transactional; - old committed segments are immutable except governed payload sanitization through GRIE; - indexes use sorted/partitioned structures (LSM/B-tree/etc.) rather than loading all records into Prolog dynamic predicates; - Prolog-facing APIs use iterators/cursors/page bounds; - namespaces/ledger IDs/sequence/time/semantic class/provenance IDs are index dimensions; - large source payloads and structural semantic objects may be separated so semantic scans do not drag full text through memory. A billion-record design cannot have `findall/3` over the whole store as its ordinary scan primitive. --- ## 4. Projection DAG + exact dependency generations Every materialized projection declares inputs and semantic dependencies: ```prolog sem_projection_descriptor( ProjectionId, ImplementationVersion, InputKinds, DependencyGenerations, IncrementalMode, CanonicalizationProfile). sem_projection_checkpoint( ProjectionId, InputCut, DependencyGenerations, StateDigest, PublishedAt, Status). ``` Dependencies include where applicable: ```text semantic schema / upcaster vocabulary identity/mapping admission redaction integrity context/bridge reasoning/epistemic profile procedure/PSC mapping ``` ### Delta processing For a projection at frontier `F` receiving committed delta `Δ`: ```text read Δ ↓ validate exact dependency generation ↓ compute output delta ↓ atomically publish changed index rows + new checkpoint ``` Simple indexes use direct delta maintenance. Recursive/iterative projections may use DBSP/differential-dataflow-style incrementalization when it wins empirically. **Do not require one universal dataflow runtime for every index.** ### Invalidation A dependency generation change invalidates only projections/partitions that declared that dependency. For huge fan-out changes (e.g. ontology/profile generation affecting enormous history), prefer: ```text generation bump + mark affected partitions stale + demanded/background bounded rebuild ``` over eagerly rewriting a billion derived rows in one transaction. Canonical history remains available while views catch up. --- ## 5. Projection freshness is explicit A query result carries at least: ```prolog projection_status( Projection, requested_cut(Requested), materialized_cut(Materialized), dependency_generations(Gens), freshness(Freshness), completeness(Completeness)). ``` Representative states: ```text exact lagging(Delta) stale_dependency rebuilding integrity_blocked redaction_blocked unavailable ``` Never answer from a lagging projection and label it simply `current`. Callers may choose policy: ```text allow_bounded_stale require_exact_cut fallback_to_demanded_ledger_evaluation ``` but the result must expose which path occurred. --- ## 6. Replication profiles are explicit and non-interchangeable ### `single_writer_local` One authoritative local ledger. Physical database WAL/backup is implementation detail. ### `consensus_replica_group` One logical ledger replicated through a consensus protocol (Raft/Paxos family or equivalent). Only committed log entries become ledger history. Minority partitions cannot fabricate committed history. ### `optimistic_replica_group` One logical object/partition family may accept concurrent versions during partition. Causality metadata (e.g. DVV) detects concurrency; reconciliation produces explicit retained branches/resolution events. This profile does **not** promise one write order during the partition. ### `mirror/archive` Copies committed history but has no authority to originate ledger commits under that identity. Hard invariant: ```text replication repair != semantic reconciliation != cross-ledger federation ``` Dynamo-style Merkle anti-entropy is appropriate for finding differing key/segment ranges inside a declared replica/mirror relation. A Merkle match proves content equality for the committed range under that digest scheme; it does not prove proposition truth or source independence. --- ## 7. Anti-entropy operates on immutable commit/segment identity Preferred repair flow: ```text exchange authenticated checkpoint/manifests ↓ compare range/segment digests ↓ identify differing ranges ↓ transfer missing immutable commits/objects ↓ validate event IDs/digests/ancestry ↓ advance replica checkpoint ``` Duplicate transfer is harmless through event/commit idempotency. If two peers claiming the same consensus/single-history ledger identity present incompatible committed ancestry, classify: ```text fork / integrity conflict ``` Do not union them automatically. Optimistic replica profiles may instead retain concurrent branches according to that profile's declared reconciliation semantics. --- ## 8. Crash behavior is deterministic at semantic boundaries Required crash points: 1. object staged, no commit -> unreachable staging garbage; not history; 2. event batch encoded, not durably committed -> absent; 3. ledger commit durable, projection not updated -> commit is authoritative; projection catches up; 4. projection rows written, checkpoint not atomically published -> those rows are invisible/discardable until a valid checkpoint; 5. checkpoint published -> readers may use it only if digest + dependency generations validate; 6. canonical segment corruption -> `integrity_failure`; projection cannot become replacement authority. Backends may use SQLite/Postgres/RocksDB/etc. WAL/transactions, but the semantic conformance suite tests the boundaries above independent of backend. --- ## 9. Partition behavior follows replica profile, never guesswork ```text consensus group without quorum -> no new committed writes; reads report exact known commit/read policy optimistic group partition -> concurrent versions allowed only if profile says so; later reconciliation explicit independent federated ledger unavailable -> local ledgers continue; federated cut/result becomes partial_remote_unavailable or incomplete_dependency ``` A remote outage cannot make unrelated local knowledge disappear, and a stale replica cannot advertise itself as globally current. --- ## 10. Governed redaction obligations become durable idempotent work GRIE/CLFM redaction obligations are canonical ledger objects. Execution state is append-only attempt/receipt history: ```prolog sem_redaction_attempt( AttemptId, ObligationId, TargetLedger, ExpectedTargetState, ActionClass, StartedAt, Result, RemoteReceipt, RetryClass). ``` Workers may retry at-least-once. The obligation ID + target + requested action make retries idempotent where the target supports it. Possible terminal/nonterminal states remain explicit: ```text pending in_progress complete(logical_redaction) complete(crypto_erased) complete(physical_purge) partial blocked failed unknown_remote_state ``` No local worker can upgrade `remote notice accepted` to `remote physical purge complete` without evidence defined by that target profile. Maintenance execution is storage/runtime work; it does not create a second Prolog-RLM plan scheduler and does not grant semantic actions host authority. --- ## 11. Billion-record assumptions / bounds The semantic contract must make these complexity expectations falsifiable rather than promise magic constants. ### Append For batch size `b`, logical append work should be `O(b)` plus backend index/log factors. It must not serialize or rewrite `O(total_history)`. ### Catch-up Projection catch-up from frontier `F` to `F+n` should consume the `n` committed deltas plus affected dependency closure. A small append must not normally force a full replay. ### Current/query indexes Point/range lookup must use persistent indexes. No requirement may imply loading all events/objects into SWI-Prolog heap. ### Federation cut Logical size is `O(k)` for the `k` participating ledger frontiers. Hierarchical immutable manifests may make *incremental storage* proportional to changed leaves/subtrees rather than copying all `k` entries per receipt. ### Causal reachability Demanded traversal cost is proportional to visited explicit edges/checkpoint summaries plus index lookup costs. Global `O(V^2)` reachability materialization is forbidden as a baseline requirement. ### Anti-entropy Merkle/range summaries should localize repair to differing ranges rather than resend complete history. Exact complexity depends on tree/segment layout; benchmark it instead of encoding a false universal bound. ### Provenance Durable provenance uses shared DAG/object references. Explanation expands only demanded paths under budgets; it does not duplicate full proof trees into every event. --- ## 12. Deterministic replay receipt A replayable result/checkpoint fingerprints: ```text ledger/federation cut exact commit/checkpoint manifests semantic object digests needed by the result schema/upcaster versions identity/mapping generations admission/redaction/integrity generations reasoning/procedural profile versions projection implementation/version canonicalization version query/options/budget class where semantically relevant ``` Canonical result hashing sorts/normalizes unordered sets before digesting. Scheduler/thread completion order must not change semantic digest. Hard differential oracle: ```text cold full rebuild == warm cached query == cache dropped + rebuild == incrementally maintained result ``` for the same exact cut + dependency generations. If a budget prevents completion, compare structured failure/completeness semantics, not a partial result masquerading as complete. --- # Falsifiable D conformance fixtures ### SAFIRE-D1 — append complexity class Append 1 event to histories of 10^3 and 10^7 records; instrumentation must prove no full-history serialization/rewrite path. ### D2 — crash before ledger commit Injected crash leaves no committed event/frontier advance. ### D3 — crash after ledger commit before projection Restart replays delta and reaches same projection digest as clean execution. ### D4 — torn/unpublished projection Projection rows without valid checkpoint are never visible as authoritative current state. ### D5 — projection corruption Corrupt projection -> reject/rebuild; canonical ledger unchanged. ### D6 — canonical segment corruption Return `integrity_failure`; projection cannot repair canonical authority. ### D7 — duplicate append retry Same idempotency key + same digest -> same commit receipt/no duplicate event. ### D8 — conflicting retry Same key + different digest -> hard idempotency conflict. ### D9 — sparse frontier Query over 3 ledgers in a federation containing 100k registered ledgers produces a cut logically naming only required participants/dependencies, not a 100k fixed vector. ### D10 — cut manifest exactness Compressed/hierarchical cut reconstructs exact same leaf frontier set and digest after restart. ### D11 — wall clock non-causality Later timestamp on independent ledger creates no causal edge. ### D12 — explicit causal import Remote event reference/import creates exactly the declared cross-ledger dependency. ### D13 — Bloom false positive safety Forced probabilistic-index false positive may add a storage probe but cannot add semantic/causal result. ### D14 — no global closure Instrumentation on sparse causal graph shows query-local traversal/cache, no mandatory global transitive-closure build. ### D15 — incremental current view One assertion append updates only affected current-view partition/dependencies. ### D16 — incremental/full oracle Incremental projection digest == clean full rebuild digest. ### D17 — cold/warm/rebuilt oracle Identical result/status/receipt semantics for cold, warm and rebuilt paths. ### D18 — dependency generation invalidation Identity/upcaster/profile generation change invalidates only declared consumers; unaffected projection digests stay valid. ### D19 — massive invalidation lazy generation Large ontology/profile change marks affected partition/generation stale without atomically rewriting all derived rows. ### D20 — stale result honesty Lagging projection reports exact materialized cut and never labels itself current. ### D21 — exact-cut query `require_exact_cut` either catches up/demand-evaluates to that cut or returns explicit inability; never silently serves older state. ### D22 — consensus minority partition Minority cannot create a committed canonical ledger event. ### D23 — optimistic concurrent updates Allowed partition writes remain concurrent branches until explicit reconciliation; no fake total order from timestamps. ### D24 — replica != corroboration Three replicas of one logical ledger contribute one source/assertion episode, not three independent evidence components. ### D25 — mirror cannot originate Mirror/archive refuses canonical commit under source-ledger identity. ### D26 — Merkle anti-entropy localization One damaged/missing segment causes transfer of differing range only; identical ranges are not resent. ### D27 — incompatible ancestry Peers claiming one single-history ledger with incompatible committed ancestry -> fork/integrity conflict, never automatic union. ### D28 — remote federation outage Local exact queries continue; cross-ledger query reports partial/incomplete cut. ### D29 — redaction retry idempotency Repeated obligation attempt cannot duplicate semantic lifecycle effect. ### D30 — redaction honesty Remote ACK without sanitization evidence remains notice/partial state, never physical-purge complete. ### D31 — erased payload resurrection defense Cache/rebuild/anti-entropy cannot restore GRIE-erased plaintext from a stale replica lacking authority to reintroduce it. ### D32 — schema/upcaster replay Historical cut under exact original interpretation and current re-interpretation remain separately reproducible where payload survives. ### D33 — billion-record cursor contract Synthetic/real large backend test proves bounded page/cursor APIs and no `findall(all_history)`/whole-store heap materialization on ordinary query path. ### D34 — provenance DAG sharing N derived results sharing one provenance subgraph store references rather than N full copied proof trees. ### D35 — deterministic concurrency Different projection-worker completion schedules produce identical canonicalized projection/result digest at same cut. ### D36 — resource limit semantics Projection/rebuild/causal traversal budget exhaustion yields `resource_limited`/lag status, not false/unknown/current. ### D37 — replica clock boundary DVV/ITC metadata inside replica profile cannot be interpreted as independent-ledger federation order. ### D38 — authority isolation Remote commit/receipt/replica metadata cannot create local host capability or principal authority. ### D39 — zero-model durable known task After semantic projection exists, restart + ledger replay + bounded retrieval + known reasoning/procedure path completes with `model_calls = 0`. ### D40 — end-to-end failure injection matrix Randomized crashes at the six semantic durability boundaries followed by restart must converge to either the last fully committed cut or a later fully committed cut, never a hybrid projection state. --- # Rejected shortcuts 1. **One global vector clock for every ledger** — metadata/federation coupling scales with participants and falsely treats independent authorities as one replication set. 2. **Wall-clock ordering as causality** — clock skew/concurrency destroys the guarantee. 3. **Global transitive causal closure** — unacceptable baseline storage/recompute cost; demanded traversal + caches suffice. 4. **One universal CRDT world model** — replica convergence is not semantic/epistemic reconciliation. 5. **One universal consensus protocol** — independent ledgers and high-availability optimistic replicas have materially different requirements. 6. **Projection tables as backup truth** — violates BSLPF/GRIE integrity boundary. 7. **Bloom/filter/index hit as semantic evidence** — probabilistic/derived index state is never authority. 8. **Eager rewrite on schema/ontology/profile change** — catastrophic fan-out; use version generations + bounded rebuild. 9. **Remote redaction ACK == sanitization proof** — false claim about another authority/storage medium. 10. **Full Prolog heap as the billion-record index** — current bootstrap technique, not scalable architecture. 11. **Automatic LLM fallback on storage/reasoning exhaustion** — resource failure remains structured failure; no provider call is implied. --- # Repository ownership / typed API boundary ### `lost-rob0t/symbolic-memory` owns - ledger/object physical adapter and logical commit API; - segment/checkpoint/frontier manifests; - replica profiles + anti-entropy implementation; - CLFM federation-link persistence/cuts; - GRIE redaction-obligation persistence/maintenance; - projection checkpoint/invalidation/freshness state; - scalable persistent indexes/cursor APIs; - storage crash/rebuild/failure-injection conformance. ### `lost-rob0t/prolog-rlm` owns - #392 semantic object meaning/schema/profile identities; - #394 epistemic/reasoning projection semantics; - #395 conformance expectations for semantic reasoning; - #401/DPEC procedural execution semantics; - consumers of bounded memory projections. Dependency stays `symbolic-memory -> prolog-rlm`. SAFIRE does not become a second reasoning semantics or plan scheduler. Conceptual storage-facing API: ```prolog memory_ledger_append(+Ledger,+Batch,+ExpectedFrontier,+Idempotency,-Receipt). memory_ledger_scan(+Ledger,+Range,+Filter,+Cursor,-Page,-NextCursor). memory_cut_open(+Scope,+Options,-Cut,-Receipt). memory_projection_status(+Projection,-Status). memory_projection_catchup(+Projection,+TargetCut,+Budget,-Receipt). memory_projection_rebuild(+Projection,+TargetCut,+Budget,-Receipt). memory_replica_sync(+ReplicaGroup,+Peer,+Budget,-Receipt). memory_redaction_process(+Obligation,+Budget,-Receipt). memory_storage_verify(+Scope,+Profile,-Receipt). ``` Exact implementation follows repo conventions; these are semantic contracts, not frozen exported predicate names. --- # Depth-5 reconciliation A/B/C/D now survive as one design: ```text BSLPF canonical bitemporal semantic history + rebuildable projections ↓ refined by GRIE payload retrievability/redaction/integrity/admission/replay capability ↓ composed by CLFM independent-ledger causality/federation/cuts/remote obligations ↓ realized by SAFIRE segmented logs + sparse cuts + projection DAG + typed replica profiles + anti-entropy + bounded indexes + deterministic replay/conformance ``` Depth 5 is therefore **design-complete only**. Implementation remains unproven. The current snapshot backend does not satisfy SAFIRE yet, and no billion-record, anti-entropy, crash-matrix, sanitization or deterministic-rebuild evidence exists merely because this design is recorded. ### Canonical handoffs required from this D pass - #397: mark Depth 5 A/B/C/D design-complete; queue #403A only. - #402: mark D complete / depth design-complete, then close as completed after reconciliation. - symbolic-memory#4: adopt BSLPF+GRIE+CLFM+SAFIRE as durable-world-model direction. - symbolic-memory#5: projection commit/checkpoint must be atomic and frontier/generation keyed. - symbolic-memory#6: implement segmented ledger/frontier/causal/projection/replica contracts rather than whole-snapshot rewrite as canonical architecture. - symbolic-memory#7/#9: query/MCP surfaces expose exact requested/materialized cut, freshness/completeness and replay/integrity/redaction status. - symbolic-memory#8: backfill/corpus ingestion uses idempotent batches, cursors/frontiers and resumable delta processing. - symbolic-memory#10: identity/mapping generation is a first-class projection invalidation dimension. ## Unresolved questions deliberately moved forward 1. first physical backend profile: SQLite/Postgres vs embedded LSM/other; 2. exact immutable segment encoding and compaction rules under GRIE redaction; 3. exact manifest tree and cut canonicalization format; 4. when to use ordinary incremental indexes versus DBSP/differential dataflow; 5. whether the first optimistic replica implementation needs DVV, ITC or can defer optimistic multi-writer replication; 6. consensus implementation/dependency choice if/when replicated logical ledgers are implemented; 7. exact anti-entropy range tree and authentication scheme; 8. benchmark thresholds for append latency, catch-up lag, rebuild throughput, point/range query latency and provenance expansion; 9. operational backup/restore and integrity-checkpoint retention policy; 10. how legal/policy deletion interacts with immutable off-site backups under deployment-specific requirements. Those are implementation/research inputs, not reasons to keep #402D open. **#402D complete. Depth 5 A/B/C/D reconciled. Exact next legal research subpass after canonical updates: #403A — retrieval / symbolic attention foundations.**
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
nsaspy/prolog-rlm#396
No description provided.