[semantic-memory] Resumable corpus/backfill ingestion for LLM logs, web research, docs, and historical memory

nsaspy commented

2026-09-08 01:57:03 +00:00

Owner

Parent: #4
Depends on: #5, #6

Goal

Make it practical to semantic-compile large existing corpora into symbolic memory without requiring one giant context or reprocessing everything on every run.

Primary targets:

historical LLM/chat logs;
agent/tool traces;
web research result archives;
documentation/manuals;
issue/PR discussions;
Org/Markdown/plain-text notes;
existing symbolic-memory source records lacking projections.

Required pipeline

corpus/source set
  -> deterministic source discovery
  -> immutable source registration/remember
  -> stable segmentation/events
  -> content-derived work IDs
  -> semantic compile through prolog-rlm#388
  -> append projection records
  -> checkpoint progress
  -> continue until exhausted

Resumability/idempotency

Track enough identity to avoid redoing completed units:

source content hash
segment/span fingerprint
semantic schema version
compiler version/config
projection generation

Interrupted ingestion resumes from uncompleted work. Re-running unchanged corpus should mostly become no-op/already-present work.

Changed source content is a new source/version and never mutates previous evidence.

Structured adapters

Where possible, extract structure deterministically before semantic compilation:

conversation/session/turn/role boundaries;
provider/model/tool-call metadata;
URLs/timestamps/title/author metadata;
document headings/sections;
issue/PR/comment identities.

That structure informs segmentation and provenance but does not replace semantic rule/procedure induction.

Large-corpus behavior

bounded chunk/window size;
overlap only when semantically useful and explicitly recorded;
stable segment identities;
bounded concurrency;
cancellation;
checkpoint/restart;
dedup identical source bytes;
progress/statistics inspection;
failures isolated per source/segment where possible;
no requirement to hold the entire corpus or resulting KB in memory.

Backfill mode

Provide a mode that walks existing durable memories and creates missing semantic projections without rewriting source records.

Conceptual API:

memory_symbolize_missing(+Scope, +Options, -Outcome).
memory_ingest_corpus(+SourceSpec, +Options, -Outcome).
memory_ingest_status(+RunId, -Outcome).
memory_ingest_resume(+RunId, +Options, -Outcome).

Acceptance

Ingest a multi-conversation LLM-log fixture and derive reusable procedural/rule knowledge across turns.
Ingest a web/docs fixture through the same generic semantic projection path.
Deterministic metadata extraction uses zero model calls where possible.
Semantic rule/procedure induction uses upstream compiler rather than a local duplicate.
Kill an ingestion halfway; resume without duplicating completed projections.
Re-run unchanged corpus and prove idempotent/no-op behavior for completed units.
Changed source creates new source/projection lineage while old evidence remains inspectable.
Backfill existing memories with no projection while preserving original IDs/source text.
Namespace/project boundaries remain intact during batch ingestion.
Progress reports actual compiled/skipped/failed/pending counts and provider usage when semantic compilation calls a model.

Parent: #4 Depends on: #5, #6 ## Goal Make it practical to semantic-compile **large existing corpora** into symbolic memory without requiring one giant context or reprocessing everything on every run. Primary targets: - historical LLM/chat logs; - agent/tool traces; - web research result archives; - documentation/manuals; - issue/PR discussions; - Org/Markdown/plain-text notes; - existing symbolic-memory source records lacking projections. ## Required pipeline ```text corpus/source set -> deterministic source discovery -> immutable source registration/remember -> stable segmentation/events -> content-derived work IDs -> semantic compile through prolog-rlm#388 -> append projection records -> checkpoint progress -> continue until exhausted ``` ## Resumability/idempotency Track enough identity to avoid redoing completed units: ```text source content hash segment/span fingerprint semantic schema version compiler version/config projection generation ``` Interrupted ingestion resumes from uncompleted work. Re-running unchanged corpus should mostly become no-op/already-present work. Changed source content is a new source/version and never mutates previous evidence. ## Structured adapters Where possible, extract structure deterministically before semantic compilation: - conversation/session/turn/role boundaries; - provider/model/tool-call metadata; - URLs/timestamps/title/author metadata; - document headings/sections; - issue/PR/comment identities. That structure informs segmentation and provenance but does not replace semantic rule/procedure induction. ## Large-corpus behavior - bounded chunk/window size; - overlap only when semantically useful and explicitly recorded; - stable segment identities; - bounded concurrency; - cancellation; - checkpoint/restart; - dedup identical source bytes; - progress/statistics inspection; - failures isolated per source/segment where possible; - no requirement to hold the entire corpus or resulting KB in memory. ## Backfill mode Provide a mode that walks existing durable memories and creates missing semantic projections without rewriting source records. Conceptual API: ```prolog memory_symbolize_missing(+Scope, +Options, -Outcome). memory_ingest_corpus(+SourceSpec, +Options, -Outcome). memory_ingest_status(+RunId, -Outcome). memory_ingest_resume(+RunId, +Options, -Outcome). ``` ## Acceptance - [ ] Ingest a multi-conversation LLM-log fixture and derive reusable procedural/rule knowledge across turns. - [ ] Ingest a web/docs fixture through the same generic semantic projection path. - [ ] Deterministic metadata extraction uses zero model calls where possible. - [ ] Semantic rule/procedure induction uses upstream compiler rather than a local duplicate. - [ ] Kill an ingestion halfway; resume without duplicating completed projections. - [ ] Re-run unchanged corpus and prove idempotent/no-op behavior for completed units. - [ ] Changed source creates new source/projection lineage while old evidence remains inspectable. - [ ] Backfill existing memories with no projection while preserving original IDs/source text. - [ ] Namespace/project boundaries remain intact during batch ingestion. - [ ] Progress reports actual compiled/skipped/failed/pending counts and provider usage when semantic compilation calls a model.

nsaspy commented

2026-09-08 01:57:27 +00:00

Author

Owner

Scope clarification from #4/#388/#392-#396: corpus/backfill ingestion must exercise heterogeneous knowledge, not only procedural extraction. Batch fixtures should include events/time, taxonomy/definitions, causal/diagnostic text, policy/defaults/exceptions, quantities/constraints, attributed conflicting claims, hypotheses/scenarios, organization/roles, and mixed documents; preserve cross-segment links and feed cross-memory reconciliation through #10 where needed.

nsaspy commented

2026-09-08 01:57:28 +00:00

Author

Owner

Machine Spirit #399A large-corpus implication

For LLM logs/web/docs backfill, do not make text-window overlap the primary cross-segment memory mechanism. The upstream compiler now needs to carry an explicit symbolic discourse workspace across bounded segments: accessible referents, entity/event candidates, speaker/quote stack, time anchors, section/genre/topic, deterministic source metadata, and unresolved semantic hypotheses.

Structured adapters should feed exact envelope metadata first (LLM role/message/tool-call/result/timestamp/provider IDs; web URL/publication/byline/update; document section/list/table structure) so the semantic model never has to rediscover metadata already known deterministically.

Checkpoint identity should therefore eventually include the upstream discourse/compiler-state version/fingerprint needed to reproduce segment interpretation. The durable corpus store still owns source/checkpoint lifecycle; the ephemeral Semantic Compilation Lattice remains upstream in Prolog-RLM.

Add a corpus fixture where an entity or speaker introduced in segment N is referenced several bounded segments later. The run must preserve identity or explicit ambiguity without loading the whole prior source into one provider context.

## Machine Spirit #399A large-corpus implication For LLM logs/web/docs backfill, do not make text-window overlap the primary cross-segment memory mechanism. The upstream compiler now needs to carry an explicit **symbolic discourse workspace** across bounded segments: accessible referents, entity/event candidates, speaker/quote stack, time anchors, section/genre/topic, deterministic source metadata, and unresolved semantic hypotheses. Structured adapters should feed exact envelope metadata first (LLM role/message/tool-call/result/timestamp/provider IDs; web URL/publication/byline/update; document section/list/table structure) so the semantic model never has to rediscover metadata already known deterministically. Checkpoint identity should therefore eventually include the upstream discourse/compiler-state version/fingerprint needed to reproduce segment interpretation. The durable corpus store still owns source/checkpoint lifecycle; the ephemeral Semantic Compilation Lattice remains upstream in Prolog-RLM. Add a corpus fixture where an entity or speaker introduced in segment N is referenced several bounded segments later. The run must preserve identity or explicit ambiguity without loading the whole prior source into one provider context.

nsaspy commented

2026-09-08 01:57:28 +00:00

Author

Owner

Upstream Machine Spirit #399C corpus-ingestion requirement

Large-corpus ingestion must treat analyzer/language/ontology heterogeneity as first-class work identity.

Extend resumable fingerprints/checkpoints to include, where applicable:

source language/genre
analyzer set + versions/configs
formalism/schema versions
mapping bundle / lexical-ontology mapping versions
translation/alignment artifact versions
semantic compiler interoperability profile

Operational requirements:

deterministic source adapters still run before semantic analyzers;
multilingual corpora may use language-specific analyzer plans rather than translating everything to English;
translation fallback is an explicit derived transform with alignment/divergence/loss provenance;
analyzer outputs and mapping transforms are independently cacheable so adding one analyzer does not force re-running all others;
mapping/version changes schedule new projection work only for affected units;
failures in one optional analyzer do not automatically invalidate usable observations from others, but required semantic dimensions must fail/degrade honestly;
progress should distinguish source parsing, analyzer observations, mapping/reconciliation, semantic export, and projection commit rather than one generic compiled counter.

See lost-rob0t/prolog-rlm#399 subpass C.

## Upstream Machine Spirit #399C corpus-ingestion requirement Large-corpus ingestion must treat analyzer/language/ontology heterogeneity as first-class work identity. Extend resumable fingerprints/checkpoints to include, where applicable: ```text source language/genre analyzer set + versions/configs formalism/schema versions mapping bundle / lexical-ontology mapping versions translation/alignment artifact versions semantic compiler interoperability profile ``` Operational requirements: - deterministic source adapters still run before semantic analyzers; - multilingual corpora may use language-specific analyzer plans rather than translating everything to English; - translation fallback is an explicit derived transform with alignment/divergence/loss provenance; - analyzer outputs and mapping transforms are independently cacheable so adding one analyzer does not force re-running all others; - mapping/version changes schedule new projection work only for affected units; - failures in one optional analyzer do not automatically invalidate usable observations from others, but required semantic dimensions must fail/degrade honestly; - progress should distinguish source parsing, analyzer observations, mapping/reconciliation, semantic export, and projection commit rather than one generic `compiled` counter. See lost-rob0t/prolog-rlm#399 subpass C.

nsaspy commented

2026-09-08 01:57:28 +00:00

Author

Owner

Upstream Machine Spirit #399D backfill execution requirement

Large-corpus/backfill ingestion should consume Prolog-RLM's Semantic Build Graph (SBG) execution receipts instead of treating segment fingerprint -> compiled yes/no as the whole resumability model.

Persist/checkpoint enough state to reuse completed immutable semantic work independently:

source adapter artifact refs/digests
per-analyzer artifact refs/digests
semantic-frontier / discourse dependency checkpoint refs
mapping/export artifact refs
compile-plan + work-key lineage
completed / dirty / failed stage refs
material semantic gaps
provider usage
projection generation

Hard behavior:

unchanged corpus reruns become stage-level cache hits/no-ops, not only whole-memory skips;
a one-span source edit invalidates the upstream-recorded dependency closure rather than automatically recompiling the entire corpus;
analyzer version bumps invalidate only that analyzer + actual descendants;
mapping/ontology bundle changes reuse native source/analyzer artifacts and recompute mapping/export descendants;
cancellation stops new work/provider calls but preserves completed immutable artifacts for resume;
partial analyzer/provider failure is isolated and resumable while preserving typed high-risk semantic gaps;
later text resolving earlier discourse ambiguity may create a bounded back-dependency and new affected projection generation;
exact replay of model-backed compilation requires retained candidate artifacts; if retention policy omitted them, a rerun is a new semantic recompile and must be labeled as such;
backfill progress should report cache hits/recomputed work/invalidation amplification in addition to compiled/skipped/failed counts.

Symbolic Memory owns the durable ingestion ledger/CAS retention policy; it should not duplicate SCL/SCC/analyzer scheduling logic from Prolog-RLM.

Full design and differential-conformance fixtures: lost-rob0t/prolog-rlm#399D.

## Upstream Machine Spirit #399D backfill execution requirement Large-corpus/backfill ingestion should consume Prolog-RLM's **Semantic Build Graph (SBG)** execution receipts instead of treating `segment fingerprint -> compiled yes/no` as the whole resumability model. Persist/checkpoint enough state to reuse completed immutable semantic work independently: ```text source adapter artifact refs/digests per-analyzer artifact refs/digests semantic-frontier / discourse dependency checkpoint refs mapping/export artifact refs compile-plan + work-key lineage completed / dirty / failed stage refs material semantic gaps provider usage projection generation ``` Hard behavior: - unchanged corpus reruns become stage-level cache hits/no-ops, not only whole-memory skips; - a one-span source edit invalidates the upstream-recorded dependency closure rather than automatically recompiling the entire corpus; - analyzer version bumps invalidate only that analyzer + actual descendants; - mapping/ontology bundle changes reuse native source/analyzer artifacts and recompute mapping/export descendants; - cancellation stops new work/provider calls but preserves completed immutable artifacts for resume; - partial analyzer/provider failure is isolated and resumable while preserving typed high-risk semantic gaps; - later text resolving earlier discourse ambiguity may create a bounded back-dependency and new affected projection generation; - exact replay of model-backed compilation requires retained candidate artifacts; if retention policy omitted them, a rerun is a new semantic recompile and must be labeled as such; - backfill progress should report cache hits/recomputed work/invalidation amplification in addition to compiled/skipped/failed counts. Symbolic Memory owns the durable ingestion ledger/CAS retention policy; it should not duplicate SCL/SCC/analyzer scheduling logic from Prolog-RLM. Full design and differential-conformance fixtures: lost-rob0t/prolog-rlm#399D.

nsaspy commented

2026-09-08 01:57:28 +00:00

Author

Owner

Machine Spirit #402A corpus/backfill durability handoff

Depth 5A (prolog-rlm#402A) makes the resumability contract more precise:

stable source/segment/compiler work identity
        ↓
retained compiler artifact where available
        ↓
atomic semantic-ledger commit with idempotency key/digest
        ↓
projection/frontier checkpoint

A corpus runner checkpoint is not proof a semantic projection is durable; the authoritative receipt is the ledger commit/frontier. After a crash, the runner reconciles work records against committed ledger identities and resumes missing work without duplicating already committed projections.

Changed source bytes remain new source lineage. Same source/work inputs may reuse #399D compiler artifacts, but semantic projection commits are independently idempotent. Progress/status views are rebuildable from durable work/commit lineage and must never manufacture completion from an uncommitted local checkpoint.

For large corpora, physical event/object segments may eventually be tiered, but canonical history cannot be destructively compacted merely to reduce index/log size. Depth 5D owns benchmark thresholds and physical backend choices.

## Machine Spirit #402A corpus/backfill durability handoff Depth 5A (`prolog-rlm#402A`) makes the resumability contract more precise: ```text stable source/segment/compiler work identity ↓ retained compiler artifact where available ↓ atomic semantic-ledger commit with idempotency key/digest ↓ projection/frontier checkpoint ``` A corpus runner checkpoint is **not** proof a semantic projection is durable; the authoritative receipt is the ledger commit/frontier. After a crash, the runner reconciles work records against committed ledger identities and resumes missing work without duplicating already committed projections. Changed source bytes remain new source lineage. Same source/work inputs may reuse #399D compiler artifacts, but semantic projection commits are independently idempotent. Progress/status views are rebuildable from durable work/commit lineage and must never manufacture completion from an uncommitted local checkpoint. For large corpora, physical event/object segments may eventually be tiered, but canonical history cannot be destructively compacted merely to reduce index/log size. Depth 5D owns benchmark thresholds and physical backend choices.

nsaspy commented

2026-09-08 01:57:28 +00:00

Author

Owner

MACHINE-SPIRIT #402C handoff — federated/backfill lineage

Corpus/backfill work that consumes remote/federated sources must checkpoint against logical remote ledger frontiers/checkpoints and federation-link/adapter versions, not merely local file position.

Required distinctions:

replica/mirror traversal must not ingest the same logical history as independent evidence;
remote retry of the same event identity+digest is idempotent;
conflicting reuse of one remote event identity/digest is an integrity failure;
an unavailable remote yields resumable/incomplete progress, not a completed scan;
adapter/mapping changes create new interpretation lineage instead of silently rewriting prior projection meaning;
backfill must preserve remote issuer/provenance while applying local admission policy separately.

Exact sparse frontier/change-feed algorithms are reserved for prolog-rlm#402D. Full composition semantics: prolog-rlm#402C.

### MACHINE-SPIRIT #402C handoff — federated/backfill lineage Corpus/backfill work that consumes remote/federated sources must checkpoint against **logical remote ledger frontiers/checkpoints and federation-link/adapter versions**, not merely local file position. Required distinctions: - replica/mirror traversal must not ingest the same logical history as independent evidence; - remote retry of the same event identity+digest is idempotent; - conflicting reuse of one remote event identity/digest is an integrity failure; - an unavailable remote yields resumable/incomplete progress, not a completed scan; - adapter/mapping changes create new interpretation lineage instead of silently rewriting prior projection meaning; - backfill must preserve remote issuer/provenance while applying local admission policy separately. Exact sparse frontier/change-feed algorithms are reserved for prolog-rlm#402D. Full composition semantics: prolog-rlm#402C.

nsaspy commented

2026-09-08 01:57:28 +00:00

Author

Owner

Machine Spirit #402D handoff: corpus/backfill ingestion should write stable idempotent event batches against expected ledger frontiers and advance resumable cursors only from committed receipts. Catch-up/projection work consumes deltas from exact frontiers; restart resumes without whole-corpus rescans. Large ingestion must use bounded cursor/page APIs, not whole-store findall/3. See prolog-rlm#402 SAFIRE D1-D8/D33/D40.

Machine Spirit #402D handoff: corpus/backfill ingestion should write stable idempotent event batches against expected ledger frontiers and advance resumable cursors only from committed receipts. Catch-up/projection work consumes deltas from exact frontiers; restart resumes without whole-corpus rescans. Large ingestion must use bounded cursor/page APIs, not whole-store `findall/3`. See prolog-rlm#402 SAFIRE D1-D8/D33/D40.

nsaspy commented

2026-09-08 01:57:28 +00:00

Author

Owner

Implementation decomposition

#8 remains the canonical large-corpus/backfill requirement. It is now split into:

#31 resumable corpus/backfill work engine
 -> #32 deterministic structured-source adapters

#22 supplies semantic projection mechanics; #23 publishes accepted semantic history. Keep ingestion orchestration out of the semantic compiler and out of transport adapters.

## Implementation decomposition #8 remains the canonical large-corpus/backfill requirement. It is now split into: ```text #31 resumable corpus/backfill work engine -> #32 deterministic structured-source adapters ``` #22 supplies semantic projection mechanics; #23 publishes accepted semantic history. Keep ingestion orchestration out of the semantic compiler and out of transport adapters.

nsaspy referenced this issue

2026-09-08 01:57:39 +00:00

[EPIC][CORE] Library-first Symbolic Memory v1: durable semantic world model, learning history, federation, and zero-model recall #38

Rows
Columns

[semantic-memory] Resumable corpus/backfill ingestion for LLM logs, web research, docs, and historical memory #41

Goal

Required pipeline

Resumability/idempotency

Structured adapters

Large-corpus behavior

Backfill mode

Acceptance

Machine Spirit #399A large-corpus implication

Upstream Machine Spirit #399C corpus-ingestion requirement

Upstream Machine Spirit #399D backfill execution requirement

Machine Spirit #402A corpus/backfill durability handoff

MACHINE-SPIRIT #402C handoff — federated/backfill lineage

Implementation decomposition