Phase 2: move reads, snapshots, and import staging directly onto Tek9 #35

New issue

Closed

opened 2026-08-17 21:32:16 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-17 21:32:16 +00:00

(Migrated from github.com)

Follow-on to #24 and Phase 1 #33.

Phase 1 makes normal committed documents, named-graph metadata/topology, workspace metadata/revision, and the authoritative journal process-durable in structured Tek9 records. The active workspace and import candidate are still materialized in SBCL, so large corpora remain heap-bound.

Goal

Remove the remaining full-corpus heap requirement without changing the Phase 1 canonical on-disk schema.

Scope

serve document reads and indexed queries directly from Tek9 where practical;
make paginated workspace/document snapshots stream from Tek9 rather than requiring a fully materialized corpus;
replace import-session full workspace copies with durable Tek9-backed staging;
write/import chunks incrementally with bounded memory and explicit backpressure;
persist import stage metadata so interrupted imports can be inspected/recovered or abandoned deterministically;
atomically promote a completed staged import into canonical workspace state at one Quasar revision;
clean abandoned/expired stages safely;
add memory/throughput/backpressure telemetry and large-corpus fixtures;
preserve existing revision conflict, transaction, referential-integrity, reconnect, and event semantics.

Hard invariants

no full workspace copy per import;
no retaining every encoded import chunk in memory;
restart during staging must not expose partial canonical state;
final promotion is atomic with workspace revision and journal update;
readers see either the old canonical revision or the fully promoted new revision;
#24 can close only when bounded-memory durable workspace/import operation is actually proven.

Parent: #24.

Follow-on to #24 and Phase 1 #33. Phase 1 makes normal committed documents, named-graph metadata/topology, workspace metadata/revision, and the authoritative journal process-durable in structured Tek9 records. The active workspace and import candidate are still materialized in SBCL, so large corpora remain heap-bound. ## Goal Remove the remaining full-corpus heap requirement without changing the Phase 1 canonical on-disk schema. ## Scope - serve document reads and indexed queries directly from Tek9 where practical; - make paginated workspace/document snapshots stream from Tek9 rather than requiring a fully materialized corpus; - replace import-session full workspace copies with durable Tek9-backed staging; - write/import chunks incrementally with bounded memory and explicit backpressure; - persist import stage metadata so interrupted imports can be inspected/recovered or abandoned deterministically; - atomically promote a completed staged import into canonical workspace state at one Quasar revision; - clean abandoned/expired stages safely; - add memory/throughput/backpressure telemetry and large-corpus fixtures; - preserve existing revision conflict, transaction, referential-integrity, reconnect, and event semantics. ## Hard invariants - no full workspace copy per import; - no retaining every encoded import chunk in memory; - restart during staging must not expose partial canonical state; - final promotion is atomic with workspace revision and journal update; - readers see either the old canonical revision or the fully promoted new revision; - #24 can close only when bounded-memory durable workspace/import operation is actually proven. Parent: #24.