Replace heap-bound imports with durable streaming workspace storage #24

New issue

Open

opened 2026-08-09 03:39:25 +00:00 by lost-rob0t · 1 comment

lost-rob0t commented

2026-08-09 03:39:25 +00:00

(Migrated from github.com)

Problem

Quasar currently keeps the canonical workspace and an atomic import candidate in SBCL memory. Large JSON corpora expand substantially after parsing, so safe imports depend on a large Lisp heap even after removing redundant journal copies. This does not scale predictably and makes process memory the durability boundary.

Goal

Move canonical workspace persistence and staged imports to a durable database-backed implementation with bounded-memory streaming and backpressure. Preserve the existing single-writer, atomic commit, revision, journal, authorization, and event contracts.

Scope

Implement a process-durable workspace-store backend suitable for canonical documents, graphs, and journals.
Stage import chunks durably instead of retaining a complete candidate workspace in the Lisp heap.
Validate incrementally with bounded queues and explicit backpressure.
Commit a completed import atomically as one revision and one event.
Clean up abandoned, rejected, conflicted, and expired staging sessions.
Add memory/throughput telemetry and configurable import budgets.
Retain the in-memory store for focused tests only.

Acceptance criteria

Import memory remains bounded as corpus size grows and does not require increasing QUASAR_DYNAMIC_SPACE_MB for supported inputs.
A failed, aborted, or disconnected import changes no authoritative workspace state and leaves no unbounded staging data.
Successful imports remain atomic, durable across process restart, and emit one committed revision/event.
Concurrent workspace authorization and revision-conflict behavior remains covered.
Automated tests exercise large imports, restart recovery, cleanup, backpressure, and failure injection.
Architecture and operational documentation describe storage, migration, recovery, and capacity limits.

Suggested first slice

Write an ADR comparing CouchDB-native staging, SQLite staging, and another transactional local store; select the backend and define the migration/recovery contract before implementation.

## Problem Quasar currently keeps the canonical workspace and an atomic import candidate in SBCL memory. Large JSON corpora expand substantially after parsing, so safe imports depend on a large Lisp heap even after removing redundant journal copies. This does not scale predictably and makes process memory the durability boundary. ## Goal Move canonical workspace persistence and staged imports to a durable database-backed implementation with bounded-memory streaming and backpressure. Preserve the existing single-writer, atomic commit, revision, journal, authorization, and event contracts. ## Scope - Implement a process-durable `workspace-store` backend suitable for canonical documents, graphs, and journals. - Stage import chunks durably instead of retaining a complete candidate workspace in the Lisp heap. - Validate incrementally with bounded queues and explicit backpressure. - Commit a completed import atomically as one revision and one event. - Clean up abandoned, rejected, conflicted, and expired staging sessions. - Add memory/throughput telemetry and configurable import budgets. - Retain the in-memory store for focused tests only. ## Acceptance criteria - Import memory remains bounded as corpus size grows and does not require increasing `QUASAR_DYNAMIC_SPACE_MB` for supported inputs. - A failed, aborted, or disconnected import changes no authoritative workspace state and leaves no unbounded staging data. - Successful imports remain atomic, durable across process restart, and emit one committed revision/event. - Concurrent workspace authorization and revision-conflict behavior remains covered. - Automated tests exercise large imports, restart recovery, cleanup, backpressure, and failure injection. - Architecture and operational documentation describe storage, migration, recovery, and capacity limits. ## Suggested first slice Write an ADR comparing CouchDB-native staging, SQLite staging, and another transactional local store; select the backend and define the migration/recovery contract before implementation.

lost-rob0t commented

2026-08-17 21:36:56 +00:00

(Migrated from github.com)

Phase 1 Tek9 storage landed

Merged via #34 at 32ecd2995916768d7f3020f2fbe1f563dc18771e.

Completed subset of #24:

Tek9 is now Quasar's canonical local embedded process-durable document/graph store; LMDB remains private beneath Tek9.
StarIntel documents are stored individually with workspace isolation.
Named-graph metadata is stored independently from topology.
graph/v2 stores durable node/edge topology and adjacency; Quasar keeps canonical node/edge sidecars only for application fields outside Tek9's generic topology shape.
workspace schema/revision/settings metadata is versioned and durable.
journal entries are individual ordered records.
one Quasar commit atomically persists affected document/graph/meta/revision/journal records in one Tek9 write transaction before the live candidate is installed or events are broadcast.
mixed-domain failure injection proves no document, graph/adjacency, revision, journal, live-workspace, or event partial commit.
restart recovery is proven with an entirely new Tek9 store and control-plane instance.
ordinary single-document mutation is covered by a regression asserting it does not rewrite the corpus.
production startup owns one full-durability Tek9 environment for its lifetime at the XDG data path.
Quasar pins merged Tek9 ca24ef35ea6877420cbca057dd7fb702fe29a740 and CI installs/tests LMDB.

#24 remains open. Phase 1 does not make the complete workspace/import path bounded-memory yet. Remaining limitations include:

the active canonical compatibility workspace is still fully materialized in SBCL;
import sessions still copy the complete workspace candidate;
encoded import chunks are still retained in memory;
snapshots/read paths still depend on the materialized workspace;
import staging is not yet process durable;
memory growth therefore still scales with the active corpus/import candidate.

The narrow follow-on is #35: move reads/paginated snapshots and staged imports directly onto Tek9, add durable chunk staging, backpressure, restart recovery, abandoned-stage cleanup, and memory/throughput telemetry without migrating the Phase 1 on-disk canonical schema again.

## Phase 1 Tek9 storage landed Merged via #34 at `32ecd2995916768d7f3020f2fbe1f563dc18771e`. Completed subset of #24: - Tek9 is now Quasar's canonical local embedded process-durable document/graph store; LMDB remains private beneath Tek9. - StarIntel documents are stored individually with workspace isolation. - Named-graph metadata is stored independently from topology. - graph/v2 stores durable node/edge topology and adjacency; Quasar keeps canonical node/edge sidecars only for application fields outside Tek9's generic topology shape. - workspace schema/revision/settings metadata is versioned and durable. - journal entries are individual ordered records. - one Quasar commit atomically persists affected document/graph/meta/revision/journal records in one Tek9 write transaction before the live candidate is installed or events are broadcast. - mixed-domain failure injection proves no document, graph/adjacency, revision, journal, live-workspace, or event partial commit. - restart recovery is proven with an entirely new Tek9 store and control-plane instance. - ordinary single-document mutation is covered by a regression asserting it does not rewrite the corpus. - production startup owns one full-durability Tek9 environment for its lifetime at the XDG data path. - Quasar pins merged Tek9 `ca24ef35ea6877420cbca057dd7fb702fe29a740` and CI installs/tests LMDB. #24 remains open. Phase 1 does **not** make the complete workspace/import path bounded-memory yet. Remaining limitations include: - the active canonical compatibility workspace is still fully materialized in SBCL; - import sessions still copy the complete workspace candidate; - encoded import chunks are still retained in memory; - snapshots/read paths still depend on the materialized workspace; - import staging is not yet process durable; - memory growth therefore still scales with the active corpus/import candidate. The narrow follow-on is #35: move reads/paginated snapshots and staged imports directly onto Tek9, add durable chunk staging, backpressure, restart recovery, abandoned-stage cleanup, and memory/throughput telemetry without migrating the Phase 1 on-disk canonical schema again.