Phase 1: make Tek9 the durable workspace document/graph store #33

New issue

Closed

opened 2026-08-17 20:46:55 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-17 20:46:55 +00:00

(Migrated from github.com)

Parent: #24

Goal

Make Tek9 the process-durable local persistence engine behind the Quasar Common Lisp control plane for normal workspace mutations while preserving the existing single-writer, revision, atomic commit, journal, graph-integrity, reconnect, and event-ordering guarantees.

Scope for this phase

add a real Tek9-backed workspace-store;
persist StarIntel documents as individual Tek9 records with workspace isolation;
persist named-graph metadata needed for exact restoration;
persist graph topology through Tek9 graph/v2 public APIs;
persist workspace revision/settings/schema metadata;
persist journal entries individually and atomically with the affected document/graph/meta changes;
keep one Tek9 environment open for the control-plane lifetime;
restore a workspace after closing and reopening an entirely new Tek9 store instance;
keep memory-store for focused unit tests;
record the backend decision and storage schema in architecture documentation;
pin the exact tested Tek9 revision and add LMDB to reproducible CI/dev dependencies.

If the current commit-workspace interface cannot express an efficient delta, introduce the smallest typed persistence commit plan derived from already-validated Quasar operation application. Do not diff or rewrite the entire corpus per mutation.

Hard invariants

one Quasar commit atomically covers affected documents, graph topology/meta, workspace revision/meta, and journal;
a persistence failure causes no live workspace swap, revision advancement, journal append, or event;
no whole-workspace serialized blob is the durable representation;
no normal single-record mutation scans/rewrites the full document corpus;
Quasar only calls exported Tek9 APIs, never tek9:: or raw lmdb:*;
Cytoscape/PouchDB/CouchDB remain projections/sync roles rather than local mutation authority;
full Tek9 durability remains enabled.

Required recovery fixture

Create documents of multiple dtypes including a relation, named graphs, nodes/edges, graph metadata, and a multi-operation transaction; stop/close the control plane and Tek9; construct a new store instance on the same path; restore and verify exact canonical state, revision, journal, graph IDs/edge IDs and cross-workspace isolation.

Failure injection

Force a storage failure after mixed document + graph writes are prepared but before outer LMDB commit. Verify the live workspace remains at revision N and a reopened Tek9 instance contains none of the candidate document, topology, adjacency, metadata, revision, or journal changes. Then rerun successfully and prove all records appear together at N+1.

Follow-on explicitly out of scope

Keep #24 open after this lands if active workspaces/import candidates remain heap materialized. The next phase is direct Tek9-backed reads/snapshots and durable staged streaming imports with bounded memory/backpressure.

Tek9 dependencies

Cross-link the Tek9 transaction-composition and any required generic graph-lifecycle API issues/PRs. Quasar must consume an exact merged Tek9 SHA/version, not floating master.

Parent: #24 ## Goal Make Tek9 the process-durable local persistence engine behind the Quasar Common Lisp control plane for normal workspace mutations while preserving the existing single-writer, revision, atomic commit, journal, graph-integrity, reconnect, and event-ordering guarantees. ## Scope for this phase - add a real Tek9-backed `workspace-store`; - persist StarIntel documents as individual Tek9 records with workspace isolation; - persist named-graph metadata needed for exact restoration; - persist graph topology through Tek9 graph/v2 public APIs; - persist workspace revision/settings/schema metadata; - persist journal entries individually and atomically with the affected document/graph/meta changes; - keep one Tek9 environment open for the control-plane lifetime; - restore a workspace after closing and reopening an entirely new Tek9 store instance; - keep `memory-store` for focused unit tests; - record the backend decision and storage schema in architecture documentation; - pin the exact tested Tek9 revision and add LMDB to reproducible CI/dev dependencies. If the current `commit-workspace` interface cannot express an efficient delta, introduce the smallest typed persistence commit plan derived from already-validated Quasar operation application. Do not diff or rewrite the entire corpus per mutation. ## Hard invariants - one Quasar commit atomically covers affected documents, graph topology/meta, workspace revision/meta, and journal; - a persistence failure causes no live workspace swap, revision advancement, journal append, or event; - no whole-workspace serialized blob is the durable representation; - no normal single-record mutation scans/rewrites the full document corpus; - Quasar only calls exported Tek9 APIs, never `tek9::` or raw `lmdb:*`; - Cytoscape/PouchDB/CouchDB remain projections/sync roles rather than local mutation authority; - full Tek9 durability remains enabled. ## Required recovery fixture Create documents of multiple dtypes including a relation, named graphs, nodes/edges, graph metadata, and a multi-operation transaction; stop/close the control plane and Tek9; construct a new store instance on the same path; restore and verify exact canonical state, revision, journal, graph IDs/edge IDs and cross-workspace isolation. ## Failure injection Force a storage failure after mixed document + graph writes are prepared but before outer LMDB commit. Verify the live workspace remains at revision N and a reopened Tek9 instance contains none of the candidate document, topology, adjacency, metadata, revision, or journal changes. Then rerun successfully and prove all records appear together at N+1. ## Follow-on explicitly out of scope Keep #24 open after this lands if active workspaces/import candidates remain heap materialized. The next phase is direct Tek9-backed reads/snapshots and durable staged streaming imports with bounded memory/backpressure. ## Tek9 dependencies Cross-link the Tek9 transaction-composition and any required generic graph-lifecycle API issues/PRs. Quasar must consume an exact merged Tek9 SHA/version, not floating `master`.