nsaspy/quasar
1
0
Fork 0
Code Issues 19 Pull requests 6 Projects Releases Packages Wiki Activity Actions

Phase 3: make ordinary workspace mutations record-bounded #38

Open
lost-rob0t wants to merge 35 commits from agent/issue-37-record-bounded-mutations into main
pull from: agent/issue-37-record-bounded-mutations
merge into: nsaspy:main
Conversation 0 Commits 35 Files changed 25 +3587 -584
lost-rob0t commented 2026-08-18 02:49:01 +00:00 (Migrated from github.com)
Copy link

Scope

Implements #37, Phase 3 of parent #24, on top of merged Phase 2 (#35 / PR #36).

Old defect

Post-Phase-2 ordinary mutations still followed the materialized path:

workspace-for -> load-workspace -> copy-workspace -> mutate -> commit

so changing one document could retain/copy the unrelated durable corpus.
workspace.transaction had the same corpus-sized candidate behavior.

TDD red evidence

The first meaningful Phase 3 commit was tests-only: 2cdcd9a6e843913c5227d23adb2506d34ffdf61d.

CI run 32093205188 / job 95579384402 failed all six new Phase 3 tests for the intended invariant: ordinary Tek9 mutations repopulated the full workspace cache, including the real 10,000-document case. No unrelated regression obscured the red evidence.

Record-bounded architecture

Production Tek9 mutations now use a record-level mutation context:

  • durable workspace metadata + base revision;
  • a partial semantic workspace overlay;
  • lazy direct document/graph/node/edge lookups;
  • bounded graph-sidecar scans only where no reference index exists;
  • staged presence/tombstone state for read-your-own-writes;
  • existing typed persistence changes compiled from validated application operations.

The authoritative Tek9 mutation routers do not call workspace-for, load-workspace, or copy-workspace. The non-streaming memory store retains an explicitly named materialized compatibility path for focused tests.

control-plane.lisp was split into cohesive core, mutation-routing, and runtime/registry modules so Phase 3 does not depend on another load-order override. The temporary Phase 3 override file used during development was deleted.

Transaction semantics

workspace.transaction uses the same overlay as single mutations:

  • ordered child operations;
  • read-your-own-writes and tombstones;
  • duplicate/conflict/referential validation against durable base + overlay;
  • one durable commit and one revision advance;
  • one transaction journal record;
  • ordered child events sharing transaction/revision metadata;
  • a 1000-child transaction budget.

Memory may scale with the explicit touched/dependency set, never with unrelated workspace corpus size.

Referential integrity

Phase 3 directly validates required document/node/edge dependencies through Tek9-backed access. Adversarial review added/fixed coverage for:

  • durable graph references without document-corpus hydration;
  • delete-then-reference transaction ordering;
  • relation documents referenced by live graph edges;
  • edge-delete then relation-delete in the same transaction;
  • repeated document touches;
  • multi-graph delete tombstones;
  • repeated puts of a newly staged graph;
  • the revision-zero conceptual all-documents graph before its metadata is first persisted.

Atomicity model

commit-change-set opens one Tek9 write transaction and:

  1. rechecks the durable base revision;
  2. applies typed document/graph/node/edge changes in deterministic order;
  3. derives document count without corpus enumeration;
  4. writes workspace settings/metadata at exactly one committed revision;
  5. writes exactly one journal entry;
  6. commits Tek9.

Events are emitted only after that transaction succeeds. Failure injection covers pre-change, post-change, pre-revision/meta, pre-journal, and pre-commit stages and requires zero record/topology/revision/journal/event leakage. Retry after failure is covered.

Large-corpus / memory evidence

The Phase 3 suite uses real Tek9 stores, close/reopen restart boundaries, and a 10,000-document corpus. It includes:

  • one-record update/create/delete without workspace-cache hydration;
  • bounded transactions;
  • exact durable record/revision/journal/event checks;
  • 1k versus 10k retained overlay working-set comparison;
  • transaction working-set checks for N = 1, 4, 16 touched records.

The tests assert that one-record retained working state remains the same small bounded record set as unrelated corpus size grows 10x, while transaction memory grows with its explicit touched set.

Tek9 boundary

No Tek9 change or pin bump was required. Existing exported Tek9 direct/range/graph/transaction APIs were sufficient. Quasar does not use tek9::, raw lmdb:, private cursors, or copied LMDB logic.

npm run check now includes a structural record-bounded mutation guard that rejects regression of the authoritative routing/bounded modules to full-workspace materialization and rejects private Tek9/raw LMDB use.

Documentation

Added docs/ADR-RECORD-BOUNDED-MUTATIONS.md and updated README/architecture while preserving unrelated existing deployment, security, StarLang, UI migration, and external-service documentation.

Remaining limitations

  • Some document/reference safety checks use bounded primary-range scans over graph sidecars because Tek9 currently has no dedicated document-reference index. Runtime can therefore be linear in graph-sidecar count, but retained heap is bounded to matching dependencies plus the range batch and does not scale with document corpus size.
  • Explicit whole-graph replace/delete may materialize that one explicitly touched graph. It never hydrates unrelated graphs or the document corpus.

Adversarial review findings fixed

Before merge review caught and fixed:

  1. multi-delete could choose a graph tombstoned earlier in the same transaction;
  2. repeated graph.put on a newly staged graph could lose its :new state;
  3. deleting a relation document could leave a live edge reference;
  4. revision-zero workspaces could lose the conceptual default graph semantics;
  5. an over-broad documentation rewrite had compressed unrelated architecture material and was reverted to focused edits.

No unresolved PR review comments are present.

Current merge gate

Exact current head: 65b91fccec2ce57f86de075d8a688242c8c44284.

Required CI run: 32096264185 (run #662), job 95588118741.

As of the latest check the exact-head job is still queued waiting for a GitHub-hosted runner. It has not reported a code/test failure. The PR remains draft and must not merge until this exact head completes the full mandatory CI matrix green.

Closes #37 only when the exact-head gates are green and the PR is merged.

Parent: #24. Evaluate #24 separately after #37; do not close the parent merely because Phase 3 is complete if any durable bounded-memory acceptance criterion remains materially unfinished.

## Scope Implements #37, Phase 3 of parent #24, on top of merged Phase 2 (#35 / PR #36). ## Old defect Post-Phase-2 ordinary mutations still followed the materialized path: `workspace-for -> load-workspace -> copy-workspace -> mutate -> commit` so changing one document could retain/copy the unrelated durable corpus. `workspace.transaction` had the same corpus-sized candidate behavior. ## TDD red evidence The first meaningful Phase 3 commit was tests-only: `2cdcd9a6e843913c5227d23adb2506d34ffdf61d`. CI run `32093205188` / job `95579384402` failed all six new Phase 3 tests for the intended invariant: ordinary Tek9 mutations repopulated the full workspace cache, including the real 10,000-document case. No unrelated regression obscured the red evidence. ## Record-bounded architecture Production Tek9 mutations now use a record-level mutation context: - durable workspace metadata + base revision; - a partial semantic workspace overlay; - lazy direct document/graph/node/edge lookups; - bounded graph-sidecar scans only where no reference index exists; - staged presence/tombstone state for read-your-own-writes; - existing typed persistence changes compiled from validated application operations. The authoritative Tek9 mutation routers do not call `workspace-for`, `load-workspace`, or `copy-workspace`. The non-streaming memory store retains an explicitly named materialized compatibility path for focused tests. `control-plane.lisp` was split into cohesive core, mutation-routing, and runtime/registry modules so Phase 3 does not depend on another load-order override. The temporary Phase 3 override file used during development was deleted. ## Transaction semantics `workspace.transaction` uses the same overlay as single mutations: - ordered child operations; - read-your-own-writes and tombstones; - duplicate/conflict/referential validation against durable base + overlay; - one durable commit and one revision advance; - one transaction journal record; - ordered child events sharing transaction/revision metadata; - a 1000-child transaction budget. Memory may scale with the explicit touched/dependency set, never with unrelated workspace corpus size. ## Referential integrity Phase 3 directly validates required document/node/edge dependencies through Tek9-backed access. Adversarial review added/fixed coverage for: - durable graph references without document-corpus hydration; - delete-then-reference transaction ordering; - relation documents referenced by live graph edges; - edge-delete then relation-delete in the same transaction; - repeated document touches; - multi-graph delete tombstones; - repeated puts of a newly staged graph; - the revision-zero conceptual `all-documents` graph before its metadata is first persisted. ## Atomicity model `commit-change-set` opens one Tek9 write transaction and: 1. rechecks the durable base revision; 2. applies typed document/graph/node/edge changes in deterministic order; 3. derives document count without corpus enumeration; 4. writes workspace settings/metadata at exactly one committed revision; 5. writes exactly one journal entry; 6. commits Tek9. Events are emitted only after that transaction succeeds. Failure injection covers pre-change, post-change, pre-revision/meta, pre-journal, and pre-commit stages and requires zero record/topology/revision/journal/event leakage. Retry after failure is covered. ## Large-corpus / memory evidence The Phase 3 suite uses real Tek9 stores, close/reopen restart boundaries, and a 10,000-document corpus. It includes: - one-record update/create/delete without workspace-cache hydration; - bounded transactions; - exact durable record/revision/journal/event checks; - 1k versus 10k retained overlay working-set comparison; - transaction working-set checks for N = 1, 4, 16 touched records. The tests assert that one-record retained working state remains the same small bounded record set as unrelated corpus size grows 10x, while transaction memory grows with its explicit touched set. ## Tek9 boundary No Tek9 change or pin bump was required. Existing exported Tek9 direct/range/graph/transaction APIs were sufficient. Quasar does not use `tek9::`, raw `lmdb:`, private cursors, or copied LMDB logic. `npm run check` now includes a structural record-bounded mutation guard that rejects regression of the authoritative routing/bounded modules to full-workspace materialization and rejects private Tek9/raw LMDB use. ## Documentation Added `docs/ADR-RECORD-BOUNDED-MUTATIONS.md` and updated README/architecture while preserving unrelated existing deployment, security, StarLang, UI migration, and external-service documentation. ## Remaining limitations - Some document/reference safety checks use bounded primary-range scans over graph sidecars because Tek9 currently has no dedicated document-reference index. Runtime can therefore be linear in graph-sidecar count, but retained heap is bounded to matching dependencies plus the range batch and does not scale with document corpus size. - Explicit whole-graph replace/delete may materialize that one explicitly touched graph. It never hydrates unrelated graphs or the document corpus. ## Adversarial review findings fixed Before merge review caught and fixed: 1. multi-delete could choose a graph tombstoned earlier in the same transaction; 2. repeated `graph.put` on a newly staged graph could lose its `:new` state; 3. deleting a relation document could leave a live edge reference; 4. revision-zero workspaces could lose the conceptual default graph semantics; 5. an over-broad documentation rewrite had compressed unrelated architecture material and was reverted to focused edits. No unresolved PR review comments are present. ## Current merge gate Exact current head: `65b91fccec2ce57f86de075d8a688242c8c44284`. Required CI run: `32096264185` (run #662), job `95588118741`. As of the latest check the exact-head job is still `queued` waiting for a GitHub-hosted runner. It has not reported a code/test failure. The PR remains draft and must not merge until this exact head completes the full mandatory CI matrix green. Closes #37 only when the exact-head gates are green and the PR is merged. Parent: #24. Evaluate #24 separately after #37; do not close the parent merely because Phase 3 is complete if any durable bounded-memory acceptance criterion remains materially unfinished.
nsaspy changed title from WIP: Phase 3: make ordinary workspace mutations record-bounded to Phase 3: make ordinary workspace mutations record-bounded 2026-08-27 19:50:33 +00:00
This pull request has changes conflicting with the target branch.
  • control-plane/src/control-plane.lisp
  • package.json
  • systems/quasar-control.asd
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin agent/issue-37-record-bounded-mutations:agent/issue-37-record-bounded-mutations
git switch agent/issue-37-record-bounded-mutations

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff agent/issue-37-record-bounded-mutations
git switch agent/issue-37-record-bounded-mutations
git rebase main
git switch main
git merge --ff-only agent/issue-37-record-bounded-mutations
git switch agent/issue-37-record-bounded-mutations
git rebase main
git switch main
git merge --no-ff agent/issue-37-record-bounded-mutations
git switch main
git merge --squash agent/issue-37-record-bounded-mutations
git switch main
git merge --ff-only agent/issue-37-record-bounded-mutations
git switch main
git merge agent/issue-37-record-bounded-mutations
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
Something isn't working

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
priority:P1
Highest priority work

  
question
Further information is requested

  
wontfix
This will not be worked on

No labels
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
priority:P1
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
nsaspy/quasar!38
No description provided.