Add durable BBP gateway recovery journal #33

Merged

lost-rob0t merged 24 commits from agent/star-lang-bbp-runtime-journal-v1 into main

2026-07-23 18:02:09 +00:00

lost-rob0t commented

2026-07-23 15:59:25 +00:00

(Migrated from github.com)

What

add a generic append-only runtime journal port
provide in-memory and file-backed journal implementations
persist pending commands before remote delivery
persist route retries, including no-live-node retries
persist remote terminal results before in-memory completion
restore dispatcher sequence, clock, idempotency records, pending commands, and completed results
redeliver restored pending commands only after a compatible worker registers
reconnect a persistent worker to a restarted main without rebuilding its program actors
preserve deterministic terminal envelope IDs across restart
validate journal command shape, settled outcomes, and monotonic sequence/clock order
detect malformed or backward-moving append logs
add focused recovery tests, a real three-process restart test, and research documentation

Event model

:pending
  command was durably recorded before remote tell

:route-result
  routing did not defer; restore dispatcher retry or failure state

:remote-result
  worker returned a result; restore terminal or retry completion state

Each event records the dispatcher sequence and clock at its transition boundary. Replaying the journal reconstructs terminal reply/error/ack envelopes with the same deterministic IDs and timestamps.

Recovery flow

phase-one main
  -> append pending command
  -> deliver to worker
  -> exit before result

phase-two main
  -> replay file journal
  -> restore pending and dispatcher state
  -> worker reconnects with preserved actors
  -> registration triggers redelivery
  -> worker run-id cache prevents duplicate execution
  -> append remote result
  -> restore deterministic terminal replay

Verification

memory journal returns defensive copies
file journal round-trips readable events with reader evaluation disabled
pending command survives simulated and real main-gserver restarts
pending command is not sent before worker registration
worker registration triggers one redelivery
external runner executes exactly once across restart and redelivery
terminal completion is journaled before pending removal
second restart restores terminal dispatcher replay
duplicate submission replays byte-identical terminal outcomes
failed remote tell restores retry state and is not auto-redelivered
no-node retry is durable, non-pending, and can be attempted again
one deterministic node selection is used per journaled route
malformed event shape, deferred settled outcomes, non-command envelopes, backward sequence, and backward clock are rejected
dedicated remoting, idempotency, journal, live restart, and full Common Lisp workflows pass

Stack

This draft is based on agent/star-lang-bbp-run-idempotency-v1 / PR #32.

Known prototype limits

file adapter calls finish-output but does not yet expose OS-level fsync
no checksums or torn-tail repair
no compaction or snapshots
single-writer ownership is not enforced

formal dispatcher command replay identity across terminal, in-progress, and retry states
fsync-capable storage adapter
checksummed records and torn-tail recovery
compacted snapshots

## What - add a generic append-only runtime journal port - provide in-memory and file-backed journal implementations - persist pending commands before remote delivery - persist route retries, including no-live-node retries - persist remote terminal results before in-memory completion - restore dispatcher sequence, clock, idempotency records, pending commands, and completed results - redeliver restored pending commands only after a compatible worker registers - reconnect a persistent worker to a restarted main without rebuilding its program actors - preserve deterministic terminal envelope IDs across restart - validate journal command shape, settled outcomes, and monotonic sequence/clock order - detect malformed or backward-moving append logs - add focused recovery tests, a real three-process restart test, and research documentation ## Event model ```text :pending command was durably recorded before remote tell :route-result routing did not defer; restore dispatcher retry or failure state :remote-result worker returned a result; restore terminal or retry completion state ``` Each event records the dispatcher sequence and clock at its transition boundary. Replaying the journal reconstructs terminal reply/error/ack envelopes with the same deterministic IDs and timestamps. ## Recovery flow ```text phase-one main -> append pending command -> deliver to worker -> exit before result phase-two main -> replay file journal -> restore pending and dispatcher state -> worker reconnects with preserved actors -> registration triggers redelivery -> worker run-id cache prevents duplicate execution -> append remote result -> restore deterministic terminal replay ``` ## Verification - memory journal returns defensive copies - file journal round-trips readable events with reader evaluation disabled - pending command survives simulated and real main-gserver restarts - pending command is not sent before worker registration - worker registration triggers one redelivery - external runner executes exactly once across restart and redelivery - terminal completion is journaled before pending removal - second restart restores terminal dispatcher replay - duplicate submission replays byte-identical terminal outcomes - failed remote tell restores retry state and is not auto-redelivered - no-node retry is durable, non-pending, and can be attempted again - one deterministic node selection is used per journaled route - malformed event shape, deferred settled outcomes, non-command envelopes, backward sequence, and backward clock are rejected - dedicated remoting, idempotency, journal, live restart, and full Common Lisp workflows pass ## Stack This draft is based on `agent/star-lang-bbp-run-idempotency-v1` / PR #32. ## Known prototype limits - file adapter calls `finish-output` but does not yet expose OS-level `fsync` - no checksums or torn-tail repair - no compaction or snapshots - single-writer ownership is not enforced ## Next - formal dispatcher command replay identity across terminal, in-progress, and retry states - fsync-capable storage adapter - checksummed records and torn-tail recovery - compacted snapshots