Persistent operation outbox: offline-safe idempotent ingest to StarIntel Server #12

New issue

Closed

opened 2026-08-16 05:17:39 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-16 05:17:39 +00:00

(Migrated from github.com)

Problem

Hackmode operations are local-first, but there is no durable synchronization path from an operation-local Tek9 store to starintel-server. Recon must continue while the central server is offline, and synchronization must not duplicate documents or silently lose poison messages.

Current behavior

Hackmode now has a canonical operation-local Tek9 store and deterministic asset lifecycle (#6 first slice).
starintel-server already exposes POST /new/document/:dtype and POST /documents/bulk; those endpoints publish documents to the existing RabbitMQ ingest routing.
StarIntel consumers already implement explicit ACK/NACK settlement and dead-letter policy.

Hackmode should therefore reuse the server's existing ingest boundary rather than introduce another RabbitMQ transport stack.

Desired behavior

Persist outgoing StarIntel-compatible documents locally before transmission and synchronize asynchronously through an operation outbox.

Lifecycle:

asset/result -> document projection -> local operation store -> persistent outbox -> StarIntel HTTP ingest -> acknowledged

Suggested states:

queued
sending
acknowledged
retry
failed
quarantined

Architecture

Create a small transport-neutral outbox protocol in Hackmode core and an HTTP StarIntel ingest adapter.

The outbox record must include at minimum:

stable message/outbox id
StarIntel document id/dtype
operation id
serialized payload or durable document reference
provenance reference
state
attempt count
created/updated/next-attempt timestamps
last error/status

Do not delete acknowledged records immediately unless retention/compaction policy is explicit; debugging and idempotency need durable state.

The sender should be actor/job friendly and must never block the operation REPL/Emacs UI on network availability.

Idempotency

enqueue must be deterministic for the same canonical document/version or otherwise explicitly deduplicated
retrying after ambiguous network failure must not construct a new logical document id
treat successful StarIntel acceptance as acknowledgment
distinguish retryable transport/server failures from permanently malformed payloads
poison/malformed documents become inspectable quarantined records, never silently dropped

Dependencies

#4 monorepo — complete
#6 canonical operation/asset state — first runtime slice complete; document projection still needed
#9 executable Common Lisp core CI
#3 StarIntel-compatible asset/document mapping

Acceptance criteria

operation continues when StarIntel Server is unavailable
enqueue is persisted before any network call
queued records survive process restart
server restoration drains retryable records
repeated retries do not create duplicate logical StarIntel documents
exponential/bounded backoff with observable next-attempt time
failed/quarantined records can be queried and inspected
no RabbitMQ credentials required in Hackmode for this default path
tests use a fake HTTP transport and do not require a live StarIntel server

Verification

enqueue document; close/reopen operation store; record remains queued
fake server offline -> retry state with attempt/error metadata
fake server restored -> acknowledged
ambiguous failure then success -> one logical document id
malformed document -> quarantined, inspectable
bulk drain preserves individual record state
operation asset/recon writes continue while sender is failing

Cross-repo note

If the existing StarIntel HTTP endpoint cannot provide enough acknowledgment/idempotency information, open a narrowly scoped starintel-server issue to extend the existing endpoint. Do not invent a parallel Hackmode-specific ingest RPC.

## Problem Hackmode operations are local-first, but there is no durable synchronization path from an operation-local Tek9 store to `starintel-server`. Recon must continue while the central server is offline, and synchronization must not duplicate documents or silently lose poison messages. ## Current behavior - Hackmode now has a canonical operation-local Tek9 store and deterministic asset lifecycle (#6 first slice). - `starintel-server` already exposes `POST /new/document/:dtype` and `POST /documents/bulk`; those endpoints publish documents to the existing RabbitMQ ingest routing. - StarIntel consumers already implement explicit ACK/NACK settlement and dead-letter policy. Hackmode should therefore reuse the server's existing ingest boundary rather than introduce another RabbitMQ transport stack. ## Desired behavior Persist outgoing StarIntel-compatible documents locally before transmission and synchronize asynchronously through an operation outbox. Lifecycle: `asset/result -> document projection -> local operation store -> persistent outbox -> StarIntel HTTP ingest -> acknowledged` Suggested states: - `queued` - `sending` - `acknowledged` - `retry` - `failed` - `quarantined` ## Architecture Create a small transport-neutral outbox protocol in Hackmode core and an HTTP StarIntel ingest adapter. The outbox record must include at minimum: - stable message/outbox id - StarIntel document id/dtype - operation id - serialized payload or durable document reference - provenance reference - state - attempt count - created/updated/next-attempt timestamps - last error/status Do not delete acknowledged records immediately unless retention/compaction policy is explicit; debugging and idempotency need durable state. The sender should be actor/job friendly and must never block the operation REPL/Emacs UI on network availability. ## Idempotency - enqueue must be deterministic for the same canonical document/version or otherwise explicitly deduplicated - retrying after ambiguous network failure must not construct a new logical document id - treat successful StarIntel acceptance as acknowledgment - distinguish retryable transport/server failures from permanently malformed payloads - poison/malformed documents become inspectable `quarantined` records, never silently dropped ## Dependencies - #4 monorepo — complete - #6 canonical operation/asset state — first runtime slice complete; document projection still needed - #9 executable Common Lisp core CI - #3 StarIntel-compatible asset/document mapping ## Acceptance criteria - operation continues when StarIntel Server is unavailable - enqueue is persisted before any network call - queued records survive process restart - server restoration drains retryable records - repeated retries do not create duplicate logical StarIntel documents - exponential/bounded backoff with observable next-attempt time - failed/quarantined records can be queried and inspected - no RabbitMQ credentials required in Hackmode for this default path - tests use a fake HTTP transport and do not require a live StarIntel server ## Verification 1. enqueue document; close/reopen operation store; record remains queued 2. fake server offline -> retry state with attempt/error metadata 3. fake server restored -> acknowledged 4. ambiguous failure then success -> one logical document id 5. malformed document -> quarantined, inspectable 6. bulk drain preserves individual record state 7. operation asset/recon writes continue while sender is failing ## Cross-repo note If the existing StarIntel HTTP endpoint cannot provide enough acknowledgment/idempotency information, open a narrowly scoped `starintel-server` issue to extend the existing endpoint. Do not invent a parallel Hackmode-specific ingest RPC.