[P0-06] Make CouchDB ingest/update persistence and downstream publication crash-safe #14

New issue

Closed

opened 2026-07-22 00:09:59 +00:00 by lost-rob0t · 2 comments

lost-rob0t commented

2026-07-22 00:09:59 +00:00

(Migrated from github.com)

Confirmed data-loss windows

New document ingest

handle-new-document currently performs:

insert into CouchDB
publish documents.new.<dtype>
ACK the ingest delivery

If publication fails after the insert, the delivery is requeued. On retry, CouchDB returns conflict; the conflict handler NACKs with requeue nil, so the downstream documents.new event is permanently lost. A process crash in the same interval has the same failure mode.

Document update

handle-update performs the same unsafe sequence:

persist the update
publish documents.updated.<dtype>
ACK the update delivery

A publish failure or crash after persistence can cause a retry/conflict loop, duplicate mutation attempts, or a persisted update with no downstream event.

Required changes

Choose and document a recoverable pattern, preferably a CouchDB-backed outbox or an idempotent mutation record with publication state.

Requirements:

a stable message/document mutation idempotency key
persistence and downstream-event state committed together or reconstructable
conflict handling that distinguishes duplicate success from invalid conflicting content
retry-safe publication for both new and updated documents
explicit at-least-once semantics and downstream deduplication
recovery scan for unpublished persisted mutations
no ACK until recoverable state is durable
outbox/event ordering rules per document id where ordering matters

Acceptance tests

Force publish failure after a successful insert; retry eventually emits exactly one logical documents.new event.
Force publish failure after a successful update; recovery eventually emits exactly one logical documents.updated event for that mutation.
Crash/restart between persistence and publish recovers the event.
Duplicate ingest/update deliveries do not create duplicate logical mutations or events.
Invalid conflicting content is quarantined rather than treated as a successful duplicate.
Recovery preserves required per-document event order.

## Confirmed data-loss windows ### New document ingest `handle-new-document` currently performs: 1. insert into CouchDB 2. publish `documents.new.<dtype>` 3. ACK the ingest delivery If publication fails after the insert, the delivery is requeued. On retry, CouchDB returns conflict; the conflict handler NACKs with `requeue nil`, so the downstream `documents.new` event is permanently lost. A process crash in the same interval has the same failure mode. ### Document update `handle-update` performs the same unsafe sequence: 1. persist the update 2. publish `documents.updated.<dtype>` 3. ACK the update delivery A publish failure or crash after persistence can cause a retry/conflict loop, duplicate mutation attempts, or a persisted update with no downstream event. ## Required changes Choose and document a recoverable pattern, preferably a CouchDB-backed outbox or an idempotent mutation record with publication state. Requirements: - a stable message/document mutation idempotency key - persistence and downstream-event state committed together or reconstructable - conflict handling that distinguishes duplicate success from invalid conflicting content - retry-safe publication for both new and updated documents - explicit at-least-once semantics and downstream deduplication - recovery scan for unpublished persisted mutations - no ACK until recoverable state is durable - outbox/event ordering rules per document id where ordering matters ## Acceptance tests - Force publish failure after a successful insert; retry eventually emits exactly one logical `documents.new` event. - Force publish failure after a successful update; recovery eventually emits exactly one logical `documents.updated` event for that mutation. - Crash/restart between persistence and publish recovers the event. - Duplicate ingest/update deliveries do not create duplicate logical mutations or events. - Invalid conflicting content is quarantined rather than treated as a successful duplicate. - Recovery preserves required per-document event order.

lost-rob0t commented

2026-07-26 15:00:12 +00:00

(Migrated from github.com)

Active implementation: PR #65 (agent/issue-14-couchdb-outbox).

The branch adds an embedded CouchDB transactional outbox, stable mutation/event IDs, duplicate-versus-conflict detection, separate ingest/update and downstream routes, pending-event recovery, per-document sequence ordering, and forced-failure/restart regression coverage. Waiting on required fixture and CLOS/outbox conformance CI.

Active implementation: PR #65 (`agent/issue-14-couchdb-outbox`). The branch adds an embedded CouchDB transactional outbox, stable mutation/event IDs, duplicate-versus-conflict detection, separate ingest/update and downstream routes, pending-event recovery, per-document sequence ordering, and forced-failure/restart regression coverage. Waiting on required fixture and CLOS/outbox conformance CI.

lost-rob0t commented

2026-07-26 15:10:37 +00:00

(Migrated from github.com)

Completed via merged PR #65.

Validation:

fixture-contract: passed
clos-conformance: passed
forced publish failure after new/update persistence: passed
restart recovery: passed
duplicate delivery and conflicting idempotency-key handling: passed
missing update target rejection: passed
per-document event ordering: passed

Merge commit: 98d9fff0caa0d10f3f1140d2ec94484ede8f02c6.

Completed via merged PR #65. Validation: - fixture-contract: passed - clos-conformance: passed - forced publish failure after new/update persistence: passed - restart recovery: passed - duplicate delivery and conflicting idempotency-key handling: passed - missing update target rejection: passed - per-document event ordering: passed Merge commit: `98d9fff0caa0d10f3f1140d2ec94484ede8f02c6`.