[P2] Repair observation/accounting events after local crash windows #82

New issue

Closed

opened 2026-08-17 19:55:28 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-17 19:55:28 +00:00

(Migrated from github.com)

Discovered by

Post-merge adversarial review of PR #78 / #57.

Problem

Recording an authoritative observation is intentionally ordered as separate durable writes:

persist immutable observation
-> append observed attempt revision
-> append observation_recorded event with usage/provenance

This ordering correctly protects the observation itself, but it leaves a local crash window after the observation/observed revision is durable and before the accounting/history event is appended.

After restart:

decision_for_existing/2 sees the immutable observation and immediately returns replay;
an already-observed attempt is not repaired by settle_observed_status/1;
no path guarantees the missing observation_recorded event is reconstructed exactly once.

The result can be an authoritative successful observation whose usage/provenance is absent from the event stream that the current docs say #44/#45 may consume.

The same general question applies to lifecycle events around other multi-write transitions, but observation usage/provenance is the important accounting case.

Required invariant

The durable observation is authoritative. Event/history materialization must be recoverable/idempotent from authoritative state after any local crash between persistence writes.

Do not solve this by making the external effect run again.

Suggested direction

give lifecycle/accounting events a stable event identity derived from attempt + revision/type (or equivalent);
on replay/status/restart, detect and append a missing derived event idempotently;
or make #44/#45 consume authoritative observation/attempt state directly and explicitly downgrade the event stream from authoritative accounting source;
whichever design is chosen, document the source of truth and prove crash recovery.

Required tests

Use fresh SWI processes and explicit crash markers, not sleeps:

observation persisted, process killed before final observed revision;
observed revision persisted, process killed before observation event;
restart repairs state without remote resubmission;
usage/provenance is present exactly once after recovery;
repeated replay/status calls do not duplicate accounting events.

Refs #57 #44 #45 #80

## Discovered by Post-merge adversarial review of PR #78 / #57. ## Problem Recording an authoritative observation is intentionally ordered as separate durable writes: ```text persist immutable observation -> append observed attempt revision -> append observation_recorded event with usage/provenance ``` This ordering correctly protects the observation itself, but it leaves a local crash window after the observation/observed revision is durable and before the accounting/history event is appended. After restart: - `decision_for_existing/2` sees the immutable observation and immediately returns replay; - an already-`observed` attempt is not repaired by `settle_observed_status/1`; - no path guarantees the missing `observation_recorded` event is reconstructed exactly once. The result can be an authoritative successful observation whose usage/provenance is absent from the event stream that the current docs say #44/#45 may consume. The same general question applies to lifecycle events around other multi-write transitions, but observation usage/provenance is the important accounting case. ## Required invariant The durable observation is authoritative. Event/history materialization must be recoverable/idempotent from authoritative state after any local crash between persistence writes. Do not solve this by making the external effect run again. ## Suggested direction - give lifecycle/accounting events a stable event identity derived from attempt + revision/type (or equivalent); - on replay/status/restart, detect and append a missing derived event idempotently; - or make #44/#45 consume authoritative observation/attempt state directly and explicitly downgrade the event stream from authoritative accounting source; - whichever design is chosen, document the source of truth and prove crash recovery. ## Required tests Use fresh SWI processes and explicit crash markers, not sleeps: 1. observation persisted, process killed before final observed revision; 2. observed revision persisted, process killed before observation event; 3. restart repairs state without remote resubmission; 4. usage/provenance is present exactly once after recovery; 5. repeated replay/status calls do not duplicate accounting events. Refs #57 #44 #45 #80