[P1] Bind effect identity to a durable store namespace and safe store lifecycle #81

New issue

Closed

opened 2026-08-17 19:55:13 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-17 19:55:13 +00:00

(Migrated from github.com)

Discovered by

Post-merge adversarial review of PR #78 / #57.

This should be resolved before #79 routes real provider/tool/MCP effects through the substrate at scale.

Problem 1: independent stores derive identical remote idempotency keys

Current automatic identity is content-derived:

call_id = hash(kind, normalized request, semantics)
attempt_id = hash(call_id, fingerprint, sequence, parent, mode)
idempotency_key = hash(attempt_id)

No durable store/project/runtime namespace participates in the identity.

Therefore two independent ledger files executing the same initial request/semantics derive the same attempt_id and provider idempotency_key. If a remote provider scopes idempotency keys more broadly than one local ledger, an operation in store B can be deduplicated against an unrelated operation from store A.

The same problem appears after explicit pruning: recreating the same initial logical call derives the old idempotency key again even though local replay state was intentionally removed.

Problem 2: store switching is not bound to an in-flight executor attempt

effect_persist_open/1 supports switching the globally attached store, while adapter submit runs outside the persistency mutex after dispatching is recorded.

A concurrent host store switch can therefore occur between:

store A: durable dispatching
-> remote adapter executes
-> host switches global store to B
-> observation/status write resumes against B

At best this loses the immediate observation and leaves A requiring reconciliation. If B contains the same deterministic attempt ID, the combination with Problem 1 can cross-contaminate attempt/observation state between otherwise independent ledgers.

Required design

Choose and document one coherent model. A likely direction is:

each effect store owns a durable generated namespace/UUID persisted with the ledger;
execution attempt/provider idempotency identity includes the durable store namespace (or an equivalent durable call-instance namespace) while remaining stable across process restart of that same store;
pruning has explicit semantics for whether a later same-payload execution receives a fresh call/attempt epoch and remote idempotency key;
an in-flight executor attempt is bound to the store generation/namespace it prepared/admitted against;
store close/switch cannot silently redirect later lifecycle writes for an in-flight attempt to a different attached store;
restart/reconciliation preserves the original store/adapter identity.

Do not solve this with process-local random IDs that change on restart.

Required tests

two independent stores with identical requests produce distinct provider idempotency keys while restart of one store preserves its key;
reopening the same store through path aliases preserves the same durable namespace;
pruning semantics are explicit and tested for subsequent same-payload execution;
deterministic in-flight store-switch fixture cannot write an observation into another store;
two stores containing otherwise identical requests cannot cross-replay/cross-reconcile each other's observations.

Non-goals

no distributed consensus requirement;
no multi-host CAS claim;
no #79 provider/tool/MCP adoption in this issue.

Refs #57 #79 #80

## Discovered by Post-merge adversarial review of PR #78 / #57. This should be resolved before #79 routes real provider/tool/MCP effects through the substrate at scale. ## Problem 1: independent stores derive identical remote idempotency keys Current automatic identity is content-derived: ```text call_id = hash(kind, normalized request, semantics) attempt_id = hash(call_id, fingerprint, sequence, parent, mode) idempotency_key = hash(attempt_id) ``` No durable store/project/runtime namespace participates in the identity. Therefore two independent ledger files executing the same initial request/semantics derive the same `attempt_id` and provider `idempotency_key`. If a remote provider scopes idempotency keys more broadly than one local ledger, an operation in store B can be deduplicated against an unrelated operation from store A. The same problem appears after explicit pruning: recreating the same initial logical call derives the old idempotency key again even though local replay state was intentionally removed. ## Problem 2: store switching is not bound to an in-flight executor attempt `effect_persist_open/1` supports switching the globally attached store, while adapter submit runs outside the persistency mutex after `dispatching` is recorded. A concurrent host store switch can therefore occur between: ```text store A: durable dispatching -> remote adapter executes -> host switches global store to B -> observation/status write resumes against B ``` At best this loses the immediate observation and leaves A requiring reconciliation. If B contains the same deterministic attempt ID, the combination with Problem 1 can cross-contaminate attempt/observation state between otherwise independent ledgers. ## Required design Choose and document one coherent model. A likely direction is: - each effect store owns a durable generated namespace/UUID persisted with the ledger; - execution attempt/provider idempotency identity includes the durable store namespace (or an equivalent durable call-instance namespace) while remaining stable across process restart of that same store; - pruning has explicit semantics for whether a later same-payload execution receives a fresh call/attempt epoch and remote idempotency key; - an in-flight executor attempt is bound to the store generation/namespace it prepared/admitted against; - store close/switch cannot silently redirect later lifecycle writes for an in-flight attempt to a different attached store; - restart/reconciliation preserves the original store/adapter identity. Do not solve this with process-local random IDs that change on restart. ## Required tests - two independent stores with identical requests produce distinct provider idempotency keys while restart of one store preserves its key; - reopening the same store through path aliases preserves the same durable namespace; - pruning semantics are explicit and tested for subsequent same-payload execution; - deterministic in-flight store-switch fixture cannot write an observation into another store; - two stores containing otherwise identical requests cannot cross-replay/cross-reconcile each other's observations. ## Non-goals - no distributed consensus requirement; - no multi-host CAS claim; - no #79 provider/tool/MCP adoption in this issue. Refs #57 #79 #80