[P0-25] Add target-lease expiry, crash recovery, and race-condition test matrix #33

New issue

Open

opened 2026-07-22 00:14:46 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-07-22 00:14:46 +00:00

(Migrated from github.com)

Objective

Prove lease behavior under the failures that make distributed locks dangerous.

Required scenarios

two clients acquire simultaneously
renew races with expiry
release races with reacquire
client timeout after backend success, followed by retry
service process crashes after acquire
actor crashes while holding a lease
KV connection loss during renew
KV restart/failover
delayed stale result arrives after a newer fencing token exists
privileged revocation while work is running
clock skew between clients and server
repeated idempotency keys with identical and conflicting payloads

Recovery requirements

Runtime startup reconciles persisted target/job state with active leases.
Orphaned leases expire without manual cleanup.
Jobs detect lease loss and stop side effects.
Audit history explains ownership changes.
Metrics expose active leases, conflicts, renew failures, expirations, revocations, and stale-fence rejections.

Acceptance criteria

Deterministic unit tests cover the state machine.
Real Valkey integration tests cover atomicity and TTL.
Multi-process tests demonstrate one accepted writer per target generation.
Chaos/restart tests run in CI or a required scheduled workflow and fail on stale-write acceptance.

## Objective Prove lease behavior under the failures that make distributed locks dangerous. ## Required scenarios - two clients acquire simultaneously - renew races with expiry - release races with reacquire - client timeout after backend success, followed by retry - service process crashes after acquire - actor crashes while holding a lease - KV connection loss during renew - KV restart/failover - delayed stale result arrives after a newer fencing token exists - privileged revocation while work is running - clock skew between clients and server - repeated idempotency keys with identical and conflicting payloads ## Recovery requirements - Runtime startup reconciles persisted target/job state with active leases. - Orphaned leases expire without manual cleanup. - Jobs detect lease loss and stop side effects. - Audit history explains ownership changes. - Metrics expose active leases, conflicts, renew failures, expirations, revocations, and stale-fence rejections. ## Acceptance criteria - Deterministic unit tests cover the state machine. - Real Valkey integration tests cover atomicity and TTL. - Multi-process tests demonstrate one accepted writer per target generation. - Chaos/restart tests run in CI or a required scheduled workflow and fail on stale-write acceptance.