[P0-23] Implement the initial Valkey target-lease backend with atomic fencing scripts #31

New issue

Closed

opened 2026-07-22 00:14:16 +00:00 by lost-rob0t · 1 comment

lost-rob0t commented

2026-07-22 00:14:16 +00:00

(Migrated from github.com)

Decision

Use Valkey as the first deployable KV backend behind the lease-store protocol. Keep the protocol backend-neutral so a stronger-consistency backend can be added without changing target or HTTP code.

Required implementation

One atomic acquire script: create lease only when free/expired and increment a monotonic fencing counter.
Atomic renew script requiring matching lease id, owner, and fencing token.
Atomic compare-and-delete release.
TTL stored by the backend; server time/TTL result is authoritative.
Request-id result cache for idempotent retries.
Namespaced keys suitable for standalone and clustered deployments.
Connection pooling, authentication, TLS configuration, deadlines, reconnect/backoff, and health checks.
No use of unsafe GET then SET races.
Document failover/replication assumptions and where fencing is enforced downstream.

Acceptance tests

Backend contract suite passes against a real Valkey service.
100 concurrent acquire attempts yield one owner and one fencing token generation.
Expiry followed by reacquire produces a strictly larger token.
Old owner cannot renew, release, or commit after reacquire.
Script interruption/retry is idempotent by request id.
Credentials and lease metadata are redacted from logs where required.

## Decision Use Valkey as the first deployable KV backend behind the lease-store protocol. Keep the protocol backend-neutral so a stronger-consistency backend can be added without changing target or HTTP code. ## Required implementation - One atomic acquire script: create lease only when free/expired and increment a monotonic fencing counter. - Atomic renew script requiring matching lease id, owner, and fencing token. - Atomic compare-and-delete release. - TTL stored by the backend; server time/TTL result is authoritative. - Request-id result cache for idempotent retries. - Namespaced keys suitable for standalone and clustered deployments. - Connection pooling, authentication, TLS configuration, deadlines, reconnect/backoff, and health checks. - No use of unsafe `GET` then `SET` races. - Document failover/replication assumptions and where fencing is enforced downstream. ## Acceptance tests - Backend contract suite passes against a real Valkey service. - 100 concurrent acquire attempts yield one owner and one fencing token generation. - Expiry followed by reacquire produces a strictly larger token. - Old owner cannot renew, release, or commit after reacquire. - Script interruption/retry is idempotent by request id. - Credentials and lease metadata are redacted from logs where required.

lost-rob0t commented

2026-08-16 14:59:47 +00:00

(Migrated from github.com)

Acceptance complete. PR #95 merged into canonical master as 7834d6614378e01232a1c810249d9ae83cec25c5 from verified head 652b148c233646af4f2e4b46dd6289ae73ef3c26.

Current-head evidence before merge:

nix flake check --show-trace — successful in Smoke Tests.
nix build .#default --no-link --print-build-logs — successful in Smoke Tests.
nix run .#star-unit-tests — successful in Smoke Tests.
nix run .#star-integration-tests — successful; the runner boots real authenticated plain + TLS Valkey and runs required suites that fail on zero tests, partial execution, skips, or failures.
./scripts/stack-test.sh — successful in Container Stack.
python -m unittest discover -s tests -p 'test_*.py' -v and git diff --check — successful in Operational salvage.
Canonical schema lock workflow — successful.

Exact recorded suite counts:

LEASE-STORE-CONTRACT-TESTS: 8 discovered / 8 executed / 8 passed / 0 failed / 0 skipped.
VALKEY-LEASE-INTEGRATION-TESTS: 22 / 22 / 22 / 0 / 0.
COUCHDB-VIEW-INTEGRATION-TESTS: 7 / 7 / 7 / 0 / 0.
HTTP-API-TESTS: 28 / 28 / 28 / 0 / 0.
Operational salvage: 12/12.
Container stack: passed.
Schema lock: verified.

The final acceptance regressions include atomic per-key list validation across release/reacquire, missing-key PTTL=-2 rejection, and a corrupt unsupported record_version that exercises the Lisp deserialization fail-closed boundary. All review threads were resolved after verifying the current implementation and tests.

Acceptance complete. PR #95 merged into canonical `master` as `7834d6614378e01232a1c810249d9ae83cec25c5` from verified head `652b148c233646af4f2e4b46dd6289ae73ef3c26`. Current-head evidence before merge: - `nix flake check --show-trace` — successful in Smoke Tests. - `nix build .#default --no-link --print-build-logs` — successful in Smoke Tests. - `nix run .#star-unit-tests` — successful in Smoke Tests. - `nix run .#star-integration-tests` — successful; the runner boots real authenticated plain + TLS Valkey and runs required suites that fail on zero tests, partial execution, skips, or failures. - `./scripts/stack-test.sh` — successful in Container Stack. - `python -m unittest discover -s tests -p 'test_*.py' -v` and `git diff --check` — successful in Operational salvage. - Canonical schema lock workflow — successful. Exact recorded suite counts: - LEASE-STORE-CONTRACT-TESTS: 8 discovered / 8 executed / 8 passed / 0 failed / 0 skipped. - VALKEY-LEASE-INTEGRATION-TESTS: 22 / 22 / 22 / 0 / 0. - COUCHDB-VIEW-INTEGRATION-TESTS: 7 / 7 / 7 / 0 / 0. - HTTP-API-TESTS: 28 / 28 / 28 / 0 / 0. - Operational salvage: 12/12. - Container stack: passed. - Schema lock: verified. The final acceptance regressions include atomic per-key list validation across release/reacquire, missing-key `PTTL=-2` rejection, and a corrupt unsupported `record_version` that exercises the Lisp deserialization fail-closed boundary. All review threads were resolved after verifying the current implementation and tests.