Add opt-in LMDB map auto-growth with safe reopen/replay #10

New issue

Open

opened 2026-09-02 17:27:30 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-02 17:27:30 +00:00

Owner

Goal

Add an explicit Tek9 setting that lets an LMDB environment grow its map capacity automatically instead of permanently failing when the configured map is exhausted.

Current master inspected: a9f5b595f5d965163d2b7c518c72a2efd9be13fe.

Current behavior:

+default-map-size+ is 16 GiB of virtual address space;
new-database accepts :max-size;
open-database passes that value to lmdb:open-env :map-size;
there is no automatic growth/reopen policy.

Proposed public configuration

Minimum API:

(new-database "canonical"
  :path #P"/var/lib/app/tek9/"
  :max-size (* 16 1024 1024 1024)
  :auto-grow t)

The implementation may expose additional bounded tuning only if needed, e.g. a growth factor and optional hard ceiling. :auto-grow nil should preserve the current fail-closed behavior.

LMDB constraint

Do not attempt to resize an active LMDB transaction. The Common Lisp LMDB boundary changes map capacity when the environment is opened/reopened. A map-full write must unwind/abort first, then Tek9 may grow/reopen and retry the complete logical top-level write.

The retry boundary must reacquire LMDB DBI handles after reopen. Current write paths often cache a DBI before entering with-database, so simply catching LMDB-MAP-FULL-ERROR inside call-with-database-transaction and rerunning the same closure is not sufficient if that closure captured a now-stale handle.

Required invariants

opt-in only unless a later compatibility decision changes the default;
never resize from inside an active read/write transaction;
failed map-full transaction leaves zero partial canonical writes;
reopen/grow/retry happens at a boundary that reacquires all DBI handles;
secondary-index and graph mutations retain their existing atomicity;
database/index handle caches are invalidated/rebuilt through the existing environment lifecycle;
growth is serialized under Tek9 environment ownership in-process;
multi-process MDB_MAP_RESIZED/externally grown-map behavior is handled explicitly rather than interpreted as corruption;
exact same logical mutation is retried, not synthesized as a new mutation;
no durability profile is weakened to make growth work.

RED-first proof

Start with a deliberately small LMDB map and a write larger than the remaining capacity.

:auto-grow nil -> deterministic map-full failure and no committed partial mutation.
:auto-grow t -> first attempt reaches map-full, Tek9 aborts it, grows/reopens the environment, reacquires DBIs, retries, and commits exactly once.
Reopen the database and prove the value/graph/index state is present exactly once.
Prove the effective map size increased from its initial value.
A multi-record transaction that runs out of map after earlier records were staged still commits all-or-nothing after retry.

Also cover an externally enlarged environment/reopen case so a second process does not remain stuck on stale map metadata.

Non-goals

no second storage engine;
no background compactor;
no silent downgrade of durability;
no unbounded retry loop;
no partial transaction resume from the failure point.

## Goal Add an explicit Tek9 setting that lets an LMDB environment grow its map capacity automatically instead of permanently failing when the configured map is exhausted. Current `master` inspected: `a9f5b595f5d965163d2b7c518c72a2efd9be13fe`. Current behavior: - `+default-map-size+` is 16 GiB of virtual address space; - `new-database` accepts `:max-size`; - `open-database` passes that value to `lmdb:open-env :map-size`; - there is no automatic growth/reopen policy. ## Proposed public configuration Minimum API: ```lisp (new-database "canonical" :path #P"/var/lib/app/tek9/" :max-size (* 16 1024 1024 1024) :auto-grow t) ``` The implementation may expose additional bounded tuning only if needed, e.g. a growth factor and optional hard ceiling. `:auto-grow nil` should preserve the current fail-closed behavior. ## LMDB constraint Do not attempt to resize an active LMDB transaction. The Common Lisp LMDB boundary changes map capacity when the environment is opened/reopened. A map-full write must unwind/abort first, then Tek9 may grow/reopen and retry the complete logical top-level write. The retry boundary must reacquire LMDB DBI handles after reopen. Current write paths often cache a DBI before entering `with-database`, so simply catching `LMDB-MAP-FULL-ERROR` inside `call-with-database-transaction` and rerunning the same closure is not sufficient if that closure captured a now-stale handle. ## Required invariants - opt-in only unless a later compatibility decision changes the default; - never resize from inside an active read/write transaction; - failed map-full transaction leaves zero partial canonical writes; - reopen/grow/retry happens at a boundary that reacquires all DBI handles; - secondary-index and graph mutations retain their existing atomicity; - database/index handle caches are invalidated/rebuilt through the existing environment lifecycle; - growth is serialized under Tek9 environment ownership in-process; - multi-process `MDB_MAP_RESIZED`/externally grown-map behavior is handled explicitly rather than interpreted as corruption; - exact same logical mutation is retried, not synthesized as a new mutation; - no durability profile is weakened to make growth work. ## RED-first proof Start with a deliberately small LMDB map and a write larger than the remaining capacity. 1. `:auto-grow nil` -> deterministic map-full failure and no committed partial mutation. 2. `:auto-grow t` -> first attempt reaches map-full, Tek9 aborts it, grows/reopens the environment, reacquires DBIs, retries, and commits exactly once. 3. Reopen the database and prove the value/graph/index state is present exactly once. 4. Prove the effective map size increased from its initial value. 5. A multi-record transaction that runs out of map after earlier records were staged still commits all-or-nothing after retry. Also cover an externally enlarged environment/reopen case so a second process does not remain stuck on stale map metadata. ## Non-goals - no second storage engine; - no background compactor; - no silent downgrade of durability; - no unbounded retry loop; - no partial transaction resume from the failure point.

nsaspy referenced this issue

2026-09-04 22:28:14 +00:00

test: require opt-in LMDB map auto-growth #14