Configurable LMDB map-size autogrow with safe retry #9

New issue

Open

opened 2026-09-02 17:27:30 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-02 17:27:30 +00:00

Owner

Outcome

Add opt-in LMDB map-size autogrowth to Tek9 so long-lived Hackmode graph/KB workloads can recover from MDB_MAP_FULL without operator intervention or partial writes.

Owning RAGE lane

Tek9/database.

Fence

This is Tek9 storage-engine behavior. Do not implement Hackmode provider/runtime/Hackpert behavior here and do not add a second persistence layer.

Required configuration

autogrow enabled/disabled explicitly;
initial map size remains configurable;
configurable growth increment and/or growth factor;
configurable hard maximum map size;
bounded retry count;
fixed-map behavior remains unchanged when autogrow is disabled.

Transaction semantics

Research LMDB resize semantics before implementation. On map-full:

abort/leave the failed transaction without partial commit;
serialize map growth correctly across concurrent writers/processes;
increase the environment map size within configured bounds;
retry the logical write safely;
fail with a typed/inspectable error when maximum size or retry budget is reached.

Do not spin forever and do not treat a failed commit as successful.

Observability

Expose enough structured/logged metadata to diagnose growth: old size, new size, trigger, environment/database identity, attempt count, and max-size exhaustion. Never log stored payload contents merely to report growth.

RED-first tests

autogrow disabled preserves current map-full failure;
enabled mode grows and retries one transaction successfully;
repeated growth over multiple boundaries;
configured maximum reached fails closed;
concurrent writers do not race incompatible resizes;
no partial write remains from failed pre-growth attempt;
reopen after growth sees the expanded map and committed data;
pre-existing databases remain compatible.

Acceptance proof

Create a deliberately tiny LMDB environment, fill it until map-full, prove disabled mode fails as before, then enable autogrow and prove the same logical transaction completes exactly once after a serialized resize. Repeat through multiple growths and max-size exhaustion.

Non-goals

automatic disk allocation unrelated to LMDB map sizing;
unbounded growth;
Hackmode-specific graph semantics inside Tek9;
shadow databases or alternate storage engines.

## Outcome Add opt-in LMDB map-size autogrowth to Tek9 so long-lived Hackmode graph/KB workloads can recover from `MDB_MAP_FULL` without operator intervention or partial writes. ## Owning RAGE lane Tek9/database. ## Fence This is Tek9 storage-engine behavior. Do not implement Hackmode provider/runtime/Hackpert behavior here and do not add a second persistence layer. ## Required configuration - autogrow enabled/disabled explicitly; - initial map size remains configurable; - configurable growth increment and/or growth factor; - configurable hard maximum map size; - bounded retry count; - fixed-map behavior remains unchanged when autogrow is disabled. ## Transaction semantics Research LMDB resize semantics before implementation. On map-full: 1. abort/leave the failed transaction without partial commit; 2. serialize map growth correctly across concurrent writers/processes; 3. increase the environment map size within configured bounds; 4. retry the logical write safely; 5. fail with a typed/inspectable error when maximum size or retry budget is reached. Do not spin forever and do not treat a failed commit as successful. ## Observability Expose enough structured/logged metadata to diagnose growth: old size, new size, trigger, environment/database identity, attempt count, and max-size exhaustion. Never log stored payload contents merely to report growth. ## RED-first tests - autogrow disabled preserves current map-full failure; - enabled mode grows and retries one transaction successfully; - repeated growth over multiple boundaries; - configured maximum reached fails closed; - concurrent writers do not race incompatible resizes; - no partial write remains from failed pre-growth attempt; - reopen after growth sees the expanded map and committed data; - pre-existing databases remain compatible. ## Acceptance proof Create a deliberately tiny LMDB environment, fill it until map-full, prove disabled mode fails as before, then enable autogrow and prove the same logical transaction completes exactly once after a serialized resize. Repeat through multiple growths and max-size exhaustion. ## Non-goals - automatic disk allocation unrelated to LMDB map sizing; - unbounded growth; - Hackmode-specific graph semantics inside Tek9; - shadow databases or alternate storage engines.

nsaspy referenced this issue from a commit

2026-09-02 17:27:31 +00:00

Merge pull request #9 from lost-rob0t/merge/issue-7-nix-package

nsaspy referenced this issue

2026-09-04 22:28:47 +00:00

Nix: expose Tek9 as a consumable ASDF package #16