Bounded secondary-index range must enumerate duplicate postings #8

New issue

Closed

opened 2026-09-02 17:27:30 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-02 17:27:30 +00:00

Owner

Consumer blocker

lost-rob0t/llm-log#15 needs a bounded outcome-dataset query over a non-unique Tek9 secondary index. Live integration evidence showed that two durable documents sharing the exact indexed value are both visible through index-fetch, but an exact-key select-index-range query does not reliably enumerate both postings, so limit + 1 cannot be used for truthful truncation.

Current source

At current master / pinned consumer revision a9f5b595f5d965163d2b7c518c72a2efd9be13fe:

non-unique indexes are LMDB DUPSORT and correctly store multiple document IDs per key;
index-fetch / index-document-ids explicitly enumerate duplicate values, but have no result bound;
select-index-range is the bounded primitive, but advances with generic cursor-next, which loses the exact duplicate-aware guarantee required by a non-unique index range.

This is a generic Tek9 query primitive defect, not llm-log-specific schema behavior.

Required contract

RED first in tests/test-indexes.lisp:

register a non-unique string index;
persist at least three documents where two share key a and one has key b;
prove exact select-index-range db index "a" :end "a" returns both a documents;
prove :limit 1 returns exactly one and :limit 2 returns exactly two duplicate postings;
prove an a..b range retains all postings in deterministic key/document order;
preserve existing unique-index behavior.

Implementation boundary

Fix the existing public select-index-range primitive rather than adding an llm-log-specific escape hatch. Use LMDB duplicate-aware cursor movement so one seek can traverse (key, duplicate-id) postings while respecting the global result limit and inclusive end key.

No full scan and no unbounded materialization.

Acceptance

Tek9 test suite GREEN on exact head, then llm-log must pin that exact tested Tek9 revision and its #15 outcome-dataset integration fixture must independently go GREEN after expert-host restart.

## Consumer blocker `lost-rob0t/llm-log#15` needs a bounded outcome-dataset query over a non-unique Tek9 secondary index. Live integration evidence showed that two durable documents sharing the exact indexed value are both visible through `index-fetch`, but an exact-key `select-index-range` query does not reliably enumerate both postings, so `limit + 1` cannot be used for truthful truncation. ## Current source At current `master` / pinned consumer revision `a9f5b595f5d965163d2b7c518c72a2efd9be13fe`: - non-unique indexes are LMDB DUPSORT and correctly store multiple document IDs per key; - `index-fetch` / `index-document-ids` explicitly enumerate duplicate values, but have no result bound; - `select-index-range` is the bounded primitive, but advances with generic `cursor-next`, which loses the exact duplicate-aware guarantee required by a non-unique index range. This is a generic Tek9 query primitive defect, not llm-log-specific schema behavior. ## Required contract RED first in `tests/test-indexes.lisp`: 1. register a non-unique string index; 2. persist at least three documents where two share key `a` and one has key `b`; 3. prove exact `select-index-range db index "a" :end "a"` returns both `a` documents; 4. prove `:limit 1` returns exactly one and `:limit 2` returns exactly two duplicate postings; 5. prove an `a..b` range retains all postings in deterministic key/document order; 6. preserve existing unique-index behavior. ## Implementation boundary Fix the existing public `select-index-range` primitive rather than adding an llm-log-specific escape hatch. Use LMDB duplicate-aware cursor movement so one seek can traverse `(key, duplicate-id)` postings while respecting the global result limit and inclusive end key. No full scan and no unbounded materialization. ## Acceptance Tek9 test suite GREEN on exact head, then llm-log must pin that exact tested Tek9 revision and its #15 outcome-dataset integration fixture must independently go GREEN after expert-host restart.

nsaspy closed this issue

2026-09-02 17:27:30 +00:00

nsaspy referenced this issue

2026-09-04 22:28:14 +00:00

Nix: expose Tek9 as a consumable ASDF package #15