RAGE slice 1: canonical document/chunk model + public embedding library API #3

New issue

Open

opened 2026-09-02 17:25:21 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-02 17:25:21 +00:00

Owner

Parent: #2

Why first

The current implementation is still organized around Org parsing and a concrete vector client. Before Tree-sitter, MCP, gptel QA, or the Emacs makeover, establish a generic library contract so every frontend uses the same types and behaviors.

Canonical ARADR document: rage/org-vector-v2-aradr.org on aradr/org-vector-v2.

Mode: AUTO RAGE. Red-first TDD required. Do not merge automatically.

Goal

Introduce canonical source/document/chunk/search-result types plus a small public library facade that can index and search generic text without requiring an Org file, while adapting the current Org/Chroma implementation underneath it.

Required behavior

Typed canonical representations for source provenance, document, chunk, and search result.
Stable deterministic identity rules are explicit and tested.
Generic in-memory text/document ingestion is possible through the public library API.
Org ingestion maps into the same canonical model.
Public low-level search returns typed results with source metadata.
No LLM/gptel/MCP dependency in the low-level API.
Existing CLI behavior and aliases remain compatible.
Existing incremental sync/deletion/reindex invariants remain green.
Fake embedder/backend tests run offline.

RED first

Add a contract test that attempts to:

instantiate the public library with fake embedder/index dependencies,
index a generic text Document that has no filesystem Org representation,
search it,
receive a typed SearchResult pointing back to the canonical source/document.

The test must fail on current master for the intended missing API/model, not because of downloads or environment problems.

Attack questions before implementation

Are document IDs content-derived, caller-provided, source-derived, or layered?
How are chunk IDs derived so re-chunking intentionally replaces stale chunks?
Which metadata is core/stable versus adapter-specific?
Can whole-document and chunk records coexist without ambiguous IDs?
Does the facade expose backend concepts accidentally?
Can the current CLI be expressed entirely as an adapter over the new public API?

Gates

New contract tests prove RED then GREEN.
Existing Python tests remain green.
Existing ERT tests remain green.
py_compile, pyright, black, flake8, and nix build .# pass at exact head.
Record exact-head evidence back into the Org ARADR/RAGE document.

Parent: #2 ## Why first The current implementation is still organized around Org parsing and a concrete vector client. Before Tree-sitter, MCP, gptel QA, or the Emacs makeover, establish a generic library contract so every frontend uses the same types and behaviors. Canonical ARADR document: `rage/org-vector-v2-aradr.org` on `aradr/org-vector-v2`. Mode: **AUTO RAGE**. Red-first TDD required. Do not merge automatically. ## Goal Introduce canonical source/document/chunk/search-result types plus a small public library facade that can index and search generic text without requiring an Org file, while adapting the current Org/Chroma implementation underneath it. ## Required behavior - Typed canonical representations for source provenance, document, chunk, and search result. - Stable deterministic identity rules are explicit and tested. - Generic in-memory text/document ingestion is possible through the public library API. - Org ingestion maps into the same canonical model. - Public low-level search returns typed results with source metadata. - No LLM/gptel/MCP dependency in the low-level API. - Existing CLI behavior and aliases remain compatible. - Existing incremental sync/deletion/reindex invariants remain green. - Fake embedder/backend tests run offline. ## RED first Add a contract test that attempts to: 1. instantiate the public library with fake embedder/index dependencies, 2. index a generic text `Document` that has no filesystem Org representation, 3. search it, 4. receive a typed `SearchResult` pointing back to the canonical source/document. The test must fail on current `master` for the intended missing API/model, not because of downloads or environment problems. ## Attack questions before implementation - Are document IDs content-derived, caller-provided, source-derived, or layered? - How are chunk IDs derived so re-chunking intentionally replaces stale chunks? - Which metadata is core/stable versus adapter-specific? - Can whole-document and chunk records coexist without ambiguous IDs? - Does the facade expose backend concepts accidentally? - Can the current CLI be expressed entirely as an adapter over the new public API? ## Gates - New contract tests prove RED then GREEN. - Existing Python tests remain green. - Existing ERT tests remain green. - `py_compile`, pyright, black, flake8, and `nix build .#` pass at exact head. - Record exact-head evidence back into the Org ARADR/RAGE document.