RAGE slice 3: generic Org/text adapters + whole-document indexing #1

New issue

Open

opened 2026-09-02 17:25:21 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-02 17:25:21 +00:00

Owner

Parent: #2

Mode: AUTO RAGE — no human gate. Research/design may proceed immediately; implementation follows dependency readiness and RED-first TDD.

Canonical ARADR docs live on aradr/org-vector-v2.

Goal

Move Org ingestion behind the generic document/source contracts and add generic text ingestion plus explicit whole-document/chunk/both indexing policy.

Required behavior

Org parsing becomes an adapter into canonical Document/Chunk types.
Plain text adapter indexes arbitrary .txt/text-like files.
Whole-document indexing is configurable as document, chunks, or both.
both must not create duplicate/noisy retrieval ambiguity; records carry explicit granularity/type metadata.
Oversized whole-document content has an explicit policy and may not silently truncate without metadata.
Existing Org heading IDs and deterministic fallback IDs remain compatible.
Reindexing replaces stale representations.
No LLM dependency in parsing/indexing.
Source locations/provenance remain jumpable from search results.

RED first

Add offline fixture tests proving current master lacks a unified adapter contract and whole-document granularity behavior. Tests should cover one Org note, one plain text document, changed content, document/chunks/both, and deterministic IDs.

Dependency

Requires #3 canonical content model and #4 source configuration/discovery. Research/design may run ahead; implementation must consume those contracts rather than bypass them.

Gates

RED -> GREEN, existing Python and ERT tests green, py_compile/pyright/black/flake8/nix build exact-head evidence recorded in Org docs.

Parent: #2 Mode: **AUTO RAGE — no human gate.** Research/design may proceed immediately; implementation follows dependency readiness and RED-first TDD. Canonical ARADR docs live on `aradr/org-vector-v2`. ## Goal Move Org ingestion behind the generic document/source contracts and add generic text ingestion plus explicit whole-document/chunk/both indexing policy. ## Required behavior - Org parsing becomes an adapter into canonical `Document`/`Chunk` types. - Plain text adapter indexes arbitrary `.txt`/text-like files. - Whole-document indexing is configurable as `document`, `chunks`, or `both`. - `both` must not create duplicate/noisy retrieval ambiguity; records carry explicit granularity/type metadata. - Oversized whole-document content has an explicit policy and may not silently truncate without metadata. - Existing Org heading IDs and deterministic fallback IDs remain compatible. - Reindexing replaces stale representations. - No LLM dependency in parsing/indexing. - Source locations/provenance remain jumpable from search results. ## RED first Add offline fixture tests proving current master lacks a unified adapter contract and whole-document granularity behavior. Tests should cover one Org note, one plain text document, changed content, `document/chunks/both`, and deterministic IDs. ## Dependency Requires #3 canonical content model and #4 source configuration/discovery. Research/design may run ahead; implementation must consume those contracts rather than bypass them. ## Gates RED -> GREEN, existing Python and ERT tests green, py_compile/pyright/black/flake8/nix build exact-head evidence recorded in Org docs.