RAGE slice 2: multi-source configuration + generic path discovery #2

New issue

Open

opened 2026-09-02 17:25:21 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-02 17:25:21 +00:00

Owner

Parent: #2

Mode: AUTO RAGE — no human gate between research, design, RED, and implementation. Merge remains separate unless explicitly authorized.

Canonical ARADR document: rage/org-vector-v2-aradr.org on aradr/org-vector-v2.

Goal

Replace the single-directory Org assumption with a reusable named-source configuration and path-discovery layer that can feed the generic library API.

This slice must not depend on Tree-sitter or any LLM. It should establish the configuration/source substrate needed by later Org/text/code adapters.

Required behavior

Support multiple named sources in one config.
Each source may contain one or more root paths.
Per-source recursive traversal.
Include/exclude glob rules.
Per-source collection/namespace.
Per-source parser/chunker/model/granularity/watch metadata preserved even if some fields are consumed by later slices.
Path normalization and ~ expansion.
Deterministic source identity independent of discovery order.
Discovery returns typed source/file candidates rather than CLI-shaped strings.
Directories are never emitted as files just because their names match a suffix/glob.
Symlink policy is explicit and deterministic.
Missing/unreadable roots fail per-source with structured diagnostics rather than crashing discovery for all other sources.
Existing [service] dir=... configuration remains accepted through a compatibility translation path.
No filesystem scanning API may require Chroma, sentence-transformers, gptel, MCP, or an LLM.

Configuration target

Illustrative, not frozen:

[index]
path = "~/.cache/org-vector"
backend = "chroma"
model = "sentence-transformers/all-MiniLM-L6-v2"

[[source]]
name = "notes"
paths = ["~/Documents/Notes"]
recursive = true
include = ["**/*.org", "**/*.md", "**/*.txt"]
exclude = ["**/.git/**", "**/archive/**"]
collection = "notes"
watch = true
granularity = "both"

[[source]]
name = "code"
paths = ["~/Documents/Projects"]
recursive = true
include = ["**/*.py", "**/*.el", "**/*.lisp", "**/*.pl", "**/*.js", "**/*.ts", "**/*.rs", "**/*.nim"]
exclude = ["**/.git/**", "**/node_modules/**", "**/target/**", "**/.direnv/**"]
parser = "tree-sitter"
collection = "code"
watch = false
granularity = "chunks"

RED first

Add deterministic tempdir-backed tests that fail on current master because the multi-source API/config does not exist.

Minimum RED scenario:

Create two roots (notes, code) with nested files plus excluded paths.
Load one config containing both named sources.
Discover candidates.
Assert only included regular files are returned.
Assert each candidate carries stable source name/id, normalized path, collection, and parser/granularity metadata.
Assert discovery order is deterministic.
Assert a missing third source produces a structured diagnostic without suppressing candidates from the valid sources.

Also add compatibility RED/contract coverage for translating the legacy [service] dir=... config into one named source.

Attack questions

Are globs evaluated relative to each configured root or absolute paths? Prefer relative-to-root semantics unless evidence argues otherwise.
What is the source identity when a named source has multiple roots?
Do symlinks default to no-follow, follow-files-only, or configurable follow? Avoid traversal loops.
How do duplicate physical files reached through overlapping roots deduplicate?
Does config validation reject duplicate source names?
Which fields belong to global defaults vs per-source override?
How are unreadable paths represented in status APIs later?

Gates

RED is demonstrated for the intended missing source/config API.
GREEN uses offline deterministic tests only.
Existing incremental indexing tests remain green.
Existing CLI aliases remain green.
Existing ERT tests remain green.
py_compile, pyright, black, flake8, and nix build .# pass at exact head.
Exact-head evidence and decisions are written back into the ARADR Org document.

Dependency

This slice is downstream of the canonical document/source model in #3. AUTO RAGE may research/design this slice in parallel, but implementation should use the canonical types from #3 rather than inventing a second source model.

Parent: #2 Mode: **AUTO RAGE — no human gate between research, design, RED, and implementation.** Merge remains separate unless explicitly authorized. Canonical ARADR document: `rage/org-vector-v2-aradr.org` on `aradr/org-vector-v2`. ## Goal Replace the single-directory Org assumption with a reusable named-source configuration and path-discovery layer that can feed the generic library API. This slice must not depend on Tree-sitter or any LLM. It should establish the configuration/source substrate needed by later Org/text/code adapters. ## Required behavior - Support multiple named sources in one config. - Each source may contain one or more root paths. - Per-source recursive traversal. - Include/exclude glob rules. - Per-source collection/namespace. - Per-source parser/chunker/model/granularity/watch metadata preserved even if some fields are consumed by later slices. - Path normalization and `~` expansion. - Deterministic source identity independent of discovery order. - Discovery returns typed source/file candidates rather than CLI-shaped strings. - Directories are never emitted as files just because their names match a suffix/glob. - Symlink policy is explicit and deterministic. - Missing/unreadable roots fail per-source with structured diagnostics rather than crashing discovery for all other sources. - Existing `[service] dir=...` configuration remains accepted through a compatibility translation path. - No filesystem scanning API may require Chroma, sentence-transformers, gptel, MCP, or an LLM. ## Configuration target Illustrative, not frozen: ```toml [index] path = "~/.cache/org-vector" backend = "chroma" model = "sentence-transformers/all-MiniLM-L6-v2" [[source]] name = "notes" paths = ["~/Documents/Notes"] recursive = true include = ["**/*.org", "**/*.md", "**/*.txt"] exclude = ["**/.git/**", "**/archive/**"] collection = "notes" watch = true granularity = "both" [[source]] name = "code" paths = ["~/Documents/Projects"] recursive = true include = ["**/*.py", "**/*.el", "**/*.lisp", "**/*.pl", "**/*.js", "**/*.ts", "**/*.rs", "**/*.nim"] exclude = ["**/.git/**", "**/node_modules/**", "**/target/**", "**/.direnv/**"] parser = "tree-sitter" collection = "code" watch = false granularity = "chunks" ``` ## RED first Add deterministic tempdir-backed tests that fail on current master because the multi-source API/config does not exist. Minimum RED scenario: 1. Create two roots (`notes`, `code`) with nested files plus excluded paths. 2. Load one config containing both named sources. 3. Discover candidates. 4. Assert only included regular files are returned. 5. Assert each candidate carries stable source name/id, normalized path, collection, and parser/granularity metadata. 6. Assert discovery order is deterministic. 7. Assert a missing third source produces a structured diagnostic without suppressing candidates from the valid sources. Also add compatibility RED/contract coverage for translating the legacy `[service] dir=...` config into one named source. ## Attack questions - Are globs evaluated relative to each configured root or absolute paths? Prefer relative-to-root semantics unless evidence argues otherwise. - What is the source identity when a named source has multiple roots? - Do symlinks default to no-follow, follow-files-only, or configurable follow? Avoid traversal loops. - How do duplicate physical files reached through overlapping roots deduplicate? - Does config validation reject duplicate source names? - Which fields belong to global defaults vs per-source override? - How are unreadable paths represented in status APIs later? ## Gates - RED is demonstrated for the intended missing source/config API. - GREEN uses offline deterministic tests only. - Existing incremental indexing tests remain green. - Existing CLI aliases remain green. - Existing ERT tests remain green. - `py_compile`, pyright, black, flake8, and `nix build .#` pass at exact head. - Exact-head evidence and decisions are written back into the ARADR Org document. ## Dependency This slice is downstream of the canonical document/source model in #3. AUTO RAGE may research/design this slice in parallel, but implementation should use the canonical types from #3 rather than inventing a second source model.

nsaspy referenced this issue

2026-09-02 17:25:21 +00:00

RAGE slice 1: canonical document/chunk model + public embedding library API #3