EPIC: org-vector v2 — generic embedding library + Emacs/MCP retrieval platform #4

New issue

Open

opened 2026-09-02 17:25:21 +00:00 by nsaspy · 1 comment

nsaspy commented

2026-09-02 17:25:21 +00:00

Owner

Goal

Turn org-vector from an Org-roam-oriented vector utility into a reusable embedding/retrieval library with a polished Emacs interface and service APIs.

This is an ARADR seed running in AUTO RAGE mode. Research/design must be written to Org files. Implementation should proceed as small TDD/red-first slices on branches after the design for that slice is materially defined. Do not silently merge; merge remains a separate operator action unless explicitly authorized.

Product direction

org-vector should become a generic local embedding platform where Org is a first-class adapter, not the architecture boundary.

Required capabilities

Reusable Python library API for indexing, embedding, deleting, syncing, searching, fetching metadata, and inspecting index state.
CLI remains supported and backwards compatible (search/query, embed/update, serve).
Configurable recursive indexing of multiple paths, with include/exclude patterns, parser/chunker selection, collection/namespace, model, watch mode, and metadata policy.
Ability to index entire notes/documents as whole-document vectors as well as chunk-level vectors.
Generic ingestion pipeline able to accept arbitrary text/document adapters rather than only .org files.
Org adapter retains heading-aware parsing and stable IDs.
Tree-sitter-aware code indexing for supported programming languages, chunking primarily on syntax/semantic units (functions, methods, classes, modules, declarations) with deterministic fallback chunking.
Emacs transient for ad-hoc embedding/search:
- current region
- current buffer
- current file
- arbitrary configured path
- reindex configured sources
- low-level vector search
- AI question answering
Polished Emacs results UI with source metadata, score, collection, location/range, open/jump actions, and re-run/refine controls.
gptel integration with separate low-level retrieval and grounded QA tools.
MCP server exposing indexing/search/QA/status capabilities to external agents.
Two retrieval layers:
1. raw search: direct vector/hybrid retrieval returning ranked chunks/documents + metadata, no LLM required.
2. ask: retrieve context and answer questions with an LLM, returning grounded citations/source references.
No implicit LLM dependency in the low-level library/search path. AI QA is an explicit higher layer.
Designed to scale to large corpora: incremental indexing, batching, bounded memory, concurrency, durable metadata, deletion/tombstone handling, stable IDs, observability, and configurable backend/index strategy.

Research questions

Define the package boundaries between ingestion, parsing/chunking, embedding, storage/index backend, retrieval/reranking, QA, service, CLI, and Emacs.
Determine a stable content/document/chunk model that supports Org, source code, plain text, Markdown, and future adapters.
Determine how whole-document embeddings coexist with chunk embeddings without duplicate/noisy retrieval.
Define Tree-sitter chunking strategy, language discovery, parser availability, malformed-code fallback, and metadata schema.
Determine backend strategy for large corpora. Preserve current Chroma compatibility initially, but evaluate whether the library boundary should allow Qdrant/pgvector/LanceDB/SQLite-vector/FAISS or another backend without rewriting ingestion.
Define configuration schema for multiple named sources/paths and per-source overrides.
Define local daemon/service protocol and MCP interface.
Define gptel tool contracts for search vs ask.
Define scale tests/benchmarks: corpus size, indexing throughput, update latency, query latency, memory ceiling, deletion correctness, and restart behavior.
Define migration/compatibility guarantees for the current CLI, existing Chroma collection, and Emacs commands.

Initial architecture target

org_vector.core / public library surface
org_vector.documents canonical document/chunk types
org_vector.sources source discovery + configured paths
org_vector.parsers adapters (org/text/markdown/tree-sitter code/...)
org_vector.chunkers
org_vector.embedders
org_vector.indexes backend interface + current Chroma implementation
org_vector.retrieval raw search + optional reranking
org_vector.qa explicit LLM/RAG layer
org_vector.service long-running index/watch/search service
org_vector.mcp MCP server
org_vector.cli
lisp/org-vector.el polished Emacs client/transient/gptel integration

Names are research targets, not frozen API.

Non-negotiable compatibility/invariants

Preserve existing working behavior unless the design explicitly migrates it:

query aliases search.
update aliases incremental embed.
unchanged sources are skipped.
removed sources are removed from the index.
reindexing replaces stale chunks.
stable deterministic IDs.
retrieval diversity should avoid flooding results from one file.
existing Emacs commands continue working or have documented compatibility wrappers.
low-level search must never require an LLM.

TDD / RAGE gates

For every implementation slice:

Materialize the design/acceptance criteria in the Org research document.
Write a failing deterministic test first.
Prove RED for the intended missing behavior.
Implement the smallest coherent slice.
Prove GREEN locally with offline tests where possible.
Run relevant type/lint/ERT/build gates.
Record exact-head evidence in the Org RAGE document.
Recurse into the next highest-value slice.

Suggested slice order

Canonical library API + generic document/chunk model.
Multi-source configuration and generic path discovery.
Generic text/Org adapters + whole-document indexing.
Tree-sitter code adapter/chunker.
Storage backend abstraction + scale-safe metadata/index lifecycle.
Emacs transient/UI makeover + ad-hoc embedding.
MCP server.
gptel raw-search tool + AI ask tool.
Scale/benchmark harness and hardening.
Migration docs + package polish.

ARADR artifact

Canonical seed document lives at:

rage/org-vector-v2-aradr.org

Canonical research/design branch:

aradr/org-vector-v2

## Goal Turn `org-vector` from an Org-roam-oriented vector utility into a reusable embedding/retrieval library with a polished Emacs interface and service APIs. This is an **ARADR seed** running in **AUTO RAGE mode**. Research/design must be written to Org files. Implementation should proceed as small TDD/red-first slices on branches after the design for that slice is materially defined. Do not silently merge; merge remains a separate operator action unless explicitly authorized. ## Product direction `org-vector` should become a generic local embedding platform where Org is a first-class adapter, not the architecture boundary. ### Required capabilities - Reusable Python library API for indexing, embedding, deleting, syncing, searching, fetching metadata, and inspecting index state. - CLI remains supported and backwards compatible (`search/query`, `embed/update`, `serve`). - Configurable recursive indexing of **multiple paths**, with include/exclude patterns, parser/chunker selection, collection/namespace, model, watch mode, and metadata policy. - Ability to index **entire notes/documents** as whole-document vectors as well as chunk-level vectors. - Generic ingestion pipeline able to accept arbitrary text/document adapters rather than only `.org` files. - Org adapter retains heading-aware parsing and stable IDs. - Tree-sitter-aware code indexing for supported programming languages, chunking primarily on syntax/semantic units (functions, methods, classes, modules, declarations) with deterministic fallback chunking. - Emacs transient for ad-hoc embedding/search: - current region - current buffer - current file - arbitrary configured path - reindex configured sources - low-level vector search - AI question answering - Polished Emacs results UI with source metadata, score, collection, location/range, open/jump actions, and re-run/refine controls. - gptel integration with separate low-level retrieval and grounded QA tools. - MCP server exposing indexing/search/QA/status capabilities to external agents. - Two retrieval layers: 1. **raw search**: direct vector/hybrid retrieval returning ranked chunks/documents + metadata, no LLM required. 2. **ask**: retrieve context and answer questions with an LLM, returning grounded citations/source references. - No implicit LLM dependency in the low-level library/search path. AI QA is an explicit higher layer. - Designed to scale to large corpora: incremental indexing, batching, bounded memory, concurrency, durable metadata, deletion/tombstone handling, stable IDs, observability, and configurable backend/index strategy. ## Research questions 1. Define the package boundaries between ingestion, parsing/chunking, embedding, storage/index backend, retrieval/reranking, QA, service, CLI, and Emacs. 2. Determine a stable content/document/chunk model that supports Org, source code, plain text, Markdown, and future adapters. 3. Determine how whole-document embeddings coexist with chunk embeddings without duplicate/noisy retrieval. 4. Define Tree-sitter chunking strategy, language discovery, parser availability, malformed-code fallback, and metadata schema. 5. Determine backend strategy for large corpora. Preserve current Chroma compatibility initially, but evaluate whether the library boundary should allow Qdrant/pgvector/LanceDB/SQLite-vector/FAISS or another backend without rewriting ingestion. 6. Define configuration schema for multiple named sources/paths and per-source overrides. 7. Define local daemon/service protocol and MCP interface. 8. Define gptel tool contracts for `search` vs `ask`. 9. Define scale tests/benchmarks: corpus size, indexing throughput, update latency, query latency, memory ceiling, deletion correctness, and restart behavior. 10. Define migration/compatibility guarantees for the current CLI, existing Chroma collection, and Emacs commands. ## Initial architecture target - `org_vector.core` / public library surface - `org_vector.documents` canonical document/chunk types - `org_vector.sources` source discovery + configured paths - `org_vector.parsers` adapters (org/text/markdown/tree-sitter code/...) - `org_vector.chunkers` - `org_vector.embedders` - `org_vector.indexes` backend interface + current Chroma implementation - `org_vector.retrieval` raw search + optional reranking - `org_vector.qa` explicit LLM/RAG layer - `org_vector.service` long-running index/watch/search service - `org_vector.mcp` MCP server - `org_vector.cli` - `lisp/org-vector.el` polished Emacs client/transient/gptel integration Names are research targets, not frozen API. ## Non-negotiable compatibility/invariants Preserve existing working behavior unless the design explicitly migrates it: - `query` aliases `search`. - `update` aliases incremental `embed`. - unchanged sources are skipped. - removed sources are removed from the index. - reindexing replaces stale chunks. - stable deterministic IDs. - retrieval diversity should avoid flooding results from one file. - existing Emacs commands continue working or have documented compatibility wrappers. - low-level search must never require an LLM. ## TDD / RAGE gates For every implementation slice: 1. Materialize the design/acceptance criteria in the Org research document. 2. Write a failing deterministic test first. 3. Prove RED for the intended missing behavior. 4. Implement the smallest coherent slice. 5. Prove GREEN locally with offline tests where possible. 6. Run relevant type/lint/ERT/build gates. 7. Record exact-head evidence in the Org RAGE document. 8. Recurse into the next highest-value slice. ## Suggested slice order 1. Canonical library API + generic document/chunk model. 2. Multi-source configuration and generic path discovery. 3. Generic text/Org adapters + whole-document indexing. 4. Tree-sitter code adapter/chunker. 5. Storage backend abstraction + scale-safe metadata/index lifecycle. 6. Emacs transient/UI makeover + ad-hoc embedding. 7. MCP server. 8. gptel raw-search tool + AI `ask` tool. 9. Scale/benchmark harness and hardening. 10. Migration docs + package polish. ## ARADR artifact Canonical seed document lives at: `rage/org-vector-v2-aradr.org` Canonical research/design branch: `aradr/org-vector-v2`

nsaspy commented

2026-09-02 17:25:22 +00:00

Author

Owner

AUTO RAGE policy override: no human gate between ARADR research, design, RED test construction, implementation, verification, documentation, and progression to the next coherent slice. This is now recorded in rage/auto-rage-policy.org on aradr/org-vector-v2. Automatic merge is still not inferred; merge-on-green remains a separate operator authorization.

AUTO RAGE policy override: **no human gate** between ARADR research, design, RED test construction, implementation, verification, documentation, and progression to the next coherent slice. This is now recorded in `rage/auto-rage-policy.org` on `aradr/org-vector-v2`. Automatic merge is still not inferred; merge-on-green remains a separate operator authorization.