Large-corpus query engine: replace raw analytics scans with indexed Tek9/capture-offset queries #96

New issue

Open

opened 2026-09-20 01:19:24 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-20 01:19:24 +00:00

Owner

Parent: corpus workbench epic.

Problem

The current generic analytics path iterates events.jsonl for each query. It is streaming rather than whole-file materialization, but an O(50GB) scan per interactive request is not acceptable.

Goal

Make interactive analytics/query cost proportional to result window/index range, not total corpus size.

Design requirements

Tek9 owns indexed expert/query projections.
Keep immutable events.jsonl as evidence, not the interactive analytics database.
Add/extend durable indexes for time, session, task, provider, model, classification, outcome and usage/cost dimensions.
Preserve stable event ID plus raw-capture locator (line/offset/segment) so selected rows can lazily fetch exact evidence.
All list APIs are cursor/page based with bounded page size.
Aggregation jobs process finite chunks and can persist resumable derived rollups.
Version every index migration and backfill exactly once.
Backfill streams from source with checkpoint; never read the whole corpus.
Define memory/RSS regression tests against a large synthetic sparse corpus.

Acceptance

Interactive queries over a synthetic large corpus do not invoke the raw full-file scan and stay below an explicit memory ceiling. Pagination/cursors remain stable across restart.

Parent: corpus workbench epic. ## Problem The current generic analytics path iterates events.jsonl for each query. It is streaming rather than whole-file materialization, but an O(50GB) scan per interactive request is not acceptable. ## Goal Make interactive analytics/query cost proportional to result window/index range, not total corpus size. ## Design requirements - Tek9 owns indexed expert/query projections. - Keep immutable events.jsonl as evidence, not the interactive analytics database. - Add/extend durable indexes for time, session, task, provider, model, classification, outcome and usage/cost dimensions. - Preserve stable event ID plus raw-capture locator (line/offset/segment) so selected rows can lazily fetch exact evidence. - All list APIs are cursor/page based with bounded page size. - Aggregation jobs process finite chunks and can persist resumable derived rollups. - Version every index migration and backfill exactly once. - Backfill streams from source with checkpoint; never read the whole corpus. - Define memory/RSS regression tests against a large synthetic sparse corpus. ## Acceptance Interactive queries over a synthetic large corpus do not invoke the raw full-file scan and stay below an explicit memory ceiling. Pagination/cursors remain stable across restart.

nsaspy referenced this issue

2026-09-20 01:19:24 +00:00

Session economics: stable session/task attribution + per-session token/cost analytics #97