Corpus workbench epic: 50GB-safe queries, session economics, Emacs/OpenCode and KB learning #98

New issue

Open

opened 2026-09-20 01:19:24 +00:00 by nsaspy · 1 comment

nsaspy commented

2026-09-20 01:19:24 +00:00

Owner

Goal

Turn llm-log into the operator-facing corpus workbench for its existing immutable capture + Tek9/SWI-Prolog expert plane.

Hard invariant: never materialize the corpus

The production corpus is expected to reach ~50 GB and beyond. No CLI, API, Emacs mode, OpenCode command, report job, Prolog query or LLM learner may read the entire corpus into memory or materialize the entire Tek9 KB into Prolog.

Allowed patterns:

streaming one-pass backfill/import with checkpoints;
bounded indexed Tek9 lookups/range scans;
cursor/page-based query/export;
explicit finite batch windows;
joins from selected expert IDs back to exact raw-capture offsets/IDs.

Disallowed:

read_text()/json.load() of the whole corpus;
unbounded list accumulation;
full Tek9 document walks in interactive/query paths;
feeding the full corpus/KB to an LLM.

Existing substrate to preserve

Current llm-log already has:

append-only events.jsonl source evidence;
checkpointed streaming expert backfill;
bounded maintenance queries;
Tek9 durable expert records and versioned index migrations;
SWI-Prolog classification/outcome rules;
task/cost accounting substrate;
paginated outcome dataset export.

Build on those paths instead of creating a second database/rule engine.

Child work

session-level economics + analytics everywhere;
indexed 50GB-scale corpus query engine;
rich CLI/TUI-ish presentation;
Emacs corpus major mode + org-roam report export;
systemd user query/backfill/report jobs;
OpenCode-native query/report commands and skill surfaces;
bounded LLM-assisted KB mining using the configured 27B model at llm.starintel.actor;
standalone installer for non-Nix hosts, while the operator desktop remains Home Manager-managed.

Acceptance

A 50GB synthetic corpus can be queried, browsed, backfilled, reported and mined with a documented bounded-memory ceiling. No acceptance test may depend on enough RAM to hold the corpus.

## Goal Turn llm-log into the operator-facing corpus workbench for its existing immutable capture + Tek9/SWI-Prolog expert plane. ## Hard invariant: never materialize the corpus The production corpus is expected to reach ~50 GB and beyond. No CLI, API, Emacs mode, OpenCode command, report job, Prolog query or LLM learner may read the entire corpus into memory or materialize the entire Tek9 KB into Prolog. Allowed patterns: - streaming one-pass backfill/import with checkpoints; - bounded indexed Tek9 lookups/range scans; - cursor/page-based query/export; - explicit finite batch windows; - joins from selected expert IDs back to exact raw-capture offsets/IDs. Disallowed: - read_text()/json.load() of the whole corpus; - unbounded list accumulation; - full Tek9 document walks in interactive/query paths; - feeding the full corpus/KB to an LLM. ## Existing substrate to preserve Current llm-log already has: - append-only events.jsonl source evidence; - checkpointed streaming expert backfill; - bounded maintenance queries; - Tek9 durable expert records and versioned index migrations; - SWI-Prolog classification/outcome rules; - task/cost accounting substrate; - paginated outcome dataset export. Build on those paths instead of creating a second database/rule engine. ## Child work - session-level economics + analytics everywhere; - indexed 50GB-scale corpus query engine; - rich CLI/TUI-ish presentation; - Emacs corpus major mode + org-roam report export; - systemd user query/backfill/report jobs; - OpenCode-native query/report commands and skill surfaces; - bounded LLM-assisted KB mining using the configured 27B model at llm.starintel.actor; - standalone installer for non-Nix hosts, while the operator desktop remains Home Manager-managed. ## Acceptance A 50GB synthetic corpus can be queried, browsed, backfilled, reported and mined with a documented bounded-memory ceiling. No acceptance test may depend on enough RAM to hold the corpus.

nsaspy referenced this issue

2026-09-20 01:20:03 +00:00

Expert: task graph + per-task token/cost accounting #13

nsaspy commented

2026-09-20 02:07:49 +00:00

Author

Owner

Child slices created from this epic:

#98 session economics
#99 indexed 50GB-scale query engine
#100 Emacs corpus mode + org-roam export
#101 systemd query/backfill/report jobs
#102 bounded 27B KB miner
#103 Rich CLI + OpenCode + standalone installer
#104 current Python admission/rate/queue/429 path
#105 current Python OpenCode profile + opt-in queue-status SSE

The operator dotfiles consumer is being wired separately so the already-landed checkpointed backfill/query/export path is available immediately while the new slices land.

Child slices created from this epic: - #98 session economics - #99 indexed 50GB-scale query engine - #100 Emacs corpus mode + org-roam export - #101 systemd query/backfill/report jobs - #102 bounded 27B KB miner - #103 Rich CLI + OpenCode + standalone installer - #104 current Python admission/rate/queue/429 path - #105 current Python OpenCode profile + opt-in queue-status SSE The operator dotfiles consumer is being wired separately so the already-landed checkpointed backfill/query/export path is available immediately while the new slices land.