[P1 reliability] Import pipeline buffers and duplicates the entire corpus in browser memory #138

New issue

Open

opened 2026-07-28 23:06:51 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-07-28 23:06:51 +00:00

(Migrated from github.com)

Finding

The import path is whole-buffered at every stage.

parseFile() loads each file completely:

const text = await file.text();

JSONL then allocates an additional full line array:

text.split(/\r?\n/).forEach((line, index) => {
  documents.push(JSON.parse(line));
});

collectImportDocuments() retains every parsed file in a map and then copies all documents/origins into combined arrays. importFiles() creates additional records, candidateRecords, candidates, and origins arrays. The batch layer then creates validated copies, ID maps, key arrays, existing-document maps, and write arrays before calling one large bulkDocs().

There is no total-byte, record-count, per-record-size, cancellation, progress, or backpressure boundary.

For large StarIntel JSONL corpora, peak memory can be several times the input size and all parsing/validation occurs on the UI thread. A sufficiently large import can freeze or crash the tab before any useful error or recovery state is persisted.

Required fix

Build a streaming, cancellable import pipeline in a worker with bounded chunks and connect it to the logical batch journal from the atomicity issue.

Stream JSONL

async function* jsonLines(file, { signal } = {}) {
  const reader = file.stream()
    .pipeThrough(new TextDecoderStream())
    .getReader();

  let buffer = "";
  let line = 0;

  try {
    for (;;) {
      signal?.throwIfAborted?.();
      const { value = "", done } = await reader.read();
      buffer += value;

      let boundary;
      while ((boundary = buffer.indexOf("\n")) !== -1) {
        const raw = buffer.slice(0, boundary).replace(/\r$/, "");
        buffer = buffer.slice(boundary + 1);
        line += 1;
        if (raw.trim()) yield { line, value: JSON.parse(raw) };
      }

      if (done) break;
      if (buffer.length > MAX_RECORD_BYTES) {
        throw new RangeError(`JSONL record exceeds ${MAX_RECORD_BYTES} bytes`);
      }
    }

    if (buffer.trim()) yield { line: line + 1, value: JSON.parse(buffer) };
  } finally {
    reader.releaseLock();
  }
}

Validate and commit bounded chunks

const CHUNK_SIZE = 500;
let chunk = [];

for await (const record of parseImportFile(file, { signal })) {
  chunk.push(record);
  if (chunk.length < CHUNK_SIZE) continue;

  await stageImportChunk(batchId, chunk, { signal });
  reportProgress({ parsed, staged, bytesRead });
  chunk = [];
}

if (chunk.length) await stageImportChunk(batchId, chunk, { signal });
await commitLogicalBatch(batchId);

Do not expose staged chunks to normal corpus queries until the batch commit marker is durable.

Worker protocol

worker.postMessage({ type: "start", files, options });
worker.onmessage = ({ data }) => {
  if (data.type === "progress") updateProgress(data);
  if (data.type === "error") showRecordError(data);
  if (data.type === "complete") finalizeImport(data);
};

Use transferable streams where supported; otherwise keep parsing in the worker and send bounded structured-clone chunks.

Enforce explicit limits

At minimum:

const LIMITS = {
  maxFiles: 256,
  maxTotalBytes: 4 * 1024 ** 3,
  maxRecordBytes: 8 * 1024 ** 2,
  maxDocuments: 5_000_000,
  maxErrorsRetainedInMemory: 1_000
};

Limits should be configurable for desktop deployments, with excess errors streamed to an export rather than retained in RAM.

Acceptance criteria

JSONL/NDJSON import memory remains bounded relative to chunk size, not corpus size.
Parsing and schema validation do not block the UI thread.
Imports expose progress, throughput, current file/line, cancellation, and resumable failure state.
A cancelled/failed import exposes no partial logical batch.
Per-file, total-byte, record-size, document-count, and retained-error limits are enforced.
Tests import multi-gigabyte synthetic streams without constructing a multi-gigabyte in-memory string or document array.
Manifest-bundle resolution works without retaining every parsed document twice.

## Finding The import path is whole-buffered at every stage. `parseFile()` loads each file completely: ```js const text = await file.text(); ``` JSONL then allocates an additional full line array: ```js text.split(/\r?\n/).forEach((line, index) => { documents.push(JSON.parse(line)); }); ``` `collectImportDocuments()` retains every parsed file in a map and then copies all documents/origins into combined arrays. `importFiles()` creates additional `records`, `candidateRecords`, `candidates`, and `origins` arrays. The batch layer then creates validated copies, ID maps, key arrays, existing-document maps, and write arrays before calling one large `bulkDocs()`. There is no total-byte, record-count, per-record-size, cancellation, progress, or backpressure boundary. For large StarIntel JSONL corpora, peak memory can be several times the input size and all parsing/validation occurs on the UI thread. A sufficiently large import can freeze or crash the tab before any useful error or recovery state is persisted. ## Required fix Build a streaming, cancellable import pipeline in a worker with bounded chunks and connect it to the logical batch journal from the atomicity issue. ### Stream JSONL ```js async function* jsonLines(file, { signal } = {}) { const reader = file.stream() .pipeThrough(new TextDecoderStream()) .getReader(); let buffer = ""; let line = 0; try { for (;;) { signal?.throwIfAborted?.(); const { value = "", done } = await reader.read(); buffer += value; let boundary; while ((boundary = buffer.indexOf("\n")) !== -1) { const raw = buffer.slice(0, boundary).replace(/\r$/, ""); buffer = buffer.slice(boundary + 1); line += 1; if (raw.trim()) yield { line, value: JSON.parse(raw) }; } if (done) break; if (buffer.length > MAX_RECORD_BYTES) { throw new RangeError(`JSONL record exceeds ${MAX_RECORD_BYTES} bytes`); } } if (buffer.trim()) yield { line: line + 1, value: JSON.parse(buffer) }; } finally { reader.releaseLock(); } } ``` ### Validate and commit bounded chunks ```js const CHUNK_SIZE = 500; let chunk = []; for await (const record of parseImportFile(file, { signal })) { chunk.push(record); if (chunk.length < CHUNK_SIZE) continue; await stageImportChunk(batchId, chunk, { signal }); reportProgress({ parsed, staged, bytesRead }); chunk = []; } if (chunk.length) await stageImportChunk(batchId, chunk, { signal }); await commitLogicalBatch(batchId); ``` Do not expose staged chunks to normal corpus queries until the batch commit marker is durable. ### Worker protocol ```js worker.postMessage({ type: "start", files, options }); worker.onmessage = ({ data }) => { if (data.type === "progress") updateProgress(data); if (data.type === "error") showRecordError(data); if (data.type === "complete") finalizeImport(data); }; ``` Use transferable streams where supported; otherwise keep parsing in the worker and send bounded structured-clone chunks. ### Enforce explicit limits At minimum: ```js const LIMITS = { maxFiles: 256, maxTotalBytes: 4 * 1024 ** 3, maxRecordBytes: 8 * 1024 ** 2, maxDocuments: 5_000_000, maxErrorsRetainedInMemory: 1_000 }; ``` Limits should be configurable for desktop deployments, with excess errors streamed to an export rather than retained in RAM. ## Acceptance criteria - JSONL/NDJSON import memory remains bounded relative to chunk size, not corpus size. - Parsing and schema validation do not block the UI thread. - Imports expose progress, throughput, current file/line, cancellation, and resumable failure state. - A cancelled/failed import exposes no partial logical batch. - Per-file, total-byte, record-size, document-count, and retained-error limits are enforced. - Tests import multi-gigabyte synthetic streams without constructing a multi-gigabyte in-memory string or document array. - Manifest-bundle resolution works without retaining every parsed document twice.