PERF: stop reprocessing the entire accumulated response on every streaming chunk #32

New issue

Open

opened 2026-08-24 00:54:04 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-24 00:54:04 +00:00

(Migrated from github.com)

Finding

The current streaming log path repeatedly processes the entire accumulated message for each chunk:

helpers/log.py::LogItem.stream() does roughly:

self.update(content=self.content + content)

Log._update_item() then secret-masks and truncation-checks that full cumulative string before storing it. For a response of total length N delivered in many chunks this trends toward O(N²) character work on the backend, plus repeated allocation/copying.

The WebUI receives repeated updated log records containing the growing full content. The message renderer's markdown path performs conversions, marked.parse() and HTML sanitization over the content when rendering/updating. This can create the same cumulative-reparse pattern in the browser for long streamed Markdown/code output.

The response log limit permits up to 250,000 characters, so this is not only theoretical.

Direction

Make streaming incremental while preserving final sanitization/security semantics.

Backend candidates:

append-buffer/chunk representation during active streaming
incremental secret masker with overlap sufficient for secrets spanning chunk boundaries, or safe bounded re-scan window
avoid allocating old_content + chunk and re-masking the entire prefix per chunk
finalize/truncate once or at bounded checkpoints

Frontend candidates:

render streaming text incrementally where safe
debounce expensive Markdown parsing/sanitization to frame/bounded intervals
preserve raw stream and do authoritative full Markdown render at completion
avoid repeatedly reparsing stable prefixes of large code blocks

Security requirement: optimization must not permit secrets split across chunk boundaries to flash unmasked, and HTML must remain sanitized before insertion.

Acceptance

Backend CPU for a streamed response scales approximately linearly with final response size/chunk count rather than cumulative prefix length.
Browser main-thread Markdown/sanitize work is bounded during a long stream.
Secrets split across arbitrary chunk boundaries are never emitted unmasked to logs/state/WebUI.
Final rendered Markdown/LaTeX/file-link behavior remains equivalent.
Completed response gets a deterministic authoritative final render.
Benchmarks cover 10k/100k/250k-character outputs with small and large chunk sizes.
Capture backend CPU/allocation, websocket bytes, browser long tasks/frame drops, and finalization latency.

## Finding The current streaming log path repeatedly processes the **entire accumulated message** for each chunk: `helpers/log.py::LogItem.stream()` does roughly: ```python self.update(content=self.content + content) ``` `Log._update_item()` then secret-masks and truncation-checks that full cumulative string before storing it. For a response of total length N delivered in many chunks this trends toward O(N²) character work on the backend, plus repeated allocation/copying. The WebUI receives repeated updated log records containing the growing full content. The message renderer's markdown path performs conversions, `marked.parse()` and HTML sanitization over the content when rendering/updating. This can create the same cumulative-reparse pattern in the browser for long streamed Markdown/code output. The response log limit permits up to 250,000 characters, so this is not only theoretical. ## Direction Make streaming incremental while preserving final sanitization/security semantics. Backend candidates: - append-buffer/chunk representation during active streaming - incremental secret masker with overlap sufficient for secrets spanning chunk boundaries, or safe bounded re-scan window - avoid allocating `old_content + chunk` and re-masking the entire prefix per chunk - finalize/truncate once or at bounded checkpoints Frontend candidates: - render streaming text incrementally where safe - debounce expensive Markdown parsing/sanitization to frame/bounded intervals - preserve raw stream and do authoritative full Markdown render at completion - avoid repeatedly reparsing stable prefixes of large code blocks Security requirement: optimization must **not** permit secrets split across chunk boundaries to flash unmasked, and HTML must remain sanitized before insertion. ## Acceptance - [ ] Backend CPU for a streamed response scales approximately linearly with final response size/chunk count rather than cumulative prefix length. - [ ] Browser main-thread Markdown/sanitize work is bounded during a long stream. - [ ] Secrets split across arbitrary chunk boundaries are never emitted unmasked to logs/state/WebUI. - [ ] Final rendered Markdown/LaTeX/file-link behavior remains equivalent. - [ ] Completed response gets a deterministic authoritative final render. - [ ] Benchmarks cover 10k/100k/250k-character outputs with small and large chunk sizes. - [ ] Capture backend CPU/allocation, websocket bytes, browser long tasks/frame drops, and finalization latency.