[P2] ZARA-027 — Stream LLM and agent output with cancellation-safe text events #28

New issue

Closed

opened 2026-07-18 19:25:13 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-07-18 19:25:13 +00:00

(Migrated from github.com)

Objective

Expose model output as an async stream so Zara can begin responding before the full answer is generated.

Required implementation

Add one provider-neutral async streaming interface that emits typed events such as text_delta, sentence_ready, tool_call_started, tool_result, completed, cancelled, and failed.
Implement streaming for configured Anthropic, OpenAI-compatible, and Ollama providers where supported; retain a tested non-streaming adapter fallback.
Integrate LangGraph/tool execution without speaking incomplete tool-call syntax or hidden protocol content.
Buffer deltas into speakable phrase/sentence chunks using punctuation, length, and maximum-wait thresholds.
Maintain exact final text for conversation history while tracking which chunks were actually delivered to speech output.
Honor turn cancellation promptly and close provider streams/sessions.
Do not expose private reasoning or raw provider protocol blocks.

Required tests

Fake provider streams with tiny, delayed, malformed, tool-call, error, and cancellation sequences.
Verify sentence chunking handles abbreviations, code/URLs, no punctuation, very long clauses, and Unicode.
Verify tool calls pause user-facing speech until a safe user-facing result exists.
Add scripts/test-streaming-llm.sh with local fake servers and extend latency benchmarks for first token/first sentence.

Acceptance

First text delta is observable without waiting for the complete model response.
Cancellation stops future deltas and closes resources.
Persisted final history is correct and excludes unsent protocol content.

Branch

perf/zara-027-streaming-llm

Dependencies

ZARA-016, ZARA-023, ZARA-026.

## Objective Expose model output as an async stream so Zara can begin responding before the full answer is generated. ## Required implementation - Add one provider-neutral async streaming interface that emits typed events such as `text_delta`, `sentence_ready`, `tool_call_started`, `tool_result`, `completed`, `cancelled`, and `failed`. - Implement streaming for configured Anthropic, OpenAI-compatible, and Ollama providers where supported; retain a tested non-streaming adapter fallback. - Integrate LangGraph/tool execution without speaking incomplete tool-call syntax or hidden protocol content. - Buffer deltas into speakable phrase/sentence chunks using punctuation, length, and maximum-wait thresholds. - Maintain exact final text for conversation history while tracking which chunks were actually delivered to speech output. - Honor turn cancellation promptly and close provider streams/sessions. - Do not expose private reasoning or raw provider protocol blocks. ## Required tests - Fake provider streams with tiny, delayed, malformed, tool-call, error, and cancellation sequences. - Verify sentence chunking handles abbreviations, code/URLs, no punctuation, very long clauses, and Unicode. - Verify tool calls pause user-facing speech until a safe user-facing result exists. - Add `scripts/test-streaming-llm.sh` with local fake servers and extend latency benchmarks for first token/first sentence. ## Acceptance - First text delta is observable without waiting for the complete model response. - Cancellation stops future deltas and closes resources. - Persisted final history is correct and excludes unsent protocol content. ## Branch `perf/zara-027-streaming-llm` ## Dependencies ZARA-016, ZARA-023, ZARA-026.