[P2] ZARA-028 — Stream phrase-sized TTS into a persistent low-latency audio player #29

New issue

Open

opened 2026-07-18 19:25:30 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-07-18 19:25:30 +00:00

(Migrated from github.com)

Objective

Begin audible response from the first safe text phrase instead of waiting for the complete assistant answer and a complete synthesized audio file.

Required implementation

Accept sentence_ready/phrase events from the streaming LLM path and synthesize them in order.
Use native provider streaming when supported. For non-streaming providers, pipeline bounded phrase-sized synthesis requests while preserving order.
Add or use a streaming endpoint for the configured Qwen TTS server if its current API only returns complete WAV files; keep a non-streaming compatibility fallback.
Maintain one persistent playback process/stream per session instead of running which, launching a player, and decoding an entire file for every response.
Define the audio chunk/container/PCM contract and perform format conversion once at the provider boundary.
Apply bounded jitter buffering so playback starts quickly without frequent underruns.
Support immediate flush/cancel for barge-in and turn replacement.
Prevent gaps, duplicate words, and chunk reordering at phrase boundaries.

Required tests

Fake streaming and non-streaming providers with variable delays, empty chunks, format changes, errors, cancellation, and out-of-order completion.
Audio-boundary tests proving phrase concatenation is ordered and nonduplicating.
Persistent-player lifecycle tests for startup, reuse, underrun, failure, restart, and shutdown.
Enforce deterministic first-text-phrase-to-first-audio p95 <= 500 ms.
Add scripts/test-streaming-tts.sh and benchmark coverage.

Acceptance

Audible playback begins before the full model response is complete.
The player is reused across turns and can be flushed within the barge-in budget.
Provider failures do not deadlock later phrases or later turns.

Branch

perf/zara-028-streaming-tts

Dependencies

ZARA-010, ZARA-026, ZARA-027.

## Objective Begin audible response from the first safe text phrase instead of waiting for the complete assistant answer and a complete synthesized audio file. ## Required implementation - Accept `sentence_ready`/phrase events from the streaming LLM path and synthesize them in order. - Use native provider streaming when supported. For non-streaming providers, pipeline bounded phrase-sized synthesis requests while preserving order. - Add or use a streaming endpoint for the configured Qwen TTS server if its current API only returns complete WAV files; keep a non-streaming compatibility fallback. - Maintain one persistent playback process/stream per session instead of running `which`, launching a player, and decoding an entire file for every response. - Define the audio chunk/container/PCM contract and perform format conversion once at the provider boundary. - Apply bounded jitter buffering so playback starts quickly without frequent underruns. - Support immediate flush/cancel for barge-in and turn replacement. - Prevent gaps, duplicate words, and chunk reordering at phrase boundaries. ## Required tests - Fake streaming and non-streaming providers with variable delays, empty chunks, format changes, errors, cancellation, and out-of-order completion. - Audio-boundary tests proving phrase concatenation is ordered and nonduplicating. - Persistent-player lifecycle tests for startup, reuse, underrun, failure, restart, and shutdown. - Enforce deterministic first-text-phrase-to-first-audio p95 <= 500 ms. - Add `scripts/test-streaming-tts.sh` and benchmark coverage. ## Acceptance - Audible playback begins before the full model response is complete. - The player is reused across turns and can be flushed within the barge-in budget. - Provider failures do not deadlock later phrases or later turns. ## Branch `perf/zara-028-streaming-tts` ## Dependencies ZARA-010, ZARA-026, ZARA-027.