[P2] ZARA-024 — Replace fixed audio chunks with streaming VAD, incremental STT, and fast endpointing #25

New issue

Closed

opened 2026-07-18 19:24:25 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-07-18 19:24:25 +00:00

(Migrated from github.com)

Objective

Move speech recognition from fixed multi-second recordings to a continuous streaming pipeline that produces partial and final transcripts with low endpoint latency.

Current limitation

Wake mode transcribes fixed passive chunks and buffers an entire active utterance before invoking Whisper. Dictation uses 10-second chunks despite claiming lower latency. This guarantees noticeable delay and prevents partial routing or early acknowledgement decisions.

Required implementation

Feed sequenced audio frames continuously from the audio actor into a dedicated VAD/STT actor.
Use a configurable frame size suitable for real-time speech detection, typically 10–30 ms at the capture boundary.
Emit typed events for speech_started, partial_transcript, speech_ended, and final_transcript.
Maintain a bounded rolling pre-speech buffer so the first phoneme is not clipped.
Implement adaptive endpointing with separate minimum speech, trailing silence, maximum utterance, and no-speech deadlines.
Prevent stale partial/final results from crossing turn IDs.
Preserve the ability to run faster-whisper locally; do not replace the configured STT provider/model without approval.
Expose a stable transcriber interface so a genuinely streaming backend can be added later without changing the coordinator.

Required tests

Fixture tests for clipped-onset prevention, short command, long command, pauses inside speech, background noise, false VAD start, no speech, continuous speech, and cancellation.
Verify partial transcripts are monotonic enough for display but only final text triggers irreversible command execution.
Benchmark warm end-of-speech-to-final-transcript latency against ZARA-022 metrics.
Add scripts/test-streaming-stt.sh and extend scripts/benchmark-voice.py.

Acceptance

No runtime path waits for a fixed 3-, 8-, or 10-second chunk before processing speech.
Short fixture commands reach a final transcript within the configured p95 budget after speech ends.
Audio buffers and event queues remain bounded during a long stream.

Branch

perf/zara-024-streaming-stt

Dependencies

ZARA-022, ZARA-023.

## Objective Move speech recognition from fixed multi-second recordings to a continuous streaming pipeline that produces partial and final transcripts with low endpoint latency. ## Current limitation Wake mode transcribes fixed passive chunks and buffers an entire active utterance before invoking Whisper. Dictation uses 10-second chunks despite claiming lower latency. This guarantees noticeable delay and prevents partial routing or early acknowledgement decisions. ## Required implementation - Feed sequenced audio frames continuously from the audio actor into a dedicated VAD/STT actor. - Use a configurable frame size suitable for real-time speech detection, typically 10–30 ms at the capture boundary. - Emit typed events for `speech_started`, `partial_transcript`, `speech_ended`, and `final_transcript`. - Maintain a bounded rolling pre-speech buffer so the first phoneme is not clipped. - Implement adaptive endpointing with separate minimum speech, trailing silence, maximum utterance, and no-speech deadlines. - Prevent stale partial/final results from crossing turn IDs. - Preserve the ability to run faster-whisper locally; do not replace the configured STT provider/model without approval. - Expose a stable transcriber interface so a genuinely streaming backend can be added later without changing the coordinator. ## Required tests - Fixture tests for clipped-onset prevention, short command, long command, pauses inside speech, background noise, false VAD start, no speech, continuous speech, and cancellation. - Verify partial transcripts are monotonic enough for display but only final text triggers irreversible command execution. - Benchmark warm end-of-speech-to-final-transcript latency against ZARA-022 metrics. - Add `scripts/test-streaming-stt.sh` and extend `scripts/benchmark-voice.py`. ## Acceptance - No runtime path waits for a fixed 3-, 8-, or 10-second chunk before processing speech. - Short fixture commands reach a final transcript within the configured p95 budget after speech ends. - Audio buffers and event queues remain bounded during a long stream. ## Branch `perf/zara-024-streaming-stt` ## Dependencies ZARA-022, ZARA-023.