[P2] ZARA-026 — Implement true barge-in and cancel stale LLM/TTS work on user speech #27

New issue

Closed

opened 2026-07-18 19:24:56 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-07-18 19:24:56 +00:00

(Migrated from github.com)

Objective

Allow the user to interrupt Zara naturally at any point, including acknowledgement, model generation, synthesis, and playback.

Required implementation

Continue microphone/VAD monitoring while Zara is speaking.
On confirmed user speech, stop or duck current playback immediately and emit cancel(turn_id) through the coordinator.
Propagate cancellation to active LLM streams, tool loops, TTS synthesis, player buffers, and pending notifications.
Prevent cancelled/stale text or audio from resuming after the new turn starts.
Distinguish user speech from Zara’s own speaker output. Implement a practical first version using playback-aware gating/reference energy and configurable thresholds; keep the interface open for acoustic echo cancellation.
Preserve explicit stop phrases, but do not require a stop phrase for barge-in.
Define how interrupted assistant text is represented in conversation history; do not persist unsaid content as if spoken.

Required tests

Synthetic duplex tests for interruption during acknowledgement, before first LLM token, mid-token stream, TTS synthesis, and buffered playback.
False-interruption tests using Zara’s own output and background noise.
Cancellation-race tests where a provider returns after cancellation.
Enforce deterministic speech-onset-to-playback-stop p95 <= 200 ms.
Add scripts/test-barge-in.sh and benchmark coverage.

Acceptance

User speech stops current audible output within budget.
No cancelled turn can later speak, notify, execute a delayed side effect, or overwrite the new turn.
Zara’s own voice does not repeatedly self-interrupt or self-trigger under fixture tests.

Branch

perf/zara-026-barge-in

Dependencies

ZARA-023, ZARA-024, ZARA-025.

## Objective Allow the user to interrupt Zara naturally at any point, including acknowledgement, model generation, synthesis, and playback. ## Required implementation - Continue microphone/VAD monitoring while Zara is speaking. - On confirmed user speech, stop or duck current playback immediately and emit `cancel(turn_id)` through the coordinator. - Propagate cancellation to active LLM streams, tool loops, TTS synthesis, player buffers, and pending notifications. - Prevent cancelled/stale text or audio from resuming after the new turn starts. - Distinguish user speech from Zara’s own speaker output. Implement a practical first version using playback-aware gating/reference energy and configurable thresholds; keep the interface open for acoustic echo cancellation. - Preserve explicit stop phrases, but do not require a stop phrase for barge-in. - Define how interrupted assistant text is represented in conversation history; do not persist unsaid content as if spoken. ## Required tests - Synthetic duplex tests for interruption during acknowledgement, before first LLM token, mid-token stream, TTS synthesis, and buffered playback. - False-interruption tests using Zara’s own output and background noise. - Cancellation-race tests where a provider returns after cancellation. - Enforce deterministic speech-onset-to-playback-stop p95 <= 200 ms. - Add `scripts/test-barge-in.sh` and benchmark coverage. ## Acceptance - User speech stops current audible output within budget. - No cancelled turn can later speak, notify, execute a delayed side effect, or overwrite the new turn. - Zara’s own voice does not repeatedly self-interrupt or self-trigger under fixture tests. ## Branch `perf/zara-026-barge-in` ## Dependencies ZARA-023, ZARA-024, ZARA-025.