#132 - P0 — stream live voice over `ZARA/1` with client-owned audio devices and cross-process barge-in - nsaspy/zara

lost-rob0t commented

2026-08-21 23:41:59 +00:00

(Migrated from github.com)

Parent epic: #127
Depends on: #129, #131
Integrates with: #25, #27, #28, #29, #30 and existing latency tracing
Research: docs/research/001-daemon-zeromq-voice-service.md

Goal

Move the live voice interaction boundary across the daemon protocol without creating a second voice stack. Clients own microphone capture and playback; the daemon owns Zara's existing VAD/STT/routing/LLM/tool/TTS runtime work.

Audio protocol

Implement the reserved v1 message families:

audio.input.start
audio.input.chunk
audio.input.commit
audio.input.cancel

audio.output.start
audio.output.chunk
audio.output.done

Initial deterministic codec contract:

input baseline: PCM signed 16-bit little-endian, mono, 16 kHz;
output PCM rate/channel format negotiated in hello/hello.ok;
binary audio lives in multipart payload frames, never base64 JSON;
every stream has stream_id + monotonic seq;
chunk duration/bytes are bounded;
gaps, duplicates, late chunks and chunks after commit/cancel have explicit behavior;
Opus or another compressed remote codec may be added later only behind capability negotiation.

Client device boundary

microphone capture belongs to the client process/device;
speaker playback belongs to the client process/device;
wake-word detection should remain local/client-side where available and begin a daemon voice session only after wake/manual activation;
daemon must not assume one process-global microphone or speaker;
multiple clients/users can speak independently without sharing audio device state.

Runtime bridge

Reuse the existing Zara pipeline:

streaming VAD/STT from #25;
canonical TurnCoordinator turn IDs/cancellation;
barge-in/stale-work cancellation from #27;
real LLM streaming from #28 when available;
phrase TTS streaming from #29 when available;
current runtime event contract from #82/#83.

Do not implement a daemon-only STT, LLM, TTS, or cancellation path.

Interruption semantics

When client speech begins while Zara is speaking:

client stops local playback immediately;
client sends the interruption/cancel signal tied to the canonical turn/stream;
daemon cancels stale LLM/tool/TTS work through existing turn semantics;
late audio.output.chunk and assistant deltas for the cancelled turn are rejected/ignored;
new input starts a new canonical turn;
the transcript/history reflects only the audio/text actually accepted by runtime policy.

Preserve latency trace_id from client audio ingress through STT, routing, model, TTS and output playback events.

Backpressure

per-stream bounded ingress/egress queues;
never buffer unbounded microphone audio when STT/model work stalls;
slow client playback cannot block another client;
output overrun/cancellation is explicit and measurable;
audio data is never included in ordinary logs or security audit records.

Tests

deterministic audio chunk framing/golden fixtures;
two concurrent principals with independent audio streams;
input gap/duplicate/out-of-order/late-after-commit behavior;
client disconnect mid-utterance;
daemon restart mid-session;
barge-in stops client playback and cancels stale daemon turn;
stale audio output cannot speak into a newer turn;
slow client/backpressure isolation;
no raw audio bytes in logs/audit/errors;
preserve trace/turn/stream correlation;
voice latency benchmark extends the existing Zara benchmark instead of inventing a second metric system;
focused non-interactive voice transport script plus full repo/Nix gate.

Acceptance

A client can keep one authenticated daemon session open, stream microphone audio, receive partial/final transcript and streaming assistant text/audio events, hear phrase-level output, and interrupt Zara naturally without sharing audio state with other users.

Non-goals

no video/screen streaming requirement in v1;
no continuous server-side capture of every client's microphone;
no duplicated voice provider implementations;
no base64 audio in JSON.

Parent epic: #127 Depends on: #129, #131 Integrates with: #25, #27, #28, #29, #30 and existing latency tracing Research: `docs/research/001-daemon-zeromq-voice-service.md` ## Goal Move the live voice interaction boundary across the daemon protocol without creating a second voice stack. Clients own microphone capture and playback; the daemon owns Zara's existing VAD/STT/routing/LLM/tool/TTS runtime work. ## Audio protocol Implement the reserved v1 message families: ```text audio.input.start audio.input.chunk audio.input.commit audio.input.cancel audio.output.start audio.output.chunk audio.output.done ``` Initial deterministic codec contract: - input baseline: PCM signed 16-bit little-endian, mono, 16 kHz; - output PCM rate/channel format negotiated in `hello`/`hello.ok`; - binary audio lives in multipart payload frames, never base64 JSON; - every stream has `stream_id` + monotonic `seq`; - chunk duration/bytes are bounded; - gaps, duplicates, late chunks and chunks after commit/cancel have explicit behavior; - Opus or another compressed remote codec may be added later only behind capability negotiation. ## Client device boundary - microphone capture belongs to the client process/device; - speaker playback belongs to the client process/device; - wake-word detection should remain local/client-side where available and begin a daemon voice session only after wake/manual activation; - daemon must not assume one process-global microphone or speaker; - multiple clients/users can speak independently without sharing audio device state. ## Runtime bridge Reuse the existing Zara pipeline: - streaming VAD/STT from #25; - canonical TurnCoordinator turn IDs/cancellation; - barge-in/stale-work cancellation from #27; - real LLM streaming from #28 when available; - phrase TTS streaming from #29 when available; - current runtime event contract from #82/#83. Do not implement a daemon-only STT, LLM, TTS, or cancellation path. ## Interruption semantics When client speech begins while Zara is speaking: 1. client stops local playback immediately; 2. client sends the interruption/cancel signal tied to the canonical turn/stream; 3. daemon cancels stale LLM/tool/TTS work through existing turn semantics; 4. late `audio.output.chunk` and assistant deltas for the cancelled turn are rejected/ignored; 5. new input starts a new canonical turn; 6. the transcript/history reflects only the audio/text actually accepted by runtime policy. Preserve latency `trace_id` from client audio ingress through STT, routing, model, TTS and output playback events. ## Backpressure - per-stream bounded ingress/egress queues; - never buffer unbounded microphone audio when STT/model work stalls; - slow client playback cannot block another client; - output overrun/cancellation is explicit and measurable; - audio data is never included in ordinary logs or security audit records. ## Tests - deterministic audio chunk framing/golden fixtures; - two concurrent principals with independent audio streams; - input gap/duplicate/out-of-order/late-after-commit behavior; - client disconnect mid-utterance; - daemon restart mid-session; - barge-in stops client playback and cancels stale daemon turn; - stale audio output cannot speak into a newer turn; - slow client/backpressure isolation; - no raw audio bytes in logs/audit/errors; - preserve trace/turn/stream correlation; - voice latency benchmark extends the existing Zara benchmark instead of inventing a second metric system; - focused non-interactive voice transport script plus full repo/Nix gate. ## Acceptance A client can keep one authenticated daemon session open, stream microphone audio, receive partial/final transcript and streaming assistant text/audio events, hear phrase-level output, and interrupt Zara naturally without sharing audio state with other users. ## Non-goals - no video/screen streaming requirement in v1; - no continuous server-side capture of every client's microphone; - no duplicated voice provider implementations; - no base64 audio in JSON.

lost-rob0t commented

2026-08-24 04:56:47 +00:00

(Migrated from github.com)

Auto-RAGE #134 release research disproved this completed slice's visible-transcript acceptance, so #132 is reopened rather than patched inside the final release-gate branch.

Concrete current-head evidence:

zara/streaming_stt.py already produces SpeechStarted, PartialTranscript, SpeechEnded, and FinalTranscript.
zara/voice_runtime.py::RuntimeVoiceIngress._run only reacts to FinalTranscript; speech-start, partial, and speech-end events are discarded. The final transcript is converted directly into SubmitTurn by _submit_final instead of being emitted to the authenticated client.
zara/protocol.py has no speech/STT partial/final transcript message vocabulary.
zara/protocol_runtime.py has no mapping for these STT events.

That conflicts with #132's Acceptance text requiring the client to “receive partial/final transcript”. Keep this repair in #132: extend the existing runtime/event boundary and ZARA/1/ZaraClient path; do not create a daemon-only STT stack. Preserve principal/conversation/turn/stream/trace correlation, cancellation/stale-event behavior, bounds, and the existing client-owned audio-device boundary.

#134 remains a proof/composition gate and will re-enter ADADR only after this regression and the other owned streaming/release prerequisites are resolved. Evidence is preserved in PR #190 under rage/134-daemon-release-research.org and rage/134-daemon-release-design.org.

Auto-RAGE #134 release research disproved this completed slice's visible-transcript acceptance, so #132 is reopened rather than patched inside the final release-gate branch. Concrete current-head evidence: - `zara/streaming_stt.py` already produces `SpeechStarted`, `PartialTranscript`, `SpeechEnded`, and `FinalTranscript`. - `zara/voice_runtime.py::RuntimeVoiceIngress._run` only reacts to `FinalTranscript`; speech-start, partial, and speech-end events are discarded. The final transcript is converted directly into `SubmitTurn` by `_submit_final` instead of being emitted to the authenticated client. - `zara/protocol.py` has no speech/STT partial/final transcript message vocabulary. - `zara/protocol_runtime.py` has no mapping for these STT events. That conflicts with #132's Acceptance text requiring the client to “receive partial/final transcript”. Keep this repair in #132: extend the existing runtime/event boundary and ZARA/1/ZaraClient path; do not create a daemon-only STT stack. Preserve principal/conversation/turn/stream/trace correlation, cancellation/stale-event behavior, bounds, and the existing client-owned audio-device boundary. #134 remains a proof/composition gate and will re-enter ADADR only after this regression and the other owned streaming/release prerequisites are resolved. Evidence is preserved in PR #190 under `rage/134-daemon-release-research.org` and `rage/134-daemon-release-design.org`.

lost-rob0t commented

2026-08-24 07:42:26 +00:00

(Migrated from github.com)

Reopened acceptance regression is repaired and fully gated.

Implementation PR #192 merged exact tested candidate d63bdf350e99ca5c8fc06c4e1239b8a94e7ae51e as 30b165e4b3d08aeac2ea5be0934d63cd296ebdb5 after local 9-phase/Nix/build gates and exact-head GitHub Actions run 32700582029 passed.

Append-only RAGE closeout PR #207 recorded that immutable evidence and merged exact tested closeout head 99c0913e4bb0d502903416343ca7d7ea65c6180d as a41877367f7e0af8d936e91982e88e03912dd5c2 after exact-head GitHub Actions run 32701942715 passed.

The repaired path is canonical: streaming STT -> RuntimeVoiceIngress -> principal RuntimeSupervisor/RuntimeEventBus -> closed ZARA/1 -> ZaraClient typed events. Cancellation fences later STT publication, provider/runtime-class internals stay off wire, and final transcript remains the sole SubmitTurn authority.

Reopened acceptance regression is repaired and fully gated. Implementation PR #192 merged exact tested candidate `d63bdf350e99ca5c8fc06c4e1239b8a94e7ae51e` as `30b165e4b3d08aeac2ea5be0934d63cd296ebdb5` after local 9-phase/Nix/build gates and exact-head GitHub Actions run `32700582029` passed. Append-only RAGE closeout PR #207 recorded that immutable evidence and merged exact tested closeout head `99c0913e4bb0d502903416343ca7d7ea65c6180d` as `a41877367f7e0af8d936e91982e88e03912dd5c2` after exact-head GitHub Actions run `32701942715` passed. The repaired path is canonical: streaming STT -> RuntimeVoiceIngress -> principal RuntimeSupervisor/RuntimeEventBus -> closed ZARA/1 -> ZaraClient typed events. Cancellation fences later STT publication, provider/runtime-class internals stay off wire, and final transcript remains the sole SubmitTurn authority.