[P2] ZARA-022 — Instrument end-to-end voice latency and define enforceable budgets #23

New issue

Closed

opened 2026-07-18 19:22:48 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-07-18 19:22:48 +00:00

(Migrated from github.com)

Objective

Measure the actual voice-turn critical path before optimizing it.

Required implementation

Add monotonic timestamped events for:
- audio frame received;
- wake detected;
- acknowledgement requested/first audio;
- speech start/end;
- final transcript;
- route selected;
- Prolog result;
- LLM request/first token/final token;
- TTS request/first chunk/first playback/final playback;
- interruption detected/cancellation completed.
Generate a turn/trace ID and propagate it through every stage.
Emit structured JSONL metrics without logging raw secrets or full audio.
Add percentile reporting for p50/p95/p99.
Define initial local budgets in config/tests. Suggested starting gates:
- wake-to-ack first audio: p95 <= 350 ms with pre-generated acknowledgement;
- end-of-speech-to-final transcript: p95 <= 700 ms for short fixture utterances;
- LLM request-to-first token: measured/provider-specific, not globally hard-coded;
- first text chunk-to-first TTS audio: p95 <= 500 ms for local fixture server;
- barge-in speech onset-to-playback stop: p95 <= 200 ms.

Required tests

Fake-clock unit tests for event ordering and duration calculation.
Add scripts/benchmark-voice.py and scripts/test-latency-metrics.sh using deterministic fixture providers.
Benchmarks must distinguish cold and warm runs.

Acceptance

One report identifies the dominant latency stage for each turn.
Missing/out-of-order events fail validation.
CI stores benchmark output but only enforces deterministic fixture budgets.

Branch

perf/zara-022-latency-instrumentation

Dependencies

ZARA-021.

## Objective Measure the actual voice-turn critical path before optimizing it. ## Required implementation - Add monotonic timestamped events for: - audio frame received; - wake detected; - acknowledgement requested/first audio; - speech start/end; - final transcript; - route selected; - Prolog result; - LLM request/first token/final token; - TTS request/first chunk/first playback/final playback; - interruption detected/cancellation completed. - Generate a turn/trace ID and propagate it through every stage. - Emit structured JSONL metrics without logging raw secrets or full audio. - Add percentile reporting for p50/p95/p99. - Define initial local budgets in config/tests. Suggested starting gates: - wake-to-ack first audio: p95 <= 350 ms with pre-generated acknowledgement; - end-of-speech-to-final transcript: p95 <= 700 ms for short fixture utterances; - LLM request-to-first token: measured/provider-specific, not globally hard-coded; - first text chunk-to-first TTS audio: p95 <= 500 ms for local fixture server; - barge-in speech onset-to-playback stop: p95 <= 200 ms. ## Required tests - Fake-clock unit tests for event ordering and duration calculation. - Add `scripts/benchmark-voice.py` and `scripts/test-latency-metrics.sh` using deterministic fixture providers. - Benchmarks must distinguish cold and warm runs. ## Acceptance - One report identifies the dominant latency stage for each turn. - Missing/out-of-order events fail validation. - CI stores benchmark output but only enforces deterministic fixture budgets. ## Branch `perf/zara-022-latency-instrumentation` ## Dependencies ZARA-021.