P1 — check in the representative real speech corpus and run VAD/STT -> dialogue/capability regression tests #168

New issue

Open

opened 2026-08-22 21:54:41 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-22 21:54:41 +00:00

(Migrated from github.com)

Parent epic: #151
Depends on: #167, #132, #165

Goal

Use the recorder to create a representative human-spoken corpus and make those files exercise Zara's real voice ingress/STT semantic path in repeatable CI/local tests. This is Level B verification; it does not replace the live human gate.

Required corpus

At minimum cover: complete timer; bare timer + Twenty seconds; correction; cancellation; open Firefox; bare open + follow-up; server search; device screenshot semantic request; safe memory fixture; safe fake tool/service; create simple user command; invoke it; parameterized authored command with follow-up; describe/edit/delete command.

Test contract

use checked-in audio and manifest expectations;
feed canonical voice/VAD/STT path where deterministic/local supported;
compare normalized transcript with declared tolerance/policy, then exact semantic/dialogue/capability expectations;
preserve declared multi-turn timing semantics without real sleeps;
no microphone, external network or paid provider in default CI;
separate model/version-sensitive STT expectations from semantic authority so a transcription model upgrade requires explicit corpus re-baseline, not silent weakened assertions.

Failure cases

Noise/silence/clipping fixture; STT error; VAD no-speech/timeout; partial/final disagreement; cancellation between utterances; stale old-turn transcript; two concurrent principal fixture streams remain isolated.

Acceptance

The repository has a real user-recorded voice corpus that catches regressions from waveform through STT into the semantic/dialogue/provider result, with strong assertions rather than execution-only coverage.

Parent epic: #151 Depends on: #167, #132, #165 ## Goal Use the recorder to create a representative human-spoken corpus and make those files exercise Zara's real voice ingress/STT semantic path in repeatable CI/local tests. This is Level B verification; it does not replace the live human gate. ## Required corpus At minimum cover: complete timer; bare timer + `Twenty seconds`; correction; cancellation; open Firefox; bare open + follow-up; server search; device screenshot semantic request; safe memory fixture; safe fake tool/service; create simple user command; invoke it; parameterized authored command with follow-up; describe/edit/delete command. ## Test contract - use checked-in audio and manifest expectations; - feed canonical voice/VAD/STT path where deterministic/local supported; - compare normalized transcript with declared tolerance/policy, then exact semantic/dialogue/capability expectations; - preserve declared multi-turn timing semantics without real sleeps; - no microphone, external network or paid provider in default CI; - separate model/version-sensitive STT expectations from semantic authority so a transcription model upgrade requires explicit corpus re-baseline, not silent weakened assertions. ## Failure cases Noise/silence/clipping fixture; STT error; VAD no-speech/timeout; partial/final disagreement; cancellation between utterances; stale old-turn transcript; two concurrent principal fixture streams remain isolated. ## Acceptance The repository has a real user-recorded voice corpus that catches regressions from waveform through STT into the semantic/dialogue/provider result, with strong assertions rather than execution-only coverage.