P1 release gate — prove transcript normalization preserves intent, privacy and voice latency #215

New issue

Open

opened 2026-08-28 03:23:52 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-08-28 03:23:52 +00:00

Owner

Parent: #215
Depends on: #219
Integrates with: #23, #30, #31, #134, #168

Goal

Prove the optional normalization stage improves messy ASR text without becoming a semantic-corruption, privacy, or latency regression.

Correctness matrix

Run paired raw -> normalized fixtures covering:

fillers and hesitations;
false starts;
self-corrections;
repetitions;
punctuation/capitalization;
numbers/dates/times/currency;
email addresses;
lists;
already-clean utterances;
short deterministic commands;
conversational questions;
negation/cancellation (never mind, don't ...);
tool/device requests where changing one token could change side effects.

Assert both text-level expectations and downstream semantic equivalence/intent correctness where appropriate. Do not score only whether output "looks cleaner."

Adversarial transformation cases

Prove bounded behavior when input contains:

instructions attempting to redirect the normalizer;
text that resembles system/tool commands;
URLs/code/shell fragments;
profanity/slang;
ambiguous corrections;
long repeated speech;
malformed Unicode/control characters;
unsupported/non-English input;
empty/silence-derived final text;
outputs that expand, summarize, answer, or add unsupported facts.

The normalizer output remains user text and cannot increase authorization.

Privacy matrix

local endpoint path emits no transcript to external network;
remote endpoint configuration is explicit and diagnosable;
raw/normalized text is absent from ordinary latency/security audit logs;
crash/error/diagnostic exports redact transcript content according to existing Zara policy;
two-principal daemon tests prove no cross-user transcript/provider-result leakage.

Latency / resource gate

Extend #23/#30 measurements with:

final_transcript -> normalization_start
normalization_start -> normalization_done
normalization_done -> route_selected

Record p50/p95/p99 where sample sizes justify it. Benchmark:

off identity cost;
fake deterministic provider in CI;
optional real local S1-mini warm/cold runs;
repeated turns for CPU/RSS/thread/socket growth;
provider unavailable/timeout path;
cancellation while inference is active.

Do not invent a latency budget before measuring, but the final report must state whether the added stage materially harms Zara's existing end-of-speech-to-action/response target and whether warm local inference is acceptable.

Real-speech integration

Extend #168-style recorded speech cases so at least some fixtures travel through:

WAV -> VAD/STT -> raw transcript -> selected normalizer -> semantic/dialogue result

Keep STT-model-sensitive transcript expectations explicit; do not weaken semantic assertions to hide normalization failures.

Release decision

Produce an evidence-based default recommendation among:

keep default off, selectable S1-mini;
enable by default only for dictation;
enable by default for supported English voice paths;
retain experimental status due to measured corruption/latency.

The gate may not silently change defaults; any default change requires a focused follow-up decision/change.

Automation

Add one non-interactive transcript-normalization release command composed into the full repository/Nix gate. Default CI uses deterministic fixtures and no network/model download; real S1-mini benchmark profile is optional but documented/reproducible.

Acceptance

The selectable S1-mini path has adversarial text, downstream intent, cancellation, privacy, multi-user, resource and latency evidence strong enough to decide where it is safe to enable without weakening Zara's existing voice release gates.

Parent: #215 Depends on: #219 Integrates with: #23, #30, #31, #134, #168 ## Goal Prove the optional normalization stage improves messy ASR text without becoming a semantic-corruption, privacy, or latency regression. ## Correctness matrix Run paired raw -> normalized fixtures covering: - fillers and hesitations; - false starts; - self-corrections; - repetitions; - punctuation/capitalization; - numbers/dates/times/currency; - email addresses; - lists; - already-clean utterances; - short deterministic commands; - conversational questions; - negation/cancellation (`never mind`, `don't ...`); - tool/device requests where changing one token could change side effects. Assert both text-level expectations and downstream semantic equivalence/intent correctness where appropriate. Do not score only whether output "looks cleaner." ## Adversarial transformation cases Prove bounded behavior when input contains: - instructions attempting to redirect the normalizer; - text that resembles system/tool commands; - URLs/code/shell fragments; - profanity/slang; - ambiguous corrections; - long repeated speech; - malformed Unicode/control characters; - unsupported/non-English input; - empty/silence-derived final text; - outputs that expand, summarize, answer, or add unsupported facts. The normalizer output remains user text and cannot increase authorization. ## Privacy matrix - local endpoint path emits no transcript to external network; - remote endpoint configuration is explicit and diagnosable; - raw/normalized text is absent from ordinary latency/security audit logs; - crash/error/diagnostic exports redact transcript content according to existing Zara policy; - two-principal daemon tests prove no cross-user transcript/provider-result leakage. ## Latency / resource gate Extend #23/#30 measurements with: ```text final_transcript -> normalization_start normalization_start -> normalization_done normalization_done -> route_selected ``` Record p50/p95/p99 where sample sizes justify it. Benchmark: - `off` identity cost; - fake deterministic provider in CI; - optional real local S1-mini warm/cold runs; - repeated turns for CPU/RSS/thread/socket growth; - provider unavailable/timeout path; - cancellation while inference is active. Do not invent a latency budget before measuring, but the final report must state whether the added stage materially harms Zara's existing end-of-speech-to-action/response target and whether warm local inference is acceptable. ## Real-speech integration Extend #168-style recorded speech cases so at least some fixtures travel through: ```text WAV -> VAD/STT -> raw transcript -> selected normalizer -> semantic/dialogue result ``` Keep STT-model-sensitive transcript expectations explicit; do not weaken semantic assertions to hide normalization failures. ## Release decision Produce an evidence-based default recommendation among: - keep default `off`, selectable S1-mini; - enable by default only for dictation; - enable by default for supported English voice paths; - retain experimental status due to measured corruption/latency. The gate may not silently change defaults; any default change requires a focused follow-up decision/change. ## Automation Add one non-interactive transcript-normalization release command composed into the full repository/Nix gate. Default CI uses deterministic fixtures and no network/model download; real S1-mini benchmark profile is optional but documented/reproducible. ## Acceptance The selectable S1-mini path has adversarial text, downstream intent, cancellation, privacy, multi-user, resource and latency evidence strong enough to decide where it is safe to enable without weakening Zara's existing voice release gates.

nsaspy referenced this issue

2026-08-28 03:23:52 +00:00

P1 — wire transcript normalization into voice, dictation and semantic routing before #122 #216

nsaspy referenced this issue

2026-08-28 03:23:52 +00:00

P1 — add transcript-normalizer configuration, selection, status and diagnostics #217

nsaspy referenced this issue

2026-08-28 03:23:53 +00:00

P1 — implement local S1-mini transcript normalizer backend #218

nsaspy referenced this issue

2026-08-28 03:23:53 +00:00

P1 — define provider-neutral transcript normalization contract and raw/normalized runtime semantics #219