nsaspy/zara

Fork 0

[EPIC] Principal-scoped programmable voice commands and real speech regression corpus #151

New issue

Open

opened 2026-08-22 21:50:28 +00:00 by lost-rob0t · 1 comment

lost-rob0t commented

2026-08-22 21:50:28 +00:00

(Migrated from github.com)

Goal

Make Zara programmable by voice without creating an unsafe macro language or a second command system. A user must be able to create, inspect, edit, parameterize, test, undo, and delete commands through normal Zara dialogue; authored commands compile into the same semantic intent/slot/capability model owned by #150.

At the same time, establish a repository-native real speech fixture acquisition system so every voice-applicable intent/tool/capability can be regression-tested with realistic user-recorded audio, while keeping final live human acceptance separate and genuinely manual.

Dependencies

Depends on #150 semantic IntentFrame/dialogue/capability architecture.
Voice transport/integration depends on #132.
Supported daemon clients depend on #133.
Multi-user private persistence depends on #131.
Final daemon release remains #134; this epic adds a later programmable-command gate and must not weaken #134.
No Prolog-RLM.

Core principle

User-authored commands are declarative semantic programs, not arbitrary shell/Python/Prolog text dictated by the user.

Bad:

trigger: work mode
shell: "emacs & firefox"

Target concept:

trigger: work_mode
actions:
  - capability: open_app
    args: {app: emacs}
    location: initiating_device
  - capability: open_app
    args: {app: firefox}
    location: initiating_device

The persisted representation must contain only versioned, validated intent/slot/capability data and references to registered providers. It may not smuggle executable source through generic fields.

Voice authoring contract

Zara must support semantic authoring intents at minimum:

create_command;
edit_command;
delete_command;
list_commands;
describe_command;
test_command / dry-run where useful;
undo_command_change for the most recent safe mutation.

Example interaction:

User: Zara, create a new command.
Zara: What should you say?
User: Work mode.
Zara: What should work mode do?
User: Open Emacs and Firefox.
Zara: Run those actions on the device that invoked the command?
User: Yes.
Zara: Work mode will open Emacs and Firefox on the initiating device. Save it?
User: Yes.
Zara: Created command "work mode".

The authoring dialogue must itself use the typed missing-slot/state machinery from #150. No separate ad-hoc conversation state.

Parameterized commands

Authored commands may define typed slots, for example:

"focus timer for {duration}"

or a trigger with a required slot that can be collected later:

User: Zara, focus timer.
Zara: How long?
User: Thirty five minutes.

The same duration parser/validator used by built-in timer intents should fill the authored command. User-command slots must not implement parallel parsing logic.

Required slot types should include only types with existing or explicitly added validators: duration, bounded string/text, contact/entity reference, URI where appropriate, number, enum, and other reviewed types. Never silently infer an unsafe raw-command/string execution type.

Principal/device ownership

definitions are principal-owned;
one principal must not enumerate/infer/use another principal's private authored commands;
device-local variants/grants may be scoped below the principal where required;
command IDs are stable and not bearer capabilities;
reconnect/restart preserves durable definitions;
guest/ephemeral principals cannot accidentally create durable commands unless an explicit promotion flow exists;
deletion/export affect only the owning principal.

Conflict and mutation policy

Prove safe behavior for:

exact trigger collision with built-ins;
aliases that create ambiguous resolution;
collision between two user commands;
edits that invalidate stored slot/action schema;
provider removed/unavailable after command creation;
permissions revoked after creation;
partial write/crash during mutation;
rollback/undo;
schema migration across command-definition versions.

Protected built-ins must not be silently shadowed. Zara should explain the conflict and offer a safe alternative trigger or explicit supported override policy if research justifies one.

Hot reload

A successfully persisted command should become usable without restarting the daemon/client. Reload must be atomic: readers see either the old valid registry or the new valid registry, never a partially parsed definition set.

Future teach-by-demonstration compatibility

Do not require demonstration recording in the first implementation, but preserve an architecture where a future authoring flow can capture typed capabilities that Zara observed executing, then propose a semantic command. Never record literal arbitrary shell as the demonstration artifact.

Real speech fixture system

Add a developer tool, working name:

zara-record-voice-fixtures

The recorder is a fixture acquisition tool, not the test oracle.

It must read declarative test-case metadata and guide a human through recording the exact utterances the automated corpus expects.

Example:

Case: timer_missing_duration

Get ready...
Say: "Zara, set a timer."
[record]

Wait about 1.8 seconds...
Say: "Twenty minutes."
[record]

Saved and validated.

Declarative voice fixture manifest

The same source of truth should drive both recorder prompts and tests. It must encode, subject to RAGE research:

stable case/version id;
one or more spoken utterances;
expected normalized transcript where deterministic enough;
expected semantic intent/slots/missing slots;
expected capability/provider location;
realistic pre-turn/inter-turn/post-speech delay or bounded range;
audio filename(s);
sample rate/channels/encoding;
expected dialogue transitions;
optional expected action/result fixture;
tags such as single-turn, multi-turn, cancellation, correction, device, server, tool, authored-command.

Do not duplicate the phrase corpus in shell scripts.

Recorder timing behavior

The tool should make recordings sound like actual assistant use rather than concatenated lab tokens:

configurable initial get-ready delay;
realistic inter-turn delay, with deterministic/default target plus allowed jitter/range metadata;
post-utterance silence sufficient for VAD fixtures;
visible/audible countdown or cue where appropriate;
repeat/re-record one step without destroying previous accepted fixtures;
bounded total case time;
no network requirement.

Tests must not depend on wall-clock sleeps where a virtual clock/recorded timing metadata can deterministically model the protocol/dialogue behavior. The sleep is for human recording guidance, not flaky CI.

Audio validation

Before committing a fixture, validate at least:

non-empty recording;
supported PCM/container format;
expected sample rate/channel contract or deterministic conversion;
plausible duration bounds;
peak/RMS sanity;
clipping/saturation warning/failure threshold;
near-silence warning/failure threshold;
no NaN/invalid PCM;
bounded file size;
explicit replacement confirmation for existing fixture.

Keep raw fixture audio free of secrets/private conversations: the tool prompts only the declared synthetic/test phrase.

Fixture classes

Support at least:

single-utterance fixtures;
multi-turn fixtures recorded as separate utterances while preserving timing metadata;
optional continuous-session recordings for VAD/dialogue integration;
corrections/cancellation;
authored-command invocation;
server tool/API capability commands;
device action commands;
unavailable/denied capability cases where no dangerous real action is required.

Mandatory example corpus

At minimum record/test representative versions of:

Zara, set a timer for twenty seconds.
Zara, set a timer. -> later Twenty seconds.
Actually, make that five minutes.
Never mind.
Zara, open Firefox.
Zara, open. -> later Firefox.
Zara, search for ZeroMQ CURVE authentication.
Zara, take a screenshot.
Zara, what do you remember about the test fact?
a safe fake/tool fixture;
command creation dialogue;
immediate invocation of the newly created command;
parameterized user command with missing-slot follow-up;
edit/describe/delete command flows.

Every newly shipped voice-applicable intent, capability, tool, or authored-command behavior must add representative text fixtures and a recorder-consumable case unless there is a documented reason voice cannot apply.

Three-level verification model

Level A: text/property CI

Deterministic text fixtures prove parser/dialogue/capability semantics without STT noise.

Level B: checked-in recorded fixture CI

Human-recorded repository fixtures exercise the real VAD/STT -> semantic path using deterministic/local models or approved fixture transcription strategies. No microphone required in CI.

Level C: manual live human gate

This remains intentionally manual. The user runs real Zara on real hardware. An assistant/test guide supplies exact phrases one at a time plus expected observable behavior. Results/evidence may be recorded afterward, but the test must not be relabeled automated merely because a script prints prompts.

Final human voice acceptance gate

The final epic gate must include a documented script/matrix for a human to perform at least:

create a simple command entirely by voice;
invoke it immediately;
create or use a parameterized command;
trigger a missing-slot question and answer it naturally;
correct an argument;
cancel a pending command;
inspect/describe the authored command;
edit it;
restart client/server and prove persistence;
delete it;
prove another principal does not receive it;
prove a device capability executes on the correct device, not inside the server/container;
prove server-side capability executes server-side;
prove unavailable/denied capability fails explicitly;
verify realistic voice latency and no stale TTS/action after cancellation.

The final gate should emit a human-readable checklist/evidence record but must not fabricate a PASS if the human did not perform it.

RAGE/TDD requirements

Each child issue follows repository RAGE: exact immutable starting commit and consumed issue in the Org log; adversarial research first; evidence-derived design; tests first with expected red; minimum coherent implementation; focused/full Nix gates; changed-code coverage inspection; exact-head Actions; merge only green/mergeable exact head.

Ordered slices

versioned principal-scoped user-command schema/persistence;
validated compiler from authored definition -> #150 semantic/capability structures;
voice/text CRUD authoring dialogue + dry-run/confirmation/undo;
parameterized commands, collisions, hot reload, migration/revocation behavior;
declarative real-speech fixture manifest;
zara-record-voice-fixtures recorder/validator;
checked-in representative recorded corpus and deterministic STT/dialogue integration gate;
manual live human programmable-command voice release gate.

Completion rule

A user can teach Zara a safe semantic command by voice, use it immediately and after restart, parameterize/clarify/edit/delete it, and have that behavior proven by text tests, real recorded speech fixtures, and an explicitly human-operated final voice gate without introducing arbitrary code execution or cross-principal state leakage.

## Goal Make Zara **programmable by voice** without creating an unsafe macro language or a second command system. A user must be able to create, inspect, edit, parameterize, test, undo, and delete commands through normal Zara dialogue; authored commands compile into the same semantic intent/slot/capability model owned by #150. At the same time, establish a repository-native **real speech fixture acquisition system** so every voice-applicable intent/tool/capability can be regression-tested with realistic user-recorded audio, while keeping final live human acceptance separate and genuinely manual. ## Dependencies - Depends on #150 semantic IntentFrame/dialogue/capability architecture. - Voice transport/integration depends on #132. - Supported daemon clients depend on #133. - Multi-user private persistence depends on #131. - Final daemon release remains #134; this epic adds a later programmable-command gate and must not weaken #134. - No Prolog-RLM. ## Core principle User-authored commands are **declarative semantic programs**, not arbitrary shell/Python/Prolog text dictated by the user. Bad: ```text trigger: work mode shell: "emacs & firefox" ``` Target concept: ```text trigger: work_mode actions: - capability: open_app args: {app: emacs} location: initiating_device - capability: open_app args: {app: firefox} location: initiating_device ``` The persisted representation must contain only versioned, validated intent/slot/capability data and references to registered providers. It may not smuggle executable source through generic fields. ## Voice authoring contract Zara must support semantic authoring intents at minimum: - `create_command`; - `edit_command`; - `delete_command`; - `list_commands`; - `describe_command`; - `test_command` / dry-run where useful; - `undo_command_change` for the most recent safe mutation. Example interaction: ```text User: Zara, create a new command. Zara: What should you say? User: Work mode. Zara: What should work mode do? User: Open Emacs and Firefox. Zara: Run those actions on the device that invoked the command? User: Yes. Zara: Work mode will open Emacs and Firefox on the initiating device. Save it? User: Yes. Zara: Created command "work mode". ``` The authoring dialogue must itself use the typed missing-slot/state machinery from #150. No separate ad-hoc conversation state. ## Parameterized commands Authored commands may define typed slots, for example: ```text "focus timer for {duration}" ``` or a trigger with a required slot that can be collected later: ```text User: Zara, focus timer. Zara: How long? User: Thirty five minutes. ``` The same duration parser/validator used by built-in timer intents should fill the authored command. User-command slots must not implement parallel parsing logic. Required slot types should include only types with existing or explicitly added validators: duration, bounded string/text, contact/entity reference, URI where appropriate, number, enum, and other reviewed types. Never silently infer an unsafe raw-command/string execution type. ## Principal/device ownership - definitions are principal-owned; - one principal must not enumerate/infer/use another principal's private authored commands; - device-local variants/grants may be scoped below the principal where required; - command IDs are stable and not bearer capabilities; - reconnect/restart preserves durable definitions; - guest/ephemeral principals cannot accidentally create durable commands unless an explicit promotion flow exists; - deletion/export affect only the owning principal. ## Conflict and mutation policy Prove safe behavior for: - exact trigger collision with built-ins; - aliases that create ambiguous resolution; - collision between two user commands; - edits that invalidate stored slot/action schema; - provider removed/unavailable after command creation; - permissions revoked after creation; - partial write/crash during mutation; - rollback/undo; - schema migration across command-definition versions. Protected built-ins must not be silently shadowed. Zara should explain the conflict and offer a safe alternative trigger or explicit supported override policy if research justifies one. ## Hot reload A successfully persisted command should become usable without restarting the daemon/client. Reload must be atomic: readers see either the old valid registry or the new valid registry, never a partially parsed definition set. ## Future teach-by-demonstration compatibility Do not require demonstration recording in the first implementation, but preserve an architecture where a future authoring flow can capture **typed capabilities that Zara observed executing**, then propose a semantic command. Never record literal arbitrary shell as the demonstration artifact. ## Real speech fixture system Add a developer tool, working name: ```text zara-record-voice-fixtures ``` The recorder is a fixture acquisition tool, not the test oracle. It must read declarative test-case metadata and guide a human through recording the exact utterances the automated corpus expects. Example: ```text Case: timer_missing_duration Get ready... Say: "Zara, set a timer." [record] Wait about 1.8 seconds... Say: "Twenty minutes." [record] Saved and validated. ``` ## Declarative voice fixture manifest The same source of truth should drive both recorder prompts and tests. It must encode, subject to RAGE research: - stable case/version id; - one or more spoken utterances; - expected normalized transcript where deterministic enough; - expected semantic intent/slots/missing slots; - expected capability/provider location; - realistic pre-turn/inter-turn/post-speech delay or bounded range; - audio filename(s); - sample rate/channels/encoding; - expected dialogue transitions; - optional expected action/result fixture; - tags such as single-turn, multi-turn, cancellation, correction, device, server, tool, authored-command. Do not duplicate the phrase corpus in shell scripts. ## Recorder timing behavior The tool should make recordings sound like actual assistant use rather than concatenated lab tokens: - configurable initial get-ready delay; - realistic inter-turn delay, with deterministic/default target plus allowed jitter/range metadata; - post-utterance silence sufficient for VAD fixtures; - visible/audible countdown or cue where appropriate; - repeat/re-record one step without destroying previous accepted fixtures; - bounded total case time; - no network requirement. Tests must not depend on wall-clock sleeps where a virtual clock/recorded timing metadata can deterministically model the protocol/dialogue behavior. The sleep is for **human recording guidance**, not flaky CI. ## Audio validation Before committing a fixture, validate at least: - non-empty recording; - supported PCM/container format; - expected sample rate/channel contract or deterministic conversion; - plausible duration bounds; - peak/RMS sanity; - clipping/saturation warning/failure threshold; - near-silence warning/failure threshold; - no NaN/invalid PCM; - bounded file size; - explicit replacement confirmation for existing fixture. Keep raw fixture audio free of secrets/private conversations: the tool prompts only the declared synthetic/test phrase. ## Fixture classes Support at least: 1. single-utterance fixtures; 2. multi-turn fixtures recorded as separate utterances while preserving timing metadata; 3. optional continuous-session recordings for VAD/dialogue integration; 4. corrections/cancellation; 5. authored-command invocation; 6. server tool/API capability commands; 7. device action commands; 8. unavailable/denied capability cases where no dangerous real action is required. ## Mandatory example corpus At minimum record/test representative versions of: - `Zara, set a timer for twenty seconds.` - `Zara, set a timer.` -> later `Twenty seconds.` - `Actually, make that five minutes.` - `Never mind.` - `Zara, open Firefox.` - `Zara, open.` -> later `Firefox.` - `Zara, search for ZeroMQ CURVE authentication.` - `Zara, take a screenshot.` - `Zara, what do you remember about the test fact?` - a safe fake/tool fixture; - command creation dialogue; - immediate invocation of the newly created command; - parameterized user command with missing-slot follow-up; - edit/describe/delete command flows. Every newly shipped voice-applicable intent, capability, tool, or authored-command behavior must add representative text fixtures and a recorder-consumable case unless there is a documented reason voice cannot apply. ## Three-level verification model ### Level A: text/property CI Deterministic text fixtures prove parser/dialogue/capability semantics without STT noise. ### Level B: checked-in recorded fixture CI Human-recorded repository fixtures exercise the real VAD/STT -> semantic path using deterministic/local models or approved fixture transcription strategies. No microphone required in CI. ### Level C: manual live human gate This remains intentionally manual. The user runs real Zara on real hardware. An assistant/test guide supplies exact phrases one at a time plus expected observable behavior. Results/evidence may be recorded afterward, but the test must not be relabeled automated merely because a script prints prompts. ## Final human voice acceptance gate The final epic gate must include a documented script/matrix for a human to perform at least: 1. create a simple command entirely by voice; 2. invoke it immediately; 3. create or use a parameterized command; 4. trigger a missing-slot question and answer it naturally; 5. correct an argument; 6. cancel a pending command; 7. inspect/describe the authored command; 8. edit it; 9. restart client/server and prove persistence; 10. delete it; 11. prove another principal does not receive it; 12. prove a device capability executes on the correct device, not inside the server/container; 13. prove server-side capability executes server-side; 14. prove unavailable/denied capability fails explicitly; 15. verify realistic voice latency and no stale TTS/action after cancellation. The final gate should emit a human-readable checklist/evidence record but must not fabricate a PASS if the human did not perform it. ## RAGE/TDD requirements Each child issue follows repository RAGE: exact immutable starting commit and consumed issue in the Org log; adversarial research first; evidence-derived design; tests first with expected red; minimum coherent implementation; focused/full Nix gates; changed-code coverage inspection; exact-head Actions; merge only green/mergeable exact head. ## Ordered slices 1. versioned principal-scoped user-command schema/persistence; 2. validated compiler from authored definition -> #150 semantic/capability structures; 3. voice/text CRUD authoring dialogue + dry-run/confirmation/undo; 4. parameterized commands, collisions, hot reload, migration/revocation behavior; 5. declarative real-speech fixture manifest; 6. `zara-record-voice-fixtures` recorder/validator; 7. checked-in representative recorded corpus and deterministic STT/dialogue integration gate; 8. manual live human programmable-command voice release gate. ## Completion rule A user can teach Zara a safe semantic command by voice, use it immediately and after restart, parameterize/clarify/edit/delete it, and have that behavior proven by text tests, real recorded speech fixtures, and an explicitly human-operated final voice gate without introducing arbitrary code execution or cross-principal state leakage.

lost-rob0t commented

2026-08-22 21:59:50 +00:00

(Migrated from github.com)

Canonical child issue map

Consume in order after the #150 semantic contracts are available:

#162 — principal-scoped versioned user-command persistence
#163 — validated compiler into ordinary IntentFrame/capability structures
#164 — voice/text create/list/describe/edit/delete/dry-run/undo dialogue
#165 — parameterized commands, collisions, revocation and atomic hot reload
#166 — declarative realistic voice-fixture manifest
#167 — zara-record-voice-fixtures guided recorder with realistic timing/audio validation
#168 — checked-in real speech corpus through VAD/STT -> dialogue/capability regression path
#169 — explicitly human live-voice programmable-command release gate

The checked-in audio gate never substitutes for #169. Do not invent a parallel macro/recording implementation from this parent when a child owns it.

## Canonical child issue map Consume in order after the #150 semantic contracts are available: - [ ] #162 — principal-scoped versioned user-command persistence - [ ] #163 — validated compiler into ordinary `IntentFrame`/capability structures - [ ] #164 — voice/text create/list/describe/edit/delete/dry-run/undo dialogue - [ ] #165 — parameterized commands, collisions, revocation and atomic hot reload - [ ] #166 — declarative realistic voice-fixture manifest - [ ] #167 — `zara-record-voice-fixtures` guided recorder with realistic timing/audio validation - [ ] #168 — checked-in real speech corpus through VAD/STT -> dialogue/capability regression path - [ ] #169 — explicitly human live-voice programmable-command release gate The checked-in audio gate never substitutes for #169. Do not invent a parallel macro/recording implementation from this parent when a child owns it.