P0 blocker — voice agent turn crashes the wake listener: LatencyTrace is not msgpack serializable in the LangGraph checkpoint #225

New issue

Closed

opened 2026-09-02 17:27:43 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-02 17:27:43 +00:00

Owner

Observed

nix run .# -- --wake --device rocm on an AMD machine: wake word detected, voice turn routed to the agent after Prolog, LLM answered, then the entire listener died:

TypeError: Type is not msgpack serializable: LatencyTrace

Crash site: langgraph/checkpoint/memory/__init__.py → InMemorySaver.put → ormsgpack _msgpack_enc, reached via zara/wake.py query_with_fallback_async → AgentManager.process_async → run_conversation_loop (zara/agent/graph.py:741).

Root cause

#209 added a per-turn InMemorySaver checkpointer to run_conversation_loop for tool-approval interrupts.
AgentManager.process_async (zara/agent/__init__.py:212) puts the live LatencyTrace object (threading locks, clock callables, sink) into the graph state under latency_trace.
Every checkpoint put serializes every AgentState channel, and ormsgpack cannot serialize LatencyTrace.

Consequences: every voice turn that reaches the agent fallback path crashes the whole wake process at the end of the turn. Text paths pass latency_trace=None and survive, which is why this hid until voice use.

Reproduced independently at the current installed dependency set with a minimal run_conversation_loop call carrying a LatencyTrace in state — identical TypeError.

Expected

A wake voice turn completes normally; the listener keeps running.
The LatencyTrace is per-turn runtime context, not durable graph state: it must never enter checkpoint serialization. Checkpoint behavior for approvals (#209) is unchanged and thread cleanup still happens on every terminal path.
Latency events (llm_request, llm_first_token, llm_final_token, …) are still recorded exactly as today.

Acceptance criteria

Regression test: run_conversation_loop invoked with a LatencyTrace in the state completes and returns a response (currently raises TypeError).
Regression test: the turn records the same latency events as before the fix.
Existing direct-node callers that pass state["latency_trace"] (e.g. test_streaming_node_records_genuine_latency_boundaries) keep working unchanged.
Full repository gate (scripts/test-all.sh, nix flake check, nix build) green at the exact candidate head; GitHub Actions test green for the pushed SHA.

## Observed `nix run .# -- --wake --device rocm` on an AMD machine: wake word detected, voice turn routed to the agent after Prolog, LLM answered, then the entire listener died: ``` TypeError: Type is not msgpack serializable: LatencyTrace ``` Crash site: `langgraph/checkpoint/memory/__init__.py` → `InMemorySaver.put` → ormsgpack `_msgpack_enc`, reached via `zara/wake.py` `query_with_fallback_async` → `AgentManager.process_async` → `run_conversation_loop` (`zara/agent/graph.py:741`). ## Root cause - #209 added a per-turn `InMemorySaver` checkpointer to `run_conversation_loop` for tool-approval interrupts. - `AgentManager.process_async` (`zara/agent/__init__.py:212`) puts the live `LatencyTrace` object (threading locks, clock callables, sink) into the graph state under `latency_trace`. - Every checkpoint put serializes every `AgentState` channel, and ormsgpack cannot serialize `LatencyTrace`. Consequences: every voice turn that reaches the agent fallback path crashes the whole wake process at the end of the turn. Text paths pass `latency_trace=None` and survive, which is why this hid until voice use. Reproduced independently at the current installed dependency set with a minimal `run_conversation_loop` call carrying a `LatencyTrace` in state — identical TypeError. ## Expected - A wake voice turn completes normally; the listener keeps running. - The `LatencyTrace` is per-turn runtime context, not durable graph state: it must never enter checkpoint serialization. Checkpoint behavior for approvals (#209) is unchanged and thread cleanup still happens on every terminal path. - Latency events (`llm_request`, `llm_first_token`, `llm_final_token`, …) are still recorded exactly as today. ## Acceptance criteria 1. Regression test: `run_conversation_loop` invoked with a `LatencyTrace` in the state completes and returns a response (currently raises `TypeError`). 2. Regression test: the turn records the same latency events as before the fix. 3. Existing direct-node callers that pass `state["latency_trace"]` (e.g. `test_streaming_node_records_genuine_latency_boundaries`) keep working unchanged. 4. Full repository gate (`scripts/test-all.sh`, `nix flake check`, `nix build`) green at the exact candidate head; GitHub Actions `test` green for the pushed SHA.

nsaspy closed this issue

2026-09-02 17:27:43 +00:00

nsaspy referenced this issue from a commit

2026-09-04 22:28:49 +00:00

Merge pull request #225 from lost-rob0t/docs/rage-17-merge-outcome