Expert plane never receives responses: observe_response/observe_usage are declared in the protocol but never called by the proxy #66

New issue

Open

opened 2026-09-08 17:42:07 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-08 17:42:07 +00:00

Owner

Summary

llm_log/expert_protocol.py declares observe_response and observe_usage as allowed observation operations, but proxy.py only ever calls observe_request + classify_request (_expert_ingest, proxy.py:183-207, both HTTP and WS paths). The expert plane's graph (LMDB graph/v2 store) therefore accumulates request-side intent observations with zero outcome evidence — no response, no usage, no completion status, ever.

Why this matters (24h corpus, 6,430 events)

Sessions span days: a tool call emitted 2026-09-05T01:25Z (5e61eb9b…) had its failing result (SBCL fatal ERROR) consumed in a request >26h later. A request-only graph cannot connect "what was asked" to "what happened".
The 24h failure surface is 23 upstream resets, 11 × 403 budget-limit 403s, 7 incomplete/error SSE streams, 24 identical-retry recovery chains — none of it is observable to the expert plane today.
The outcome-labeling effort (#15) needs exactly this evidence; until the proxy sends it, labels would be derived from... nothing on the response side.

Suggested fix

After recorder.record(event) in the HTTP path (and WS success path), call observe_response with a safe payload mirroring _expert_base_payload + response_status, latency_ms, response_sha256, and (when parseable) finish_reason / usage — no raw bodies by default, consistent with #60's payload contract.
Call observe_usage when the final SSE chunk carries usage (already parsed trivially from the last frame).
Keep it fire-and-forget like _expert_ingest (exceptions swallowed, never block the relay), and make sure the WS path uses the fixed expert_plane threading from #66.

Regression fixture

Proxy a mocked streaming upstream end-to-end with the expert subprocess stubbed; assert one observe_request, one classify_request, one observe_response (and one observe_usage when usage is present) per request, and none on the client-abort path.

## Summary `llm_log/expert_protocol.py` declares `observe_response` and `observe_usage` as allowed observation operations, but `proxy.py` only ever calls `observe_request` + `classify_request` (`_expert_ingest`, proxy.py:183-207, both HTTP and WS paths). The expert plane's graph (LMDB `graph/v2` store) therefore accumulates request-side intent observations with **zero outcome evidence** — no response, no usage, no completion status, ever. ## Why this matters (24h corpus, 6,430 events) - Sessions span days: a tool call emitted 2026-09-05T01:25Z (`5e61eb9b…`) had its failing result (SBCL `fatal ERROR`) consumed in a request >26h later. A request-only graph cannot connect "what was asked" to "what happened". - The 24h failure surface is 23 upstream resets, 11 × 403 budget-limit 403s, 7 incomplete/error SSE streams, 24 identical-retry recovery chains — none of it is observable to the expert plane today. - The outcome-labeling effort (#15) needs exactly this evidence; until the proxy sends it, labels would be derived from... nothing on the response side. ## Suggested fix - After `recorder.record(event)` in the HTTP path (and WS success path), call `observe_response` with a safe payload mirroring `_expert_base_payload` + response_status, latency_ms, response_sha256, and (when parseable) finish_reason / usage — **no raw bodies** by default, consistent with #60's payload contract. - Call `observe_usage` when the final SSE chunk carries `usage` (already parsed trivially from the last frame). - Keep it fire-and-forget like `_expert_ingest` (exceptions swallowed, never block the relay), and make sure the WS path uses the fixed expert_plane threading from #66. ## Regression fixture Proxy a mocked streaming upstream end-to-end with the expert subprocess stubbed; assert one `observe_request`, one `classify_request`, one `observe_response` (and one `observe_usage` when usage is present) per request, and none on the client-abort path.