Transport error classifier: distinguish client disconnects, upstream resets, router failures and model anomalies #56

New issue

Open

opened 2026-09-07 03:14:17 +00:00 by nsaspy · 1 comment

nsaspy commented

2026-09-07 03:14:17 +00:00

Owner

Goal

Turn proxy/journal failures into typed error evidence instead of noisy generic tracebacks.

Observed evidence

The 2026-09-06 one-hour capture contains repeated aiohttp.client_exceptions.ClientConnectionResetError: Cannot write to closing transport while streaming to localhost, plus two captured HTTP 502 events whose upstream body is [Errno 104] Connection reset by peer.

These are materially different failure domains and must not be confused with quantization/model-quality anomalies.

Classes

At minimum:

downstream/client disconnected while llm-log was writing;
upstream provider connection reset;
OpenRouter/router error with metadata;
HTTP 4xx/5xx provider error;
stream framing/protocol error;
model/structured-output anomaly from #58;
unknown transport failure.

Acceptance

client disconnect does not dump an unhandled traceback for an otherwise expected cancellation path;
upstream reset remains a recorded failed capture with exact class and status;
error records link to event/provider/model/router attempts where available;
alert severity/rate policy can differ by class;
transport failures never get mislabeled as quantization detection.

## Goal Turn proxy/journal failures into typed error evidence instead of noisy generic tracebacks. ## Observed evidence The 2026-09-06 one-hour capture contains repeated `aiohttp.client_exceptions.ClientConnectionResetError: Cannot write to closing transport` while streaming to localhost, plus two captured HTTP 502 events whose upstream body is `[Errno 104] Connection reset by peer`. These are materially different failure domains and must not be confused with quantization/model-quality anomalies. ## Classes At minimum: - downstream/client disconnected while llm-log was writing; - upstream provider connection reset; - OpenRouter/router error with metadata; - HTTP 4xx/5xx provider error; - stream framing/protocol error; - model/structured-output anomaly from #58; - unknown transport failure. ## Acceptance - client disconnect does not dump an unhandled traceback for an otherwise expected cancellation path; - upstream reset remains a recorded failed capture with exact class and status; - error records link to event/provider/model/router attempts where available; - alert severity/rate policy can differ by class; - transport failures never get mislabeled as quantization detection.

nsaspy commented

2026-09-08 17:43:30 +00:00

Author

Owner

Full-day census to size each class (24h slice 2026-09-06T03:39Z → 09-07T03:39Z, 6,430 events, all POST /api/v1/chat/completions via OpenRouter):

class	count	where it surfaces
downstream/client disconnect	426 journal tracebacks (`ClientConnectionResetError` at proxy.py:486)	journal only — zero CaptureEvents (raise path skips recording)
upstream pre-connect reset	23 events, 502, body `[Errno 104] Connection reset by peer`	events.jsonl
in-stream provider error	1 event: OpenRouter error frame `{"code":504,"message":"Upstream idle timeout exceeded"}` with `finish_reason:"error"` inside a 200	events.jsonl
silent SSE truncation	5 events: status 200, no `[DONE]`, no finish_reason, clean EOF mid-generation (no journal exception at those timestamps)	events.jsonl
rate/credit limiting	11 × 403 `Budget limit exceeded (weekly limit)` — burst 18:59–19:00Z hitting two different models in the same minute ⇒ account-scoped, not provider	events.jsonl
identical-retry recovery	24 chains: byte-identical `request_sha256` ≤7s later, 100% recovered on another provider	cross-event

Key attribution traps for the classifier:

Client disconnects are invisible in events.jsonl by construction — they must be ingested from the journal (a separate evidence pool), and must never enter provider failure rates.
Silent SSE truncation (class above) is currently indistinguishable from a completed 200 without body-parsing; needs the completeness flag from #67.
A recorded 502 has provider=null (pre-connect) — a recorded 200 with an in-stream error frame has the real provider name per-chunk. Same "upstream error", two shapes.
The 403 burst is account-level: scoping it to a model/provider would be wrong.
Retries are the router's fallback signal: retry-succeeded ⇒ self-healed (log), retry-failed ⇒ alert-worthy.

Detector proposals with per-event IDs, confidence and FP analysis are in the 24h research report (staged at /tmp/opencode/llmlog24h/llm-log-failure-research.md).

Full-day census to size each class (24h slice 2026-09-06T03:39Z → 09-07T03:39Z, 6,430 events, all `POST /api/v1/chat/completions` via OpenRouter): | class | count | where it surfaces | |---|---|---| | downstream/client disconnect | 426 journal tracebacks (`ClientConnectionResetError` at proxy.py:486) | **journal only — zero CaptureEvents** (raise path skips recording) | | upstream pre-connect reset | 23 events, 502, body `[Errno 104] Connection reset by peer` | events.jsonl | | in-stream provider error | 1 event: OpenRouter error frame `{"code":504,"message":"Upstream idle timeout exceeded"}` with `finish_reason:"error"` **inside a 200** | events.jsonl | | silent SSE truncation | 5 events: status 200, no `[DONE]`, no finish_reason, clean EOF mid-generation (no journal exception at those timestamps) | events.jsonl | | rate/credit limiting | 11 × 403 `Budget limit exceeded (weekly limit)` — burst 18:59–19:00Z hitting two different models in the same minute ⇒ **account-scoped**, not provider | events.jsonl | | identical-retry recovery | 24 chains: byte-identical `request_sha256` ≤7s later, 100% recovered on another provider | cross-event | Key attribution traps for the classifier: 1. Client disconnects are invisible in events.jsonl by construction — they must be ingested from the journal (a separate evidence pool), and must never enter provider failure rates. 2. Silent SSE truncation (class above) is currently indistinguishable from a completed 200 without body-parsing; needs the completeness flag from #67. 3. A recorded 502 has `provider=null` (pre-connect) — a recorded 200 with an in-stream error frame has the real provider name per-chunk. Same "upstream error", two shapes. 4. The 403 burst is account-level: scoping it to a model/provider would be wrong. 5. Retries are the router's fallback signal: retry-succeeded ⇒ self-healed (log), retry-failed ⇒ alert-worthy. Detector proposals with per-event IDs, confidence and FP analysis are in the 24h research report (staged at `/tmp/opencode/llmlog24h/llm-log-failure-research.md`).