Add observe-only quant anomaly detection and safe alerts #62

Closed

nsaspy wants to merge 4 commits from rage/58-60-quant-anomaly-alerts-r3 into rage/59-init-files-r3

nsaspy commented

2026-09-07 03:44:56 +00:00

Owner

Part of #58 and #60. Stacked on #63.

Policy

Quant anomaly detection is disabled by default in llm-log core. This slice observes and emits evidence; it does not change OpenRouter routing and does not block requests.

Detectors

invalid UTF-8 / replacement-character corruption;
unexpected control characters in assistant output;
repeated exact token-block reasoning loops;
unknown tool names relative to the request tool set;
malformed streamed tool argument JSON;
tool arguments that decode to a non-object.

Streamed tool calls are reconstructed by tool_call.index before validation so parallel calls do not generate the false malformed-JSON signal seen with naive chunk concatenation.

Derived evidence

anomalies.jsonl: versionable detector evidence linked by event ID;
alerts.jsonl: safe metadata only — event ID, severity/category, model, provider/selected provider, disclosed quantization, detector IDs;
no prompt/completion/auth/tool-argument contents in desktop-alert payloads.

Exact precision is never inferred from output behavior. If router metadata discloses FP4/INT4/etc. it is grounded; otherwise quantization remains unknown and behavioral anomalies stay a separate signal.

RED-first structure

Detector + safe-alert contracts were committed before implementation.

Validation status

Branch/diff structure is audited. Full local unittest and Nix gates are still required before leaving draft.

Part of #58 and #60. Stacked on #63. ## Policy Quant anomaly detection is **disabled by default in llm-log core**. This slice observes and emits evidence; it does not change OpenRouter routing and does not block requests. ## Detectors - invalid UTF-8 / replacement-character corruption; - unexpected control characters in assistant output; - repeated exact token-block reasoning loops; - unknown tool names relative to the request tool set; - malformed streamed tool argument JSON; - tool arguments that decode to a non-object. Streamed tool calls are reconstructed by `tool_call.index` before validation so parallel calls do not generate the false malformed-JSON signal seen with naive chunk concatenation. ## Derived evidence - `anomalies.jsonl`: versionable detector evidence linked by event ID; - `alerts.jsonl`: safe metadata only — event ID, severity/category, model, provider/selected provider, disclosed quantization, detector IDs; - no prompt/completion/auth/tool-argument contents in desktop-alert payloads. Exact precision is never inferred from output behavior. If router metadata discloses FP4/INT4/etc. it is grounded; otherwise quantization remains `unknown` and behavioral anomalies stay a separate signal. ## RED-first structure Detector + safe-alert contracts were committed before implementation. ## Validation status Branch/diff structure is audited. Full local unittest and Nix gates are still required before leaving draft.

nsaspy added 4 commits

2026-09-07 03:44:57 +00:00