Quant observability: record selected OpenRouter endpoint + disclosed quantization #65

New issue

Open

opened 2026-09-08 13:09:50 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-08 13:09:50 +00:00

Owner

Goal

Make endpoint precision observable before attempting behavioral inference.

Evidence

A one-hour user capture on 2026-09-06 contained 252 OpenRouter events and zero requests with provider.quantizations. The same model slugs were served by multiple providers, including GMICloud, Z.AI, Phala, AtlasCloud, Crusoe, Friendli, Together and Wafer.

OpenRouter supports X-OpenRouter-Metadata: enabled to expose selected provider, candidate endpoints, retries/fallbacks and pipeline metadata. Its model-endpoints API exposes per-endpoint serving metadata including disclosed quantization.

Contract

For OpenRouter requests, llm-log should:

opt into router metadata by default for observability without changing routing policy;
record selected provider, successful attempt number, fallback attempts and router pipeline summary as derived metadata;
resolve the selected model/provider against endpoint metadata and record disclosed quantization exactly as reported;
represent missing disclosure as unknown, never infer a bit width from provider reputation;
snapshot the metadata source/time so historical captures remain auditable if a provider changes precision later;
keep raw capture immutable and persist precision/routing observations separately with provenance;
never persist the OpenRouter bearer token while querying metadata.

Acceptance

streaming and non-streaming OpenRouter responses retain router metadata;
a known quant endpoint records its exact reported quantization;
undisclosed quantization records unknown;
a retry/fallback records all attempts and the selected endpoint;
endpoint metadata is cached with bounded TTL and failure does not break proxying;
fixtures prove model slug alone is insufficient to identify serving precision.

Non-goal

Do not claim output-only detection can identify exact FP4/INT4/FP8 precision.

## Goal Make endpoint precision observable before attempting behavioral inference. ## Evidence A one-hour user capture on 2026-09-06 contained 252 OpenRouter events and zero requests with `provider.quantizations`. The same model slugs were served by multiple providers, including GMICloud, Z.AI, Phala, AtlasCloud, Crusoe, Friendli, Together and Wafer. OpenRouter supports `X-OpenRouter-Metadata: enabled` to expose selected provider, candidate endpoints, retries/fallbacks and pipeline metadata. Its model-endpoints API exposes per-endpoint serving metadata including disclosed quantization. ## Contract For OpenRouter requests, llm-log should: - opt into router metadata by default for observability without changing routing policy; - record selected provider, successful attempt number, fallback attempts and router pipeline summary as derived metadata; - resolve the selected model/provider against endpoint metadata and record disclosed quantization exactly as reported; - represent missing disclosure as `unknown`, never infer a bit width from provider reputation; - snapshot the metadata source/time so historical captures remain auditable if a provider changes precision later; - keep raw capture immutable and persist precision/routing observations separately with provenance; - never persist the OpenRouter bearer token while querying metadata. ## Acceptance - streaming and non-streaming OpenRouter responses retain router metadata; - a known quant endpoint records its exact reported quantization; - undisclosed quantization records `unknown`; - a retry/fallback records all attempts and the selected endpoint; - endpoint metadata is cached with bounded TTL and failure does not break proxying; - fixtures prove model slug alone is insufficient to identify serving precision. ## Non-goal Do not claim output-only detection can identify exact FP4/INT4/FP8 precision.

nsaspy referenced this issue

2026-09-19 05:30:22 +00:00

Upstream exhaustion failover: route depleted sub-provider proxies through OpenRouter #88