nsaspy/llm-log

Fork 0

Observe OpenRouter provider and quantization metadata #64

Open

nsaspy wants to merge 7 commits from rage/57-openrouter-quant-observability-r2 into main

nsaspy commented

2026-09-07 03:44:57 +00:00

Owner

Part of #57.

Why

A one-hour user capture on 2026-09-06 contained 252 OpenRouter calls with zero requests specifying provider.quantizations. The same model slugs were served across multiple provider endpoints, so the model slug alone cannot identify serving precision.

What this adds

opt only OpenRouter requests into X-OpenRouter-Metadata: enabled at the aiohttp middleware edge;
keep the existing proxy transport implementation unchanged;
parse normal JSON and SSE responses for the actual serving provider;
persist derived routing observations in routing.jsonl, linked by immutable capture event_id;
record quantization only when the response/router metadata actually discloses it;
use quantization = "unknown" when it is undisclosed;
never infer FP4/INT4/FP8 from provider reputation;
keep raw events.jsonl unchanged.

RED-first structure

Contract tests were committed before the corresponding implementation, including a dedicated metadata-header opt-in contract.

Validation status

Branch/diff structure has been audited. Full local python -m unittest discover -s tests -v and nix flake check still need to run in a normal checkout before this leaves draft; no green claim is made here.

Part of #57. ## Why A one-hour user capture on 2026-09-06 contained 252 OpenRouter calls with zero requests specifying `provider.quantizations`. The same model slugs were served across multiple provider endpoints, so the model slug alone cannot identify serving precision. ## What this adds - opt only OpenRouter requests into `X-OpenRouter-Metadata: enabled` at the aiohttp middleware edge; - keep the existing proxy transport implementation unchanged; - parse normal JSON and SSE responses for the actual serving `provider`; - persist derived routing observations in `routing.jsonl`, linked by immutable capture `event_id`; - record quantization only when the response/router metadata actually discloses it; - use `quantization = "unknown"` when it is undisclosed; - never infer FP4/INT4/FP8 from provider reputation; - keep raw `events.jsonl` unchanged. ## RED-first structure Contract tests were committed before the corresponding implementation, including a dedicated metadata-header opt-in contract. ## Validation status Branch/diff structure has been audited. Full local `python -m unittest discover -s tests -v` and `nix flake check` still need to run in a normal checkout before this leaves draft; no green claim is made here.

nsaspy added 7 commits

2026-09-07 03:44:57 +00:00

test: define RED OpenRouter quant observation contract 097ad466a3

feat: parse grounded OpenRouter routing observations 9a307b6811

feat: persist OpenRouter routing observations e53d1c09d7

feat: record OpenRouter routing metadata separately 6e237e4c65

test: define RED OpenRouter router-metadata opt-in contract 2832272990

feat: opt OpenRouter requests into router metadata 8f31d3bebb

feat: enable OpenRouter router metadata at proxy edge 6cc10ae904

nsaspy referenced this pull request from a commit

2026-09-11 04:25:22 +00:00

Add observe-only quant anomaly detection and safe alerts (#64)

This pull request has changes conflicting with the target branch.

llm_log/cli.py

View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Merge commit title

Merge pull request 'Observe OpenRouter provider and quantization metadata' (#64) from rage/57-openrouter-quant-observability-r2 into main

Merge commit body

Reviewed-on: https://git.starintel.actor/nsaspy/llm-log/pulls/64

Merge pull request 'Observe OpenRouter provider and quantization metadata' (#64) from rage/57-openrouter-quant-observability-r2 into main

Reviewed-on: https://git.starintel.actor/nsaspy/llm-log/pulls/64

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin rage/57-openrouter-quant-observability-r2:rage/57-openrouter-quant-observability-r2

git switch rage/57-openrouter-quant-observability-r2

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff rage/57-openrouter-quant-observability-r2

git switch rage/57-openrouter-quant-observability-r2

git rebase main

git switch main

git merge --ff-only rage/57-openrouter-quant-observability-r2

git switch rage/57-openrouter-quant-observability-r2

git rebase main

git switch main

git merge --no-ff rage/57-openrouter-quant-observability-r2

git switch main

git merge --squash rage/57-openrouter-quant-observability-r2