nsaspy/llm-log

Fork 0

Correlate router attempts and retry recovery #104

Open

nsaspy wants to merge 17 commits from fix/61-retry-routing-correlation into fix/61-transport-error-evidence

nsaspy commented

2026-09-20 06:13:36 +00:00

Owner

Tracks the remaining correlation/recovery part of #61.

Stacked on #88 (fix/61-transport-error-evidence) so routing/recovery evidence reuses the typed transport taxonomy without bloating that slice.

What changed

opt only OpenRouter upstream traffic into its documented X-OpenRouter-Metadata: enabled response metadata;
strip client-supplied copies of that OpenRouter-specific header from non-OpenRouter upstreams;
project safe append-only routing.jsonl records with stable routing:<event-id> IDs;
retain only selected provider and bounded attempt provider/status/error-code fields; raw messages, pipeline detail, endpoint blobs, model labels inside attempts and quantization data are not persisted;
parse selected provider from either the response provider field or documented openrouter_metadata.endpoints.available[].selected metadata;
link typed transport errors to a routing observation only after the routing projection is durably appended;
add a derived CorrelationActor that owns routing.jsonl and recoveries.jsonl while RecorderActor remains the owner of raw captures and typed errors;
correlate the nearest later clean success with the same request_sha256 inside a 10-second window;
persist failed/retry event IDs, routing IDs, retry delay and disclosed provider change;
label recovery explicitly with correlation_basis=byte_identical_request_sha256_within_window and confidence=heuristic so it cannot be mistaken for causal proof;
rebuild pending failures from append-only routing/recovery/error evidence after process restart;
reject false correlations for hash mismatch, expired window, client disconnects and already-consumed failures;
record the verified invariants in Org-roam and the project Prolog KB.

OpenRouter's current public API reference confirms the metadata header on Chat Completions, Messages and Responses:

RED evidence

Exact test-only head 5e7a697137f52bd51f92bda08cf5e994c579872f ran 115 tests. Exactly five new correlation expectations failed:

OpenRouter metadata opt-in was absent;
safe routing observations were absent;
typed errors had no routing link;
byte-identical retry recovery was absent;
pending failures did not recover after recorder restart.

All existing #86 stream-integrity, #87 response/usage and #88 typed-transport contracts stayed green.

GREEN evidence

Live wiring head 4fb8d76120426f33827682ca81c335661ba2159b: Analytics workflow passed after all five RED cases flipped green.
Hardened code/test head 80c9e5a4642650ec4d6236866c1c254e1f870266: Analytics run 62 and Quota run 67 both passed, including full Python regression, adversarial analytics and delivered package build. This head includes false-positive contracts, current OpenRouter endpoint-metadata parsing and explicit heuristic provenance.
Final exact head 351d11f4f8652e0bb078fa038f4cf3961faebda6: Analytics run 67 and Quota run 72 both passed after Org-roam/Prolog knowledge promotion; quota again built the delivered package.

Boundary

This slice observes router attempts and likely retry recovery; it does not perform retries. The retry relationship is deliberately heuristic, not causal identity. Alert severity/rate policy and model/structured-output anomaly detection remain separate follow-up work. No merge is performed here.

Tracks the remaining correlation/recovery part of #61. Stacked on #88 (`fix/61-transport-error-evidence`) so routing/recovery evidence reuses the typed transport taxonomy without bloating that slice. ## What changed - opt only OpenRouter upstream traffic into its documented `X-OpenRouter-Metadata: enabled` response metadata; - strip client-supplied copies of that OpenRouter-specific header from non-OpenRouter upstreams; - project safe append-only `routing.jsonl` records with stable `routing:<event-id>` IDs; - retain only selected provider and bounded attempt provider/status/error-code fields; raw messages, pipeline detail, endpoint blobs, model labels inside attempts and quantization data are not persisted; - parse selected provider from either the response provider field or documented `openrouter_metadata.endpoints.available[].selected` metadata; - link typed transport errors to a routing observation only after the routing projection is durably appended; - add a derived `CorrelationActor` that owns `routing.jsonl` and `recoveries.jsonl` while `RecorderActor` remains the owner of raw captures and typed errors; - correlate the nearest later clean success with the same `request_sha256` inside a 10-second window; - persist failed/retry event IDs, routing IDs, retry delay and disclosed provider change; - label recovery explicitly with `correlation_basis=byte_identical_request_sha256_within_window` and `confidence=heuristic` so it cannot be mistaken for causal proof; - rebuild pending failures from append-only routing/recovery/error evidence after process restart; - reject false correlations for hash mismatch, expired window, client disconnects and already-consumed failures; - record the verified invariants in Org-roam and the project Prolog KB. OpenRouter's current public API reference confirms the metadata header on Chat Completions, Messages and Responses: - https://openrouter.ai/docs/api/api-reference/chat/send-chat-completion-request - https://openrouter.ai/docs/api/api-reference/responses/create-responses ## RED evidence Exact test-only head `5e7a697137f52bd51f92bda08cf5e994c579872f` ran 115 tests. Exactly five new correlation expectations failed: 1. OpenRouter metadata opt-in was absent; 2. safe routing observations were absent; 3. typed errors had no routing link; 4. byte-identical retry recovery was absent; 5. pending failures did not recover after recorder restart. All existing #86 stream-integrity, #87 response/usage and #88 typed-transport contracts stayed green. ## GREEN evidence - Live wiring head `4fb8d76120426f33827682ca81c335661ba2159b`: Analytics workflow passed after all five RED cases flipped green. - Hardened code/test head `80c9e5a4642650ec4d6236866c1c254e1f870266`: Analytics run 62 and Quota run 67 both passed, including full Python regression, adversarial analytics and delivered package build. This head includes false-positive contracts, current OpenRouter endpoint-metadata parsing and explicit heuristic provenance. - Final exact head `351d11f4f8652e0bb078fa038f4cf3961faebda6`: Analytics run 67 and Quota run 72 both passed after Org-roam/Prolog knowledge promotion; quota again built the delivered package. ## Boundary This slice observes router attempts and likely retry recovery; it does not perform retries. The retry relationship is deliberately heuristic, not causal identity. Alert severity/rate policy and model/structured-output anomaly detection remain separate follow-up work. No merge is performed here.

nsaspy added 17 commits

2026-09-20 06:13:36 +00:00

test: define RED retry and routing correlation contract 5e7a697137

feat: add safe OpenRouter routing observations 891925f58e

feat: add restart-durable retry recovery tracker af64e62375

feat: link transport errors to routing observations 21d4c4d867

feat: serialize routing and recovery evidence a30f160732

fix: emit routing only when provider evidence exists 6185cf50ac

feat: wire routing and recovery correlation into proxy 4fb8d76120

test: reject false retry recovery correlations 49550cb893

test: cover documented OpenRouter selected endpoint metadata 0ca8c65173

fix: parse documented selected OpenRouter endpoint 1dbb7d122f

fix: mark retry recovery correlation as heuristic evidence 62bcd4f577

test: require explicit heuristic correlation provenance 80c9e5a464

docs: record retry and routing correlation invariants 6da2e11214

docs: load retry routing correlation knowledge 36c2f0d858

docs: add retry routing correlation issue note 8c62813d94

docs: add safe router retry correlation solution 7304c6e751

docs: index retry routing correlation knowledge

Analytics API / test (pull_request) Failing after 17s

Details

Quota telemetry exact-head contract / quota-contracts (pull_request) Failing after 45s

Details

Analytics API / package (pull_request) Has been skipped

Details

351d11f4f8

Analytics API / test (pull_request) Failing after 17s

Details

Quota telemetry exact-head contract / quota-contracts (pull_request) Failing after 45s

Details

Analytics API / package (pull_request) Has been skipped

Details

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin fix/61-retry-routing-correlation:fix/61-retry-routing-correlation

git switch fix/61-retry-routing-correlation

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch fix/61-transport-error-evidence

git merge --no-ff fix/61-retry-routing-correlation

git switch fix/61-retry-routing-correlation

git rebase fix/61-transport-error-evidence

git switch fix/61-transport-error-evidence

git merge --ff-only fix/61-retry-routing-correlation

git switch fix/61-retry-routing-correlation

git rebase fix/61-transport-error-evidence

git switch fix/61-transport-error-evidence

git merge --no-ff fix/61-retry-routing-correlation

git switch fix/61-transport-error-evidence

git merge --squash fix/61-retry-routing-correlation