nsaspy/llm-log

Fork 0

Upstream exhaustion failover: route depleted sub-provider proxies through OpenRouter #88

New issue

Open

opened 2026-09-19 05:30:22 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-19 05:30:22 +00:00

Owner

Parent: #94
Related: #59 #61 #65 #67
Cross-repo economics: starintel-labs/starintel-biz#110

Goal

Add a first-class upstream exhaustion fallback policy to llm-log.

When a configured direct/sub-provider upstream cannot accept more work because its quota, credits, rate window, or configured capacity is exhausted, llm-log may transparently retry the same logical inference through the configured OpenRouter upstream.

This must be implemented as a generic routing-policy primitive with OpenRouter as the initial fallback target, not as ad-hoc special-case retry code.

Core routing path

client
  |
  v
llm-log
  |
  +--> preferred direct/sub-provider proxy
          |
          +-- success -------------------------------> client
          |
          +-- classified exhaustion before commit
                    |
                    v
              routing policy
                    |
                    +--> OpenRouter
                          |
                          +--> same model/provider family where configured
                          +--> OpenRouter provider failover
                          +--> optional configured model fallback

What counts as "ran out"

Use the typed error classifier rather than raw status codes alone.

Default fallback-eligible classes:

local preflight quota/capacity exhausted
provider credit exhausted / payment-required-for-more-capacity
provider rate quota exhausted
provider concurrency/capacity limit where retrying the same direct endpoint is not useful

Typical HTTP evidence may include 402 or 429. A 403 only counts when a provider adapter/classifier proves it means quota/capacity exhaustion.

Do not treat generic authentication, permission, policy, malformed-request, context-length, or safety/moderation errors as "ran out".

5xx/network failure may be enabled by separate policy, but is not part of the default exhaustion trigger.

Never replay after downstream commit

Fallback is only legal while llm-log still owns the full response decision and has not committed response bytes to the downstream client.

For streaming/SSE/WebSocket requests:

if the preferred upstream rejects before llm-log emits the downstream response head/body, fallback is allowed;
once any response body/token/frame has been emitted, do not restart the request through OpenRouter;
record a typed mid-stream failure and let the caller decide whether to retry.

This avoids duplicate/divergent completions, tool-call duplication, and hidden double spend.

Credential boundary

Current transparent proxy behavior forwards the caller's provider credential. OpenRouter fallback requires a separate server-side credential.

Requirements:

OpenRouter API key comes from operator secret storage / environment, never TOML plaintext and never capture output;
when falling back, strip the original provider Authorization / provider-specific credential headers;
inject only the configured OpenRouter credential into the OpenRouter attempt;
never forward the direct provider's bearer token to OpenRouter;
downstream tenants never receive the operator OpenRouter key.

This should share the server-owned provider credential mechanism in #94.

Model mapping

Direct-provider model IDs are not assumed to equal OpenRouter model IDs.

Add an explicit mapping layer:

source provider + source model
        ->
OpenRouter model slug
        +
optional OpenRouter provider-routing policy

No fuzzy model-name guessing in the live path.

Example configuration shape (exact syntax may evolve):

[routing.exhaustion_fallback]
enabled = true
target = "openrouter"
max_fallback_attempts = 1
cooldown_seconds = 300

[routing.exhaustion_fallback.triggers]
quota_exhausted = true
rate_quota_exhausted = true
credit_exhausted = true
capacity_exhausted = true
network_failure = false
server_error = false

[routing.openrouter.models]
"subproxy:some-model" = "vendor/some-model"

Secrets are referenced indirectly through the provider credential registry, not embedded here.

OpenRouter provider routing

Use OpenRouter's routing layer after llm-log crosses the fallback boundary.

OpenRouter currently supports provider-level failover for the same model by default and exposes provider routing controls such as order, only, ignore, allow_fallbacks, quantizations, data_collection, zdr, and max_price.

llm-log should preserve configured policy constraints when constructing the fallback request.

Important: if llm-log's quantization policy disallows unknown/low-precision endpoints, the fallback must carry the allowed quantization constraint rather than broadening quality policy merely to obtain a response.

Request transformation

Implement fallback through provider adapters, not byte-blind URL substitution.

At minimum support:

OpenAI-compatible chat/completions/responses requests where semantics are preserved;
Anthropic-compatible messages through an explicit adapter when supported by OpenRouter;
streaming semantics;
model field rewriting via explicit model mapping;
provider-routing object injection where configured.

If the original provider-native request cannot be represented losslessly enough for the configured fallback adapter, fail explicitly instead of silently changing semantics.

Exhaustion circuit breaker

Once a sub-provider is proven exhausted:

mark a scoped temporary exhaustion state;
avoid hammering it on every new request during the cooldown/reset window;
route eligible requests directly to OpenRouter during that window;
clear on known reset, explicit operator action, or successful probe;
scope state by provider/account/model/quota bucket as appropriate.

The state must be observable and event sourced where durable state is enabled.

Attempt chain / provenance

Record every logical request as one request/run with ordered upstream attempts.

For each attempt record:

attempt number
source provider/upstream
target model
start/end timestamps
terminal class
HTTP/provider code
whether downstream was committed
normalized token usage when available
upstream cost when known
selected OpenRouter provider/endpoint metadata when available
fallback reason
policy/rule version

Conceptual projection:

routing_attempt(RequestId, N, Provider, Model, Outcome, Reason).
fallback_used(RequestId, FromProvider, ToProvider, Reason).
provider_exhausted(ProviderScope, Window, EvidenceRef).

Cost / Biz integration

Fallback is not "free retry".

Emit normalized usage/cost events so StarIntel Biz can distinguish:

direct-provider included quota / subscription cost
direct-provider failed attempt cost if any
OpenRouter fallback variable cost
total logical-request cost
attributable revenue/profit

Feed the generic provider pricebook/metering contract in starintel-labs/starintel-biz#110.

A fallback may be blocked when the caller/company budget cannot reserve the expected OpenRouter cost.

Loop prevention

max one exhaustion fallback by default;
attach an internal attempt-chain marker;
OpenRouter-originated requests must not recursively fall back to OpenRouter again;
nested llm-log proxies must preserve/inspect loop-prevention metadata without exposing secrets;
retries remain idempotent in the accounting/event layer.

Observability

Expose at least:

primary attempts
exhaustion events
fallbacks attempted
fallback success rate
fallback cost
providers currently circuit-broken
time until known quota reset when available
final provider/model/endpoint selected

Do not leak operator credentials or sensitive routing config.

Tests

Required fixtures:

preferred upstream 200 -> no fallback
preferred upstream classified 429 quota exhaustion -> OpenRouter attempt succeeds
preferred upstream 402 credit exhaustion -> OpenRouter attempt succeeds
generic 403 auth failure -> no fallback
malformed 400 -> no fallback
preflight local quota exhausted -> skip direct call and use OpenRouter
direct 429 opens circuit -> next request bypasses direct provider
cooldown/reset restores direct provider
fallback mapping missing -> explicit failure, no guessed model
original provider token never reaches fallback fixture
OpenRouter key never enters capture/evidence output
OpenRouter failure does not recurse
downstream response already committed -> no fallback/replay
partial SSE failure -> typed mid-stream failure only
accounting records one logical request with multiple attempts
budget denies OpenRouter reserve -> no paid fallback
quantization/provider policy survives fallback construction

Acceptance

typed exhaustion classifier
generic routing-policy fallback contract
OpenRouter initial fallback adapter
server-side OpenRouter credential boundary
explicit direct-model -> OpenRouter-model mapping
pre-commit-only replay rule
exhaustion circuit breaker/cooldown
loop prevention
per-attempt provenance
cost/budget hooks
OpenRouter provider/quantization policy propagation
SSE/streaming regression coverage
OpenAI-compatible smoke test through fallback
no regression to transparent single-upstream routing

Parent: #94 Related: #59 #61 #65 #67 Cross-repo economics: starintel-labs/starintel-biz#110 ## Goal Add a first-class **upstream exhaustion fallback policy** to llm-log. When a configured direct/sub-provider upstream cannot accept more work because its quota, credits, rate window, or configured capacity is exhausted, llm-log may transparently retry the same logical inference through the configured OpenRouter upstream. This must be implemented as a generic routing-policy primitive with OpenRouter as the initial fallback target, not as ad-hoc special-case retry code. ## Core routing path ```text client | v llm-log | +--> preferred direct/sub-provider proxy | +-- success -------------------------------> client | +-- classified exhaustion before commit | v routing policy | +--> OpenRouter | +--> same model/provider family where configured +--> OpenRouter provider failover +--> optional configured model fallback ``` ## What counts as "ran out" Use the typed error classifier rather than raw status codes alone. Default fallback-eligible classes: - local preflight quota/capacity exhausted - provider credit exhausted / payment-required-for-more-capacity - provider rate quota exhausted - provider concurrency/capacity limit where retrying the same direct endpoint is not useful Typical HTTP evidence may include 402 or 429. A 403 only counts when a provider adapter/classifier proves it means quota/capacity exhaustion. Do **not** treat generic authentication, permission, policy, malformed-request, context-length, or safety/moderation errors as "ran out". 5xx/network failure may be enabled by separate policy, but is not part of the default exhaustion trigger. ## Never replay after downstream commit Fallback is only legal while llm-log still owns the full response decision and has not committed response bytes to the downstream client. For streaming/SSE/WebSocket requests: - if the preferred upstream rejects before llm-log emits the downstream response head/body, fallback is allowed; - once any response body/token/frame has been emitted, **do not restart the request through OpenRouter**; - record a typed mid-stream failure and let the caller decide whether to retry. This avoids duplicate/divergent completions, tool-call duplication, and hidden double spend. ## Credential boundary Current transparent proxy behavior forwards the caller's provider credential. OpenRouter fallback requires a separate server-side credential. Requirements: - OpenRouter API key comes from operator secret storage / environment, never TOML plaintext and never capture output; - when falling back, strip the original provider `Authorization` / provider-specific credential headers; - inject only the configured OpenRouter credential into the OpenRouter attempt; - never forward the direct provider's bearer token to OpenRouter; - downstream tenants never receive the operator OpenRouter key. This should share the server-owned provider credential mechanism in #94. ## Model mapping Direct-provider model IDs are not assumed to equal OpenRouter model IDs. Add an explicit mapping layer: ```text source provider + source model -> OpenRouter model slug + optional OpenRouter provider-routing policy ``` No fuzzy model-name guessing in the live path. Example configuration shape (exact syntax may evolve): ```toml [routing.exhaustion_fallback] enabled = true target = "openrouter" max_fallback_attempts = 1 cooldown_seconds = 300 [routing.exhaustion_fallback.triggers] quota_exhausted = true rate_quota_exhausted = true credit_exhausted = true capacity_exhausted = true network_failure = false server_error = false [routing.openrouter.models] "subproxy:some-model" = "vendor/some-model" ``` Secrets are referenced indirectly through the provider credential registry, not embedded here. ## OpenRouter provider routing Use OpenRouter's routing layer after llm-log crosses the fallback boundary. OpenRouter currently supports provider-level failover for the same model by default and exposes provider routing controls such as `order`, `only`, `ignore`, `allow_fallbacks`, `quantizations`, `data_collection`, `zdr`, and `max_price`. llm-log should preserve configured policy constraints when constructing the fallback request. Important: if llm-log's quantization policy disallows unknown/low-precision endpoints, the fallback must carry the allowed quantization constraint rather than broadening quality policy merely to obtain a response. ## Request transformation Implement fallback through provider adapters, not byte-blind URL substitution. At minimum support: - OpenAI-compatible chat/completions/responses requests where semantics are preserved; - Anthropic-compatible messages through an explicit adapter when supported by OpenRouter; - streaming semantics; - model field rewriting via explicit model mapping; - provider-routing object injection where configured. If the original provider-native request cannot be represented losslessly enough for the configured fallback adapter, fail explicitly instead of silently changing semantics. ## Exhaustion circuit breaker Once a sub-provider is proven exhausted: - mark a scoped temporary exhaustion state; - avoid hammering it on every new request during the cooldown/reset window; - route eligible requests directly to OpenRouter during that window; - clear on known reset, explicit operator action, or successful probe; - scope state by provider/account/model/quota bucket as appropriate. The state must be observable and event sourced where durable state is enabled. ## Attempt chain / provenance Record every logical request as one request/run with ordered upstream attempts. For each attempt record: - attempt number - source provider/upstream - target model - start/end timestamps - terminal class - HTTP/provider code - whether downstream was committed - normalized token usage when available - upstream cost when known - selected OpenRouter provider/endpoint metadata when available - fallback reason - policy/rule version Conceptual projection: ```prolog routing_attempt(RequestId, N, Provider, Model, Outcome, Reason). fallback_used(RequestId, FromProvider, ToProvider, Reason). provider_exhausted(ProviderScope, Window, EvidenceRef). ``` ## Cost / Biz integration Fallback is not "free retry". Emit normalized usage/cost events so StarIntel Biz can distinguish: - direct-provider included quota / subscription cost - direct-provider failed attempt cost if any - OpenRouter fallback variable cost - total logical-request cost - attributable revenue/profit Feed the generic provider pricebook/metering contract in starintel-labs/starintel-biz#110. A fallback may be blocked when the caller/company budget cannot reserve the expected OpenRouter cost. ## Loop prevention - max one exhaustion fallback by default; - attach an internal attempt-chain marker; - OpenRouter-originated requests must not recursively fall back to OpenRouter again; - nested llm-log proxies must preserve/inspect loop-prevention metadata without exposing secrets; - retries remain idempotent in the accounting/event layer. ## Observability Expose at least: - primary attempts - exhaustion events - fallbacks attempted - fallback success rate - fallback cost - providers currently circuit-broken - time until known quota reset when available - final provider/model/endpoint selected Do not leak operator credentials or sensitive routing config. ## Tests Required fixtures: 1. preferred upstream 200 -> no fallback 2. preferred upstream classified 429 quota exhaustion -> OpenRouter attempt succeeds 3. preferred upstream 402 credit exhaustion -> OpenRouter attempt succeeds 4. generic 403 auth failure -> no fallback 5. malformed 400 -> no fallback 6. preflight local quota exhausted -> skip direct call and use OpenRouter 7. direct 429 opens circuit -> next request bypasses direct provider 8. cooldown/reset restores direct provider 9. fallback mapping missing -> explicit failure, no guessed model 10. original provider token never reaches fallback fixture 11. OpenRouter key never enters capture/evidence output 12. OpenRouter failure does not recurse 13. downstream response already committed -> no fallback/replay 14. partial SSE failure -> typed mid-stream failure only 15. accounting records one logical request with multiple attempts 16. budget denies OpenRouter reserve -> no paid fallback 17. quantization/provider policy survives fallback construction ## Acceptance - [ ] typed exhaustion classifier - [ ] generic routing-policy fallback contract - [ ] OpenRouter initial fallback adapter - [ ] server-side OpenRouter credential boundary - [ ] explicit direct-model -> OpenRouter-model mapping - [ ] pre-commit-only replay rule - [ ] exhaustion circuit breaker/cooldown - [ ] loop prevention - [ ] per-attempt provenance - [ ] cost/budget hooks - [ ] OpenRouter provider/quantization policy propagation - [ ] SSE/streaming regression coverage - [ ] OpenAI-compatible smoke test through fallback - [ ] no regression to transparent single-upstream routing

nsaspy referenced this issue

2026-09-20 06:13:36 +00:00

Define class-specific transport alert policy #103

nsaspy referenced this issue

2026-09-20 06:13:36 +00:00

Correlate router attempts and retry recovery #104