Add bounded provider-wide request scheduler with queue and 429 backpressure #73

New issue

Open

opened 2026-09-12 19:54:49 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-12 19:54:49 +00:00

Owner

Goal

Make llm-log an admission/scheduling layer for shared LLM credentials instead of allowing every client request to immediately consume an upstream connection.

First slice

Enforce a provider-wide maximum active request count.
Maintain a bounded FIFO queue for excess requests.
Give each queued request a maximum queue-wait budget.
Admit queued requests as active slots free up.
Return 429 Too Many Requests with Retry-After when the queue is full or the queue-wait budget is exhausted.
Never start an upstream request after its queue deadline has expired.
Release slots on success, upstream failure, and client disconnect during an active relay.
Keep the mechanism generic per provider; do not special-case Z.AI.

Transparent streaming behavior

The normal proxy path must remain transparent. It must not emit SSE response-body bytes while queued because that would commit the downstream HTTP status before the upstream responds. Opt-in queue-status/SSE behavior for smart clients is tracked separately in #73.

Configuration

Add explicit scheduler settings with safe defaults, including max active requests, max queue depth, queue timeout, and retry-after.

Tests

Cover at least:

direct admission below the active limit;
FIFO admission after a slot is released;
queue-full 429;
queue-timeout 429;
Retry-After header;
slot release on upstream error;
no active-count leak;
per-provider isolation.

This is the first implementation slice before provider/client fingerprint profiles.

## Goal Make llm-log an admission/scheduling layer for shared LLM credentials instead of allowing every client request to immediately consume an upstream connection. ## First slice - Enforce a provider-wide maximum active request count. - Maintain a bounded FIFO queue for excess requests. - Give each queued request a maximum queue-wait budget. - Admit queued requests as active slots free up. - Return `429 Too Many Requests` with `Retry-After` when the queue is full or the queue-wait budget is exhausted. - Never start an upstream request after its queue deadline has expired. - Release slots on success, upstream failure, and client disconnect during an active relay. - Keep the mechanism generic per provider; do not special-case Z.AI. ## Transparent streaming behavior The normal proxy path must remain transparent. It must not emit SSE response-body bytes while queued because that would commit the downstream HTTP status before the upstream responds. Opt-in queue-status/SSE behavior for smart clients is tracked separately in #73. ## Configuration Add explicit scheduler settings with safe defaults, including max active requests, max queue depth, queue timeout, and retry-after. ## Tests Cover at least: - direct admission below the active limit; - FIFO admission after a slot is released; - queue-full 429; - queue-timeout 429; - `Retry-After` header; - slot release on upstream error; - no active-count leak; - per-provider isolation. This is the first implementation slice before provider/client fingerprint profiles.

nsaspy referenced this issue

2026-09-12 20:08:46 +00:00

Add provider-wide request scheduling and 429 backpressure #74

nsaspy referenced this issue

2026-09-12 23:06:23 +00:00

Versioned outbound privacy/compatibility profiles for Hermes, OpenClaw and Agent Zero #72

nsaspy referenced this issue

2026-09-12 23:21:00 +00:00

Rate admission: time-based quotas shared across client aliases #77