Multi-tenant LLM gateway: upstream providers, per-user credits, billing, usage logs and OpenAI-compatible API #86

New issue

Open

opened 2026-09-18 22:04:12 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-18 22:04:12 +00:00

Owner

Goal

Evolve llm-log from transparent capture into the shared multi-tenant LLM control plane while preserving its lossless logging/Prolog analytics role.

Hosted Zara and other StarIntel clients should implement providers once against llm-log and receive a stable OpenAI-compatible downstream API.

Service identity

Private StarIntel clients use the internal service identity (for example llm-log.star.intel; infra chooses final DNS).
Public API may be exposed behind llm.starintel.actor with explicit auth/policy.

Multi-tenancy

Every request resolves:

principal/user
tenant
API key/service identity
plan/entitlements
provider
model
request/run id

No analytics, raw prompt/completion, API key, balance or usage endpoint may leak across tenants.

Upstream provider ownership

Add server-side provider credential support:

upstream keys live in operator secret storage
downstream users receive llm-log-scoped keys
provider adapters preserve existing arbitrary/custom upstream support
local llm.star.intel can be one upstream among many
OpenRouter/OpenAI/Anthropic/etc. credentials never need to enter Zara tenant runtimes

API

Provide a stable OpenAI-compatible surface at minimum:

/v1/models
chat/completion/responses surfaces supported by registered adapters
streaming/SSE
explicit error model for quota/payment/provider failure

Keep provider-native adapters where OpenAI compatibility would lose required semantics.

Billing/credits

Integrate through a narrow billing authority contract (StarIntel biz):

preflight entitlement
credit/spend-limit check
request reservation/hold when appropriate
normalized token/cost event
finalization exactly once
cancellation/error releases reservation
separate upstream_cost, customer_cost, pricing_rule_version

llm-log must not silently invent provider costs. Unknown cost remains unknown until a configured pricing rule/provider report supplies it.

Logging

Preserve llm-log's core purpose:

per-request lossless event/corpus record under retention policy
Prolog projection
normalized token counts
latency/status/provider/model
tenant-safe analytics

Add configurable privacy/retention tiers because a paid service cannot assume every customer's raw prompt should live forever.

Quotas / abuse

per-user and per-tenant concurrency
request/token/rate limits
model/provider allowlists
spend caps
prepaid-credit exhaustion
bounded queue/backpressure
no retry amplification
admin emergency disable

User dashboard API

Tenant-scoped:

current credit balance/ledger projection
usage by time/model/provider
upstream vs customer cost where policy permits
active limits
API keys
recent request status

Acceptance

two tenants using the same upstream cannot see/use each other's keys, logs or credits
concurrent last-credit race cannot overspend beyond configured tolerance
SSE cancel finalizes at most once
upstream 429/5xx retry policy does not double-bill
local StarIntel model can be selected as an upstream
arbitrary custom provider remains possible
OpenAI SDK/OpenCode/gptel-compatible smoke tests pass
existing capture/analytics behavior remains covered
security tests cover forged tenant ids, key confusion, IDOR and header injection

Routing/failover children

#95 Upstream exhaustion failover: route depleted sub-provider proxies through OpenRouter
#96 Subscription proxy adapters + worker correlation: record ChatGPT Pro/Codex and GLM Coding Plan token I/O

## Goal Evolve llm-log from transparent capture into the shared **multi-tenant LLM control plane** while preserving its lossless logging/Prolog analytics role. Hosted Zara and other StarIntel clients should implement providers once against llm-log and receive a stable OpenAI-compatible downstream API. ## Service identity Private StarIntel clients use the internal service identity (for example `llm-log.star.intel`; infra chooses final DNS). Public API may be exposed behind `llm.starintel.actor` with explicit auth/policy. ## Multi-tenancy Every request resolves: - principal/user - tenant - API key/service identity - plan/entitlements - provider - model - request/run id No analytics, raw prompt/completion, API key, balance or usage endpoint may leak across tenants. ## Upstream provider ownership Add server-side provider credential support: - upstream keys live in operator secret storage - downstream users receive llm-log-scoped keys - provider adapters preserve existing arbitrary/custom upstream support - local `llm.star.intel` can be one upstream among many - OpenRouter/OpenAI/Anthropic/etc. credentials never need to enter Zara tenant runtimes ## API Provide a stable OpenAI-compatible surface at minimum: - `/v1/models` - chat/completion/responses surfaces supported by registered adapters - streaming/SSE - explicit error model for quota/payment/provider failure Keep provider-native adapters where OpenAI compatibility would lose required semantics. ## Billing/credits Integrate through a narrow billing authority contract (StarIntel biz): - preflight entitlement - credit/spend-limit check - request reservation/hold when appropriate - normalized token/cost event - finalization exactly once - cancellation/error releases reservation - separate `upstream_cost`, `customer_cost`, `pricing_rule_version` llm-log must not silently invent provider costs. Unknown cost remains unknown until a configured pricing rule/provider report supplies it. ## Logging Preserve llm-log's core purpose: - per-request lossless event/corpus record under retention policy - Prolog projection - normalized token counts - latency/status/provider/model - tenant-safe analytics Add configurable privacy/retention tiers because a paid service cannot assume every customer's raw prompt should live forever. ## Quotas / abuse - per-user and per-tenant concurrency - request/token/rate limits - model/provider allowlists - spend caps - prepaid-credit exhaustion - bounded queue/backpressure - no retry amplification - admin emergency disable ## User dashboard API Tenant-scoped: - current credit balance/ledger projection - usage by time/model/provider - upstream vs customer cost where policy permits - active limits - API keys - recent request status ## Acceptance - two tenants using the same upstream cannot see/use each other's keys, logs or credits - concurrent last-credit race cannot overspend beyond configured tolerance - SSE cancel finalizes at most once - upstream 429/5xx retry policy does not double-bill - local StarIntel model can be selected as an upstream - arbitrary custom provider remains possible - OpenAI SDK/OpenCode/gptel-compatible smoke tests pass - existing capture/analytics behavior remains covered - security tests cover forged tenant ids, key confusion, IDOR and header injection ## Routing/failover children - [ ] #95 Upstream exhaustion failover: route depleted sub-provider proxies through OpenRouter - [ ] #96 Subscription proxy adapters + worker correlation: record ChatGPT Pro/Codex and GLM Coding Plan token I/O

nsaspy referenced this issue

2026-09-20 06:13:36 +00:00

Correlate router attempts and retry recovery #104

nsaspy referenced this issue

2026-09-20 06:13:36 +00:00

Add typed transport error evidence #105

nsaspy referenced this issue

2026-09-20 06:13:37 +00:00

Feed response and usage outcomes into expert plane #106