P1 ARDR research: OpenRouter Analytics API -> StarIntel LLM telemetry #199

New issue

Open

opened 2026-09-02 17:26:16 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-02 17:26:16 +00:00

Owner

StarIntel problem

Owning issue: starintel-labs/starintel-infra#65.

StarIntel wants OpenRouter cost/token/model/provider activity in the canonical OpenObserve llm telemetry stream. The owning issue was written when the exact upstream collection surface was uncertain and explicitly required re-checking current OpenRouter APIs rather than assuming a webhook.

Current starintel-infra/main inspected at a9de6e13ab83ba7a0d6c5994c9d8330f1271c3cf. Repository code search found no existing OpenRouter collector implementation to preserve.

ARDR state

READY_FOR_DESIGN

Implementation approval remains operator-only.

Current upstream evidence

OpenRouter announced its Activity dashboard / Analytics API on 2026-08-17. The current first-party description says the Analytics API exposes the same aggregate data as Activity and supports metrics including spend, request count, prompt/completion/reasoning/cached tokens, cache hit rate, blended cost, BYOK-vs-credit spend, latency and throughput. It can group by dimensions including model, variant, provider, API key, app, user, workspace, origin, country, data region, finish reason, context length, session, generation, custom user IDs and classifier dimensions, with minute/hour/day/week/month rollups.

OpenRouter also documents per-response usage accounting in API responses, but that surface only covers requests StarIntel directly intercepts and therefore is not a complete account/workspace reconciliation source. Current OpenRouter material also distinguishes Activity/account analytics from per-key usage totals and per-response usage objects.

Research decision

Use the OpenRouter Analytics API as the authoritative upstream accounting source for this collector. Do not scrape the Activity UI and do not invent a webhook dependency.

Per-response usage may later be correlated for low-latency request telemetry, but aggregate/account reconciliation must remain Analytics-API-backed so requests from multiple StarIntel agents/apps/keys are recoverable after collector downtime.

StarIntel requirements

Collector is a private StarIntel service/adapter feeding the canonical OpenTelemetry/OpenObserve plane from infra #56/#57; it does not create another analytics database.
Authentication uses a dedicated OpenRouter management credential from runtime secret storage only.
Collection uses bounded closed time windows plus a durable watermark.
Each imported aggregate row has a deterministic identity derived from source, workspace/account scope, metric/dimensions, rollup interval and window start/end so replay is idempotent.
Re-query a bounded trailing correction window because provider/accounting data can settle after initial observation; corrections replace/reconcile the same logical aggregate rather than incrementing totals twice.
Do not use raw prompt/completion logging as a dependency. Prompt/completion content is excluded even if upstream workspace observability has it enabled.
Default dimensions are allowlisted and low-risk: model, provider, app/agent attribution where intentionally configured, key identity as opaque/stable identifier, workspace, finish/error class, data region, and safe performance/accounting dimensions.
Human/user/custom IDs are excluded by default. Enabling them requires a separate privacy decision because they can become cross-system identity telemetry.
Origin/session/generation identifiers are not persisted by default unless a concrete debugging/correlation requirement approves them.
Collector state is small and local/durable; no raw response archive.
Rate-limit/5xx/network failures use bounded exponential backoff with jitter and do not block applications.
A failed poll does not advance the watermark.
Collector health exposes last successful window, watermark, lag, retry/error class, rows reconciled, rows rejected, and export status without credentials/content.

Normalized `llm` telemetry contract

At minimum preserve:

source=openrouter-analytics
interval start/end + rollup
model/provider/app/workspace dimensions when requested
request count
prompt/completion/reasoning/cached token counts where returned
spend/cost metric in upstream units with explicit currency/semantics
cache hit rate where returned
latency/throughput aggregates/quantiles where requested
BYOK-vs-OpenRouter spend classification where returned
collection timestamp
upstream query/schema version or collector contract version
deterministic reconciliation identity

Do not manufacture zero values for fields the API omitted; preserve missing/unknown semantics.

Acquisition strategy

Use a small bounded scheduled collector. This is an external SaaS API adapter, so an external HTTP request is justified, but no new general StarLang syntax is required. If StarIntel later represents scheduled acquisition jobs in StarLang, the collector can be invoked through that capability; the upstream protocol adapter remains ordinary service/runtime code.

Privacy boundary

Never persist/export by default:

prompts/completions/messages/tool payloads;
management/API keys or Authorization headers;
raw user identifiers;
arbitrary custom metadata/classifier payloads;
full upstream response bodies;
private request/response content.

Dimension values must be independently allowlisted; the fact that OpenRouter can group by a dimension does not make it safe for StarIntel telemetry.

RED-first research output

First implementation slice should be a pure normalization/reconciliation contract with fixtures, before any live credential/API wiring.

Mandatory initial REDs on untouched infra:

two identical Analytics rows for the same interval/dimensions must normalize to the same deterministic identity rather than two additive records;
a corrected value for the same identity must replace/reconcile, not double count;
fixture containing prompt/completion/raw user/custom metadata must prove those fields cannot enter the normalized llm record;
missing optional metrics remain absent/unknown rather than fabricated zero;
failed collection window cannot advance durable watermark.

Adversarial findings

Rejected:

scraping Activity HTML/UI;
assuming an undocumented webhook;
using only per-response usage and losing requests generated outside one instrumented proxy;
append-only aggregate import that double-counts corrections/replays;
persisting every available dimension because the API exposes it;
storing raw Analytics responses for convenience;
coupling application availability to telemetry ingestion;
using an ordinary inference API key where a scoped management credential is required.

Dependencies

infra #56 OpenObserve backend
infra #57 reusable OTLP collector plane

The pure collector contract can be designed/tested before those services are live; deployment/export integration remains dependent on their stable ingestion contract.

Decision

Evidence is sufficient for design. No human research validation is required.

Implementation approval: PENDING / AWAITING_OPERATOR_APPROVAL.

## StarIntel problem Owning issue: `starintel-labs/starintel-infra#65`. StarIntel wants OpenRouter cost/token/model/provider activity in the canonical OpenObserve `llm` telemetry stream. The owning issue was written when the exact upstream collection surface was uncertain and explicitly required re-checking current OpenRouter APIs rather than assuming a webhook. Current `starintel-infra/main` inspected at `a9de6e13ab83ba7a0d6c5994c9d8330f1271c3cf`. Repository code search found no existing OpenRouter collector implementation to preserve. ## ARDR state `READY_FOR_DESIGN` Implementation approval remains operator-only. ## Current upstream evidence OpenRouter announced its Activity dashboard / Analytics API on 2026-08-17. The current first-party description says the Analytics API exposes the same aggregate data as Activity and supports metrics including spend, request count, prompt/completion/reasoning/cached tokens, cache hit rate, blended cost, BYOK-vs-credit spend, latency and throughput. It can group by dimensions including model, variant, provider, API key, app, user, workspace, origin, country, data region, finish reason, context length, session, generation, custom user IDs and classifier dimensions, with minute/hour/day/week/month rollups. OpenRouter also documents per-response usage accounting in API responses, but that surface only covers requests StarIntel directly intercepts and therefore is not a complete account/workspace reconciliation source. Current OpenRouter material also distinguishes Activity/account analytics from per-key usage totals and per-response `usage` objects. ## Research decision Use the **OpenRouter Analytics API as the authoritative upstream accounting source for this collector**. Do not scrape the Activity UI and do not invent a webhook dependency. Per-response usage may later be correlated for low-latency request telemetry, but aggregate/account reconciliation must remain Analytics-API-backed so requests from multiple StarIntel agents/apps/keys are recoverable after collector downtime. ## StarIntel requirements 1. Collector is a private StarIntel service/adapter feeding the canonical OpenTelemetry/OpenObserve plane from infra #56/#57; it does not create another analytics database. 2. Authentication uses a dedicated OpenRouter management credential from runtime secret storage only. 3. Collection uses bounded closed time windows plus a durable watermark. 4. Each imported aggregate row has a deterministic identity derived from source, workspace/account scope, metric/dimensions, rollup interval and window start/end so replay is idempotent. 5. Re-query a bounded trailing correction window because provider/accounting data can settle after initial observation; corrections replace/reconcile the same logical aggregate rather than incrementing totals twice. 6. Do not use raw prompt/completion logging as a dependency. Prompt/completion content is excluded even if upstream workspace observability has it enabled. 7. Default dimensions are allowlisted and low-risk: model, provider, app/agent attribution where intentionally configured, key identity as opaque/stable identifier, workspace, finish/error class, data region, and safe performance/accounting dimensions. 8. Human/user/custom IDs are excluded by default. Enabling them requires a separate privacy decision because they can become cross-system identity telemetry. 9. Origin/session/generation identifiers are not persisted by default unless a concrete debugging/correlation requirement approves them. 10. Collector state is small and local/durable; no raw response archive. 11. Rate-limit/5xx/network failures use bounded exponential backoff with jitter and do not block applications. 12. A failed poll does not advance the watermark. 13. Collector health exposes last successful window, watermark, lag, retry/error class, rows reconciled, rows rejected, and export status without credentials/content. ## Normalized `llm` telemetry contract At minimum preserve: - source=`openrouter-analytics` - interval start/end + rollup - model/provider/app/workspace dimensions when requested - request count - prompt/completion/reasoning/cached token counts where returned - spend/cost metric in upstream units with explicit currency/semantics - cache hit rate where returned - latency/throughput aggregates/quantiles where requested - BYOK-vs-OpenRouter spend classification where returned - collection timestamp - upstream query/schema version or collector contract version - deterministic reconciliation identity Do not manufacture zero values for fields the API omitted; preserve missing/unknown semantics. ## Acquisition strategy Use a small bounded scheduled collector. This is an external SaaS API adapter, so an external HTTP request is justified, but no new general StarLang syntax is required. If StarIntel later represents scheduled acquisition jobs in StarLang, the collector can be invoked through that capability; the upstream protocol adapter remains ordinary service/runtime code. ## Privacy boundary Never persist/export by default: - prompts/completions/messages/tool payloads; - management/API keys or Authorization headers; - raw user identifiers; - arbitrary custom metadata/classifier payloads; - full upstream response bodies; - private request/response content. Dimension values must be independently allowlisted; the fact that OpenRouter can group by a dimension does not make it safe for StarIntel telemetry. ## RED-first research output First implementation slice should be a pure normalization/reconciliation contract with fixtures, before any live credential/API wiring. Mandatory initial REDs on untouched infra: 1. two identical Analytics rows for the same interval/dimensions must normalize to the same deterministic identity rather than two additive records; 2. a corrected value for the same identity must replace/reconcile, not double count; 3. fixture containing prompt/completion/raw user/custom metadata must prove those fields cannot enter the normalized `llm` record; 4. missing optional metrics remain absent/unknown rather than fabricated zero; 5. failed collection window cannot advance durable watermark. ## Adversarial findings Rejected: - scraping Activity HTML/UI; - assuming an undocumented webhook; - using only per-response `usage` and losing requests generated outside one instrumented proxy; - append-only aggregate import that double-counts corrections/replays; - persisting every available dimension because the API exposes it; - storing raw Analytics responses for convenience; - coupling application availability to telemetry ingestion; - using an ordinary inference API key where a scoped management credential is required. ## Dependencies - infra #56 OpenObserve backend - infra #57 reusable OTLP collector plane The pure collector contract can be designed/tested before those services are live; deployment/export integration remains dependent on their stable ingestion contract. ## Decision Evidence is sufficient for design. No human research validation is required. Implementation approval: `PENDING / AWAITING_OPERATOR_APPROVAL`.