nsaspy/starintel-auto-research

Fork 0

DESIGN_READY_FOR_OPERATOR_REVIEW: OpenRouter Analytics collector + reconciliation #198

New issue

Open

opened 2026-09-02 17:26:15 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-02 17:26:15 +00:00

Owner

Authority / state

Research: #184 — READY_FOR_DESIGN
Owning implementation issue: starintel-labs/starintel-infra#65
Dependencies: infra #56/#57 for deployed OpenObserve/OTLP export
Current infra main inspected: a9de6e13ab83ba7a0d6c5994c9d8330f1271c3cf
Design state: DESIGN_READY_FOR_OPERATOR_REVIEW
Implementation approval: PENDING / AWAITING_OPERATOR_APPROVAL — operator only

This design is not executable by RAGE until explicit operator approval exists.

Design goal

Add one bounded private collector that periodically queries the documented OpenRouter Analytics API, normalizes only approved accounting/performance fields, reconciles replay/corrections deterministically, and exports sanitized records into StarIntel's canonical llm telemetry stream.

No Activity-page scraping, no prompt/completion ingestion, no parallel analytics database.

Architecture

OpenRouter Analytics API
  -> bounded OpenRouter analytics adapter
  -> pure normalize + privacy allowlist
  -> deterministic reconciliation identity
  -> durable tiny watermark/reconciliation state
  -> OTLP exporter / #57 collector plane
  -> OpenObserve #56 `llm` stream

Language boundary

No StarLang extension is justified for the provider protocol itself. This is infra/service acquisition from an external SaaS API. Keep the normalization/reconciliation core deterministic and host-language neutral; implement in the smallest language/runtime already appropriate to starintel-infra service tooling. If a StarLang scheduled-acquisition capability later owns invocation, it calls this adapter rather than duplicating the protocol.

Collector contract

Configuration must define:

API base/endpoint from current documented OpenRouter contract;
secret reference for management credential, never literal value;
account/workspace scope;
rollup interval;
collection lag/closed-window delay;
bounded trailing correction window;
approved metric set;
approved grouping dimensions;
poll cadence;
request timeout/retry ceiling;
OTLP destination through #57;
durable state path.

Configuration validation rejects content-bearing dimensions/fields and unknown dimension names by default.

Window algorithm

Load durable watermark.
Compute latest safely closed upstream interval using configured collection lag.
Select at most the configured maximum number of intervals per run.
Include a bounded trailing correction horizon behind the watermark.
Query upstream with explicit metric/group/rollup selection.
Normalize and privacy-filter before persistence/export.
Build deterministic identity from source + scope + rollup + interval + canonical sorted dimensions + metric family.
Reconcile same identity: unchanged row is a no-op; changed aggregate supersedes/replaces previous logical value.
Commit state only after normalization and export for the window succeed according to the selected OTLP delivery contract.
Never advance past a failed window.

This gives restart recovery without unbounded historical rescans.

State model

Keep only what is required for restart/reconciliation:

watermark
last-success-at
recent correction-horizon identities + normalized hashes/values
contract/schema version

Do not store raw API responses or prompt/content data.

State writes must be atomic (write+fsync+rename or equivalent durable primitive) so a crash cannot advance the watermark without the corresponding reconciliation state.

Normalized record

Required envelope:

source
contractVersion
intervalStart
intervalEnd
rollup
scopeOpaqueId
metricFamily
dimensions{}
metrics{}
reconciliationId
observedAt

Allowed metrics include documented spend/request/token/cache/performance aggregates. Missing upstream values remain missing.

Default-safe dimensions: model, provider, app/agent identifier when intentionally configured, opaque key identifier, workspace opaque identifier, finish/error class, data region, variant where useful.

User/custom-user/origin/session/generation/arbitrary classifier dimensions are disabled by default and require a separate explicit privacy review.

Secret handling

dedicated management credential via systemd credential/runtime secret mechanism;
never Nix literal/store path;
never command-line argument;
Authorization header redacted from errors/traces;
upstream error bodies bounded and sanitized before logging;
health endpoint never reports secret-derived values.

Failure behavior

timeout/429/5xx: bounded retry with exponential backoff + jitter;
auth failure: fail collector health, do not hot-loop;
schema drift/unknown field: reject affected record/window and surface typed schema error rather than silently guessing;
OTLP/OpenObserve unavailable: bounded downstream buffering belongs to #57; collector does not create an unbounded local spool;
application/inference traffic is completely independent of collector health.

Observability

Expose safe collector metrics/state:

last successful interval;
watermark/lag;
poll duration;
upstream status/error class;
normalized/rejected row counts;
replay no-op count;
correction/reconciliation count;
export outcome;
schema/contract version.

Do not emit raw query parameters if they include sensitive dimension values.

Dependency-ordered implementation slices

All remain AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL.

Pure normalized-record schema, privacy allowlist, deterministic identity and correction/replay reconciliation fixtures.
Durable watermark/state transaction with crash/failure fixtures.
OpenRouter Analytics HTTP adapter using recorded sanitized fixtures; bounded paging/window/retry behavior.
OTLP export integration through #57 contract.
NixOS package/service/credential wiring and health/readiness.
Optional live API verification with operator-provided management credential; never required to fake GREEN for pure/unit slices.

Mandatory RED-first targets

Slice 1

Before production mutation, add fixtures proving untouched repo lacks the contract:

duplicate same logical row -> same reconciliation identity and one logical value;
corrected same logical row -> replacement/reconciliation, never additive duplicate;
content/user/custom metadata fixture -> forbidden fields absent/rejected;
omitted optional metric -> remains absent, not zero;
unknown dimension -> rejected unless explicitly allowlisted.

Slice 2

simulated failure before commit -> watermark unchanged;
crash/torn-write fixture -> prior valid state remains readable;
correction horizon remains bounded.

Slice 3

429 then success -> bounded retry;
repeated 5xx -> bounded failure, no watermark advance;
malformed/schema-drift response -> typed failure;
oversized response/page -> bounded rejection;
auth header never appears in captured logs/errors.

Slice 4

exporter unavailable does not affect any application service;
delivery/retry semantics do not duplicate logical aggregate totals.

Adversarial review

Rejected:

UI scraping: brittle and unnecessary now that a first-party Analytics API exists.
Per-response-only accounting: misses requests outside one instrumented caller and cannot reconcile downtime globally.
Append-only aggregate telemetry: replay/corrections inflate spend/tokens.
Every available dimension by default: turns operational telemetry into identity/content tracking.
Raw response cache: unnecessary sensitive retention.
Advancing watermark after partial failure: creates permanent gaps.
Unbounded backfill after long downtime: provider/API and local resource abuse.
Collector-owned unbounded disk queue: duplicates #57 responsibility.
Treating missing metric as zero: corrupts accounting semantics.
Putting management credentials in Nix/config/CLI: secret leakage.

Acceptance

documented Analytics API is the upstream accounting source;
restart/replay produces no duplicate logical totals;
bounded correction window updates prior aggregates deterministically;
prompt/completion/raw user/custom metadata cannot enter default llm telemetry;
missing values preserve unknown semantics;
failures never advance watermark incorrectly;
polling/retries/backfill/state growth are bounded;
credential is runtime-only and absent from Git/Nix store/logs/CLI;
collector outage cannot affect inference/application availability;
sanitized records export through #57 to OpenObserve #56 when those dependencies are available;
exact-head local tests/flake gates are green.

Implementation approval

PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL

Only the operator may authorize implementation.

## Authority / state - Research: #184 — `READY_FOR_DESIGN` - Owning implementation issue: `starintel-labs/starintel-infra#65` - Dependencies: infra #56/#57 for deployed OpenObserve/OTLP export - Current infra main inspected: `a9de6e13ab83ba7a0d6c5994c9d8330f1271c3cf` - Design state: `DESIGN_READY_FOR_OPERATOR_REVIEW` - **Implementation approval: PENDING / AWAITING_OPERATOR_APPROVAL — operator only** This design is not executable by RAGE until explicit operator approval exists. ## Design goal Add one bounded private collector that periodically queries the documented OpenRouter Analytics API, normalizes only approved accounting/performance fields, reconciles replay/corrections deterministically, and exports sanitized records into StarIntel's canonical `llm` telemetry stream. No Activity-page scraping, no prompt/completion ingestion, no parallel analytics database. ## Architecture ```text OpenRouter Analytics API -> bounded OpenRouter analytics adapter -> pure normalize + privacy allowlist -> deterministic reconciliation identity -> durable tiny watermark/reconciliation state -> OTLP exporter / #57 collector plane -> OpenObserve #56 `llm` stream ``` ## Language boundary No StarLang extension is justified for the provider protocol itself. This is infra/service acquisition from an external SaaS API. Keep the normalization/reconciliation core deterministic and host-language neutral; implement in the smallest language/runtime already appropriate to `starintel-infra` service tooling. If a StarLang scheduled-acquisition capability later owns invocation, it calls this adapter rather than duplicating the protocol. ## Collector contract Configuration must define: - API base/endpoint from current documented OpenRouter contract; - secret reference for management credential, never literal value; - account/workspace scope; - rollup interval; - collection lag/closed-window delay; - bounded trailing correction window; - approved metric set; - approved grouping dimensions; - poll cadence; - request timeout/retry ceiling; - OTLP destination through #57; - durable state path. Configuration validation rejects content-bearing dimensions/fields and unknown dimension names by default. ## Window algorithm 1. Load durable watermark. 2. Compute latest safely closed upstream interval using configured collection lag. 3. Select at most the configured maximum number of intervals per run. 4. Include a bounded trailing correction horizon behind the watermark. 5. Query upstream with explicit metric/group/rollup selection. 6. Normalize and privacy-filter before persistence/export. 7. Build deterministic identity from source + scope + rollup + interval + canonical sorted dimensions + metric family. 8. Reconcile same identity: unchanged row is a no-op; changed aggregate supersedes/replaces previous logical value. 9. Commit state only after normalization and export for the window succeed according to the selected OTLP delivery contract. 10. Never advance past a failed window. This gives restart recovery without unbounded historical rescans. ## State model Keep only what is required for restart/reconciliation: ```text watermark last-success-at recent correction-horizon identities + normalized hashes/values contract/schema version ``` Do not store raw API responses or prompt/content data. State writes must be atomic (write+fsync+rename or equivalent durable primitive) so a crash cannot advance the watermark without the corresponding reconciliation state. ## Normalized record Required envelope: ```text source contractVersion intervalStart intervalEnd rollup scopeOpaqueId metricFamily dimensions{} metrics{} reconciliationId observedAt ``` Allowed metrics include documented spend/request/token/cache/performance aggregates. Missing upstream values remain missing. Default-safe dimensions: model, provider, app/agent identifier when intentionally configured, opaque key identifier, workspace opaque identifier, finish/error class, data region, variant where useful. User/custom-user/origin/session/generation/arbitrary classifier dimensions are disabled by default and require a separate explicit privacy review. ## Secret handling - dedicated management credential via systemd credential/runtime secret mechanism; - never Nix literal/store path; - never command-line argument; - Authorization header redacted from errors/traces; - upstream error bodies bounded and sanitized before logging; - health endpoint never reports secret-derived values. ## Failure behavior - timeout/429/5xx: bounded retry with exponential backoff + jitter; - auth failure: fail collector health, do not hot-loop; - schema drift/unknown field: reject affected record/window and surface typed schema error rather than silently guessing; - OTLP/OpenObserve unavailable: bounded downstream buffering belongs to #57; collector does not create an unbounded local spool; - application/inference traffic is completely independent of collector health. ## Observability Expose safe collector metrics/state: - last successful interval; - watermark/lag; - poll duration; - upstream status/error class; - normalized/rejected row counts; - replay no-op count; - correction/reconciliation count; - export outcome; - schema/contract version. Do not emit raw query parameters if they include sensitive dimension values. ## Dependency-ordered implementation slices All remain **AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL**. 1. Pure normalized-record schema, privacy allowlist, deterministic identity and correction/replay reconciliation fixtures. 2. Durable watermark/state transaction with crash/failure fixtures. 3. OpenRouter Analytics HTTP adapter using recorded sanitized fixtures; bounded paging/window/retry behavior. 4. OTLP export integration through #57 contract. 5. NixOS package/service/credential wiring and health/readiness. 6. Optional live API verification with operator-provided management credential; never required to fake GREEN for pure/unit slices. ## Mandatory RED-first targets ### Slice 1 Before production mutation, add fixtures proving untouched repo lacks the contract: - duplicate same logical row -> same reconciliation identity and one logical value; - corrected same logical row -> replacement/reconciliation, never additive duplicate; - content/user/custom metadata fixture -> forbidden fields absent/rejected; - omitted optional metric -> remains absent, not zero; - unknown dimension -> rejected unless explicitly allowlisted. ### Slice 2 - simulated failure before commit -> watermark unchanged; - crash/torn-write fixture -> prior valid state remains readable; - correction horizon remains bounded. ### Slice 3 - 429 then success -> bounded retry; - repeated 5xx -> bounded failure, no watermark advance; - malformed/schema-drift response -> typed failure; - oversized response/page -> bounded rejection; - auth header never appears in captured logs/errors. ### Slice 4 - exporter unavailable does not affect any application service; - delivery/retry semantics do not duplicate logical aggregate totals. ## Adversarial review Rejected: 1. UI scraping: brittle and unnecessary now that a first-party Analytics API exists. 2. Per-response-only accounting: misses requests outside one instrumented caller and cannot reconcile downtime globally. 3. Append-only aggregate telemetry: replay/corrections inflate spend/tokens. 4. Every available dimension by default: turns operational telemetry into identity/content tracking. 5. Raw response cache: unnecessary sensitive retention. 6. Advancing watermark after partial failure: creates permanent gaps. 7. Unbounded backfill after long downtime: provider/API and local resource abuse. 8. Collector-owned unbounded disk queue: duplicates #57 responsibility. 9. Treating missing metric as zero: corrupts accounting semantics. 10. Putting management credentials in Nix/config/CLI: secret leakage. ## Acceptance - documented Analytics API is the upstream accounting source; - restart/replay produces no duplicate logical totals; - bounded correction window updates prior aggregates deterministically; - prompt/completion/raw user/custom metadata cannot enter default `llm` telemetry; - missing values preserve unknown semantics; - failures never advance watermark incorrectly; - polling/retries/backfill/state growth are bounded; - credential is runtime-only and absent from Git/Nix store/logs/CLI; - collector outage cannot affect inference/application availability; - sanitized records export through #57 to OpenObserve #56 when those dependencies are available; - exact-head local tests/flake gates are green. ## Implementation approval `PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL` Only the operator may authorize implementation.