EPIC: RAGE Qtile telemetry — restore balance and make TPM physically believable #107

New issue

Open

opened 2026-08-22 22:00:57 +00:00 by lost-rob0t · 1 comment

lost-rob0t commented

2026-08-22 22:00:57 +00:00

(Migrated from github.com)

Problem

The OpenRouter rewrite accidentally dropped the balance widget while keeping token rate + graph. Worse, the collector treats a single /analytics/query response for the last 60 wall-clock seconds as unquestionable truth, so duplicated/coarse/bucketed analytics can surface absurd spikes such as ~165M tokens/minute.

Code-review findings

The installed telemetry cluster currently contains only OpenRouterRate and OpenRouterIOGraph; balance is gone.
parse_token_totals() blindly sums every returned row.
There is no row identity/bucket validation, timestamp validation, continuity check, outlier policy, or comparison to the previous sample.
Cache state stores only the last rendered status, so it cannot distinguish a legitimate reset from a provider-side aggregation glitch.
The graph faithfully amplifies bad samples once they enter the cache.
Failure fallback is useful, but stale cache and implausible fresh data are not distinguished.

Required design

Restore OpenRouter balance as a first-class widget; keep existing rate + graph.
Read credentials from environment or an ignored per-user key file only. Never commit a key.
Validate analytics rows before summing; prefer a bounded explicit time bucket if the endpoint supports it.
Persist enough sample metadata to reject duplicate/regressive/implausible samples without freezing forever.
Use monotonic elapsed time for local rate derivation where local deltas are involved.
Never silently clamp a nonsense reading and call it accurate: mark degraded/stale and retain the last trusted sample.
Graph only trusted samples.

Tests / gates

Balance widget exists in the installed center telemetry cluster.
Balance color thresholds remain: < $5 red, $5–$10 warning, >= $10 healthy.
Normal 60-second sample renders expected TPM.
Duplicate analytics rows cannot multiply TPM.
A provider response that implies an impossible discontinuity is rejected/marked degraded.
Clock regressions / tiny elapsed intervals cannot create giant TPM.
Stale cached data is visibly marked.
Rate and graph tests cover the 165M-style regression.
Literate source and tangled files remain in parity.

RAGE rule

Do the whole correctness chain: API semantics -> collector -> cache -> widget -> graph -> tests -> docs. No cosmetic clamp-only patch.

## Problem The OpenRouter rewrite accidentally dropped the balance widget while keeping token rate + graph. Worse, the collector treats a single `/analytics/query` response for the last 60 wall-clock seconds as unquestionable truth, so duplicated/coarse/bucketed analytics can surface absurd spikes such as ~165M tokens/minute. ## Code-review findings - The installed telemetry cluster currently contains only `OpenRouterRate` and `OpenRouterIOGraph`; balance is gone. - `parse_token_totals()` blindly sums every returned row. - There is no row identity/bucket validation, timestamp validation, continuity check, outlier policy, or comparison to the previous sample. - Cache state stores only the last rendered status, so it cannot distinguish a legitimate reset from a provider-side aggregation glitch. - The graph faithfully amplifies bad samples once they enter the cache. - Failure fallback is useful, but stale cache and implausible fresh data are not distinguished. ## Required design - Restore OpenRouter balance as a first-class widget; keep existing rate + graph. - Read credentials from environment or an ignored per-user key file only. Never commit a key. - Validate analytics rows before summing; prefer a bounded explicit time bucket if the endpoint supports it. - Persist enough sample metadata to reject duplicate/regressive/implausible samples without freezing forever. - Use monotonic elapsed time for local rate derivation where local deltas are involved. - Never silently clamp a nonsense reading and call it accurate: mark degraded/stale and retain the last trusted sample. - Graph only trusted samples. ## Tests / gates - [ ] Balance widget exists in the installed center telemetry cluster. - [ ] Balance color thresholds remain: < $5 red, $5–$10 warning, >= $10 healthy. - [ ] Normal 60-second sample renders expected TPM. - [ ] Duplicate analytics rows cannot multiply TPM. - [ ] A provider response that implies an impossible discontinuity is rejected/marked degraded. - [ ] Clock regressions / tiny elapsed intervals cannot create giant TPM. - [ ] Stale cached data is visibly marked. - [ ] Rate and graph tests cover the 165M-style regression. - [ ] Literate source and tangled files remain in parity. ## RAGE rule Do the whole correctness chain: API semantics -> collector -> cache -> widget -> graph -> tests -> docs. No cosmetic clamp-only patch.

lost-rob0t commented

2026-08-22 22:20:12 +00:00

(Migrated from github.com)

RAGE slice landed in #109 (merged to master as 4a37779). Balance is restored; TPM now uses a closed minute bucket and rejects truncated, duplicated, malformed, negative, and implausible data; stale trusted data is marked instead of graphing garbage; the ~165M TPM regression is covered. Keeping the epic open for live-provider validation / future telemetry work.

RAGE slice landed in #109 (merged to `master` as `4a37779`). Balance is restored; TPM now uses a closed minute bucket and rejects truncated, duplicated, malformed, negative, and implausible data; stale trusted data is marked instead of graphing garbage; the ~165M TPM regression is covered. Keeping the epic open for live-provider validation / future telemetry work.