[P1] Make live OpenRouter streaming gate router-safe #46

New issue

Closed

opened 2026-08-16 01:22:48 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-16 01:22:48 +00:00

(Migrated from github.com)

Problem

main commit 52ca8060456a0e91b69783867d6733026b94d07e had a fully green deterministic job while Actions run 31676691726 failed in REAL OpenRouter integration.

openrouter/free routed the streaming request to nvidia/nemotron-3.5-content-safety:free. The provider path itself succeeded (HTTP 200, incremental events, [DONE], usage, final text), but the test asserted that the model must literally emit STREAM_OK; the routed model instead emitted User Safety: safe.

This is the same class of instability already handled more carefully by the benchmark layer, where provider/runtime health and instruction-quality evidence are separate signals.

Current state — 2026-08-18

The exact PR #89 head 6543853b5e0499ca4881a3991f579387596a470a later passed the full REAL OpenRouter lane, including the live core suite, structured repair, benchmark, depth 0/1/2 experiment, and one-command RLM CLI smoke.

That green run does not satisfy this issue's acceptance criteria by itself. The defect is that a healthy openrouter/free route can still fail the sentinel-sensitive streaming assertion when the selected model ignores the requested sentinel. No merged change in #89 modified that gate, so #46 remains open. The failing run above is retained as historical reproduction evidence rather than a claim about the latest CI state.

Desired behavior

keep this a genuinely live provider test; do not replace it with a fake;
do not treat arbitrary routing behind openrouter/free as deterministic model semantics;
either pin OPENROUTER_TEST_MODEL for sentinel-sensitive acceptance, or make the default free-router streaming gate assert transport/stream correctness and record sentinel compliance as quality evidence;
continue verifying real incremental SSE events, termination, selected model, HTTP status, and usage when available;
never weaken credential redaction or provider-error handling.

Acceptance criteria

deterministic CI remains unchanged and green;
the live streaming test cannot fail solely because a healthy openrouter/free request was routed to a model that ignored the requested sentinel;
a broken/truncated/non-streaming provider path still fails;
selected underlying model and sentinel-quality evidence remain visible in CI logs.

Discovered during

2026-08-15 repository/TODO reconciliation before full manual validation.

## Problem `main` commit `52ca8060456a0e91b69783867d6733026b94d07e` had a fully green deterministic job while Actions run `31676691726` failed in `REAL OpenRouter integration`. `openrouter/free` routed the streaming request to `nvidia/nemotron-3.5-content-safety:free`. The provider path itself succeeded (`HTTP 200`, incremental events, `[DONE]`, usage, final text), but the test asserted that the model must literally emit `STREAM_OK`; the routed model instead emitted `User Safety: safe`. This is the same class of instability already handled more carefully by the benchmark layer, where provider/runtime health and instruction-quality evidence are separate signals. ## Current state — 2026-08-18 The exact PR #89 head `6543853b5e0499ca4881a3991f579387596a470a` later passed the full REAL OpenRouter lane, including the live core suite, structured repair, benchmark, depth 0/1/2 experiment, and one-command RLM CLI smoke. That green run does **not** satisfy this issue's acceptance criteria by itself. The defect is that a healthy `openrouter/free` route can still fail the sentinel-sensitive streaming assertion when the selected model ignores the requested sentinel. No merged change in #89 modified that gate, so #46 remains open. The failing run above is retained as historical reproduction evidence rather than a claim about the latest CI state. ## Desired behavior - keep this a genuinely live provider test; do not replace it with a fake; - do not treat arbitrary routing behind `openrouter/free` as deterministic model semantics; - either pin `OPENROUTER_TEST_MODEL` for sentinel-sensitive acceptance, or make the default free-router streaming gate assert transport/stream correctness and record sentinel compliance as quality evidence; - continue verifying real incremental SSE events, termination, selected model, HTTP status, and usage when available; - never weaken credential redaction or provider-error handling. ## Acceptance criteria - deterministic CI remains unchanged and green; - the live streaming test cannot fail solely because a healthy `openrouter/free` request was routed to a model that ignored the requested sentinel; - a broken/truncated/non-streaming provider path still fails; - selected underlying model and sentinel-quality evidence remain visible in CI logs. ## Discovered during 2026-08-15 repository/TODO reconciliation before full manual validation.