test: pin the paid OpenRouter lane to glm-5.3-flash (drop the 120b cache pin) #374

Closed

nsaspy wants to merge 5 commits from test/paid-lane-glm-53-flash into main

nsaspy commented

2026-09-04 22:14:05 +00:00

Owner

Summary

Switches the paid OpenRouter lane off openai/gpt-oss-120b and onto the
repo variable's new value z-ai/glm-5.3-flash. No production runtime code
changes; test plumbing, workflow steps, and one doc paragraph.

test/live_compiler_cache_openrouter_test.pl no longer requires 120b:
require_pinned_cache_model/1 enforces a paid pinned model (free tiers
excluded), the cache turns use reasoning_effort(minimal) with 256-token
headroom (glm endpoints reject none with "Reasoning is mandatory"), and
the generation-metadata fetch retries on 404 — the endpoint is eventually
consistent and can lag fast completions by tens of seconds.
The cache-hit bar becomes a majority of the fourteen warm turns with a
real floor (≥5 hits). Rationale: the runtime-controlled invariant is
prefix stability — a churning prefix collapses the hit rate toward zero —
while provider upstream routing (each instance holding its own cache)
periodically misses a warm cache; measured 67%–100% across runs on glm.
paid-openrouter.yml drops the 120b equality gate; both workflows rename
the cache-acceptance step/evidence titles; docs/direct-runtime.md
updated.
Repo variable OPENROUTER_TEST_MODEL set to z-ai/glm-5.3-flash.

Evidence

Cache acceptance with glm-5.3-flash: green (100% warm hits in one run;
66.7% in another under the old 80% bar, motivating the majority bar).
glm-5.3-flash full core lane: 11/11 (verified on the merged PR #290
head); deterministic suite 1067/1067; git diff --check clean.

Note on the previous follow-up plan

The dedicated "120B planner-context" follow-up proposed during #290 review
becomes moot for CI: the lane no longer pins that model, and the planner-
context test passes with glm-5.3-flash (it exits 0 with a plunit
choicepoint warning, which is pre-existing and cosmetic).

## Summary Switches the paid OpenRouter lane off `openai/gpt-oss-120b` and onto the repo variable's new value `z-ai/glm-5.3-flash`. No production runtime code changes; test plumbing, workflow steps, and one doc paragraph. - `test/live_compiler_cache_openrouter_test.pl` no longer requires 120b: `require_pinned_cache_model/1` enforces a paid pinned model (free tiers excluded), the cache turns use `reasoning_effort(minimal)` with 256-token headroom (glm endpoints reject `none` with "Reasoning is mandatory"), and the generation-metadata fetch retries on 404 — the endpoint is eventually consistent and can lag fast completions by tens of seconds. - The cache-hit bar becomes a majority of the fourteen warm turns with a real floor (≥5 hits). Rationale: the runtime-controlled invariant is prefix stability — a churning prefix collapses the hit rate toward zero — while provider upstream routing (each instance holding its own cache) periodically misses a warm cache; measured 67%–100% across runs on glm. - `paid-openrouter.yml` drops the 120b equality gate; both workflows rename the cache-acceptance step/evidence titles; `docs/direct-runtime.md` updated. - Repo variable `OPENROUTER_TEST_MODEL` set to `z-ai/glm-5.3-flash`. ## Evidence - Cache acceptance with glm-5.3-flash: green (100% warm hits in one run; 66.7% in another under the old 80% bar, motivating the majority bar). - glm-5.3-flash full core lane: 11/11 (verified on the merged PR #290 head); deterministic suite 1067/1067; `git diff --check` clean. ## Note on the previous follow-up plan The dedicated "120B planner-context" follow-up proposed during #290 review becomes moot for CI: the lane no longer pins that model, and the planner- context test passes with glm-5.3-flash (it exits 0 with a plunit choicepoint warning, which is pre-existing and cosmetic).

nsaspy added 5 commits

2026-09-04 22:14:05 +00:00

test(live): replace the 120b cache pin with the pinned paid model a9654f4a7d

The compiler-cache acceptance no longer requires openai/gpt-oss-120b:
require_pinned_cache_model/1 enforces a paid pinned model, the cache
turns use reasoning_effort(minimal) with 256-token headroom (glm
endpoints reject effort none), and the generation-metadata fetch retries
briefly on 404 (the endpoint is eventually consistent after fast
completions). The hit bar becomes a majority of the fourteen warm turns
with a real floor (>=5): a churning prefix collapses toward zero, while
provider upstream routing periodically misses a warm cache (measured
67%-100% across runs). Workflows and docs drop the 120b naming.

fix(direct): advertise the context alias only when context ops are granted 576792a609

The direct-loop System message told every session that context content
is opaque with initial alias input, even when the session's capability
set grants no context operations — fail-closed preflight then converted
reasonable model behavior (reading the context it was invited to read)
into a failed batch. The alias sentence is now capability-gated.
Workflows: the CLI smoke grants reasoning headroom (512 tokens) and
pins reasoning_effort minimal, since glm endpoints reject effort none
and mandatory reasoning otherwise exhausts the 96-token budget.

Paid lane at this branch: core/repair/cache/benchmark/deep steps green
on glm-5.3-flash (run 33394174202); the CLI smoke was the only red step.
Local glm CLI smoke: exit 0 with all trace assertions matching.

test(live): streaming sentinel request gets reasoning headroom e9f0d211e0

glm endpoints make reasoning mandatory; at the old 256-token cap the
mandatory reasoning could truncate the streamed reply before the
STREAM_OK sentinel (CI run 33396271627: exactly 256 incremental events,
sentinel assertion failed). The live streaming request now pins
reasoning effort minimal with a 1024-token cap; the sentinel compliance
invariant is unchanged. Green 3/3 locally with the pinned model.

ci(paid): CLI smoke surfaces the CLI error payload instead of dying blind fcf17ed55d

set -e aborted the step at the swipl line, so a CLI failure whose error
lives inside the json (stderr silent) left no diagnostic at all. Capture
stderr, dump the json payload on failure, then apply the same greps.

review(319): rename stale cache-test name, tidy indentation, keep grep gate strict

Tree-sitter FFI / Direct SWI-Prolog Tree-sitter FFI (pull_request) Successful in 2m23s

Details

CI / REAL OpenRouter integration (pull_request) Has been skipped

Details

Nix flake / Flake package and clean runtime load (pull_request) Failing after 8s

Details

CI / Deterministic unit and load checks (pull_request) Successful in 3m23s

Details

Paid OpenRouter / Pinned paid OpenRouter integration (pull_request) Failing after 2m48s

Details

Clean SWI pack install / Install and load copied pack (pull_request) Successful in 4m21s

Details

033a3e785a

The paid CLI smoke must abort on the first failed trace assertion: the
diagnostic rewrite accidentally dropped set -e before the greps, letting
a failed early grep be masked by later ones. Also renames the cache
test to fifteen_fresh_... (turn count grew from ten) and fixes patch
indentation. No behavior changes otherwise.

nsaspy closed this pull request

2026-09-04 22:56:03 +00:00