Scheduler run hackmode-bbp-daily fails: unusable-response breaker stops run after ZAP startup fatal #102

New issue

Open

opened 2026-09-05 05:45:52 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-05 05:45:52 +00:00

Owner

Found during live inspection of the running a0-symbolics:local container (up 6h, healthy) on 2026-09-05.

Run: scheduler task hackmode-bbp-daily (uuid yfRKVVqX), schedule 30 3 * * * (America/New_York), last run 2026-09-05T00:32:27-04:00.

Recorded state: tasks.json has state: error with

ERROR: Agent stopped after 2 consecutive unusable model responses to prevent further API charges. Send a new message to try again.

Context: the worker was running the BBP recon pipeline (/a0/usr/projects/hackmode/scripts/bbp-daily-recon.sh). Its pipeline log ends with:

[00:39:56] Pre-cleaning stale ZAP/proxy processes
[00:39:59] Starting ZAP headless
[00:41:29] FATAL: ZAP did not come up
[00:41:29] Cleanup

So the model spent the run shepherding a pipeline that dies at ZAP startup, then produced textless/empty model turns twice in a row, tripping the circuit breaker in extensions/python/_functions/agent/Agent/hist_add_warning/end/_90_stop_unusable_response_loop.py (max_consecutive_unusable_responses).

Problems to address

The run never completes: long pipeline babysitting + a hard ZAP-start failure push the model into unusable (textless) responses and the cost breaker stops it.
bbp-daily-recon.sh treats "ZAP did not come up" as fatal with no retry/health window tuning, so a slow ZAP start kills the whole daily recon.
The task has no retry and lands in error until manually re-run.

Suggested next steps

Make the recon script fail soft when ZAP can't start (skip ZAP-dependent steps, report partial results) or add a longer startup/health-check window.
Have scheduled tasks produce a proper response tool break even on partial failure, so the circuit breaker isn't hit.

Found during live inspection of the running `a0-symbolics:local` container (up 6h, healthy) on 2026-09-05. **Run**: scheduler task `hackmode-bbp-daily` (uuid `yfRKVVqX`), schedule `30 3 * * *` (America/New_York), last run `2026-09-05T00:32:27-04:00`. **Recorded state**: `tasks.json` has `state: error` with ``` ERROR: Agent stopped after 2 consecutive unusable model responses to prevent further API charges. Send a new message to try again. ``` **Context**: the worker was running the BBP recon pipeline (`/a0/usr/projects/hackmode/scripts/bbp-daily-recon.sh`). Its pipeline log ends with: ``` [00:39:56] Pre-cleaning stale ZAP/proxy processes [00:39:59] Starting ZAP headless [00:41:29] FATAL: ZAP did not come up [00:41:29] Cleanup ``` So the model spent the run shepherding a pipeline that dies at ZAP startup, then produced textless/empty model turns twice in a row, tripping the circuit breaker in `extensions/python/_functions/agent/Agent/hist_add_warning/end/_90_stop_unusable_response_loop.py` (`max_consecutive_unusable_responses`). **Problems to address** 1. The run never completes: long pipeline babysitting + a hard ZAP-start failure push the model into unusable (textless) responses and the cost breaker stops it. 2. `bbp-daily-recon.sh` treats "ZAP did not come up" as fatal with no retry/health window tuning, so a slow ZAP start kills the whole daily recon. 3. The task has no retry and lands in `error` until manually re-run. **Suggested next steps** - Make the recon script fail soft when ZAP can't start (skip ZAP-dependent steps, report partial results) or add a longer startup/health-check window. - Have scheduled tasks produce a proper `response` tool break even on partial failure, so the circuit breaker isn't hit.

nsaspy added the

bug

label

2026-09-05 05:45:52 +00:00