Scheduler run task-steward-gate fails every 20m: unusable-response breaker after GH_TOKEN 401, invalid push credential, missing pyyaml #105

New issue

Open

opened 2026-09-05 05:45:59 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-05 05:45:59 +00:00

Owner

Found during live inspection of the running a0-symbolics:local container (up 6h, healthy) on 2026-09-05.

Run: scheduler task task-steward-gate (uuid T2xggOLO), schedule */20 * * * * (America/New_York), last run 2026-09-05T00:48:06-04:00.

Recorded state: tasks.json has state: error with

ERROR: Agent stopped after 2 consecutive unusable model responses to prevent further API charges. Send a new message to try again.

Context: chat tail (/a0/usr/chats/T2xggOLO/messages/14.txt) shows the steward gate worker hitting three independent environment failures during the run:

Missing/invalid secret — a Forgejo API call returned:
```
401 {"message":"access token does not exist [sha: §§secret(GH_TOKEN)]", ...}
```
The GH_TOKEN secret either isn't provisioned in the container or maps to a dead token.
Invalid git credential — log line: Error "user does not exist [uid: 0, name: ]" means the token is invalid/wrong secret. and error: failed to push some refs to 'https://git.starintel.actor/mara/task-steward.git'.
Missing Python module — the worker's YAML validation step crashed with ModuleNotFoundError: No module named 'yaml' in the worker venv (state/projects.yml / tasks.yml / policies.yml / handoffs.yml validation never runs).

After these failures the model produced unusable (textless) responses twice and the cost breaker (max_consecutive_unusable_responses, _90_stop_unusable_response_loop.py) stopped the agent. With a */20 cadence this cycles straight back into the same broken environment.

Problems to address

GH_TOKEN secret is not usable inside the runtime container (401 access token does not exist).
The task-steward push target git.starintel.actor/mara/task-steward.git rejects the stored credential ("user does not exist") — need a valid deploy key/token for that repo path.
pyyaml (or equivalent) is missing from the worker venv used by the gate's validation script.
The gate task has no preflight: it burns a full model run discovering missing secrets/modules every 20 minutes.

Suggested next steps

Provision/rotate the tokens as secrets and verify GH_TOKEN resolves in-container (§§secret(GH_TOKEN) must expand to a live token).
Add pyyaml to the runtime image/venv.
Add a cheap preflight (secret expansion + python3 -c "import yaml") before starting the model loop.

Found during live inspection of the running `a0-symbolics:local` container (up 6h, healthy) on 2026-09-05. **Run**: scheduler task `task-steward-gate` (uuid `T2xggOLO`), schedule `*/20 * * * *` (America/New_York), last run `2026-09-05T00:48:06-04:00`. **Recorded state**: `tasks.json` has `state: error` with ``` ERROR: Agent stopped after 2 consecutive unusable model responses to prevent further API charges. Send a new message to try again. ``` **Context**: chat tail (`/a0/usr/chats/T2xggOLO/messages/14.txt`) shows the steward gate worker hitting three independent environment failures during the run: 1. **Missing/invalid secret** — a Forgejo API call returned: ``` 401 {"message":"access token does not exist [sha: §§secret(GH_TOKEN)]", ...} ``` The `GH_TOKEN` secret either isn't provisioned in the container or maps to a dead token. 2. **Invalid git credential** — log line: `Error "user does not exist [uid: 0, name: ]" means the token is invalid/wrong secret.` and `error: failed to push some refs to 'https://git.starintel.actor/mara/task-steward.git'`. 3. **Missing Python module** — the worker's YAML validation step crashed with `ModuleNotFoundError: No module named 'yaml'` in the worker venv (`state/projects.yml` / `tasks.yml` / `policies.yml` / `handoffs.yml` validation never runs). After these failures the model produced unusable (textless) responses twice and the cost breaker (`max_consecutive_unusable_responses`, `_90_stop_unusable_response_loop.py`) stopped the agent. With a `*/20` cadence this cycles straight back into the same broken environment. **Problems to address** 1. `GH_TOKEN` secret is not usable inside the runtime container (401 `access token does not exist`). 2. The `task-steward` push target `git.starintel.actor/mara/task-steward.git` rejects the stored credential ("user does not exist") — need a valid deploy key/token for that repo path. 3. `pyyaml` (or equivalent) is missing from the worker venv used by the gate's validation script. 4. The gate task has no preflight: it burns a full model run discovering missing secrets/modules every 20 minutes. **Suggested next steps** - Provision/rotate the tokens as secrets and verify `GH_TOKEN` resolves in-container (`§§secret(GH_TOKEN)` must expand to a live token). - Add `pyyaml` to the runtime image/venv. - Add a cheap preflight (secret expansion + `python3 -c "import yaml"`) before starting the model loop.

nsaspy added the

bug

label

2026-09-05 05:45:59 +00:00