[P1] Scheduled runs need hard deadlines, restart recovery, and terminal-result validation #108

New issue

Open

opened 2026-09-06 04:15:54 +00:00 by nsaspy · 1 comment

nsaspy commented

2026-09-06 04:15:54 +00:00

Owner

Summary

Scheduled tasks can remain RUNNING indefinitely because TaskScheduler awaits agent.monologue() without a deadline. It also treats any returned string as success, even when the persisted result is a raw tool request rather than a terminal response.

Exact live evidence (2026-09-06)

Four scheduled runs are still RUNNING:

hackmode-rage-worker started 2026-09-05 10:45:27Z.
hackmode-reviewer started 2026-09-05 12:30:33Z.
symbolics-reviewer started 2026-09-05 13:30:36Z.
prologrlm-reviewer started 2026-09-05 14:10:39Z.

Each current run appended exactly one user/task log entry and no subsequent agent, tool, warning, or final-response entry. The oldest has been running over 17 hours and the newest about 14 hours.

A previous prologrlm-reviewer run was recorded as successful with last_result equal to a raw exec tool-request JSON object, not a response-tool completion.

Three other schedules did reach the unusable-response circuit breaker, but that breaker only observes consecutive misformat/repeat warnings. It does not cover blocked context compilation, transport lock waits, long tool loops, or nonterminal monologue results.

Owning code

helpers/task_scheduler.py:

line 959 awaits agent.monologue() with no timeout.
lines 961-964 mark any returned value successful.
persisted RUNNING state is not reconciled at process/container startup.

Impact

Stuck tasks never reach a terminal state and future cron occurrences are skipped.
Operators see RUNNING rather than a bounded error with a cause.
Tool/model loops can consume provider calls without a task-level cost or iteration budget.
Raw tool requests can be exposed as completed task results.

Acceptance criteria

Add a configurable wall-clock deadline per scheduled run.
Add task-level maximum model-call and iteration budgets independent of the unusable-response breaker.
Require a terminal response-tool completion before marking success; reject raw tool requests and empty/nonterminal results.
On timeout/cancellation, persist a diagnostic error that identifies the blocked phase without including prompts or secrets.
Reconcile stale RUNNING states on startup using durable run identity/heartbeat evidence.
Ensure deadline cleanup releases transport/tool resources and persists chat state.
Add tests for blocked monologue, endless valid tool calls, raw tool-call return, restart with stale RUNNING state, and normal successful completion.

## Summary Scheduled tasks can remain RUNNING indefinitely because TaskScheduler awaits agent.monologue() without a deadline. It also treats any returned string as success, even when the persisted result is a raw tool request rather than a terminal response. ## Exact live evidence (2026-09-06) Four scheduled runs are still RUNNING: - hackmode-rage-worker started 2026-09-05 10:45:27Z. - hackmode-reviewer started 2026-09-05 12:30:33Z. - symbolics-reviewer started 2026-09-05 13:30:36Z. - prologrlm-reviewer started 2026-09-05 14:10:39Z. Each current run appended exactly one user/task log entry and no subsequent agent, tool, warning, or final-response entry. The oldest has been running over 17 hours and the newest about 14 hours. A previous prologrlm-reviewer run was recorded as successful with last_result equal to a raw exec tool-request JSON object, not a response-tool completion. Three other schedules did reach the unusable-response circuit breaker, but that breaker only observes consecutive misformat/repeat warnings. It does not cover blocked context compilation, transport lock waits, long tool loops, or nonterminal monologue results. ## Owning code helpers/task_scheduler.py: - line 959 awaits agent.monologue() with no timeout. - lines 961-964 mark any returned value successful. - persisted RUNNING state is not reconciled at process/container startup. ## Impact - Stuck tasks never reach a terminal state and future cron occurrences are skipped. - Operators see RUNNING rather than a bounded error with a cause. - Tool/model loops can consume provider calls without a task-level cost or iteration budget. - Raw tool requests can be exposed as completed task results. ## Acceptance criteria - Add a configurable wall-clock deadline per scheduled run. - Add task-level maximum model-call and iteration budgets independent of the unusable-response breaker. - Require a terminal response-tool completion before marking success; reject raw tool requests and empty/nonterminal results. - On timeout/cancellation, persist a diagnostic error that identifies the blocked phase without including prompts or secrets. - Reconcile stale RUNNING states on startup using durable run identity/heartbeat evidence. - Ensure deadline cleanup releases transport/tool resources and persists chat state. - Add tests for blocked monologue, endless valid tool calls, raw tool-call return, restart with stale RUNNING state, and normal successful completion.

nsaspy added the

bug

label

2026-09-06 04:15:54 +00:00

nsaspy referenced this issue

2026-09-06 04:30:55 +00:00

[P1] Scheduled tasks must start fresh run contexts seeded by the previous terminal output #110

mara commented

2026-09-08 01:14:10 +00:00

Collaborator

Update after reviewing main up to 5116eb4d + bba5dcba (commits 0b62cccb "fix: isolate scheduler run contexts" via PR #111, and 7c057880 "fix: route harness direct/complete through the context compiler" via PR #92):

Now fixed by 0b62cccb:

Terminal-result validation: helpers/task_scheduler.py:1146-1160 — _execute_task now raises NonTerminalTaskResultError unless the monologue result matches an attested response-tool completion (recorded by extensions/python/tool_execute_after/_90_scheduler_terminal_result.py). The old behavior of recording a raw tool-request JSON as a successful last_result is gone.
Stale-run completion races: _finish_run guards promotion with task.current_run_id == run_id, so a late completion from a superseded run can't clobber state.

Still missing (this issue's remaining scope), verified at bba5dcba:

Hard deadline: await agent.monologue() at helpers/task_scheduler.py:1146 still has no timeout wrapper. The TaskRunStatus.TIMED_OUT classification at :1190-1192 only fires if something else raises TimeoutError — the scheduler itself never imposes one. Suggested: wrap the monologue in asyncio.wait_for(..., timeout=task.run_deadline_seconds) with a configurable default (e.g. 3600s), and classify as TIMED_OUT on asyncio.TimeoutError.
Restart recovery: nothing reconciles a persisted TaskState.RUNNING / last_run_status: running with current_run_id set at scheduler load time. After a container restart, such tasks stay RUNNING forever and check_schedule (:732) skips future occurrences. Suggested: on SchedulerTaskList.load(), re-run any task with state == RUNNING through _finish_run(task_uuid, task.current_run_id, TaskRunStatus.FAILED, "scheduler restarted mid-run") (or CANCELLED).
Related hygiene found during this review (filed separately as #116): per-run contexts are persisted with save_tmp_chat on every occurrence and never removed.

Happy to split 1 and 2 into separate issues if preferred.

Update after reviewing main up to `5116eb4d` + `bba5dcba` (commits `0b62cccb` "fix: isolate scheduler run contexts" via PR #111, and `7c057880` "fix: route harness direct/complete through the context compiler" via PR #92): **Now fixed by `0b62cccb`:** - Terminal-result validation: `helpers/task_scheduler.py:1146-1160` — `_execute_task` now raises `NonTerminalTaskResultError` unless the monologue result matches an attested response-tool completion (recorded by `extensions/python/tool_execute_after/_90_scheduler_terminal_result.py`). The old behavior of recording a raw tool-request JSON as a successful `last_result` is gone. - Stale-run completion races: `_finish_run` guards promotion with `task.current_run_id == run_id`, so a late completion from a superseded run can't clobber state. **Still missing (this issue's remaining scope), verified at `bba5dcba`:** 1. **Hard deadline:** `await agent.monologue()` at `helpers/task_scheduler.py:1146` still has no timeout wrapper. The `TaskRunStatus.TIMED_OUT` classification at `:1190-1192` only fires if something *else* raises `TimeoutError` — the scheduler itself never imposes one. Suggested: wrap the monologue in `asyncio.wait_for(..., timeout=task.run_deadline_seconds)` with a configurable default (e.g. 3600s), and classify as `TIMED_OUT` on `asyncio.TimeoutError`. 2. **Restart recovery:** nothing reconciles a persisted `TaskState.RUNNING` / `last_run_status: running` with `current_run_id` set at scheduler load time. After a container restart, such tasks stay RUNNING forever and `check_schedule` (`:732`) skips future occurrences. Suggested: on `SchedulerTaskList.load()`, re-run any task with `state == RUNNING` through `_finish_run(task_uuid, task.current_run_id, TaskRunStatus.FAILED, "scheduler restarted mid-run")` (or CANCELLED). 3. Related hygiene found during this review (filed separately as #116): per-run contexts are persisted with `save_tmp_chat` on every occurrence and never removed. Happy to split 1 and 2 into separate issues if preferred.