parallel tool: refresh_parallel_jobs awaits task.result() unbounded, bypassing the wait timeout (permanent loop stall on hung job) #101

New issue

Open

opened 2026-09-05 05:24:21 +00:00 by mara · 0 comments

mara commented

2026-09-05 05:24:21 +00:00

Collaborator

Summary

The parallel tool's wait loop checks its deadline only after refreshing job results, and the refresh awaits task.result() with no timeout. A background job that never completes (e.g. a wrapped tool blocked on a synchronous call) parks refresh_parallel_jobs inside a blocking concurrent.futures.Future.result() on a cross-loop future forever — the tool call never returns, the timeout guard is never reached, and the agent loop stalls permanently.

Location

helpers/parallel_tools.py:247-262 — poll loop: await await_parallel_jobs(agent) runs before the deadline check at :258.
helpers/parallel_tools.py:308-311 — for a ready task: await task.result() with no timeout.
helpers/defer.py:183-199 — DeferredTask.result() parks a blocking future.result(timeout=None) on a default-executor thread (sync-fetch over cross-loop future).
Introduced with the parallel job store (last touched in 936713bc, 2026-08-24); reviewed at main 5116eb4d.

Repro

Run the parallel tool with wait: false, tool_calls: [{tool_name: "code_execution_tool", tool_args: {runtime: "python", code: "import time; time.sleep(9999)"}}] (or any tool that blocks synchronously inside execute).
Poll with action: "await", job_ids: [...], timeout: 10.
The timeout is never honored: the refresh blocks indefinitely inside task.result() before the deadline check ever runs. Only a UI-initiated job cancel can break the stall.

Impact

One hung job freezes that chat's agent loop permanently.
Each in-flight task.result() also occupies a thread of the shared default ThreadPoolExecutor; concurrent chats doing this starve unrelated run_in_executor users framework-wide.

Suggested fix

Pass the remaining wait budget into DeferredTask.result(timeout=...) (or asyncio.wait_for around it) so the deadline check at parallel_tools.py:258 is authoritative.
Alternatively check the deadline before each refresh iteration and per-job, not after.
Optionally bound concurrent task.result() waits to avoid shared-executor starvation.

Secondary findings (same subsystem, can be split out)

Uncollected wait: false jobs leak: no teardown hook removes PARALLEL_JOBS_KEY entries or cancels running DeferredTasks at context end (parallel_tools.py:215, 273-297); orphaned coroutines keep the background loop busy and results are lost.
All parallel jobs share the single background event-loop thread (helpers/defer.py:21-26, parallel_tools.py:222), so a blocking wrapped tool head-of-line blocks sibling jobs plus memory/history background extensions.

## Summary The parallel tool's wait loop checks its deadline only *after* refreshing job results, and the refresh awaits `task.result()` with no timeout. A background job that never completes (e.g. a wrapped tool blocked on a synchronous call) parks `refresh_parallel_jobs` inside a blocking `concurrent.futures.Future.result()` on a cross-loop future forever — the tool call never returns, the `timeout` guard is never reached, and the agent loop stalls permanently. ## Location - `helpers/parallel_tools.py:247-262` — poll loop: `await await_parallel_jobs(agent)` runs **before** the deadline check at `:258`. - `helpers/parallel_tools.py:308-311` — for a ready task: `await task.result()` with **no timeout**. - `helpers/defer.py:183-199` — `DeferredTask.result()` parks a blocking `future.result(timeout=None)` on a default-executor thread (sync-fetch over cross-loop future). - Introduced with the parallel job store (last touched in 936713bc, 2026-08-24); reviewed at main `5116eb4d`. ## Repro 1. Run the parallel tool with `wait: false`, `tool_calls: [{tool_name: "code_execution_tool", tool_args: {runtime: "python", code: "import time; time.sleep(9999)"}}]` (or any tool that blocks synchronously inside `execute`). 2. Poll with `action: "await"`, `job_ids: [...]`, `timeout: 10`. 3. The `timeout` is never honored: the refresh blocks indefinitely inside `task.result()` before the deadline check ever runs. Only a UI-initiated job cancel can break the stall. ## Impact - One hung job freezes that chat's agent loop permanently. - Each in-flight `task.result()` also occupies a thread of the shared default `ThreadPoolExecutor`; concurrent chats doing this starve unrelated `run_in_executor` users framework-wide. ## Suggested fix - Pass the remaining wait budget into `DeferredTask.result(timeout=...)` (or `asyncio.wait_for` around it) so the deadline check at `parallel_tools.py:258` is authoritative. - Alternatively check the deadline *before* each refresh iteration and per-job, not after. - Optionally bound concurrent `task.result()` waits to avoid shared-executor starvation. ## Secondary findings (same subsystem, can be split out) - Uncollected `wait: false` jobs leak: no teardown hook removes `PARALLEL_JOBS_KEY` entries or cancels running `DeferredTask`s at context end (`parallel_tools.py:215, 273-297`); orphaned coroutines keep the background loop busy and results are lost. - All parallel jobs share the single background event-loop thread (`helpers/defer.py:21-26`, `parallel_tools.py:222`), so a blocking wrapped tool head-of-line blocks sibling jobs plus memory/history background extensions.

mara added the

bug

label

2026-09-05 05:24:21 +00:00