[P0] Shared Prolog worker can deadlock all model turns with full bidirectional pipes #106

New issue

Open

opened 2026-09-06 04:15:52 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-06 04:15:52 +00:00

Owner

Summary

The live Agent Zero container has a healthy HTTP/Supervisor status while its shared Prolog JSON-lines worker is deadlocked. One blocked bridge prevents every later model turn and scheduled run from producing output.

Exact live evidence (2026-09-06)

Container: agent-zero, image a0-symbolics:local, source worktree HEAD a594a6a84feb8a44a4761e54f820203b8288234b (base ec58d20376).
The four relevant source files in /a0 hash-identically match that worktree.
Supervisor reports 6/6 processes RUNNING; Docker health is healthy; OOMKilled=false.
Scheduler started hackmode-rage-worker at 2026-09-05 10:45:27Z.
At 10:47:09Z the bridge logged Prolog worker exceeded 45.000s and entered the generic retry path.
One persistent SWI worker (PID 3445) has remained alive since that run.
Parent to worker pipe: 65,536 available bytes of 65,536 capacity.
Worker to parent pipe: 65,536 available bytes of 65,536 capacity.
The parent bridge thread and the SWI main thread both wait in anon_pipe_write.
Four scheduled runs later remained running for about 14 to 17 hours. Each appended only the new user/task prompt, with zero new agent, tool, or final-response log entries.
A later interactive Worker status chat likewise has a user entry followed by an empty agent event and no response.

A fresh provider-free status smoke still passes, which demonstrates why the existing health check is false-green for this condition.

Owning code

plugins/_prolog_context_compiler/helpers/transport.py:

lines 50-55 hold one lock across exchange, worker reset, and retry.
lines 73-74 perform blocking write and flush.
the 45 second timeout begins only afterward at lines 78-83.
line 84 performs a blocking readline after readiness of only the first byte.

plugins/_prolog_rlm/helpers/bridge.py lines 60-76 caches and shares one bridge across callers.

Impact

Global head-of-line blocking: one bad context compile stalls unrelated chats and scheduler tasks.
The container remains health-green while model service is unusable.
The existing unusable-response cost breaker does not cover a blocked transport operation.

Acceptance criteria

Bound the complete request operation, including lock acquisition, request write/flush, response drain, and retry.
Use nonblocking or concurrent duplex I/O that cannot wedge when request and response exceed the 64 KiB pipe capacity.
On deadline, terminate and reap the worker and release/fail queued callers within a bounded interval.
Isolate callers or otherwise prevent one stuck compile from globally blocking unrelated chats.
Add a regression with request and response bodies larger than 64 KiB, including timeout plus retry and concurrent queued callers.
Add a liveness check for the shared bridge rather than only launching a fresh status worker.
Verify the fix against the pinned runtime and current Prolog-RLM main without paid provider calls.

Version context

The container pin is Prolog-RLM 2e1264d80d02fecfb9f946e1328caaf1053e7a3b. Current Forgejo main is e1ed6c2b5f717d9bb380fcf22d144737299197f8, 186 commits ahead. The pipe supervision code is Agent Zero-owned, so this issue belongs here.

## Summary The live Agent Zero container has a healthy HTTP/Supervisor status while its shared Prolog JSON-lines worker is deadlocked. One blocked bridge prevents every later model turn and scheduled run from producing output. ## Exact live evidence (2026-09-06) - Container: agent-zero, image a0-symbolics:local, source worktree HEAD a594a6a84feb8a44a4761e54f820203b8288234b (base ec58d2037624154c760d7e3ffd84a53c1595f4cc). - The four relevant source files in /a0 hash-identically match that worktree. - Supervisor reports 6/6 processes RUNNING; Docker health is healthy; OOMKilled=false. - Scheduler started hackmode-rage-worker at 2026-09-05 10:45:27Z. - At 10:47:09Z the bridge logged Prolog worker exceeded 45.000s and entered the generic retry path. - One persistent SWI worker (PID 3445) has remained alive since that run. - Parent to worker pipe: 65,536 available bytes of 65,536 capacity. - Worker to parent pipe: 65,536 available bytes of 65,536 capacity. - The parent bridge thread and the SWI main thread both wait in anon_pipe_write. - Four scheduled runs later remained running for about 14 to 17 hours. Each appended only the new user/task prompt, with zero new agent, tool, or final-response log entries. - A later interactive Worker status chat likewise has a user entry followed by an empty agent event and no response. A fresh provider-free status smoke still passes, which demonstrates why the existing health check is false-green for this condition. ## Owning code plugins/_prolog_context_compiler/helpers/transport.py: - lines 50-55 hold one lock across exchange, worker reset, and retry. - lines 73-74 perform blocking write and flush. - the 45 second timeout begins only afterward at lines 78-83. - line 84 performs a blocking readline after readiness of only the first byte. plugins/_prolog_rlm/helpers/bridge.py lines 60-76 caches and shares one bridge across callers. ## Impact - Global head-of-line blocking: one bad context compile stalls unrelated chats and scheduler tasks. - The container remains health-green while model service is unusable. - The existing unusable-response cost breaker does not cover a blocked transport operation. ## Acceptance criteria - Bound the complete request operation, including lock acquisition, request write/flush, response drain, and retry. - Use nonblocking or concurrent duplex I/O that cannot wedge when request and response exceed the 64 KiB pipe capacity. - On deadline, terminate and reap the worker and release/fail queued callers within a bounded interval. - Isolate callers or otherwise prevent one stuck compile from globally blocking unrelated chats. - Add a regression with request and response bodies larger than 64 KiB, including timeout plus retry and concurrent queued callers. - Add a liveness check for the shared bridge rather than only launching a fresh status worker. - Verify the fix against the pinned runtime and current Prolog-RLM main without paid provider calls. ## Version context The container pin is Prolog-RLM 2e1264d80d02fecfb9f946e1328caaf1053e7a3b. Current Forgejo main is e1ed6c2b5f717d9bb380fcf22d144737299197f8, 186 commits ahead. The pipe supervision code is Agent Zero-owned, so this issue belongs here.

nsaspy added the

bug

label

2026-09-06 04:15:52 +00:00